It was a long day in early December 2008. Thirteen hours alone on a Sunday in a windowless room of the presidential transition HQ on 6th Street in DC. The transition team that had started as a dozen people the previous summer had ballooned after the election to almost 700 people who were now responsible for planning the first hundred days of the Obama administration. It was a microcosm of the government, designing initiatives to launch the new presidency with a socially impactful and politically practical bang. Coming on the heels of the Bush Administration and plummeting rates of trust in government, it was imperative that we govern differently, not behind closed doors, but in the open. Although the iPhone had only just been invented and social media platforms, Facebook and Twitter, were still comparatively new, it was clear that the internet, especially new data science tools and methods, might make it possible to strive for more evidence-based policy-making and better solutions to public problems.

At that juncture, I was chairing the Technology, Innovation and Government Reform (TIGR) working group, a small band of people passionate about the potential for using new technology to modernise and improve the workings of government. Our policy initiatives were designed to cut across the usual topics of economy, education, foreign policy, and health to promote a different way of working. We wanted to be “one bullet point of every five” and help each of the subject-matter teams to use technology, data, and innovation to accelerate the implementation of their goals.

We had a motley array of cross-cutting suggestions to put forward to the President-elect. They included new websites, such as that would lay bare the money we were spending on the bailout after the financial crisis, and new hires, including the creation of a new Chief Technology Officer position, an expanded Chief Information Officer role, and a technology “SWAT” team that would go into each agency and assess the state of its infrastructure, as well as a new open government policy. As is now well known, that policy had three inextricably intertwined prongs: transparency, participation, and collaboration.

Inspired by the way the publication of weather data had spawned a billion-dollar forecasting industry or the sharing of government-collected genomic data had birthed the biotech revolution, we were convinced that opening up the information that government collects would accelerate solutions to public problems if designed to go beyond mere transparency to create incentives for a wide range of actors across government, academia, and industry to use information for public good.

Just as open source software development – creating code with a larger group of people often outside the confines of one organisation to accelerate the process of both writing and testing software – opening up government data could make it possible for those outside of government to scrutinise and use government information more productively than government acting on its own. Now ten years into the open data revolution, it is almost hard to remember how radical an idea open data – or transparency plus participation and collaboration – was at the time.

First, it upended 50 years of thinking about the right-to-know strategies embodied in Freedom of Information (FOI) legislation. Open data complicated our reliance on FOI as the bedrock of transparency policy by shifting the underlying theoretical understanding of the relationship between the state and the public from the adversarial to the collaborative.

FOI is an inherently confrontational tactic focused on prying secrets out of government. Open data is not. It depends upon the institution that collects the data wanting to publish it in order to attract knowledgeable and passionate members of the public who want to use it. Because governments in an open data regime must proactively publish their data with the intent that people will use it, the normative essence of open data is participation rather than litigation. The role of the public has always been to scrutinise and criticise. The idea that the public and government can work together to augment the manpower and skills in under-resourced public institutions continues to demand a major shift of mindset.

Second, many transparency and good government activists were actively hostile toward the new policy because it did not focus squarely on publishing information only about the workings of government such as budget data that is designed to produce greater government accountability. By catalysing public engagement to promote both the scrutiny of data by the public and collaboration with the public in building new analytical tools and websites, open data galvanised collaboration between institutions and the public to create value of different kinds, especially to advance solutions to hard problems.

Opening up the corpus of patent data – one of our earliest projects – while laudable, struck many as a distraction from the all-important goal of enhancing government accountability. The fact that such data could unlock our understanding of the innovation economy was not yet well understood. Similarly, the idea that open data could be a key asset in developing tools to help passengers know which flights were likely to be delayed, help patients choose between hospitals, or help parents make more informed decisions about colleges, ran contrary to what open government meant for many people. It took many years of experience with open data to temper the discontent and persuade the naysayers. Creating apps for the Health Datapalozza by using newly published datasets from Health and Human Services began to change minds. Witnessing first-hand the reforms to the criminal justice system in the United States made possible by opening up police data was a sign that the movement was maturing. Thousands of lives saved by CPR-trained bystanders responding to texts specifying the locations of people experiencing cardiac arrest, generated by a real-time open data feed of emergency 911 calls, drove home the point that open data is a vital new tool for advancing social justice. The countless examples from around the world sprinkled throughout this volume, and the over 70 countries making commitments to publishing open data as part of their participation in the Open Government Partnership, have created widespread awareness of the power of open data as a new tool in the toolkit for public problem solving.

The explosion of newly available data coupled with mounting evidence (as this book so thoroughly demonstrates) that data catalyses productive, problem-solving partnerships between government and the governed suggests that the use of open data as a tool of governing will continue to grow. If the trend continues, open data will lead to new empirically informed ways to hold government and others accountable, spurring consumer choice and expanding the range of approaches to tackling human rights and development challenges. Yet a week does not go by when I do not still have to debate with those in government about the value of opening data. Open data in many places is still under threat from the move toward more closed governments and closed societies. Even in more enlightened regimes, however, many still argue that it is better to sell than give away the data that was paid for by, and belongs to, taxpayers. I still plead with those who doubt whether people will use the open data we invest in publishing in machine-readable formats rather than PDFs. These doubts stem, in part, from the lack of data-analytical skills among public servants. We know more in 2019 than we did a decade ago about how to use data for good. But even when governments know to open and publish their data, they still often lack the ability to use the data themselves. This may slowly change as agencies like Digital Canada, the Argentinian government lab (LabGobAr), and the multi-university Coleridge Initiative in the US, train people in government in how to use data to solve problems.

To be sure, there have been times when the potential for open data has been over-hyped, especially when naively assuming that data publication, in and of itself, will solve problems, neglecting the importance of investing in the original idea that participation and collaboration are vital for getting multi-disciplinary teams of people inside and outside of government scrutinising, visualising, and using the data to create value.

But, fundamentally, the challenge for open data – and open government more broadly – is the shift in mindset it demands to embrace the original values and learn the practices of transparency, participation, and collaboration. Open government shifts the focus of transparency from monitoring government after the fact to mechanisms that encourage the public to participate actively in improving societal outcomes. Open data fosters more active citizenship and more collaborative democratic institutions that draw directly on the collective expertise of the population to solve public problems. Ultimately, open data gives us a vision for a new kind of government to strive for – not bigger or smaller – but one that ensures collaboration makes our public institutions more effective and legitimate and our democracy stronger. By taking stock of the current state of open data, this book acts as a key resource and charts a course for future action to keep open data on track as a transformative tool of more open, collaborative, innovative, and participatory governance.

Beth Simone Noveck

Professor, New York University and Director, The Governance Lab

New York City, 2019