Guest post from Bill Anderson

Bill is an information architect with Development Initiatives, leading their work on the data revolution for sustainable development data and joined-up data standards. He is also the technical lead for the International Aid Transparency Initiative and the coordinator for the newly-formed Joined-up Data Alliance. He was part of the drafting team for the Africa Data Consensus.

The Open Data movement is roughly ten years old. Starting in California in 2007 a number of attempts have been made to enshrine the principles of open data. I think it is fair comment that most of these charters have been drafted by and for constituencies in North America and Europe and have focused primarily on the relationship between western liberal governments and economies and the technically literate sections of their citizenry. Until now.

The international Open Data Charter launched in New York last month is the first globally inclusive and comprehensive manifesto of its type. It recognizes the north-south digital divide. It recognizes the challenges and needs facing developing countries. It embraces the data revolution for sustainable development data. As Jose Alonso from the Web Foundation has argued, it provides a canonical open data standard that can be embedded, adopted, and implemented by other related initiatives without duplication or fragmentation.

Of equal importance is the inclusion of the principle of “Comparability and Interoperabilityâ€? – the first time that a charter like this has moved beyond transparency to properly consider the whole point of open data: usage. Most datasets are pretty meaningless on their own. It is only when they are combined and contextualized do they generally make sense. Joined-up data creates the information we need.

You can only join up datasets if they are comparable and “speak the same language�. The Charter recognizes “that in order to be most effective and useful, data should be easy to compare within and between sectors, across geographic locations, and over time�.

And if you wish machines to do the joining up for you, the systems, formats, and definitions used by each dataset need to be interoperable: “data should be presented in structured and standardized formats to support interoperability, traceability, and effective reuseâ€?.

Most of the useful data that we need today – even that which is already open – resides in a massive Tower of Babel. Global institutions – the UN, OECD, World Bank, and IMF among them – maintain countless incompatible and competing standards covering financial, administrative, geographic, and socio-economic data. So do national governments. Most open data portals are silos in which thousands of datasets sit in isolation from each other, each speaking their own distinct language or dialect. Continuing this language metaphor, there are two solutions to the problem: either agree to speak the same language or get an interpreter.

Principle four of the Charter first calls for “increased interoperability between existing international standardsâ€? – making sure we can translate accurately between standards. It then demands that we “support the creation of common, global data standards where they do not already exist and ensure that any new data standards we create are, to the greatest extent possible, interoperable with existing standardsâ€? – in other words: standards that speak the same language.

The importance of the Charter is that it provides us with a platform from which to launch a two-pronged attack on data silos.

We need to engage with all global and regional standard-setting institutions

• to persuade them of the importance of joined-up standards;
• to get them to cooperate with each other on resolving incompatibilities; and
• to get them to agree that all future standards will be properly joined-up.

And we need to stimulate the market to build more translation tools that will enable us to generate more usable and accessible information from existing data.

Guest post from Tim Davies

Tim Davies is co-founder of Open Data Services Co-operative, and is a social researcher exploring open government and the civic implications of Internet technologies.

A few weeks back the International Open Data Charter was officially launched on the fringes of the United Nations General Assembly Meeting in New York. The Charter’s Principles, which were first previewed at the IODC15 conference in Ottawa, have been finalized over the last few months by a global network of stewards from government and civil society.

One of the final additions to the Charter was a new top-level principle on ensuring that open data is “Comparable and Interoperable”.

“Principle 4 – Comparable and Interoperable

We recognize that in order to be most effective and useful, data should be easy to compare within and between sectors, across geographic locations, and over time.

We recognize that data should be presented in structured and standardized formats to support interoperability, traceability, and effective reuse.

We will:

1. Implement consistent, open standards related to data formats, interoperability, structure, and common identifiers when collecting and publishing data;
2. Ensure that open datasets include consistent core metadata and are made available in human- and machine-readable formats;
3. Ensure that data is fully described, that all documentation accompanying data is write in clear, plain language, and that data users have sufficient information to understand the source, strengths, weaknesses, and analytical limitations of the data;
4. Engage with domestic and international standards bodies and other standard setting initiatives to encourage increased interoperability between existing international standards, support the creation of common, global data standards where they do not already exist, and ensure that any new data standards we create are, to the greatest extent possible, interoperable with existing standards; and
5. Map local standards and identifiers to emerging globally agreed standards and share the results publicly.�

This principle picks up one of the key themes to also come out from IODC 2015: the need to go beyond the publication of datasets, to also consider how data can be made re-usable across different countries and contexts—delivering data that can be joined up.

Often, this is treated solely as a conversation about technical standards, schema, and field names. But, by focussing on interoperability, the Charter taps into a much richer framework. Making data interoperable is not about agreeing a lowest-common-denominator set of fields that everyone must use for their data. Instead, it involves identifying the points of connection between different datasets, working to develop shared understanding of definitions, and building common approaches and identifier sets that make it easier to connect up diverse data. At the same time, it involves recognizing that no single schema should be imposed as the one true representation of reality. Oftentimes, interoperability requires not just standards, but also tools and processes to bridge and translate between data that was generated within different cultural and political contexts.

Concrete user needs must be at the heart of a global agenda for open data interoperability. Whether it is anti-corruption use cases that demand the ability to trace financial flows across borders, or use cases focusing on improving trade in agricultural inputs and outputs to increase sustainability and address climate change, each instance of intended data use highlights particular features that will be needed to provide joined up and interoperable data. Perfect interoperability is an impossibility; nor is it desirable. That means our efforts for interoperability must be explicitly intentional.

One live example we’re working on at Open Data Services, the workers co-operative I co-founded earlier this year to focus on supporting the implementation of open data projects, involves the creation of a database of oil, gas, and mining projects around the world with the Natural Resource Governance Institute (NRGI). Legislation around the world increasingly demands project-level reporting by extractives companies—yet definitions of projects, and the categories of payments or reserves to be disclosed vary across jurisdictions. Creating a data model and identifier schemes for this data involves considering the different users who want to know about extractives, and thinking about the different approaches, from regulation to voluntary consensus, through which shared approaches to identifiers and definitions can be developed.

As the conversation on interoperability moves forward over the coming year, we need to emphasize the importance of a human-scale and inclusive conversation. Whilst the vision of interoperable and comparable data involves the creation of ambitious new global open data infrastructures, it is vital that we ensure these infrastructures are not just shaped by one or two interested parties. Issue by issue, theme by theme, we will need to identify the stakeholders, explore priority datasets and user needs, and then negotiate the balance between standard development, tool building, and translation: critically aware of who is included and excluded from the conversation.

Through the work of the nascent Joined Up Data Alliance, the development of sector-specific packages for the resource centre of the International Open Data Charter, the conversations around a Global Partnership for Sustainable Development Data, as well as the work of thematic open data networks such as GODAN, there are many opportunities in the year ahead to develop these conversations. The energy is there: although also the risk that initiatives themselves will not be joined up. Perhaps our first challenge then is to find the right ways to build interoperability between these networks of people, to make sure we will be able to build interoperable data.

Hopefully by the time the open data policy and practitioner communities meet together again at IODC 2016 in Spain we’ll have some solid methods, approaches, and progress to build upon.