A guest blog post from the action leads reflecting on the gaps that measurement efforts around open data should address.

Open data is a growing phenomenon: in less than ten years the number of open data initiatives around the world has grown from a handful to number in the many hundreds. Governments, civil society and business are utilizing open data for their benefits – promoting economic growth, improving public services, and increasing accountability and transparency. A number of different initiatives have started to measure the work that is done by the open data ecosystem to spur data reuse and create value. While these initiatives are important to support learning around open data, and strategic actions to produce the expected impact, far too often they are not coordinated and can appear to operate in separate silos.

A year ago, in May 2014, a group of open data researchers and practitioners gathered in New York City to discuss common approaches to open data measurement. This workshop resulted in a draft Common Assessment Framework, which lay down an outline for measurement and assessment of open data. The framework describes four areas: Context, Data (technical openness), Use and Impact – all part of the data value chain. These headline categories provide a basis for further development of indicators, and deeper analysis of comparable measurements.

However, the work that was done in NYC was only the beginning. As a growing field we are constantly changing and finding new opportunities, therefore, we need to constantly question and improve our own assessment methods. Simple refinements and aligning different studies around common definitions are just the beginning of what is needed. Some of the additional questions go much deeper: how to measure and quantify social impact? How to mode from estimates to the evaluation of real impact of Open Data? We need to explore many different ways to deal with these questions.

In addition to common metrics, if we want comparable data, we need common ways to put them into practice. That requires knowledge and skill sharing across the field of assessment, and capacity building for researchers. Many approaches to assessing open datasets for example, demand extensive knowledge both of subject matter, such as fiscal transparency, and of technical features of datasets, such as machine readability, not to mention the need for a strong understanding of the context at the local government or city level where real action often takes place together with efforts to assess impact generation/. Local experts who can combine domain expertise and open data understanding are a scarce resource right now.

But capacity building around open data measurement can’t be about creating a dispassionate network of ‘observers’ watching what happens: it should be about measurement that is linked to action. Open Knowledge’s Open Data Index has shown how an open data measurement exercise can provide an important opportunity to build deeper understandings of open data, and to explore not only the current state of open data in a country, but also to build community that can drive forward future improvements in practice. The OECD Open Useful and Reusable (OUR) government data index shows variations in the existing level of governments’ commitment to support data availability, accessibility and re-use.

As part of the Measurement Action Area at IODC, and based on the work of the Common Assessment Framework workshop, we are seeking to map current gaps in open data measurement: whether at the level of key metrics, classifications, definitions, methods or operational tools to help us carry out quality and comparable research. The idea is to be able to develop a road map that shall enable us to collaborate as a community, creating better processes and practices of assessment.

To get us started, we want to hear your ideas on the current gaps in assessment methods We’ve updated the list of existing assessment methods from the Common Assessment Framework, and have published it in this Google Doc for your comment. What are we missing? What important assessment and measurement issues are these studies failing to capture? What should measuring efforts focus to assess?

We also want to raise broader questions that work on measurement must engage with. Questions like:

• How do we minimize bias in research?
• How do we ensure we look at both positive and negative effects of open data?
• How do we balance independence of measurement efforts, with creating comparable data through collaboration?
• How do we move from estimates of economic value creation to real impact assessment?

You can share your thoughts in the comments below.