4th IODC

Blog IODC 2016
Madrid. October 6-7, 2016


Developing User-Centered Methods for Measuring the Value of Open Data

May 11, 2015 by IODC4

A guest post from Johanna Walker and Mark Frank

Many existing approaches to open data measurement start from assessing ideal properties of datasets. But there is little evidence these properties capture the key things that data users need. Through an Open Data for Development grant from the Open Government Partnership Open Data Working Group, we’ve been exploring user-centred methods for measuring open data.

We started from the assumption that the purpose of open data metrics should be to determine the value of a group of open datasets to specific users. For this project, we have focussed on one kind of user:  small third sector organisations working with the homeless in the UK (Winchester) and in India (Ahmedabad)


In order to ensure we were measuring attributes of importance to users, we created a ‘reverse-hackathon’ process, where we asked users to outline problems that mattered to them and then to identify the information that could be used to solve those problems. We then located data that could be used to create this information, and through a process of workshop discussion and role-play, established the key attributes for usability of the data. From this, we identified metrics for these attributes.

We used an approach that reflected the key challenges of making effective use of the the data, but also aimed to provide  metrics which could be easily operationalised and applied. We used both subjective and objective metrics, though with a preference for objective metrics where practical. Where a suitable metric already existed, we used it.

Attributes and Metrics

The five relevant attributes derived from this process were discoverability, granularity, immediate intelligibility, authority and how linked the data is. Each attribute is scored on a points system, to build up an assessment of whether datasets on a given topic are fit for user needs.

Discoverability – Data Portal Points Measure

For any given dataset, points are awarded if the owner/published has an open data portal and desirable characteristics of the portal such as providing a list of unpublished datasets.

Granularity – Aggregation Scale

We evaluated two approaches. One was context specific and relied on a subject matter expert specifying appropriate levels of aggregation for that context e.g. country, state, city, street. The second approach looked for the presence of generic class data which can be used for aggregation such as post codes, dates, and gender.

Immediate Intelligibility – Supporting Documentation Scale

We evaluated a scale based on how quickly a user could access supporting documentation ranging from no documentation exists through to context specific information available instantly.

Trustworthiness –  7 Ws Score

Ram and Liu (2005) identified 7 questions which should be answered to make data trustworthy. We scored datasets on how many of those questions were answered in the metadata.

Linkable to other data

The Five Stars of Linked Open Data is an accepted and easily applicable measure of open data format standards which reflects the user need to be able to discover unanticipated relationships among data.

All these metrics have been piloted against data sets and assessed for validity, reliability, discrimination, transferability/comparability and efficiency. In India these were the data sets used in the workshops, in the UK they were the data sets used in the NESTA Open Data Housing Challenge.

What we’ve learnt

Although none of the user-derived attributes were completely novel, they demonstrate a different emphasis from ‘top-down’ or publisher-focused metrics widely used in the literature, and provide a way of assessing data in a given sector, rather than looking at isolated datasets.

The attributes we found to be important were similar for both the UK and Indian participants in our study. However, the features of data that matter may be very different for larger organisations, who may, for example, have more resources to invest in data discovery and understanding poorly documented data. This raises  potential discrimination issues if publishers are influenced by the concerns of larger, rather than smaller third sector organizations.

The time and resource constrained nature of our groups meant that we had to focus on quick and simple to apply metrics. We were only able to find one pre-existing appropriately efficient and ‘low-tech’ metric for these attributes, the ‘Five Stars of Open Linked Data’, and one model on which we could base a metric. Otherwise, the metrics had to be developed afresh.

Our users were digitally literate, relatively skilled in administrative IT and domain experts. Despite this, there were non-trivial blocks to using Open Data;

Discoverability and the amount of contextual knowledge required to facilitate this is a substantial challenge.

Debating measurement methods

The full findings of this research will be presented as part of the Open Data Research Symposium ahead of IODC on 27th May, and then on Friday 29th will feed into discussions in the Measurement Action Area.

What do you think about the potential for rethinking open data measurement from the bottom-up, rather than the top-down? Do these metrics make sense to you? How do we balance easy-to-operationalise measurement, with measurement that matters to users. Share your thoughts in the thread below.


  • Alan Hudson

    May 18, 2015 at 20:52

    I really like the idea of user-centred methods for assessing the value of open data. If the point of open data is to see it used to address real-world problems, then this has got to be the way to go.

    Global Integrity is working with the Mexican Institute for Competitiveness – with support from the Omidyar Network – to map the landscape of fiscal governance in Mexico and to assess the ability of citizens to follow and track the flow of public resources (to “follow the money”).

    Some info on the work that Global Integrity is doing with IMCO, in Mexico, with IFAI (now renamed INAI), available here https://www.globalintegrity.org/posts/following-the-money-and-opening-government-building-from-the-local/

    This project already had a strong user-centric flavour, but reading this note from the team at the University of Southampton has further piqued my interest in the possibility of doing user-centric assessments of the value of open data e.g. datos.gob.mx for citizens’ efforts to follow the money.

    Follow the Money Open Data Treasure Hunts maybe?!

    Alan Hudson, Executive Director, Global Integrity


  • Transparify

    May 28, 2015 at 08:33

    we indeed think the key should be to use “engaged citizens” as one key standard. This is also useful, in explaining the need to tech & communications staff. Having two different people rate, and measure the time they need to retrieve is also good. In a next step, we are thinking of recording our raters, to show to some organizations just how hard it is to find info on their websites. They often are surprised! (Our results on transparency of Open Data Con participants will go up in one hour.)


  • Pingback: Slow down with the standards talk: it’s interoperability & information quality we should focus on

  • Pingback: Follow the Money: Putting People & Problems First - new.globalintegrity.org

Leave a Reply

Your email address will not be published. Required fields are marked *

Use of cookies

This site uses cookies in order to improve your user experience. By continuing to use the site, you are agreeing to the use of cookies and accepting our cookies policy. .