OpenTED, Opening Tender Electronic Daily
This post reviews how OpenTED and OpenSpending have worked to make procurement data from the EU site, Tender Electronic Daily (TED), available as a CSV download. More than 100,000 public sector contracts are published annually in the European procurement register originating from tiny municipalities to large government agencies.
Why open up EU procurement data?
TED contains procurement data on contracts awarded from any public agency within the EU valued above the minimum threshold of EUR 200,000. In most EU countries, granular data from contract awards therefore comprises a significant share of procured and projected spending.
It is an often overlooked fact that EU procurement rules apply to all majority publicly owned companies. For this reason, the public can, for instance, access data on more than 500 contracts awarded by the Swedish state-owned Vattenfall in all EU countries of operation, such as a contract awarded from their Berlin based company, due to the fact that it is majority owned by the Swedish state.
Project and issues
Data from TED is not available as a bulk download, and so in 2011, a small data journalism project, OpenTED, began exploring the options for scraping the data in order to make it openly available. In November 2012 and May 2013, this was explored further through community hack days in London and Brussels organised by OKF, where data was retrieved, parsed, and cleaned. The full TED data is now available as CSV files EU-wide, on a country-by-country basis, and by annual breakdown.
Several challenges remain, which are primarily tied to the data quality. Additional data cleaning is still needed before it is even possible to assess to what extent the TED data actually contains sufficiently useful information. A review of data quality is needed. Preliminary findings have shown that significant data fields such as contract amounts and contractor name suffer from low reporting due to what could be an absence of mandatory reporting requirements. The community involved in advancing procurement transparency, such as Transparency International and Sunlight Foundation, should examine how disclosure practices can be improved. The data quality review of TED is an example of a dataset which can only be improved if the transparency communities across countries join forces to argue for such improvements.