The voices and stories in this report highlight the diversity of the community working at the intersection of technology and government finance. We conclude this report by summarizing the demands CSOs are making for spending data and our recommendations for how they can pursue those demands and use the data to engage citizens with public finance.
Data: demands and obstacles
It is impossible to deny that civil society is interested not only in open financial data but also in other forms of data that are necessary to make sense of public finance.
We encountered strong demand for the traditional categories of financial data, namely budgets, transactional spending data, and procurement and contract data. We also encountered interest in data on the entities referenced in spending data. Such information is necessary to contextualize and understand financial data, particularly procurement data:
- Company information. Data on companies is necessary in order to be able to answer questions related to policy and outsourcing decisions, the integrity and effectiveness of the contracting process, and the track records of companies contracting with governments, to name just a few applications. The OpenCorporates report on open corporate data in Open Government Partnership Countries provides more information on company data.
Information on people in positions of power and their business and family associations. Such data is needed to answer key questions on responsibility for political decisions, the relationships between winners of tenders and government actors, and so on.
There have been a few attempts to standardise and collect this biographical information, including Popit from MySociety and Popolo from OpenNorth and the Participatory Politics Foundation. A coordinated international effort is required in order to maintain this enormous and rapidly evolving dataset, however.
CSO demand for open spending data (and contextual data) goes well beyond simply asking for information. Our CSO interviewees have been in consensus about the technical impediments presented by existing disclosure practices and about the need for machine-readable, bulk-downloadable data. Lack of such data is an obstacle to transparency and accountability. Further details are provided in the appendix on machine-readable data.
Besides bad data publishing practices, the CSOs we interviewed reported similar problems with obstruction from governments. Privacy concerns in particular were reported as frequently cited by governments as an excuse for not releasing granular data; similarly, many governments argue that contracts are commercially confidential and for this reason only agree to disclose redacted contracts (if any at all). Our interviewees believe these concerns are often raised in bad faith as an excuse. To help counter qualms about privacy, we have prepared a privacy guide showing how technology and good data management practices make it possible to develop a workflow which allows sensitive information to be redacted prior to publication.
Recommendations: consolidation, training, strategy
From our interviews with CSOs, we have identified three areas of opportunity for enhancing CSO use of spending data. The variety of projects and diversity of skills in the CSO community present an opportunity for knowledge consolidation and knowledge sharing. We see a need for training to address both technical issues and lack of awareness of the potential of open spending data. This knowledge-sharing and training must take place in the context of a diverse and inclusive coalition, particularly a multilingual one. Finally, CSOs would benefit from articulating a shared strategy for pressuring governments to publish data.
We have observed a great diversity of CSO experience and expertise using spending data. We believe it is time for CSOs working with public finance data to share this diverse knowledge with one another. While most of the projects considered in this study developed as tailor-made solutions to context-bound problems, adapting the available data to the focus of a particular project, each of these projects could be used as a template for future projects. We hope the case studies featured in this report facilitate such reuse of good ideas.
Alongside this diversity of expertise, we have found recurring problems with key skills in the data-wrangling toolkit, namely web scraping, PDF liberation, and FOI requests. Focused trainings around these skills would prove valuable in promoting uptake of public spending data. The appendix on the ecosystem of tools provides further details on recurring issues and how they can be tackled, providing concrete anchors for technical trainings.
Technical skills are not the only opportunity for training: there is also need for training in the concept of data-driven investigation itself. In some countries where high-quality spending data has been made available, we have found no journalists or CSOs making use of it. Despite Slovenia arguably having one of the leading models for publishing government transactions since 2011, this data has seen very little uptake, and despite the existence of the Cl@rity programme in Greece, we struggled to find interested CSOs or journalists. This issue requires further attention, but it suggests that more needs to be done to bring the potential of open spending data to the attention of civil society.
Raising awareness and sharing knowledge will not happen unless a concerted effort is made to overcome the anglocentric tendency of the existing training materials and discussion around open data. This report is no exception to this tendency: besides being written in English, it focused on those CSOs who were able to conduct interviews in English. Efforts must be made to conduct trainings in the native languages of those countries were open data advocacy is taking place. Translations must also be made of key resources such as training materials and fact sheets on data standards. This will be especially necessary—and challenging—in massively multilingual countries like India.
Our final recommendation is that CSOs arm themselves with a coherent strategy for pressuring governments to publish open data based around the promotion of best practices. Countries like Slovakia and the UK present clear examples of successful open data policies that could be used by CSOs in consultations with governments in plans for Open Government Partnership meetings. Best-practice examples like these are valuable rhetorical tools in the fight for disclosure.
CSOs developing such strategies would benefit from being aware of the wide variety of factors that lead to the release of better spending data. Recent improvements to spending transparency in Greece and Slovenia, for example, appear to be driven by factors not related to public pressure for transparency. Greece’s financial crisis has boosted centralised fiscal management, with the striking result that Greece has rapidly gone from one of the least to one of the most financially transparent European states. In Slovenia, the Supervizor project has been driven by a combination of a strong independent anti-corruption commission and access to pro-bono development resources to prototype and develop the models. Attention to the concrete historical conditions responsible for success documented in this report can help CSOs develop their approach to engagement with governments.