Open Knowledge

Using technology in your work

As you move into more advanced data projects, you may find yourself in a situation where a research project turns into an effort that involves coders, designing databases and web sites. It is important that you take a step back and realize that you are now running not just an advocacy effort, but also an IT project.

There are many difficulties which CSOs face when developing software. Some common issues include:

  • Difficulties in finding qualified developers that want to contribute to your projects at a reasonable rate, as well as in the communication between CSO staff and developers.
  • Clearly communicating the requirements for the software so that both non-technical and technical staff share a vision for the outcome of the project.
  • The estimation of time and resources for particular tasks, especially how to handle projects that drastically overrun the timeframe and funding they were initially assigned.
  • Evaluating the work of developers to ensure that the product that has been delivered is according to what has been agreed, especially in small projects with only a single coder or when working with external contractors.
  • Maintaining the project after the main development period has finished

Starting from scratch vs. re-using Components

It is always going to be more costly and riskier to develop something from scratch than to customise something that already exists.

You should make the software behind your projects open source. If many other organizations do the same, this allows code to be reused across jurisdictions. Not only does this ease the financial burden, but it helps create the expectation in populaces around the globe for the high quality engagement tools that their neighbouring country has access to.

That's not to say you should never develop something new - just ask around first, and make sure that what you are asking is technically feasible.

Commissioning New Software

Purchasing software is more closely related to having a piece of clothing made than to buying chairs. You can give the designer a basic vision of what you would like, but you will always need to come back to make sure it really fits, and your thoughts may change when you see things in practice. If you have an arrangement with your tailor which allows you to first specify the general idea, and a couple of other appointments for fittings and trials, you'll probably end up with a better and more creative result than if you tried to design the whole thing and it was simply unveiled to you at the end. You'll also feel more in control and it may even be quicker to do design and implementation in parallel.

Defining requirements & Designing the solution

Define the basic components of your project and prioritise them by their importance. As the developers start working on one of these chunks, you can then break it up into more specific tasks based on your evolved understanding of the project. A popular technique for this purpose are user stories, small narrative pieces that describe each problem: "As a [web site visitor] I want to [be able to see a supplier's contracts] so that I can [understand what services they provided to government]". The key to these stories is that they describe the actual user need, not the details of the solutions that you have envisaged. While you should of course discuss those with the developers as well, defining solutions is mainly the job of the developers, not the project manager.

Implementation: iterations and milestones

A saying amongst developers goes: "Walking on water and developing software from a specification are easy if both are frozen." As your software project is progressing you will likely realize that the specifications you have given need to be revised or extended. Yet by modifying the requirements you are essentially shifting the ground on which developers are executing - meaning they will have to stop their work to adjust. To prevent such changes from freezing all development, the process of introducing changes and additional requirements needs to be structured.

Iterations are periods of a defined length - often two or four weeks - during which developers are tasked to execute a set of previously selected user stories or requirements. Before the iteration starts, developers have to pull in the work from a list of tasks (a so-called backlog) prepared by the project manager, committing themselves to delivering those tasks within the agreed period. Crucially, project managers are not allowed to extend or revise the scope of an iteration while it is ongoing (unless they want to declare it failed). This method ensures that changes are introduced in bulk and understood by the team. This approach mandates the opposite of the more common unstructured communication between managers and developers, e.g. emails with unsorted lists of change requests which tend to be ignored and lead to confusion.

Whenever you consider an additional requirement, be sure to consider if it is realistic within the resources you have available. "Scope creep", the progressive extension of a project during its development, is a common cause of project failure. By becoming more and more ambitious, the project finally ends up with no usable product at all. To avoid scope creep, make sure to have a storage area for long-term ideas. Also make sure that developers accept additional tasks through a pull process, and not by having them pushed into their workflow.

Maintenance

Make sure to budget a for ongoing maintenance after the end of your project. Who is going to guarantee that the servers stay online? Who is going to fix a typo? It is unlikely that your project will remain entirely static after its initial development, so you should have an explicit agreement with the developers regarding future support. It is also useful to collect feedback after the projects release to commission a small number of additional days when enough additional work has accumulated.

Roles and how to find developers

The key ingredient to a successful software development project is having the right people on staff or as contractors. Depending on the scope and type of your project, you may need a variety of skills - these are some of the common descriptions:

  • Web designers typically produce designs and layouts for web pages, often initially in a graphics program like Adobe Photoshop. Most, but not all, web designers then translate their designs into web markup (HTML, CSS).
  • Web developers are more technical. They produce interactive web interfaces such as search masks, browsers or specific form-based operations. They often use programming languages such as PHP, Ruby on Rails, Python or JavaScript.
  • Visualization designers develop graphics that represent quantitative information. A key distinction here is between non-interactive graphics (i.e. static images) and interactive visualizations, which often require some programming. There are still very few designers who design interactive visualizations, so rates may be relatively high.
  • Software developers are even more technical, developing backend software for data processing or acquisition. They are experienced in the use of database software (such as SQL databases) and programming languages such as Ruby, Python or Java.
  • Data scientists and statisticians produce analysis based on large sets of data, detecting tendencies and outliers in the dataset. They are not usually expected to produce front-end applications, but may produce software in the process of analysing data.
  • Usability experts and user experience (UX) designers think about the way your user will interact with your site answering questions such as 'is it obvious from the landing page what the purpose of this site is?'
  • Testers try and break things to test their robustness. This is particularly useful e.g. if you think your project will receive a lot of traffic as a result of a media campaign, you want to know your site can survive the hit

Good places to look for developers

The easiest way to meet developers is through community meetups, such as hackdays. During such events, coders meet up to cooperatively develop prototypes of new software. To meet volunteer developers who can help you make sense and unleash the power of government spending and budgets, it's wortwhile to investigate events such as Random Hacks of Kindness (http://www.rhok.org), Data Kind (http://datakind.org/) and TechCamps (http://techcampglobal.org/).

There are a few ways you can discover if there is a hackday in your area. One way is to search on Lanyrd (http://lanyrd.com/search/?q=hackday&context=future) or set up an account on that system and request that you are alerted when there is a hackday in your area. Another approach is join mailing lists of organisations that might help you find developers e.g. the Open Knowledge Foundation lists (http://lists.okfn.org/mailman/listinfo) or the Sunlight Labs mailing list (https://groups.google.com/forum/#!forum/sunlightlabs).

What do things cost?

It's impossible to give concrete guidelines on how much a project should cost. Developers' salaries are generally quite high for a country's average, but vary very strongly from country to country. Worried about your project spiraling out of control? We'd recommend agreeing on a price per iteration, and it may be a good idea to draw up a contract which allows you to break it off if you are not happy with the work at the end of an iteration. Plus, you can generally also find a friendly developer to glance over a quote from a company for a sanity check.