The quantity of data available is increasing at an exponential rate, a trend likely to continue in years to come. This explosion of data is driven partially by exponentially increasing computing power, lower costs and increased speed of broadband connections, but also by novel data sources such as sensors used by the Internet of Things, drones, nano-satellites and mobile phones.

Increasing affordability of devices and connectivity has spurred the near ubiquity of mobile phones and mobile networks. As of the end of 2016, approximately 95 per cent of the global population is now living in places covered by at least a 2G mobile signal.

The increasing quantities of data and novel data sources available are likely to have both positive and negative implications for development. On the one hand, more data and data sources provide us with increasing opportunities to understand social and natural phenomena. They allow us to measure and uncover things more accurately and faster than before. Moreover, these data sources provide more granular pictures allowing development practitioners to pinpoint exactly where efforts should be focused.

On the other hand, there are also potential negative implications requiring action to mitigate. Although increased data sources and new ways to gather data can make data samples more representative, we still do not live in a world where ‘digital’ data can be considered representative.

If adopted effectively by the development sector, new data sources and technologies could help overcome data gaps and allow development practitioners to gain a better understanding of contexts before, during and after interventions. However, digital data over-represents those already connected and more data does not automatically translate to use. The challenge is to ensure inclusiveness, especially in triggering data demand and use.

The availability of digital data from different sources and the emergence of new analytical tools has shifted the way we do data analysis and thus how we make sense of things. In the past, it was common to create new models based on pre-existing historical data, to get as much out of the data as possible. Today, there is so much recent data that the focal point has shifted to mining the data to figure out what it tells us.

New technology has made it easier to garner insights from troves of data at speeds previously unimaginable. This offers the opportunity to see things we were never able to see, to see them faster, to open things up to be seen and scrutinised by more people, as well as to democratise the way they are seen and what gets seen. These new ways of visualising problems allow us to understand issues in new ways and thus respond to them differently. Algorithms make it increasingly possible to uncover potential issues, predict when they are likely to happen, estimate their consequences, and prescribe solutions based on causal inferences.

The increased use of new types of technologies has both potentially positive and negative implications on the development sector. Because digital data is collected more continuously, information can be disseminated through more fluid visualisation techniques and visualise problems in new ways to allow development organisations to tackle them using novel methods. Also, digital data analytic tools can be used to induce agile adaptive programming. They provide development organisations with the ability to describe and understand situations better, predict what will happen, and prescribe improved actions based on data.

However, there are potential negative effects that require attention. For instance, many of the algorithms that help provide new ways of making sense of data were developed in the private sector. Because private companies seek to protect their intellectual property, the code that underpins how algorithms process data, analyse it and come to conclusions remains out of sight and non-transparent. A lack of algorithmic transparency can be especially worrisome when poorly constructed algorithms are underpinned by biased processes and data, which lead to biased results and decisions.

The complexity of challenges in development not only requires new data sources and tools, but also new actors and partnerships. In data for development, at least two kinds of partnerships have emerged, data philanthropy and data collaboratives. The term data philanthropy describes corporations that share their data for the public good, while data collaboratives share data assets and combine their expertise and/or tools to solve specific public problems. Non-traditional actors have also emerged, such as data and innovation labs, and data analytics companies, which increasingly play a role in the development sector.

On the one hand, the combination of specialised expertise and resources opens up new ways of addressing complex problems. It also challenges the current structure of the development enterprise as more stakeholders become involved in using data to achieve development outcomes. Joint platform projects make possible the productive conversations between and among different stakeholders who are habitually working in isolation with limited opportunities for data and information exchange; and this has opened up opportunities for future collaborative work.

On the other hand, it also raises a lot of issues. For example, one of the experts interviewed pointed to the possibility of blurring the lines between public and private initiatives that can potentially affect development outcomes, with private actors likely to be less transparent in the use of algorithms, for example. It also has the tendency to cause the unintended effect of creating silos of collaborators, potentially excluding others not within these circles.

Ethical considerations, in particular protection of privacy, are likely to receive growing attention in the coming years. Since legal frameworks that govern personal data protection are non-existent, outdated or poorly implemented in most developing countries, the challenge will be to either invest in the development or improvement of such regulatory frameworks or come up with alternative (legal) instruments. While the various responsible data and data privacy guidelines of organisations can inform and guide their individual efforts, a coordinated and inclusive approach to rule setting will be needed that involves a broad range of stakeholders to identify context-appropriate provisions.

There are at least two broad directions in which the trend could evolve over the next five years. The first sees the key players in the data for development domain realise that protecting individual privacy while using personal data to tackle development challenges is virtually impossible. In an extreme case, this realisation could result not only in a negative scenario of citizens in developing countries being used as ‘data farms’ for business interests and/or enhanced government control, but also as sources of data for development initiatives that have no intent to give people agency through a fair, agreed and equitable exploitation of the data they produce.

At the same time, when it comes to openness of government data, the protection of personal privacy might be used as an excuse to withhold relevant public sector data which could be used by citizens to advocate for better public service provision, to hold governments accountable or to tackle corruption in the public sector.

The second, arguably more positive, direction the trend could take features an inclusive multi-stakeholder approach to identifying workable solutions to privacy challenges. From mainstreaming organisational responsible data practices to institutionalising privacy guidelines and reworking entire regulatory frameworks, this work would need to be closely linked to investments on cyber security, digital rights and the protection of human rights online.

The challenge to achieve sustained and meaningful results, along with the struggle experienced when attempting to replicate programmes, scale them, or even just sustain them over time, have led to widespread awareness among the data for development community of the limitations of solely relying on quantitative data and the need to pay greater attention to contextual factors and local, granular – often qualitative – information.

A key driver of this trend is the growing evidence that show the limitations of data in capturing and explaining social phenomena. Data related to social interactions always carries some bias, because of what and who is included in the datasets and, more importantly, because of what and who is left out. Datasets are not just a reproduction of reality, but are affected by who gathered the data and how it was harvested.

Future data initiatives are likely to consider various data types, driven by the contexts and problems that need to be tackled. Data for development initiatives will increasingly be ‘human-centric’ and aim for long term empowerment of local communities, governments and civil society organizations, in ways that enable them to cooperate to use data to improve their situations. A bottom-up involvement will be required to achieve sustained impacts on the ground, which frequently involve incremental, slow changes in mindsets and institutional cultures.

Existing information inequalities are likely to be exacerbated in the coming years as some actors are in a better position than others to harness the opportunities arising from the greater availability of digital data. Because initial conditions are already unequal, the ability to fully participate in and benefit from the positive developments arising from data will be differentiated. There will surely be losers and winners in the process unless equalising conditions are present. Organisations, government agencies included, that have more resources at their disposal – expertise, financial capacity and sophisticated technology – will have more capacity to use data for their benefit, to the detriment of ill-resourced players and stakeholders.

A key driver of this trend is the current data ownership regime in which private companies collect vast amounts of data from customers and subscribers. They use these data assets not only to evaluate product offerings, but also to forecast new consumption patterns, define potential products and institutionalise marketing programmes based on buyer behaviour. Governments, on the other hand, also make these types of investments, but for an entirely different purpose. They collect personal information of citizens and engage in citizen identification schemes in order to manage public services, receive citizen feedback, forecast future social security spending, automate elections, deliver public goods, and plan settlements, production areas, and recreation facilities, among others.

Unless there is a drive to diminish this inequality, those with the capacity to handle large volumes of data will be in an advantageous position, while others will lose out. Data for development projects must consider the fundamental institutional structure, culture and priorities, with the end-view of changing incentive structures, power relations, and patterns of influence so that the poor and marginalised will have a voice in meaning-making and solution-building.