25.05.2016 Views

UNDP-Session2.1-lokanathan-160524

Create successful ePaper yourself

Turn your PDF publications into a flip-book with our unique Google optimized e-Paper software.

(Towards) Real-Time Responsiveness:<br />

Leveraging both new & tradi1onal data<br />

Sriganesh Lokanathan<br />

#2030NOW: Foresight & Innova;on for Sustainable Human Development<br />

Colombo, 24 May 2016<br />

Some of the work men;oned was carried out with the aid of a grant from the Interna;onal Development Research<br />

Centre, Canada and the Department for Interna;onal Development UK.


2


3


Data is essen;al to developing<br />

real-;me insight<br />

4


What’s the challenge with tradi;onal data?<br />

• Despite recent advances, exis;ng data has problems:<br />

– The cost means data is not collected frequently enough<br />

– Collected data is not released fast enough<br />

– Marginalized popula;ons are oYen not counted<br />

– The data is oYen not sufficient for the needs<br />

So where do we look for new data to give us<br />

better insights, more frequently and at higher<br />

spatial resolution?<br />

5


What is big data and where can you find it?<br />

What are the benefits?<br />

What are the challenges?<br />

What is needed to get towards real-?me<br />

responsiveness?<br />

6


What is big data and where can you find it?<br />

What are the benefits?<br />

What are the challenges?<br />

What is needed to get towards real-?me<br />

responsiveness?<br />

7


What is big data?<br />

• An all-encompassing term for any collec;on<br />

of data sets so large and complex that it becomes<br />

difficult to process using tradi;onal data processing<br />

applica;ons<br />

– Qualita;vely different from previous forms of data<br />

• Challenges include: analysis, capture, cura;on,<br />

search, sharing, storage, transfer, visualiza;on, and<br />

privacy viola;ons<br />

• Examples:<br />

– 100 million Call Detail Records are generated per day by<br />

by Sri Lanka mobile operators<br />

– 45 Terabytes of data from Hubble Telescope<br />

8


So where will the data come from?<br />

• Administra;ve data<br />

– E.g., digi;zed medical records, insurance records, tax records<br />

• Commercial transac;ons (transac;on-generated data)<br />

– E.g., Stock exchange data, bank transac;ons, credit card records,<br />

supermarket transac;ons connected by loyalty card number<br />

• Sensors and tracking devices<br />

– E.g., road and traffic sensors, climate sensors, equipment &<br />

infrastructure sensors, mobile phones communica;ng with base<br />

sta;ons, satellite/ GPS devices<br />

• Online ac;vi;es/ social media<br />

– E.g., online search ac;vity, online page views, blogs/ FB/ twifer posts<br />

9


Currently only mobile network big data has<br />

broad popula;on coverage<br />

Mobile SIMs/100 Internet users/100 Facebook users/100<br />

Myanmar 50 2 12<br />

Bangladesh 76 10 9<br />

Pakistan 73 14 11<br />

India 73 18 9<br />

Sri Lanka 107 26 16<br />

Philippines 112 40 41<br />

Indonesia 125 17 25<br />

Thailand 143 35 49<br />

Source: ITU Measuring Information Society 2015; Facebook advantage portal<br />

10


What can we do with mobile data?<br />

11


Spa;ally granular & high-frequency es;mates of<br />

popula;on density<br />

DSD population density from<br />

2012 census<br />

DSD population density<br />

estimate from MNBD<br />

Voronoi cell population<br />

density estimate from MNBD<br />

Source:LIRNEasia<br />

12


Popula?on density changes in Colombo region: weekday/ weekend<br />

Pictures depict the change in popula?on density at a par?cular ?me rela?ve to midnight<br />

Weekday<br />

Time 06:30<br />

Time 12:30<br />

Time 18:30<br />

Sunday<br />

Decrease in Density<br />

Increase in Density<br />

13


Finding displaced popula;ons post disaster<br />

The figure shows the<br />

number of people<br />

estimated to have<br />

been in Port-au-Prince<br />

(PaP) on the day of the<br />

2010 Haiti earthquake,<br />

but outside the capital<br />

19 days later. The<br />

circles represent the<br />

numbers of people<br />

who were displaced.<br />

This map was<br />

produced on the basis<br />

of mobile-network data<br />

to show the potential of<br />

big data in tracking<br />

population movements<br />

Source: Bengtsson, L., Lu, X., Thorson, A., Garfield, R., & von Schreeb, J. (2011). Improved<br />

response to disasters and outbreaks by tracking population movements with mobile phone network<br />

data: a post-earthquake geospatial study in Haiti. PLoS Medicine, 8(8), e1001083. doi:10.1371/<br />

journal.pmed.1001083<br />

14


Granular & high-frequency es;mates of poverty and<br />

socio-economic levels<br />

Lighter color denotes higher poverty<br />

Multidimensional poverty<br />

index (MPI) for 11 regions<br />

MPI estimated from diversity of<br />

communication diversity<br />

between 11 regions<br />

Much finer MPI estimation<br />

based on tower-to-tower<br />

communication diversity<br />

Source: Smith, C., Mashadi, A., & Capra, L. (2013). Ubiquitous Sensing for Mapping Poverty in<br />

Developing Countries Submitted to D4D @ NetMob 2013. In Data for Development: Net Mobi 2013.<br />

Cambridge, MA.<br />

15


We can develop new proxy measures of economic ac;vity<br />

Es;mated log(wage)<br />

High Es;mated<br />

Wage<br />

Low Es;mated<br />

Wage<br />

16


…and many more<br />

• Understand labor market mobility<br />

• Understand seasonal and long term migra;on<br />

• Iden;fy displaced communi;es post disaster<br />

• Model the spread of communicable diseases<br />

• Es;mate informal economic ac;vity<br />

• Facilitate financial inclusion for the unbanked<br />

through new proxy measures for “creditworthiness”<br />

• Geo-demographics<br />

These insights will come from ‘meshing’ data<br />

including extant official statistics, using new<br />

techniques<br />

17


Mobile network big data + other data è rich, ;mely insights<br />

that serve private as well as public purposes<br />

Mobile network big data<br />

(CDRs, Internet access<br />

usage, airtime recharge<br />

records)<br />

Construct Behavioral<br />

Variables<br />

1. Mobility variables<br />

2. Social variables<br />

3. Consumption variables<br />

Traditional data<br />

1. Data from Dept. of<br />

Census & Statistics<br />

2. Transportation data<br />

3. Health data<br />

4. Financial data<br />

5. Etc.<br />

insights<br />

Public purposes<br />

1. Transportation & Urban<br />

planning<br />

2. Crises response +<br />

DRR<br />

3. Health services<br />

4. Poverty mapping<br />

5. Financial inclusion<br />

18


What is big data and where can you find it?<br />

What are the benefits?<br />

What are the challenges?<br />

What is needed to get towards real-?me<br />

responsiveness?<br />

19


So what’s the real advantage of such new<br />

data for the data revolu;on?<br />

Characteris?cs<br />

High frequency<br />

High resolu;on<br />

Near real ;me<br />

(Almost) universal<br />

coverage<br />

Benefits<br />

• Can complement surveys and censuses with more frequent<br />

measurements (e.g. commu;ng and seasonal migra;on)<br />

• “New” findings (e.g. hourly popula;on density, informal<br />

economic ac;vity)<br />

• Insights at fine geographic resolu;on<br />

• Create near real-;me proxy indicators (e.g., economic<br />

ac;vity, socio-economic levels.)<br />

• Respond to real-;me events<br />

• Allow extrapola;on of surveys (e.g. income & expenditure<br />

surveys, labor force surveys) to larger samples<br />

• Capture informal (economic) ac;vi;es<br />

20


What is big data and where can you find it?<br />

What are the benefits?<br />

What are the challenges?<br />

What is needed to get towards real-?me<br />

responsiveness?<br />

21


Challenge: accessing government data<br />

• Data from government<br />

– Requires greater ‘datafica;on’<br />

– Data oYen sits in silos<br />

22


Challenge: accessing private sector data<br />

• Not amenable to open data ini;a;ves<br />

• There can be compe;;ve implica;ons to data<br />

release, even if anonymized<br />

Potential opportunities for accessing data:<br />

• The value of ‘mashing’ data (both private as well as<br />

government data) is greater than the sum of the parts<br />

• The state of the art is still developing enabling unique<br />

collaborations e.g. UN Global Pulse innovation labs,<br />

Orange Data for Development challenges, UP Singapore,<br />

LIRNEasia<br />

• There is a potential role for intermediaries that interface<br />

between private business interests and public uses e.g.<br />

ongoing work at LIRNEasia<br />

23


Challenge: new skills<br />

• Sta;s;cs can work differently in a big data world<br />

• Its not just about data science skills: domain<br />

knowledge equally important<br />

Opportunity:<br />

• New collaborations<br />

• Inter-disciplinary teams will be essential if we are<br />

to leverage such data<br />

24


Challenge: protec;ng privacy<br />

• The difficulty is that:<br />

– The higher the resolu;on of your insights, the greater the privacy<br />

implica;ons<br />

– Mixing non-personal data with other sources can reveal personal<br />

afributes<br />

– Informed consent is meaningless in a big data world<br />

– People oYen don’t know what their generalizable privacy needs are or<br />

how their preferences might evolve<br />

Potential solutions:<br />

• Privacy by design<br />

• Different levels of disaggregation of shared data<br />

• Technical solutions (e.g. anonymization, adding noise)<br />

• Evolution from a rights based approach to a harms based<br />

approach<br />

25


What is big data and where can you find it?<br />

What are the benefits?<br />

What are the challenges?<br />

What is needed to get towards real-?me<br />

responsiveness?<br />

26


What is needed<br />

• Breaking data silos in government<br />

• Private sector ‘Data philanthropy’<br />

• New partnerships to foster innova;ons<br />

– With government, private sector, academia, &<br />

non-governmental research ins;tutes<br />

27


Thank You.<br />

Sriganesh Lokanathan<br />

Team Leader – Big Data Research, LIRNEasia<br />

sriganesh@lirneasia.net<br />

hfp://lirneasia.net/projects/bd4d/<br />

28

Hooray! Your file is uploaded and ready to be published.

Saved successfully!

Ooh no, something went wrong!