CONFERENCE PROCEEDINGS - IPSC - Europa

C O N F E R E N C E P R O C E E D I N G S
EUR 24948 EN - 2011

VALgEO 2011
Workshop Proceedings
JRC, Ispra, 18-19 19 October 2011
Edited by
C. Corbane, D. Carrion,
M. Broglia and M. Pesaresi

The mission of the JRC-IPSC is to provide research results and to support EU policy-makers in
their effort towards global security and towards protection of European citizens from accidents,
deliberate attacks, fraud and illegal actions against EU policies.
European Commission
Joint Research Centre
Institute for the Protection and Security of the Citizen
Contact information
Address: JRC - TP 267 - Via E. Fermi, 2749 - 21027 Ispra (VA), Italy
E-mail: marco.broglia@jrc.ec.europa.eu
Tel.: +39 0332 785435
Fax: +39 0332 785154
http://ipsc.jrc.ec.europa.eu/
http://www.jrc.ec.europa.eu/
Legal Notice
Neither the European Commission nor any person acting on behalf of the Commission
is responsible for the use which might be made of this publication.
Europe Direct is a service to help you find answers
to your questions about the European Union
Freephone number (*):
00 800 6 7 8 9 10 11
(*) Certain mobile telephone operators do not allow access to 00 800 numbers or these calls may be billed.
A great deal of additional information on the European Union is available on the Internet.
It can be accessed through the Europa server http://europa.eu/
JRC66899
EUR 24948 EN
ISBN 978-92-79-21379-3 (print)
ISSN 1018-5593 (print)
ISBN 978-92-79-21380-9 (PDF)
ISSN 1831-9424 (online)
doi:10.2788/73045
Luxembourg: Publications Office of the European Union
© European Union, 2011
Reproduction is authorised provided the source is acknowledged
Printed in Italy

CONTENTS
RATIONALE.................................................................................................................................................................................1
AGENDA .....................................................................................................................................................................................3
FORWARD ..................................................................................................................................................................................7
SESSION I ....................................................................................................................................................................................9
THE ROLE OF VALIDATION IN INFORMATION AND COMMUNICATION TECHNOLOGIES FOR CRISIS MANAGEMENT ...........9
Potential applications of tracking macro-trends within Crisis Management.................................................................11
The OpenStreetMap way to data creation and validation in emergency preparedness and response .........................13
Impact Opportunities and Methodology Challenges: Crisis Mapping and Geoanalytics in Human Rights Research....15
Web and mobile emergencies network to real-time information and geodata management. .....................................17
An Integrated Quality Score for Volunteered Geographic Information on Forest Fires .................................................29
SESSION II .................................................................................................................................................................................35
VALIDATION OF REMOTE SENSING DERIVED EMERGENCY SUPPORT PRODUCTS ...............................................................35
Definition of a reference data set to assess the quality of building information extracted automatically from VHR
satellite images ...............................................................................................................................................................37
On the Validation of An Automatic Roofless Building Counting Process .......................................................................47
Evacuation plans : interest and limits.............................................................................................................................55
Outside the Matrix, a review of the interpretation of Error Matrix results....................................................................57
On the complexity of validation in the security domain – experiences from the G-MOSAIC project and beyond .........69
SESSION III................................................................................................................................................................................71
USABILITY OF WEB BASED DISASTER MANAGEMENT PLATFORMS AND READABILITY OF CRISIS INFORMATION .............71
Emergency Support System: Spatial Event Processing on Sensor Networks ..................................................................73
Near‐real‐time monitoring of volcanic emissions using a new web‐based, satellite‐data‐driven, reporting system:
HotVolc Observing System (HVOS) .................................................................................................................................81
Image interpreters and interpreted images: an eye tracking study applied to damage assessment. ...........................83
Crisis maps readability: first results of an experiment using the eye-tracker ................................................................93
SESSION IV ...............................................................................................................................................................................95
TOWARDS ROUTINE VALIDATION AND QUALITY CONTROL OF CRISIS MAPS ......................................................................95
A methodological framework for qualifying new thematic services for an implementation into SAFER emergency
response and support services........................................................................................................................................97
A methodology for a user oriented validation of satellite based crisis maps...............................................................105
Quality policy implementation: ISO certification of a Rapid Mapping production chain.............................................107
AUTHORS INDEX ....................................................................................................................................................................109

RATIONALE
Over the past decade, the international community has responded to an increasing number of major
natural and man-made disasters. In parallel, the emergency management has become increasingly complex
and specialized due to the necessity for various authorities and organizations to cooperate during emergency,
and to the emergence of disasters of an unexpected or unknown nature. With these growing challenges, the
need for more sophisticated tools for the production, sharing and integration of geospatial information
without prejudice to the usability of end-user products, has given rise to a rapid development of geoinformational
technologies to assist in crisis management operations. Recent events such as the earthquake in
Japan, the flooding in Australia and the crisis in the Middle East and North African region showed that Earth
observation, ICT and Web-mapping technologies are now playing a vital role in crisis management efforts,
especially during the preparedness and response phases.
No matter what the origin of crises, their geographical context and the dimension of their impacts, there is a
common need by all actors involved in crisis management for timely, relevant, usable and most of all reliable
information. For the community concerned with validation of geo-information, this poses new challenges in
terms of having access to methodologies that can address the increasing variety and amount of data, and that
help to render validation closer to a routine process.
Following the two successful VALgEO workshops held in 2009 and 2010, we are pleased to announce the
organization of the 3rd edition of the international workshop on validation of geo-information for crisis
management. The annual VALgEO workshop sets out to act as an integrative agent between the needs of
practitioners in situation centers and in the field guiding the Research and Development community, with a
special focus on the quality of information.
The following topics will be addressed in four main sessions:
• The role of validation in Information and Communication Technologies (ICT) for crisis
management
• Validation of Remote Sensing derived emergency support products
• Usability of Web based disaster management platforms and readability of crisis
information
• Towards routine validation and quality control of crisis maps
1

Workshop chair
Martino Pesaresi, Joint Research Centre, Italy
martino.pesaresi@jrc.ec.europa.eu
Organizing committee
Christina Corbane, Joint Research Centre, Italy
Daniela Carrion, Joint Research Centre, Italy
Marco Broglia, Joint Research Centre, Italy
Barbara Secreto, Joint Research Centre, Italy
christina.corban@jrc.ec.europa.eu
daniela.carrion@jrc.ec.europa.eu
marco.broglia@jrc.ec.europa.eu
barbara.secreto@ec.europa.eu
Scientific committee
Michael Judex, German Federal Office of Civil Protection, Germany
Daniel Stauffacher, ICT4Peace Foundation, Switzerland
Peter Zeil, University of Salzburg, Austria
Tom De Groeve, Joint Research Centre, Italy
Marco Broglia, Joint Research Centre, Italy
Daniela Carrion, Joint Research Centre, Italy
Christina Corbane, Joint Research Centre, Italy
michael.judex@bbk.bund.de
daniel_stauffacher@ict4peace.org
peter.zeil@sbg.ac.at
tom.de-groeve@jrc.ec.europa.eu
marco.broglia@jrc.ec.europa.eu
daniela.carrion@jrc.ec.europa.eu
christina.corban@jrc.ec.europa.eu
2

AGENDA
TUESDAY, October 18 th 2011
9:00 Workshop Opening and Welcome Address
Delilah Al Khudhairy – Joint Research Centre (Head, Global Security and Crisis Management Unit)
Martino Pesaresi- Joint Research Centre (ISFEREA Action Leader)
9:40 Torsten Redlinger- European Commission (DG ENTR, GMES Bureau)
SESSION I– THE ROLE OF VALIDATION IN INFORMATION AND COMMUNICATION TECHNOLOGIES (ICT) FOR
CRISIS MANAGEMENT
Chair: Daniel Stauffacher. ICT4Peace Foundation, Geneva, Switzerland
10:00 Invited Speaker: Daniel Stauffacher. ICT4Peace Foundation
10:20 Potential applications of tracking macro-trends within Crisis Management
Invited Speaker: Douglas Hubbard. Hubbard Decision Research, United States
10:40 The OpenStreetMap way to data creation and validation in emergency preparedness and response
Invited Speaker: Nicolas Chavent, Open Street Map, France
11:00 Coffee Break
11: 10 Impact Opportunities and Methodology Challenges: Crisis Mapping and Geoanalytics in Human
Rights Research
Scott Edwards & Koettl C. – George Washington University/Amnesty International
11:30 Web and mobile emergencies network to real-time information and geodata management
Elena Rapisardi 1-3 , Lanfranco M. 2-3 , Dilolli A. 4 & Lombardo D. 4
1
Openresilience
2
Doctoral School in Strategic Sciences, SUISS, University of Turin
3
NatRisk, Interdepartmental Centre for Natural Risks, University of Turin
4
Vigili del Fuoco, Comando Provinciale di Torino
11: 50 An Integrated Quality Score for Volunteered Geographic Information on Forest Fires
12:10 Lunch
Ostermann Frank & Spinsanti L.
European Commission, Joint Research Centre, SDI Unit, IES
3

SESSION II – VALIDATION OF REMOTE SENSING DERIVED EMERGENCY SUPPORT PRODUCTS
Chair : Dirk Tiede , University of Salzburg, Austria
13:40 Recent experiences on the use of remote sensing for damage assessment and its validation: 2010
Pakistan flood, 2011 Tohoku Tsunami and 2011 (February) Christchurch earthquake
Keiko Saito 1,6 , R. Eguchi 2 , G. Lemoine 3 , L. Barrington 4 , J. Bevington 2 , S. Gill 5 , A. King 6 , A. Lin 4 , M. Green 7 ,
R. Spence 1 & P. Wood 8
1
Cambridge Architectural Research Ltd, UK - 2 ImageCat Inc, USA ; ImageCat Ltd, UK- 3 Joint Research
Centre, EU - 4 Tomnod, USA- 5 Global Facility for Disaster Risk Reduction, The World Bank- 6 GNS, New
Zealand - 7 EERI, USA - 8 New Zealand Society for Earthquake Engineering
14:00 Definition of a reference dataset to assess the quality of building information extracted
automatically from VHR satellite images
Annett Wania, Kemper T., Ehrlich D., Soille P. & Gallego J.
Joint Research Centre
14:20 On the Validation of An Automatic Roofless Building Counting Process
Lionel Gueguen, Pesaresi M. & Soille P.
Joint Research Centre
14: 40 Evacuation plans : interest and limits
Alix Roumagnac & Moreau K.
PREDICT Services
15: 00 Outside the Matrix, a review of the interpretation of Error Matrix results
Pablo Vega Ezquieta 1 , Tiede D. 2 , Joyanes G. 1 , Gorzynska M. 1 & Ussorio A. 1
1
European Union Satellite Centre- 2 Z-GIS Research, University of Salzburg
15: 20 On the complexity of validation in the security domain – experiences from the G-MOSAIC project
and beyond
Thomas Kemper, Wania A. & Blaes X.
Joint Research Centre
15: 40 Coffee Break
SESSION III– USABILITY OF WEB BASED DISASTER MANAGEMENT PLATFORMS AND READABILITY OF CRISIS
INFORMATION
Chair : Tom De Groeve, Joint Research Centre
15:50 Emergency Support System: Spatial Event Processing on Sensor Networks
Roman Szturc, Horáková B., Janiurek D. & Stankovič J.
Intergraph CS
16:10 Near-real-time monitoring of volcanic emissions using a new web-based, satellite-data-driven,
reporting system: HotVolc Observing System (HVOS)
Mathieu Gouhier 1 , Labazuy P. 1 , Harris A. 1 , Guéhenneux Y. 1 , Cacault P. 2 , Rivet S. 2 & Bergès J-C. 3
1
Laboratoire Magmas et Volcans, CNRS, IRD, Observatoire de Physique du Globe de Clermont-Ferrand,
Université Blaise Pascal- 2 Observatoire de Physique du Globe de Clermont-Ferrand, CNRS, Université
Blaise Pascal - 3 PRODIG, UMR 8586, CNRS, Université Paris 1
16:30 1- Image interpreters/interpreted images, study of recognition mechanisms
2- Crisis maps readability: first results of an experiment using the eye-tracker
Roberta Castoldi, Carrion D., Corbane C., Broglia M. & Pesaresi, M.
Joint Research Centre
16: 50 Closure of DAY 1
20:00 Social dinner at Hotel Belvedere
4

WEDNESDAY, October 19 th 2011
SESSION IV–TOWARDS ROUTINE VALIDATION AND QUALITY CONTROL OF CRISIS MAPS
Chair: Michael Judex, German Federal Office of Civil Protection, Germany
9:00 Validation, standardisation, innovation and ability to respond to user requirements
Joshua Lyons - UNITAR/UNOSAT
9:20 A methodological framework for qualifying new thematic services for an implementation into
SAFER emergency response and support services
Hannes Römer , Zwenzner H. , Gähler M. & Voigt S.- German Aerospace Center (DLR)
9:40 A methodology for a user oriented validation of satellite based crisis maps
Michael Judex 1 , Sartori G. 2 , Santini, M. 3 , Guzmann, R. 3 , Senegas, O. 4 & Schmitt, T. 5
1
Federal Office of Civil Protection and Disaster Assistance, Germany- 2 World Food Programme, Italy-
3
Dipartimento della Protezione Civile- 4 United Nations Institute for Training and Research (UNOSAT)-
5
Ministère de l’intérieur, Direction de la Sécurité Civile
10:00 Quality policy implementation: ISO certification of a Rapid Mapping production chain
Bernard Allenbach 1 , Rapp JF. 1 , Fontannaz D. 2 & Chaubet JY. 3
1
SERTIT- 2 CNES - 3 APAVE
10:20 Introduction to the LIVE EXERCISE AT THE CRISIS ROOM
10:30 Coffee break & transfer to the crisis room
SESSION V- LIVE EXERCISE AT THE CRISIS ROOM
Coordinator: Alessandro Annunziato, Joint Research Centre, Italy
11:00 Presentation of crisis room and related tools
Tom de Groeve & Galliano D.
European Commission, Joint Research Centre
11:20 Simulation of a case study
All
12:30 Lunch
SESSION VI- WRAP UP SESSION
14:00 Panel discussion
Scientific committee of VALgEO 2011
15:00 Recommandations and conclusion
All
5

SESSION V- LIVE EXERCISE AT THE CRISIS ROOM
Coordinator: Alessandro Annunziato, Joint Research Centre, Italy
11:00 Presentation of the crisis room and the related tools
Tom De Groeve & Alessandro Annunziato
European Commission, Joint Research Centre (JRC)
11:10 Presentation of collected data & discussion of interoperable mobile applications for data collection
iphone application: Beate Stollberg (JRC)
Field Reporting tool: Daniele Galliano (JRC)
11:30 Presentation and demonstration of the PDNA suite
Daniele Galliano (JRC)
11:45 End user tailored interface application for collaboration in GIS environment - solution example
Michał Krupiński (Space Research Centre Polish Academy of Sciences)
Piotr Koza (Astri Polska)
12:00 IQ demonstration for automatic image information extraction
Lionel Gueguen & Vasileios Syrris (JRC)
12:20 Questions and Answers
12:30 Lunch
6

FORWARD
This third edition of the international workshop on validation of geo-information for crisis
management confirms that the topics we are addressing are important and that we need this sort of platform
to regularly discuss the new challenges we face as a result of continuous evolution in technology, especially ICT
(including space), which is impacting both the quality and quantity of information relevant to crisis
management.
2010 marks the start of a new era in the way ICT is beng used in crisis management. I would like to begin with
the Haiti earthquake in 2010 which even though it was not a typical disaster, it marked a new epoch in the way
various novel and traditional ICT solutions were used in an integrated manner by the emergency response
communities as well as professional organizations and voluntary intiatives. But it also confirmed that we have
yet to apply lessons learned from past major disasters. The rapid advances in ICT, including space, are not
necessarily facilitating the work of the international humanitarian relief, emergency rescue and post-disaster
recovery/reconstruction communities. On the contrary, today we are facing an increasing deluge of
information, with the risk that only a small fraction of it is relevant to, or reliable enough for, effective crisis
management.
Sanjana Hattotuwa (2010) captures this helplessness very well in the quotation “Where is the knowledge we
have lost in information?”
In Haiti alone, there were hundreds of email messages exchanged amongst the disaster response community
and hundreds of information products in the form of maps being produced by various entities on a daily basis.
How much rich and relevant knowledge was present in the deluge of information which cost significant
amount of resources to produce and disseminate….? Can we measure this?
Furthermore, Haiti showed that in addition to the contributions of traditional communities engaged in crisis
response, the citizens of impacted countries were also making potentially important contributions, thereby
shifting the balance we have become familiar with from impacted communities that are at the receiving end of
assistance and information, to one in which the impacted communities are empowered and are becoming
increasingly responsible for themselves through actively engaging in the crisis management process. It is only a
matter of time when the type of community engagement we saw in Haiti will become familiar as opposed to
exceptional.
The crowd sourcing/social media and collaborative analysis and mapping technologies we saw being used in
Haiti at different levels mark the beginning of a new era in crisis management. The era of “citizen or
community crisis management”.
Time will tell if this new marriage between technology and the community will have a sustainable future. Some
experts reckon it will take a decade. And by then, we could expect to live in a world whereby citizens will
become important sources of local and regional human observations who can supply information to the
disaster response communities that are not readily met by increasingly improved remotely sensed data
including space and airborne. Equally important, we expect that citizens will likewise become increasingly
responsible for guiding their recovery and reconstruction. I agree with the predictions of these experts.
Moreover I think there will be a future between technology and communities in crisis management. In a
decade, children born between the 1990s and 2000s will be between their late teens and late 20s. These
young adults would have grown up in a world where the digital camera and the internet are things that have
7

always been there. So, they will be equipped to contribute and participate in a future, where they will be able
to contribute directly to help themselves in the event of a crisis. This is something we, as an older generation,
can only begin to imagine and contemplate.
Today the crisis management community is living in an increasingly complex information and technological
world. Interactive and real-time type platforms are edging in on static maps, but many challenges and
problems will have to overcome before the static maps with which we have become familiar in crisis
management become relics of the past. We cannot afford to have less effective responsiveness to disasters,
whilst advances in technologies and the way they are being used outweigh the benefits they can bring.
In other words, today, we face important questions such as: Are rapid advances in ICT and new ways of using
ICT helping us make improvements with regard to producing relevant and trusted information and making it
available in a reliable and timely manner to the stakeholders engaged in crisis management? Not necessarily.
For effective crisis management we do not need a fast growing and enormous amount of information available
through a variety of media. With the increasing number of information sources and contributing actors
(specialists, citizens, mapping and ICT volunteers) in crisis management, we risk even less knowledge and value
at the expense of more information. Validation and trusted analysis have become more critical than before to
creating value and trust in knowledge in crisis management.
This is why we need a platform like VALgEO to bring us together to discuss the elements of validation that will
result in value and trust in knowledge for crisis management. But in our discussions during this workshop and
subsequent ones we already have to think 10 years ahead. We have to discuss and identify the ‘validation and
trust’ elements that accommodate not only the traditional geo-information products and information sources
that are being used by today’s crisis management communities, but we have to already think ahead in terms
of the eventual regular use of new information sources such as the citizen as well as community / participatory
ICT and mapping volunteers. Without agreed principles and standards related to validation and trusted
analysis, we risk having even less content and trusted knowledge in the future at the expense of yet ever more
tools, services and technologies, to the dis-benefit of the disaster affected communities and the disaster
response and post-response communities.
The VALgEO community which you have helped to establish at our first and second workshops and through
your participation this year, can make important steps in developing recommendations for agreed principles
and standards as well as other important components of validation in a new crisis management information
landscape in which the information sources are extending beyond traditional remote and in-situ sources, and
in which the community or the citizen will become increasingly engaged.
We look forward to a vibrant and exciting workshop, and we are optimistic that we, the VALgEO Community,
can make progress both at this year’s workshop and in future workshops towards achieving these goals. We
are entering a very exciting period in which we have never had it so good in terms of the variety of sources and
technologies which can be used to produce and disseminate crisis relevant information and knowledge. Let us
now take the time to reflect and understand this landscape in order to come up with recommendations and
principles that will benefit the crisis management process in the longer-term.
AL-KHUDHAIRY D.
Head, Global Security and Crisis Management Unit
European Commission - Joint Research Centre, Institute for the Protection and Security of the Citizen (IPSC)
8

SESSION I
THE ROLE OF VALIDATION IN INFORMATION AND
COMMUNICATION TECHNOLOGIES FOR CRISIS
MANAGEMENT
Chair: Daniel Stauffacher
The contemporary global crisis management environment is increasingly relying on ICT
as a source for critical, timely decision-making information. Effective crisis management requires
not only quick decisions for an immediate response but most of all a co-ordinated reaction. In
order to reach a coherent action at all levels, crisis management organizations need to rely on
accurate information that must be produced, transmitted and shared with speed and precision.
This places the challenges of ICT less on technical capacities rather than on the effective
management and integration of an optimum amount of quality information.
The challenge of ICT for crisis management today is in building trust in both the systems used to
process the information and the people handling it. Validation and multiple checking of
information flows are therefore essential to avoid the risk of having less knowledge at the
expense of more information. The workshop aims to i) assess the needs for a formal validation
within ICT for crisis management, ii) help in understanding the attitudes of the end-users towards
these technologies and finally iii) define an agenda for research on valid methods and measures
to assess the quality and accuracy of the information.
9

ABSTRACT
Potential applications of tracking macro-trends within Crisis
Management
HUBBARD D.
Hubbard Decision Research
dwhubbard@hubbardresearch.com
Abstract:
The objective of this workshop is to have a discussion exploring how crisis management might
benefit from adding other macro-trend tracking to geolocation data. Douglas Hubbard, the author
of Pulse: The New Science of Harnessing Internet Buzz to Track Threats and Opportunities will
facilitate a discussion about how tools like Google Trends, Facebook, and Twitter might be used to
track trends relevant to crisis management including when does not include specific geolocation
data. Methods of analyzing social networks have been developed that would not only track but
forecast the transmission of disease throughout a population. Just as Twitter and Facebook helped
to mobilize social unrest in the Middle East, they can also be used to forecast social upheavals
before they become a humanitarian crisis. The possibility exists for more elaborate models that
use multiple data sources that could actively track a macroscopic picture of certain kinds of risks.
11

ABSTRACT
The OpenStreetMap way to data creation and validation in
emergency preparedness and response
CHAVENT N.
Open Street Map, France
nicolas.chavent@gmail.com
Abstract:
This presentation will look back at past activations of the Humanitarian OpenStreetMap Team
(HOT) since the Haiti Earthquake January 2010 featuring remote and on-the-ground work of the
OpenStreetMap (OSM) project in the context of emergency preparedness and emergency response
to discuss how this wiki approach to geodata management had been and is currently addressing
“the challenge of ICT for crisis management today [which] is in building trust in both the systems
used to process the information and the people handling it”.
This discussion will feature the following elements:
• The HOT/OSM approach to geodata creation in emergency preparedness and response
to be a source for “critical, timely decision-making information”.
• The way that HOT/OSM ensure coordination with the humanitarian system through the
emergency preparedness and response cycle to help contributing to crisis management as a coordinated
reaction.
• The typology of validation flows emerging from past operational contexts depending on
the intensity of the remote mapping work, the strength of the local OSM groups, the level of
coordination and interaction between OSM (remote and on the ground) and the humanitarian
response system.
We feel that the analysis of those use cases of the OSM work in emergency preparedness and
response are likely to contribute in a significative manner to the goals of the workshop
i) assess the needs for a formal validation within ICT for crisis management,
ii) help in understanding the attitudes of the end-users towards these technologies and finally
iii) define an agenda for research on valid methods and measures to assess the quality and
accuracy of the information.
13

ABSTRACT
Impact Opportunities and Methodology Challenges:
Crisis Mapping and Geoanalytics in Human Rights Research
EDWARDS S. 1 and KOETTL C. 2
1 George Washington University/Amnesty International, USA
2 Amnesty International, USA
sedwards@aiusa.org
Abstract
Crises are inherently complex—with the intertwining of multiple interdependent causal processes
and emergence of properties at differing levels of societal aggregation. This complexity is especially
challenging when crises are approached from a right-based perspective. As in disaster relief, the
ability to source timely, geo-referenced information in human rights emergencies provides—at
minimum—critical situational awareness for researchers in the midst of great need, and
overwhelming complexity.
Further—and based on cursory evaluation instances of web-based crowd maps—it is likely that
these tools offer the ability to capture representative human rights data above current legal
research methodologies, in many contexts. By layering crowd-derived events data into GIS analytic
products, human rights researchers and advocates may demonstrate the constituent elements of
grave crimes, such as qualities of “widespread” or “systematic” in the case of Crimes Against
Humanity. Additionally, the layering of events data into analytic tools can allow human rights
researchers to offer policy recommendations with greater technical specificity, and thus with
greater effect.
In the context of human rights research, this paper will evaluate current opportunities and
challenges as it relates to the integration of mapping and GIS research and analytic tools
increasingly used in disaster response. Challenges related to the verification of events data entail—
for most human rights organizations—serious risk to the credibility of reporting, and thus to policy
impact. These and related challenges will be explored, as well as analytic measures that can be
employed to minimize them, particularly in the context of crisis.
15

SHORT PAPER
Web and mobile emergencies network to real-time information
and geodata management.
RAPISARDI E. 1-3 , LANFRANCO M. 2-3 , DILOLLI A. 4 and LOMBARDO D. 4
1 Openresilience, http://openresilience.16012005.com/
2 Doctoral School in Strategic Sciences, SUISS, University of Turin, Italy.
3 NatRisk, Interdepartmental Centre for Natural Risks, University of Turin, Italy; www.natrisk.org.
4 Vigili del Fuoco, Comando Provinciale di Torino, Italy.
e.rapisardi@gmail.com
Abstract:
Major and minor disasters are part of our environment. The challenge we all have to face is to
switch from relief to preparedness. Recent events from Haiti to Japan revealed a new scenario:
web and mobile technologies can play a crucial role to manage the disasters, increasing and
improving the information flow between the different actors and players - citizens, civil protection
bodies, local and central governments, volunteers, media. In this perspective, “the post-Gutemberg
revolution” is changing our communication framework and practices. Mobile devices and advanced
web data management may ameliorate preparedness and boost crises response in the shadow of
natural and man-made disasters, and are defining new approaches and operational models. Key
words are: crowdsourcing, geolocation, geomapping. A full integration of web and mobile solutions
allows geopositioning and geolocalization, video and photo sharing, voice and data
communications, and guarantees accessibility anytime and anywhere. This can also give the direct
push to set up an effective operational dual side system to “inform, monitor and control”. Starting
from the international experiences, Open Resilience Network and Italian Firefighters have carried
out tabletop and full scale exercises to test tools and procedures and experiment the use of new
technologies to better manage information flow from/to different actors. The paper will focus the
ongoing experimental work on missing person emergency, led by Italian Firefighters TAS team -
Andrea Di Lolli and Davide Lombardo - and supported by a multi-competences team from Open
Resilience Network and University of Turin - Elena Rapisardi and Massimo Lanfranco. The aim of
the paper is to share methods and technologies used, and to show the operational results of the
exercise carried out during PROTEC2011, in order to stimulate comments that will be taken into
account in the further research steps.
Keywords: missing person, disaster relief, crowdsourcing, geolocation, geomapping
17

1. Introduction
Disasters preparedness and relief operations have been widely studied and debated in the last 20 years.
“At risk”, edited by Ben Winser (Winser et al., 1994), expands the disaster consequences management to the
preemptive measures linked to social vulnerability, switching from a “war against nature” (hazard reduction)
to a “fight against poverty” (risk reduction), that the same year led UNDP to the human security concept
introduced in the Human Development Report (UNDP, 1994).
Quarantelli (1998) drafted a comprehensive review of previous works, implementing the technical point of
view with a sociological approach that lead to a full spectrum approach to Disaster Risk Management.
9/11 Twin Towers attack boost and refreshed studies on disasters: the “war against terror” it’s a new paradigm
that remind the “fight” against natural disasters (struggling the effects rather than the root causes), but some
authors (Alexander, 2001) pointed out that effects management of natural and anthropogenic disasters have
the same operational needs and procedures.
On the other hand, also the well defined “disaster cycle” (fig. 1) has been investigated by sociological
approach, that led to the community based risk reduction and the resilience concept. These concepts fit well
with UN efforts to overrun the simple humanitarian relief, which became more and more costly in last 10
years.
Figure 1. The disaster cycle: a life long work to web/mobile technologies
Web access and mobile devices seem to be the key for achieving all the goals that scholars and practitioners
were debating in the last 20 years at global and local levels:
- Citizens engagement in preparedness, planning, relief, rebuilding;
- Faster relief with improved situational awareness;
- Communication strategy with a Bottom/Up and Top/Down merge (two way data exchange);
- Resilience enhancing with local storytelling.
UN Foundation (HHI, 2011) points up mobile technologies involvement during Haiti Earthquake, drawing the
state of the art situation.
Since early 2000, the “GeoSITLab” (GIS and Geomatics Laboratory) at the University of Turin started to
enhance the application of Geomatics technologies for geothematic mapping and for geological and
geomorphological field activities (Giardino et al., 2004). These activities were implemented at NatRisk
Interdepartmental Centre (natural risks) and at Strategic Sciences School (man-made risks) with different
approach relates to “natural sciences” and “social” approaches.
18

In the shadows of Haiti earthquake, GeoSITLab developed a mobile GIS application based on ArcPad software
for direct mapping and damages assessing with smartphones and deliver it on the ground with AGIRE NGO
(Giardino et al., 2010). Data collected by NGO operators in Haiti were immediately transmitted to Italian
Operational Centre for retrofitting / rebuilding cost evaluation and donors search.
OpenResilence, whose members started working in VGI with Ushahidi and Crisis Mappers Net, offer to
professional and practitioners of forest fire fighting the next step, meshing mobile technologies and Webmapping
2.0 (http://openforesteitaliane.crowdmap.com/) .
The aim of our research is to come up with ideas that should link and connect governmental emergency
operators and citizens (Civil Protection 2.0), both on the side of collaborative mapping (data exchange) and
information dissemination (http://www.slideshare.net/elenis/protec-informing-the-public).
2. The Talent of the Crowd in face of emergency and disasters
In 1455 the Gutemberg revolutionary printing system changed the institutionalized information model and
lowered the production costs, increased the books production, favored the democratic access to knowledge,
stimulated literacy and contributed to the critical thinking.
“For more than 150 years, modern complex democracies have depended in large measure on an industrial
information economy for these basic functions. In the past decade and a half, we have begun to see a radical
change in the organization of information production. Enabled by technological change, we are beginning to
see a series of economic, social, and cultural adaptations that make possible a radical transformation of how
we make the information environment we occupy as autonomous individuals, citizens, and members of
cultural and social groups.” (Benkler, 2006).
In this scenario, we are individuals with multiple and crossing socio-cultural-economic memberships, where
information could be seen as the channel of the Simmel’s “Intersection of Social Circles”; a sociological
concept, that in some ways Google+ recently transformed in a social media, with a distinctive approach with
respect to Facebook and Twitter.
The first Web 2.0 Conference, on October 2004, could be taken as the turning point to a new approach to the
information: Web 2.0 (O’Reilly, 2007) introduced a set of principles and practices that tie together a veritable
solar system of sites, where the first one principle was: “The web as platform” [Tim O’Reilly].
This stream of thoughts and actions proposes a new approach that consider the collective
knowledge/intelligence as superior to the single individual knowledge/intelligence. Web 2.0 radically changed
the basis and the ways in which information is created, spread and consumed. In the post-Gutemberg
revolution “with advances in technology, the gap between professionals and amateurs has narrowed, paving
the way for companies to take advantage of the talent of the public.” [Darren Gilbert].
Apart from the light and shadows of the “social media” success, we can state that the post-Gutemberg
revolution is “The end of institutionalised mediation models” [Richard Stacy], and the crowdsourcing as a
participatory approach.
#share, #collaborate, #communicate, #cooperate, #support, #include - e.g. #diversity.
Key words that would be appreciated by the society models of the utopian socialism the first quarter of the
19th century. In 2011 Web 2.0 has become an everyday reality, web 2.0 has an impact also in emergency and
disaster response.
When a disaster or an emergency occurs, it is crucial to collect and analyze volumes of data and to distil from
the chaos the critical information needed to target the rescue mission most efficiently.
19

Since the Haiti earthquake, a completely new “engagement” took place “For the first time, members of the
community affected by the disaster issued pleas for help using social media and widely available mobile
technologies. Around the world, thousands of ordinary citizens mobilized to aggregate, translate, and plot
these pleas on maps and to organize technical efforts to support the disaster response. In one case, hundreds
of geospatial information systems experts used fresh satellite imagery to rebuild missing maps of Haiti and plot
a picture of the changed reality on the ground. This work—done through OpenStreetMap—became an
essential element of the response, providing much of the street-level mapping data that was used for logistics
and camp management.” (HHI, 2011).
“Without information sharing there can be no coordination. If we are not talking to each other and sharing
information then we go back 30 years.” [Ramiro Galvez, UNDAC].
This is a clear and undoutable effect of the post- Gutemberg revolution on the emergency and crisis response,
that is leading to the creation of Volunteer and Technical Communities (VTCs) working on disaster and conflict
management. This 2.0 world wide community is allowing the setting up of technical development community
and operational processes/procedures, that are changing risk and crisis management as focused on “citizens as
sensors” and on “preparedness”. On the other hand, the VTCs communities are now facing the issue of trust
and reliability of a wide information flow involving the “crowd” and the emergency bodies.
3. Italian Civil Protection system
Italian Civil Protection National Service is based on horizontal and vertical coordination of central and local
bodies (Regions, Provinces, municipalities, national and local public agencies, and any other public and private
institution and organisation). One of the backbone of the Italian Civil Protection System are the civil protection
volunteering organizations, whose duties and roles differ on regional basis. The Civil Protection Volunteers are
called to action during small emergencies and major disasters. The Abruzzo earthquake, in 2009, highlighted
the need of a more efficient communication flow between the volunteers organizations and professionals, and
of common shared protocols and tools to manage information. As a matter of fact, the “diversity” in managing
information causes a sort of “friction” and a weak collaboration in terms of data and information sharing.
Despite the adoption of softwares and device (radio), there is a low level of awareness on how the web 2.0
usage, in line with the web 2.0 litteracy of the internet population. The mobile phones and web penetration
(Italy has the European record for mobiles per capita with 122 phones for 100 inhabitants, 70% of population
with internet access, 13% of population with mobile internet access) and the social network “fever”, can be
considered as a driving factor to raise awareness and develop skills so to allow a wider adoption of web 2.0
solutions and tools. Moreover the volunteers organisation have to cope with small budgets that should
include equipments first. In this perspective the free and open tools (e.g. android market, content sharing
platforms) are a concrete opportunity to increase the web 2.0 penetration and develop acknowledged
practices to implement web 2.0 information sharing in C3 activities (Command, Control, Communications).
Fire and rescue services are provided by Vigili del Fuoco (VVF - Fire Fighters), a national government
department ruled by Ministry of Interior. Territorial divisions are based on provincial Fire Departments with
operational teams at biggest municipal level. Fire Fighters are also the primary emergency response agency for
HAZMAT and CBRN accidents.
According to the national legal framework, fire and rescue departments have the duty to operate as first
responders under a well-defined command structure providing 24-hour emergency response. Unlike law
enforcement, which operates individually for most duties, fire departments operate under a highly organized
20

team structure with the close supervision of a commanding officer. Fire departments also act at the direction
of the Prefect (Ministry of Interior local coordinator) during major disasters.
Fulltime professional personnel staff fire and rescue departments but volunteers provide reinforcement at
minor municipality’s stations.
Recently, after a big failure in procedures for search of a kidnapped girl, Fire Fighters were assigned to the
overall coordination of search for missed persons.
TAS Teams (Topografia Appicata al Soccorso - Topography Applied to Rescue) were set up during L’Aquila
Earthquake (April 2009) to support relief operation and damage assessment, through the use of GIS
technology. The TAS teams work to coordinate Fire and Rescue teams from Operational Room (SO115) and to
guide tactical activities from a mobile Incident Command Post (UCL - Unità di Comando Locale – Local
Command Unit) placed on special vans.
4. The Real Time Data Management
The use of digital base maps in relief operations can be considered as the first step towards an innovation of
practices and procedures of the TAS teams, and in a broader sense of the relief activities as a whole. As stated
in the previous paragraph, any emergency requires an information flow between different actors , physically
located in different places.
Starting from other experiences in the field, specifically the one of Centro Intercomunale di Protezione Civile
Colline Marittime e Bassa Val di Cecina [COI] 1 , a joint research group [the authors of this article] has been set
up to test and experiment open and free web solutions in order to guarantee sharing and collaboration on
geographical data. Despite the budget lacking, the choice to use easy and common tools and web solutions
available for free on the internet, although used in other scenarios and with diverse purposes, gave the
possibility to start trials. The concrete experiences of the wider VTCs community played a fundamental role to
benefit from, avoiding to start from scratch.
After some testing, the team focused the testing phase on two different tools: Ushahidi (to ensure the
participation of the citizens - crowdsourcing) and Google Maps (see also Google Crisis response).
On the 27 th of June in the town of Carignano (TO), for the first time during a real rescue mission for a missing
person emergency, the TAS used a geodata software to acquire and record the geolocated information related
to the occurence. The processing of geographic data through the use of GIS software used by staff of the TAS
Turin Provincial Fire Department, have been published on the web using Google My Maps, so to be shared by a
restricted number of users, as the Operational Rooms (SO115) in Turin and Aosta, the Municipal Police Station
of Carignano and the local media.
This process allowed a real-time information flow from the incident area: data and physical condition of the
missed person, the zoning of area of operation, the point of last sighting, the geolocation of search units, the
geolocation of discovered personal effects.
These were basic information but very useful to the immediate reconstruction of the incident scenarios also
for Judicial Police activities.
1 During the exercise, the team used the tools and solutions tested and adopted by the Centro Intercomunale di Protezione
Civile Colline Marittime e Bassa Val di Cecina (COI), to manage and share geolocated information between, volunteers
teams, COI Operational room, and COC (Centro Operativo Comunale - municipal operational centre). These solutions,
including a blog website to inform in real time the population and media representatives, have been successfully tested
during a missing person intervention in Cecina.
21

Missed person search procedure provide the locating of an ICP, based on UCL van when possible, where TAS
personnel must:
1. zone the search area,
2. upload GPS devices with appropriate maps and search routes or areas,
3. settle Search And Rescue (SAR) teams area of operation (AO) and tune radio devices (TETRA system for
VVF teams),
4. monitor communication, facilitate cooperation and head operations,
5. download GPS tracks (once SAR teams come back) to check not covered areas,
6. inform Operational Room (SO115) on activities.
A common platform to share information uploaded by different organizations professionals (Fire Fighters,
municipal and national police forces, Civil Protection volunteers, specialized SAR teams) should improve
dramatically operations efficiency.
Information sharing on web 2.0 platform would be used for missed person search as for every emergency
operation.
Nevertheless this is a goal not only for Italian Fire Fighters internal procedures, that linked ICP to field teams
and SO115, but also for all public bodies involved in emergency and disaster management.
The platform is suitable to coordinate different emergency operation and major disasters relief.
Real time information sharing is proper to address, for example, technical support by geologists during severe
storms that lead to floods and landslides or by air analysts during CBRN terrorist attacks.
At the same time the platform would facilitate information dissemination to media and directly to citizens.
5. Protec2011 Exercice
The Protec2011 Exercise was based on a missing person search scenario and it was carried out during Protec
2011 Exibition (http://www.protec-italia.it/indexk.php). This might allow to involve the conference attendees
as VGI’s sensors and to get independent feedbacks on procedures and activities.
The TAS team was interested to test interaction among GPS devices and data formats, radios, mobile phones
and geo mapping software and also to verify the IT infrastructure capacities.
OpenResilience aimed to test VGI platforms like Ushahidi, Google Maps and Twitter to see if they satisfy the
requirements related to the rescue operations. We are also involved to see the results of real time translation
among different GIS data formats (shp, KML, wpt, GPX, PLT) and different software platforms using GIS or
web-GIS (OziExplorer, ArcPad, Google-maps, Ushahidi, Global Mapper).
Usually each format or platform is used for a specific purpose, this creates many difficulties in emergency
management (U.S. House of Representatives, 2006). The winning idea is to develop a “black box” able to
contain and share different information from different actors and make them available to everyone.
An extra test is the opportunity offered by open source software for smartphones, with automatic delivery of
georeferenced informations (SMS, MMS, photos, videos) to an emergency service number (like US 911 or EU
112), that would allow a more effective rescue response.
As the exercice location was ideal (full Wi-Fi, WiMAX, cellular phone, TETRA coverage), the interaction among
different infratructures and the device switch among them was to be tested too.
This will allow better exercise tuning before country tests within difficult terrain. Moreover the urban search
give interesting data to future improvement for fire operations, earthquake USAR and damage assesment,
HAZMAT pollution and CBRN contamination.
22

The exercise focuses on the test of web technologies and mapping instruments for the emergency
management of information fluxes among different actors and aims to open a two-way communication
channel with citizens.
3.1. The scenario
Mrs. Paola Bianchi, 75 years old, affected by Alzheimer’s disase, is missed from her house during the morning.
His family raised alarm at 2:00 pm. The Police department calls up, as protocol, the Fire Department drills
Prefect and Civil Protection volunteers responsible.
At Operational Room (COC, placed inside Protec2011 Green Room) a Command Post is activated.
TAS Team join the last seeing area with the UCL van (Photo 2), that will be used as ICP and technical rescue
management centre (as decreed by Italian Law). A TAS professional will receive search area zoning ruled by OR
and upload GPS devices, while a second professional will facilitate information exchange between SAR teams
and OR.
3.2. The crew
OpenResilience and TAS Team planned the exercise and partecipate as described in Table 1. Turin and Aosta
Fire Departments provided SAR personnel and K9 teams, while students from the University of Turin played as
civil protection volunteers, media reporters and citizens. A UNITO technician was a perfect Paola Bianchi,
whose photo was published on exercise blog (http://esercitazioneprotec.wordpress.com/). Some Protec2011
conference attendees partecipate as witness.
[1 ] UCL DiLolli A. & Lombardo D. (search coordinators) + 1 VVF + 2 Prisma
Engineering (LSUnet)
[2] Operational Room Rapisardi E. (exercice coordinator) + 2 web 2.0 specialists + 2 VVF
[3] Search team 1 2 VVF + K9 unit
[4] Search team 2 2 VVF + K9 unit
[5] Search team 3 Lanfranco M. (devices tester) + 2 GIS specialists (UNITO students)
[6] Civil Protection
Volunteers
UNITO students
[7] Citizens UNITO students + Protec 2011 attendees
[8] Audio / Video Operators 2 VVF + 2 UNITO students
[9] Media Observer http://www.ilgiornaledellaprotezionecivile.it/
Table 1. Crew composition
3.3. Communication Infrastructure
Commercial GSM/UMTS cellular network
Lingotto Fiere internal Wi-Fi (plus an outdoor ad-hoc exercise network)
Fire Department WiMAX
23

The whole Turin Province is covered by a WiMAX 5GB network, utilized by Provincial Command Centre for data
exchange among Detachments and SO115. This network can be also used for terminals connecting within
urban and sub-urban zones, through a fixed antennas network and “on demand” mobile repeaters. The access
is secured by password.
Fire Department radio network
Italian Fire Department own a nation wide radio network. The radio network link rescue teams to Provincial
Operational Rooms while backbones link Regional Commands with National Crisis Room. The VVF radio
network never failed during disasters since the Friuli Erthquake, back in 1976. TAS teams are able to geolocate
VHF vehicle devices and some UHF personnel radio.
GSM/UMT cellular network: Prisma Engeenering (http://www.prisma-eng.com/lsu_net.html)
LSUnet can carry (in a backpack or with a trolley) a GSM (or UMTS) wherever necessary. Disaster, often,
undermine mobile networks directly (i.e. interrupting the power supply) or indirectly (network congestion due
to an excess of information exchange among people involved).
A LSUnet emergency network allows first responders to restore a cellular coverage in a short time (10 minutes)
to use standard phones or smartphones to coordinate relief efforts and/or to get a two way contact with
affected citizens.
Photo 1. COC (Operational Room)
Photo 2. UCL (Incident Command Centre)
6. Discussion
The Protec2011 Exercise has been an important test to highlight how the VVF procedures could be transferred
in a web 2.0 environment and which are the strength and weak points of the adopted solutions .
From a wider perspective, the excercise underlined that geolocated information sharing is perceived as a need
in any rescue or relief operation, as real time communication, e.g. between the UCL and the COC, at least
allows the situational awareness and remote tactical control.
The citizens involvement [crowdsourcing] has been undoubtable considered a plus, never experimented
before. The new emerging geolocation tools and platforms, althought considered “poor” and low-reliable by
academia community, represent a new challenge in a world where stakeholders’ information and geolocated
informations needs are rapidly increasing and are the expression of a democratization of geodata access, that
24

eflects a collaborative and proactive approach to cope with risks and disasters [“Towards a more resilient
society” - Third Civil Protection Forum, 2009].
However, to have a more reliable data, the post-Gutenberg map makers should acquire some sort of
competence on mobile applications or to be prepared through specific information campaign (web and
mobile litteracy). The challenge is to “design a more robust collaborative mapping strategy” (Kerle, 2011)
defining common guidelines.
From the technological point of view, crowdmapping should take into account that geolocation accuracy is
highly depending on the device’s GPS quality [in tested commercial cellulars - iphone, blackberry, HTC,
Samsung - GPS chips showed different level of accuracy].
One more feature to be introduced is the sms channel to allow citizens to send reports, even though the sms
has no geolocated information.
On the back-end side, we are aware that we should focus more on the capability of the VVF and COC operators
to interpret and validate citizen’s reports. Ushahidi experience teaches that a validation processes must be set
up and should follow specific rules: this means that the personnel in charge of the validation should be trained
on this specific issue and should develop some experience in the field.
During debriefing, the participants underlined that the whole system should use a unique platform in order to
have all data in the same map: trackings, citizens reports, VVF operations.
On the connectivity side, the internal Wi-Fi infrastructure (used by COC and UCL) was not appropriate for the
purpose and the WiMAX did’t work inside, but the test of the LSUnet by Prisma Engineering for cellular voice
communication was extremely positive; however this communication network would not support any public
web platforms as, in this exercise, it sets up only a local voice channel .
7. Step forward
Collaborative mapping is the crucial need in any rescue and relief operation. Our recent experience lead us to
focus the research on the development of a unique platform [web and mobile] that allows different levels of
geolocated information sharing, on a “user permissions” base [anonymus user, registered user level 1, ….]. Our
approach is to use the solutions that are free and open [such as Google Maps, Google Earth, Google 3d,
Ushahidi, OpenStreetMap, or Android apps for route tracking] and to develop a stable tool through the
integration of diverse solutions ensuring a high level of sharing and collaboration among different players.
Step by step
The next steps of our research team, apart from the crucial fundraising task, should start with a more strict
evaluation of the information formats and standards used by the different civil protection players and bodies,
and with an analysis of the information flow in some sample operations (e.g. ,missing persons search, critical
infrastructure crippling, USAR).
Further on we will carry out a platform/projects/solutions review to draw a bigger picture, so to acquire the
necessary information to design and implement the whole web/mobile system, that will be tested in TAS
Team exercises and operations during next winter season.
A vademecum and the setting up of training package targeting different users, will complete the basic
research and highlight the spread of adopted platform to all other Fire Departments and further involvement
of local government.
25

Acknowledgements
The project has been started on behalf of the Turin Provincial Chief of Fire Fighters (Ing. Silvio Saffiotti) and
with the Ministry of Interior authorization. The Lingotto Fiere crew strongly support our activities during
PROTEC2011, also with ad-hoc Wi-Fi network. Prisma Engineering gave us LSUnet station. Student from
University of Turin feed volunteers team. Barbara Bersanti and Antonio Campus from Centro Intercomunale
Protezione Civile Colline Marittime e Bassa Val di Cecina ruled Operational Room.
References
ALEXANDER D.E, 2001, Nature's Impartiality, Man's Inumanity. Disasters 26(1), pp.1-9.
BENKLER Y., 2006. The wealth of networks. http://cyber.law.harvard.edu/wealth_of_networks/
BURNINGHAM K., FIELDING J., THRUSH D., 2008, "It'll never happen to me": understanding public awareness of local
flood risk. doi:10.1111/j.0361-3666.2007.01036.x., pp. 216-238.
DRABEK T.E., 1999, Understanding Disaster Warning Responses. The Social Sciences Jounal 36(3), pp. 515-523.
GIARDINO, M., GIORDAN, D., AND AMBROGIO, S., 2004. GIS technologies for data collection, management and
visualization of large slope instabilities: two applications in the Western Italian Alps. Natural Hazards and Earth
System Sciences 4, pp. 197–211.
GIARDINO M., PEROTTI L., LANFRANCO M., PERRONE G., 2010. GIS and Geomatics for disaster management and
Emergency relief: a proactive response to natural hazards. Proceeding of Gi4DM 2010 Conference – Geomatics
for Crisis Management. Turin (I).
HARVARD HUMANITARIAN INITIATIVE, 2011, Disaster Relief 2.0: The Future of Information Sharing in Humanitarian
Emergencies. Washington, D.C. and Berkshire, UK: UN Foundation & Vodafone Foundation Technology
Partnership.
KERLE N., 2011., Remote Sensing Based Post-Disaster Damage Mapping - Ready for a collaborative approach?,
www.earthmagazine.org.
MECHLER R., KUNDZEWICZ Z.W., 2010, Assesing Adaptation to Extreme Wheater Events in Europe - Editorial. Mitig
Adapt Strateg Glob Change 15(7), pp. 611-620.
MCCLEARY P., 2011, Battlefield 411. Defense Technology International 6, vol. 5, p. 48.
MCCLEARY P., 2011, Small-Unit Comms. Defense Technology International 7, vol. 5, p. 47.
PEEK L.A., SUTTON J.N, 2003, An Explorating Comparision of Disasters, Riots and Terrorism Acts. Disasters (27)4,
pp. 319-335.
PERRY R.W., LINDELL M.K., 2003, Preparadness for emergency Response: Guidelines for the Emergency Planning
Process. Disasters 27(4), pp. 336-350.
PLOTNICK L., WHITE C., PLUMMER M., 2009, The Design of an Online Social Network for Emergency Management:
A One-Stop Shop. In: Proceeding of 15 ACIS, San Francisco (USA).
QUARANTELLI, E.L. (ed.), 1998. What is a Disaster? Perspectives on the Question. Routledge, Oxon (UK).
26

O'REILLY T., 2007. What Is Web 2.0: Design Patterns and Business Models for the Next Generation of Software.
http://oreilly.com/web2/archive/what-is-web-20.html
UN/ISDR, 2005, Hyogo Framework for Action 2005-2015: Building the resilience of nations and communities to
disasters (HFA). United Nations International Strategy for Disaster Reduction, Kobe, Hyogo (J).
UNDP, 1994. Human Development Report. http://hdr.undp.org/en/reports/global/hdr1994/
U.S. HOUSE OF REPRESENTATIVES, 2006. A Filure of Initiative. Final Report of the Select Bipartisan Committee to
Investigate the Preparation for and Response to Hurricane Katrina.
http://www.gpoaccess.gov/katrinareport/mainreport.pdf
WINSER, B., BLAIKIE P., CANNON T., DAVIS I., 2005, At Risk. 2nd edition, Routledge, Oxon (UK).
WHITE C., PLOTNICK L., KUSHMA J., HILTZ S.R., TUROFF M., 2009, An Online Social Network for Emergency
Management. In: Proceeding of 6 ISCRAM Conference, Gotheburg (S).
SHEN X., 2010, Flood Risk Perception and Communication within Risk Management in Different Cultural
Context. Graduate Research Series 1, UNU-EHS, Bonn (D).
ZLATANOVA S., DILO A., 2010, A data Model for Operational and Situational information in Emergency Response:
the Dutch Case. in: Proceeding of Gi4DM2010, Turin (I).
27

SHORT PAPER
An Integrated Quality Score for Volunteered Geographic
Information on Forest Fires
OSTERMANN F. and SPINSANTI L.
Joint Research Centre of the European Commission, Italy
frank.ostermann@jrc.ec.europa.eu
Abstract: The paper presents the most recent developments in an exploratory research project
that investigates the potential utility of volunteered geographic information (VGI) for fighting forest
fires. As social media platforms change the way people communicate and share information in
crisis situations, we focus on the value and options to integrate VGI with existing spatial data
infrastructures (SDI) and crisis response procedures. Two major obstacles to using VGI in crisis
situations are (1) a lack of quality control and (2) an increasing amount of information.
Consequently, the overall quality and fitness-for-use of VGI needs assessment first. One year ago,
we proposed a workflow for automatically processing and assessing the quality and the accuracy of
VGI in the context of forest fires. This contribution presents the advancements since then, focusing
on the approach to define and implement a measure of the overall quality/fitness-for-use of the
content analyzed. A proposed integrated quality score (IQS) consists of two main criteria, i.e.
relevance and credibility. For both criteria, we have identified several contributing components.
The geographic context of a message has crucial significance, since we argue that it allows
assessing both relevance and credibility. However, the geographic context is difficult to establish,
since a single piece of VGI can contain multiple types of geographic references, each of varying
quality itself.
Keywords: VGI, Forest fire, quality measure, crisis management, social networks.
29

1. Introduction
There is already a substantial amount of information provided by the general public on/during natural and
man-made disasters (Palen & Liu, 2007a). However, the expected future development and adoption of
integrated mobile devices such as smart phones makes it likely that the amount of near-real time,
geographically referenced volunteered information will increase manifold during the coming years. In our case
study, as it is possible to observe in the Table 1 in next section, 2011 retrieved data is more than doubled with
respect to 2010 one.
This development is going to change the way information is collected, distributed and used. The uni-directional
vertical flow of information from officials to public via traditional broadcasting media like radio or television of
the past is overcome by horizontal peer-to-peer(s) communication. The lines between public and official
already blur when official administrative agencies (e.g. in British Columbia 2 ) use regular accounts of private
companies like Facebook or Twitter for communication services. However, until now these newly created
back-channels do not yet integrate well with traditional established emergency response protocols. Clearly,
there are a lot of open questions to be investigated, and recent examples for research on the role of
volunteered information during concrete disasters includes wildfires (De Longueville et al., 2010; De
Longueville, Smith, & Luraschi, 2009; Hudson-Smith, Crooks, Gibin, Milton, & Batty, 2009; Liu & Palen, 2010).
In the case of volunteered information on crisis events, its potential utility depends on the possibility to
georeference the information - if we cannot locate the user-generated content, it is impossible to act on it.
While some volunteered information is explicitly geographic by itself (e.g. OpenStreetMap), other is only
implicitly geographic, since it mentions a place or has geographic coordinates as meta-data. We group both
types under the label of Volunteered Geographic Information (VGI). Another notable aspect is that this VGI is
intrinsically heterogeneous as it is provided by different people, using different media such as photographs,
text, or video, and authors often overcome device and software limitations in imaginative and unpredictable
ways.
The work presented here is part of an exploratory research project with the objectives to (i) to develop, test,
and deploy workflows able to quality control volunteered geographic information and (ii) to assess the value of
volunteered geographic information in supporting both early warning, and local impact assessments of forest
fires. For more details see(Spinsanti and Ostermann 2010).
As test cases for a proof-of-concept implementation, we decided to analyze two different social networks:
Twitter and Flickr. The first is a micro-blogging network while the second is a photo sharing network. The
research aims to study the two separately to individuate their specific characteristics but also to investigate
how the two can complement each other.
We started harvesting VGI at the beginning of the forest fire season in July 2010, and continued until end of
September 2010. At the moment of writing, we are collecting the 2011 season data. Using the public Twitter
streaming API with a filter of 12-17 wildfire related keywords (e.g. fire, forest, evacuation) in 8 different
languages, we collected a total of 24.5 GB of data, equaling around 8 million Tweets for 2010. Using a similar
2 British Columbia Forest Fire Information - http://www.bcforestfireinfo.gov.bc.ca/
The Twitter profile http://twitter.com/#!/BCGovFireInfo
The Facebook profile http://www.facebook.com/group.php?gid=2290613964
30

set of keywords for searching Flickr, we retrieved meta data for around 700 thousand images for 2010. For
2011 we can see (Table 1) the trend of retrieved information is increased of more than +200% for both Twitter
and Flickr. All these VGI are potentially related to a forest fire. The large amount of information leads to the
essential need of an automatic methodology to assess the quality and the accuracy of the VGI. First, however,
we have to geocode any location information.
2. Creating VGI - geocoding user-generated content
As we have defined in the previous section, VGI is information (text, image, video, etc.) with one or more
geographic references, which can be explicit (coordinates), or implicit in the form of placenames (toponyms).
The explict georeferences can be generated in two ways: either automatically by the device if it has Global
Positioning System (GPS) hardware and then added by the software used to transmit the information; or
alternatively, some platforms offer the user to select a place from a list (or to select a point on a map), which is
then added to the message meta-data in the form of text or coordinates. Implicit geo-references are created
when the contributor uses toponyms in the message content or adds them as tags. As we show below, even
already explicitly geocoded information needs to be examined for toponyms and possible re-geocoding.
Looking at the Twitter/Flickr for August 2010/2011 we can observe that the number of explicit geographic
information is very low compared with the large amount of retrieved information.
Table1: number of retrieved VGI and explicit geographic information
TWITTER FLICKR 3
August 2010 August 2011 August 2010 August 2011
Number of retrieved VGI 2,904,065 7,996,228 7,991 17,850
Percentage with 35% 27% 53% 50%
toponym
Percentage with geocode 1.1% 0.92% 20% 21%
Yet a simple string matching search for toponyms using a large database of toponyms reveals that much higher
percentage of messages potentially contains implicit geographic reference: for Twitter the implicit georeferences
are around 30% against 1% of the explicit ones; for Flickr the implicit geo-references are around
50% against 20% of the explicit ones. For both data type, in the 28% of the cases the implicit geographic
reference is the unique reference. In order to make this implicit VGI usable for crisis management, we have to
made explicit it first (i.e. geocode) and this can be done using different applications and strategies.
Independently from the methodologies we can choose, we have to consider the different types of geographic
reference we are likely to encounter. Let’s consider in the next two examples and the information we want to
analyze: the part of VGI that describes a location, the associated real world object or event that is located and
the geo-referencing of this location information.
In the Tweet example in the left part of Figure 2, the geographic reference is contained in the text in the
toponym “Funchal”, more precisely the mountains behind the city of Funchal. The information refers to a
forest fire event. Suppose the Tweet message itself has some coordinates originating from the source GPS
device: this represents the location of the message. Nevertheless, we can suppose that the user was sending
the message safely far away from the forest fire, so the tree different locations are not overlapping. In the
Flickr example in the right part of Figure 2, consider a person taking a picture with a camera or a smartphone
3 Retrieved without English keywords
31

with GPS integrated system. The person and the device coordinate overlap, while the subject of the photo
coordinate has a distance from the camera. This distance could be considerable. Let’s imagine that the content
represented in the photo is the Mount Everest. The user with the camera is necessarily far from the mountain
peak to include into the photo. As shown in Fig.1 on the right side, the mountain and the camera could be
consistently far one from the other. Moreover the user can set the photo coordinates manually: in this case
the precision depends on several factors such as his/her surrounding knowledge or the application interface.
Figure 2. Geographic Information in VGI
These examples illustrate that there is a discrepancy between the location of the content (the registered
device location at the time the message is sent or the photo is taken) and the geographic content contained in
the message itself. The location of the device is not necessarily equal to the location of the reported content:
they can overlap or be far away as in the examples. This inconsistency is not of technological nature, but will
always include semantic aspects.
Because the number of explicit geographic information in VGI is low, but also because of the semantic
uncertainty of this geographic information, the geocoding of the VGI is a crucial step.
3. Geocoding VGI and context information
Geocoding is the process of finding associated geographic coordinates (often expressed as latitude and
longitude) from other geographic data, such as street addresses, or zip codes (postal codes). In our case we are
looking for place names to be associated with a fire event. In natural language, place names (toponyms) are
used to refer to these locations without having to mention the actual geographic coordinates. We select a
granularity level of the communes names in Europe because the European Forest Fire Information System
(EFFIS) use this level for the forest fire database collection and at the end of the workflow we want to be able
to integrate data. We decided to limit the proof of concept experiments to four Mediterranean countries:
Italy, France, Spain and Portugal. This choice was driven by several reasons: they are the most probable place
for forest fires in summer, the language of this country use Latin characters and exclude most of the USA
users. We used communes names extracted from GISCO 4 , an Eurostat service which promotes and stimulates
the use of GIS within the European Statistical System and the Commission. There are more than 57,000
communes in the four counties. We search for the presence of the commune name in the title and tags for the
photos and in the text for the Tweets. Toponym disambiguation (a.k.a. toponym resolution) is the task of
determining which real location is referred to by a certain instance of a name. Toponyms, as with named
entities in general, are highly ambiguous. From (Habib, M.B. and van Keulen (2011)) can be observed that
around 46% of toponyms have two or more, 35% three or more, and 29% four or more references. In natural
4 http://epp.eurostat.ec.europa.eu/portal/page/portal/gisco_Geographical_information_maps/introduction
32

language, humans rely on the context to disambiguate a toponym. In human communication, the context used
for disambiguation is broad: not only the surrounding text matters, but also the author and recipient, their
background knowledge, the activity they are currently involved in, even the information the author has about
the background knowledge of the recipient, and much more. Moreover, in our task, we are dealing with short
text messages (often with poor grammar and syntax) and tags: the methods implemented for larger text
corpora are often not valid. Our approach is to use several existing systems and combine the final result in a
geographic score. The first step is to identify VGI that can potentially be geocoded, by filtering it using the
commune names labels. This brute approach ensures that VGI without any geographic reference in the text
are discarded. The remaining VGI are sent to the Europe Media Monitor (EMM) geographic module for
toponym extraction. The results are compared with the previous one to reinforce or decrease the confidence
in the retrieved place name. For the uncertain one another step is generate sending the VGI to
Yahoo!Placemaker service and compare again with the previous retrieved. At the end a geocoding score is
generated and used in the Integrated Quality Score (IQS) as described in the next section.
4. Integrated Quality Score
The aim of assigning a score to each VGI has to deal with several facets each of them contributing to the final
value. We aim to score each of these facets independently, and then arrive at an integrated quality score
combining all aspects. While the idea of additive quality score is not new (Friberg, Prödel, and Koch 2011), we
intend to focus on the spatial context of the information, which to our knowledge has not been attempted so
far. The following figure shows the sequence of procedure: After the information has been successfully
geocoded (see previous section), we gather and assess information on the geospatial context, and rate the
degree of topicality, i.e. the likelyhood of the information being about forest fires. We intend to plug-in further
modules dealing with source credibility later.
VGI
Geocoding
Geo-Spatial
Context
information
Topicality
Integrated Quality
Score
Figure 3. Several aspects of integrated quality score
In more detail, the Geo-Spatial Context considers land use and land cover, population density, and distance to
known hot spots discovered by satellite imagery.
Topicality is about the content of the message: is the VGI referring to a forest fire? To calculate this score each
keyword gets assigned a value based on statistical analysis made on VGI evaluated by hand as ground truth
basis.
These values are combined in a weighted sum: IQS(io j ) = ∑ N i=1w i v i (s ji ) with w being weight for criterion i, v
being the value function for criterion i and s being the score for the information object j.
In our specific case IQS = (topicality*weight1) + (geocoding*weight2) + (context*weight3).
5. Conclusion
In this paper, we have argued that the emerging use of social media by members of the public will become an
important pathway for communication during a crisis event, while the notion of citizens as sensors can provide
the decision-makers of the crisis management team with important information. A large part of volunteered
33

information has a geographic component and the number will increase constantly. However, the increasing
usage will also amplify two main challenges: the volume of information will need some sort of filtering, and in
order to be useful for official emergency response, its quality, relevance and credibility needs to be assessed.
Humans have carried out both tasks so far, but the tasks need to be automatized to cope with the expected
increase of volunteered information. We propose an integrated quality score for VGI assessment from social
media. The geographic component will play an important part in assessing the data and in clustering the data
to fit specific (sub-) events. The next steps will include the integration of VGI with official spatial data
infrastructures, and its evaluation for fire events.
Acknowledgements
This work was partially funded by the exploratory research project "Next Generation Digital Earth: Engaging
the citizens in forest fire risk and impact assessment" from the Institute for Environment and Sustainability of
the European Commission – Joint Research Centre.
Thanks to the EFFIS and EMM teams for their contribution and support.
References
De Longueville, B. D., Luraschi, G., Smits, P., Peedell, S., & Groeve, T. D. (2010). Citizens as Sensors for Natural
Hazards: A VGI integration Workflow. Geomatica, 64(1), 355-363.
De Longueville, B. D., Smith, R. S., & Luraschi, G. (2009). "OMG, from here, I can see the flames!": a use case of
mining location based social networks to acquire spatio-temporal data on forest fires. Proceedings of the 2009
International Workshop on Location Based Social Networks. doi:http://doi.acm.org/10.1145/1629890.1629907
Friberg, Therese, Stephan Prödel, and Rainer Koch. 2011. Information Quality Criteria and their Importance for
Experts in Crisis Situations. In Proceedings of the 8th International ISCRAM Conference. Lisbon.
Habib, M.B. and van Keulen, M. (2011) Named Entity Extraction and Disambiguation: The Reinforcement
Effect. In: Proceedings of the 5th International Workshop on Management of Uncertain Data, MUD 2011, 29
Aug 2011, Seatle, USA. pp. 9-16. CTIT Workshop Proceedings Series WP11-02. Centre for Telematics and
Information Technology, University of Twente. ISSN 0929-0672
Hudson-Smith, A., Crooks, A., Gibin, M., Milton, R., & Batty, M. (2009). NeoGeography and Web 2.0: concepts,
tools and applications. Journal of Location Based Services, 3(2), 118 - 145.
Liu, S. B., & Palen, L. (2010). The New Cartographers: Crisis Map Mashups and the Emergence of
Neogeographic Practice. Cartography and Geographic Information Science, 37, 69-90.
Palen, L., & Liu, S. B. (2007a). Citizen Communications in Crisis: Anticipating a Future of ICT-Supported Public
Participation. In CHI 2007 Proceedings (pp. 727-726). Presented at the Computer Human Interaction 2007, San
Jose, USA.
Spinsanti, Laura, and Frank O. Ostermann (2010). Validation and Relevance Assessment of Volunteered
Geographic Information in the Case of Forest Fires. In Proceedings of the 2nd International Workshop On
Validation Of Geo-Information Products For Crisis Management, ed. Christina Corbane, Daniela Carrion, Marco
Broglia, and M. Pesaresi, 101-108. Ispra, Italy: Publications Office of the European Union, Luxembourg.
34

SESSION II
VALIDATION OF REMOTE SENSING DERIVED
EMERGENCY SUPPORT PRODUCTS
Chair: Dirk Tiede
New generation remote sensing technologies open today new application areas
and demonstrate their effectiveness providing geo-information in support to all the
phases of the crisis management cycle. The presentations in this session establish a
consistent framework on the use of remotely sensed data and its validation during the
preparedess phase (e.g. baseline data on built-up areas), the early warning phase (e.g
evacuation plans), the post-disaster phase (e.g. damage assessment) and the
reconstruction phase (e.g. monitoring reconstruction). A particular emphasis is given to
the technical (e.g. sampling issues), the practical (e.g. validation in the secutiry domain)
and even the theoretical challenges (e.g. review of the error matrix) related to the
collection of reference data and their use within the different validation methodologies.
35

SHORT PAPER
Definition of a reference data set to assess the quality of building
information extracted automatically from VHR satellite images
WANIA A., KEMPER T., EHRLICH D., SOILLE P. and GALLEGO J.
Joint Research Centre of the European Commission, Italy
Annett.wania@jrc.ec.europa.eu
Abstract
Rapid urbanisation continues to be an issue with one third of the world’s urban population living
under poor living conditions in informal settlements or shanty towns. Improving the lives of this
population, which is one objective of the United Nations’ Millennium Development Goals, requires
knowledge about the areas under concern. This information is currently still collected in intensive
field studies. Earth observation data could be an alternative source of information that can support
the process of information collection and on longer terms also serves for monitoring the evolution
of those areas. Today’s sub-meter resolution satellites provide very detailed information allowing
identification of different urban pattern. However, the huge amount of data requires an automatic
information extraction to derive relevant information in a fast and consistent way.
Reference data is crucial for the assessment of those algorithms but in case of absence of a
relevant data set, which is especially the case in developing countries, an alternative database
needs to be defined. In this paper we present a robust approach to produce a reference data set
with limited field surveys for the city of Harare (Zimbabwe). This data set is defined to serve two
objectives: first, quality assessment of the automatically extracted information and second, further
analysis to identify built-up pattern.
Two reference data sets are defined using systematic and cluster sampling. The building stock of
the city is systematically sampled by visual image interpretation of regular grid points covering the
entire image area. Cluster sampling in several stages is performed to define a small sample of
buildings that are surveyed in the field. The first stage consists of constructing clusters for the
entire city area based on building density and height. From each of these clusters, a sample is
randomly selected and in each a sample of buildings is finally randomly selected.
Keywords: reference data set, sampling, building stock, VHR, quality assessment.
37

1. Introduction
The monitoring and evaluation of global urban conditions and trends has become an important issue against
the background of rapid, continuous urbanisation worldwide and the high percentage of urban citizens living
under poor conditions. UN-HABITAT has established the Global Urban Observatory (GUO) to address the
urgent need to improve the world-wide base of urban knowledge by helping governments and local
authorities and organisations to develop and apply urban indicators, statistics and other urban information.
Earth observation data, and especially today’s generation of sub-meter resolution satellites could be one
source of information that allows collecting information about the physical characteristics of the building stock
systematically and world-wide. It could complement field studies and could be used to optimise field sampling.
With the recurring acquisition of images it could also support the monitoring of urban areas.
Against this background, the Joint Research Centre is currently developing a workflow to extract information
on the building stock (Kemper et al. 2009). The ultimate goal is to develop a workflow that can be applied to
any image in any region of the world, with a parameterisation that is consistent and as much as possible
independent from local conditions. The information is extracted from very high resolution optical imagery
(panchromatic resolution ≤ 1 m) and provides information at fine scale relating to homogeneous built-up
structures. Since its first set-up at the end of last year, the workflow is continuously improving and was run for
several cities on different continents. The first results were discussed with potential users from the GUO
network which outlined the potential of the derived urban indicators. By that time the quality of the results
was assessed using manually digitised buildings which were available only for two of the ten analysed cities.
The data sets were used to validate information which relates to the footprint of buildings. No information
about the height was available. Furthermore it was problematic that one of the data sets was older than the
image used.
In the frame of the GMES project G-MOSAIC the same workflow was recently run on the city of Harare,
Zimbabwe. As there is no ground truth available for the quality assessment, we decided to build a systematic
reference data set of building samples. The sample set should serve two purposes. First, it will be used to
assess the quality of the building stock extracted using the automatic workflow. Second, the building samples
will be used to classify the entire city. The aim here is to identify spatial clusters according to type, usage,
height of buildings and the housing quality (poor, rich).
This paper presents an approach for the sample definition, which combines systematic and cluster sampling.
Systematic sampling is used to define a data set for the entire city area. Systematic sampling with a random
origin is used because it provides a smaller variance than random sampling in spatially correlated populations
(Bellhouse, 1988, Dunn and Harrison, 1993). The main drawback of spatial sampling is that there are no
unbiased estimators for their variance. Available estimators are usually conservative in the sense that the
variance is overestimated (Osborne, 1942, Wolter, 1984). We have considered that this is not a major problem
for this application. Clustering points does not give a major advantage for the first sampling phase (points to
be photo-interpreted), but is necessary to reduce working time in the in-situ survey (second phase sample).
The data set is created by visual interpretation of the input image. Cluster sampling is used to define a small
subset of buildings that are surveyed in the field. While the visual interpretation allows collecting very limited
two-dimensional building characteristics, the field survey provides the opportunity to collect height
information and characteristics that require a close or front view on the sampled object.
38

2. Study area and data
The study area covers 340 km 2 of the larger urban zone of Harare, which is the capital of Zimbabwe and the
centre of industrial production and commerce in the country (Figure 1 left). In comparison to other Sub-
Saharan countries, Harare with its approx. 1.6 million inhabitants (2007) does not follow the general trend of
fast urban growth in mostly slum settlements. According to UN-HABITAT (2008) only 6.3 % of the population of
Harare were living in slum conditions (sub-Saharan average 63%). However, with limited expansion of Harare’s
housing stock, backyard shacks and illegal extensions to formal housing units are a dominant feature for the
poor and much of the middle class whose incomes do not qualify them for private sector housing (UN-HABITAT
2008).
For this area, GeoEye-1 satellite data with 0.5 m spatial resolution for the panchromatic band and 2.0 m for
the multispectral bands were acquired on 26 April 2010. The data were delivered already as a pan-sharpened
multispectral data set with 4 bands in the visible and near infrared spectrum.
The ultimate goal of the project is the characterisation of the building stock of Harare achieved with an
automated information extraction. This is achieved with a processing chain based on morphological image
analysis taking into account the local image contrast and shadows. The processing chain provides information
related to the building height derived from the shadow length, building size based on the contrast and the
vegetation density based on a vegetation index. More detailed information on the methodology is provided in
Kemper et al. 2009.
Figure 1. GeoEye-1 image over the city of Harare (left) and automatically extracted building stock (right).
39

3. Methodology
The sampling scheme for the definition of the reference data set of buildings has two phases: in the first phase
a grid of points covering the entire image is defined for visual interpretation. In the second phase a small set of
buildings is selected for the field survey. In the following paragraphs we will first specify which building
characteristics are collected in each of those sample sets and second, describe the sampling method.
a. Building characteristics
In view of building a reference data set for quality assessment and further analysis of the building stock,
several building attributes were defined for both the visual interpretation and the field survey. Table 1 shows
the attributes and categories for each and whether the attribute is collected in the visual interpretation, in the
field survey or in both. For the visual interpretation a procedure was implemented in ESRI ArcGIS to collect the
information. In the field survey the information is collected on print-out evaluation sheets.
Table 1. Building attributes and values for visual interpretation (VI) and/or field survey (FS).
Attribute VI FS Values
Functional use x x Industrial
Commercial/business
Education/government/hospital
Mixed use
Residential
Level of income
(only for residential use)
x x High or medium
Low
Very low (squatter)
Size x x Area of the digitised building footprint
Height x Number of storeys
Arrangement x Attached buildings
Single buildings
Main construction material x Concrete
Bricks
Corrugated iron
Assembled material
Other
Degree of planning x Planned
Not planned (informal)
b. Master sample grid and systematic sampling by visual image interpretation
The basis for the sampling is a regularly spaced 400 m grid which covers the image mosaic in its full extension
as shown in Figure 2 (master grid). From the initial grid (1988 cells), cells which are not fully covered by the
image and those which are to a large extend affected by cloud cover were removed (see “No data” cells in
Figure 2). Likewise, cells affected by the associated cloud shadow were removed as the thematic information
extracted in those areas is not reliable. In total, the final sample grid is composed by 1662 cells covering an
area of approximately 266 km 2 .
40

In each of the 400 m grid cells the four centroids of the 200 m sub-grid are selected (see Figure 2 inlet). Those
6648 points are used for the visual interpretation of the building stock. If the point falls on a building, three
attributes are collected: functional use, likely income level (for residential use) and size (see Table 1).
Figure 2. Sampling design: the main figure shows the sampling grid over the image extent with the stratification into
three classes and the location of the 50 sample cells (clusters) for the field survey. The inlet shows the example of one
grid cell with the four sample points (systematic) for the visual interpretation.
c. Cluster sampling for field survey
A smaller sample of points was defined for the field survey. Field surveys provide the opportunity to observe
more than the three attributes from visual image interpretation due to the vicinity to and the multidimensional
view on the object of interest. In addition to the attributes from the visual interpretation, the field
survey aims at collecting also the height, arrangement, main construction material, and the degree of planning
(see Table 1).
We applied cluster sampling to reduce the survey cost and to obtain, at the same time, a set of buildings which
are representative for the building stock of Harare. The sampling was performed in three steps:
1. Stratification of the automatically extracted building stock (classification of the 1662 cells from the 400
m x 400 m grid)
2. Stratified sampling of 50 cells
3. Sampling of single buildings.
41

In the first step, the 1662 cells were stratified on the basis of the automatically extracted built-up information.
Three strata were defined based on the relative built-up area and the average height per cell. Table 2 shows
the cluster definition and the number of cells included in each.
Table 2. Stratification of the 1662 cells and definition of the subsample of 50 master grid cells for the building sample
definition.
Built-up
Nb of
area (%)
cells
Cluster
Average
height
(m)
Percentage
of cells
(%)
Built-up
area
(km 2 )
Percentage
of total
built-up
area (%)
Nb
sampling
cells
Percentage
of
sampling
cells
1

Figure 3. Final step of the cluster sampling to identify buildings for the field survey.
In some cases, this method led to the selection of less than five buildings. This was the case either in cells
where only few buildings were present or where buildings were very small and dispersed or concentrated in
one area. In those cases the following was applied:
• All buildings were included in case there were equal or less than five buildings.
• Probability Proportional to Size sampling (using the building size) was applied in case of more than five
buildings. For this purpose all present buildings were digitised and the area computed. The method is
most useful when the sampling units vary considerably in size because it assures that those in larger
sites have the same probability of getting into the sample as those in smaller sites, and vice versa.
Besides the coordinates of the sample buildings, the person performing the field survey is equipped with one
map for each of the selected 50 cells showing the relative location in the city area and the exact location of the
sample buildings. The building characteristics are recorded on an evaluation sheet (see Figure 4).
43

Figure 4. Material for the field survey. Left: Example of the maps that were provided for each of the 50 cells showing
their location in the city area and the location of the sample buildings. Right: evaluation form.
4. Results
The visual interpretation was accomplished by four different interpreters that spent approximately 34 hours
on the visual interpretation. To assure that they followed the same criteria, they were equipped with an
interpretation key providing also examples. In addition interpreters marked doubtful cases and discussed them
together. Finally, the data were randomly checked for consistency.
Out of the 6648 points that were visually interpreted on the image, 562 buildings were identified and the
footprint of each of them was digitised. Table 3 below summarises the main characteristics of the building
stock. More than half of the mapped buildings were identified as residential buildings and the majority of the
residential buildings (78%) were marked as high or medium income levels. These groups are also clearly
distinguishable by the dwelling size.
Table 3. Summary of the results of the visual interpretation of 6648 points in Harare.
Residential
high or
medium
income
Residential
low income
Industrial Commercial/
business
Education/
government
Mixed
use
Number 249 70 160 38 31 12
Percentage 44 12 29 7 6 2
Average size [m 2 ] 221 164 3237 973 742 1505
At the time of writing the field assessment was not yet available. Hence, the quality of the visual interpretation
could not be assessed.
5. Discussion and conclusion
The methodology described in this paper was designed to allow collecting information on the building stock of
a city in a systematic and repeatable way. This is important in situations where alternative reference data is
missing or out-dated. The information collected remotely from the visual interpretation can be used to
44

validate automated information extraction procedures. This can be based on standard confusion matrices with
information about omission, commission and overall quality, or using more sophisticated tools, which take into
account also the area and/or shape of structures (Congalton & Green, 2009).
The experience from the visual interpretation shows that determining the usage categories is ambiguous in
some cases (especially between different residential classes). The definition of the classes requires local
knowledge to be as objective as possible in the definition of the classes. The same, the definition of a
building‘s footprint is not always straight forward for complex buildings or complex roof structures (e.g.
industrial buildings with non-flat roof segments). In both cases it is important to establish clear interpretation
keys and to exchange with the interpreters during the process. In this context, it might be important to
consider also what type of processing the data are compared with. Will the procedure map built-up areas,
which may include also open spaces (e.g. Pesaresi et al. 2008) or will it extract building footprints?
One advantage of our sampling approach lies in the partial overlap of the two data sets. As such it allows
evaluating the consistency of the reference data sets. Building characteristics, which are collected by two
different persons (field surveyor and interpreter) can be compared and as such assessed. Such comparison will
not only allow drawing conclusions on the information of the quality of the reference data set but also on the
design of future samplings.
Acknowledgements
The research is conducted within the frame of the FP7 GMES project G-MOSAIC (GMES Services for
Management of Operations, Situation Awareness and Intelligence for Regional Crises). The GeoEye-1 image
was acquired in the frame of DAP ID DAP_MG2b_23. We would like to thank Xavier Blaes from JRC for his
contribution in the visual image interpretation.
References
BELLHOUSE, D.R., 1988, Systematic sampling, Handbook of Statistics, vol. 6, ed. P.R. Krisnaiah, C.R. Rao, pp. 125-
146, North-Holland, Amsterdam.
CONGALTON, R.G. and GREEN, K., 2009, Assessing the accuracy of remotely sensed data: principles and practices,
second edition. CRC Taylor & Francis, Boca Raton, 183 p.
DUNN, R. and HARRISON, A.R., 1993, Two-dimensional systematic sampling of land use. Journal of the Royal
Statistical Society Series C: Applied Statistics 42 ( 4), pp. 585-601.
KEMPER, T., WANIA, A. and PESARESI, M., 2009, Supporting slum mapping using very high resolution satellite data.
In Conference Proceedings: 33rd International Symposium on Remote Sensing of Environment. Tuscon,
Arizona: International Center for Remote Sensing of Environment (ICRSE); p. 480-483.
OSBORNE, J.G., 1942, Sampling errors of systematic and random surveys of cover-type areas, Journal of the
American Statistical Association 37, pp. 256-264.
PESARESI, M., GERHARDINGER, A. and KAYITAKIRE, F., 2008, A robust built-up area presence index by anisotropic
rotation-invariant textural measure. IEEE J. Sel. Topics Appl. Earth Observ. Remote Sens. 1 (3), pp. 180-192
WOLTER, K.M., 1984, An investigation of some estimators of variance for systematic sampling. Journal of the
American Statistical Association 79 (388), pp. 781-790.
UN-HABITAT (2008), The state of African Cities 2008, a framework for addressing urban challenges in Africa.
Assessed 12.09.2011: http://www.unhabitat.org/pmss/getElectronicVersion.aspx?nr=2574&alt=1
45

SHORT PAPER
On the Validation of An Automatic Roofless Building Counting
Process
GUEGUEN L., PESARESI M. and SOILLE P.
Joint Research Centre of the European Commission, Italy
lionel.gueguen@jrc.ec.europa.eu
Abstract: Roofless buildings are encountered in case of conflict and disasters as well as
construction sites. Thanks to the recent progress in image information extraction methodologies,
characterizing and counting roofless buildings can now be obtained by automatic process. The
result is a map of roofless building fuzzy membership, which can be used to estimate the number
of roofless buildings. While being less accurate than photo-interpretation, such methodology
enables to assess a crisis situation in a very short period of time (some minutes). In this paper, the
validity of such automatic products is assessed and compared to a photo-interpretation based
roofless map. The validation is conducted on a WorldView1 panchromatic image representing the
city of Tskhinvali, Georgia.
Keywords: automatic detection, ROC.
1. Introduction
The contribution of space technologies was demonstrated to be effective for regional/continental damage
assessment using low- or medium-resolution remotely sensed data input (ranging from 30m to 1 km), and
both automatic and manual interpretation approaches have been successfully used for extraction of
information [1]. With new image products providing detailed scene description, information extracted at high
resolution (ranging from 5m to 10m) is crucial for calibration and estimation of the reliability of low- and
medium-resolution assessment, planning logistics for relief action in the field immediately after the event, and
planning the resources needed for recovery and reconstruction.
Local or detailed damage assessment can be addressed using very-high-resolution (VHR) satellite data with a
spatial resolution ranging from 0.5 to 2.5m. At this level, the operational methodology for extraction of the
information is based on manual photo-interpretation of the satellite data which are processed on the screen
by the photo-interpreter as for any other aerial imagery. The drawbacks of traditional photo-interpretation
methodology are linked first to the time and cost needed for manual processing of the data and second to the
difficulty in maintaining coherent interpretation criteria in case there are large numbers of photo-interpreters
working in parallel for interpretation of wide areas in a short time. In order to tackle the problem, automatic
processes for detecting damaged buildings were presented in the literature, either exploiting optical sensors
[2]-[4] or SAR images [5], [6]. Nevertheless, these methods are generally dedicated to one type of damage and
47

equire pre- and post-damage images. Following some particular event, like an armed conflict or a hurricane,
the observable damages are roofless buildings as illustrated in Figure 1.
a) b) c) d)
Figure 1. Fig. 1. Example of roofless buildings observed in VHR optical images. (a) Following conflict in
Georgia.(WorldView1) (b) Following conflict in Sri Lanka.(WorldView2) (c) Following conflict in Nagorno-
Karabakh.(QuickBird) (d) Construction site in Haiti.(Aerial sensor)
Some automatic methods were proposed in literature [7] for characterising the roofless buildings from VHR
panchromatic images. The output products is a fuzzy membership map which assigns to each pixel an index
values between 0 and 1 representing its belonging to the semantic class roofless building. Such method are
based on the aggregation of morphological characteristics. More recently a method for estimating the number
of roofless buildings was presented in [8].
The validation of such methods is crucial in understanding the reliability of the given products. This paper
adresses the problem of validating automatic characterization and counting of roofless buildings from VHR
optical images. In order to perform the validation, a WorldView 1 panchromatic image has been photointerpreted,
resulting in geolocalized points on top of the roofless buildings. First, such collection of points
enables to assess the fuzzy membership map validity through a receiver operational characteristics analysis,
estimating the probabilities of false alarms and missed detections (commissions, omissions). Secondly, a
method for estimating the global number of roofless buildings is reminded [8]. Such method exploits the
availability of partial ground truth in order to learn an optimal parameter, then giving the number estimate.
The biais and variance of this estimate are experimentally derived giving understanding about the product
reliability.
Approximate ROC Analysis
A VHR image can be formally represented by a map function I(x) from the grid space to the measurement
space. By an automatic processing of the image I(x), an automatic detection of relevant structures can be
derived. Let be the roofless building fuzzy membership function associating a confidence to each pixel .
Such function takes its values in the interval [0,1]. Due to the underlying process the fuzzy membership
contains blobs around roofless buildings, as depicted in Figure 2.
a) b)
Figure 2. a) A subregion of the VHR image I(x). b) The corresponding fuzzy membership function .
48

The photo-interpretation and digitalization being a time consuming task, roofless buildings are generally
identified by geolocalized points. The comparison of points to blobs is not trivial and require some
approximations. Instead of reducing the fuzzzy membership map to points, a mask of roofless buildings is
synthesized from the points. It is assumed that the roofless buildings have an average shape corresponding to
a disk of diameter 15 meters, such that a disk is centered at each geolocalized point, as depicted in Figure 2.
Such a mask can formally be represented by the function .
In order to validate the automatic process, both functions and are compared by computing the
Receiver Operational Characteristics pixelwise, providing an approximation of the false alarm and missed
detection
probabilities:
(1)
(2)
where is the threshold of . Varying the threshold , gives the parametric function
which is commonly called ROC curve. As the ground truth is synthesized from geolocalized
points, the ROCs are approximations of the real errors.
Figure 3. A WorldView panchromatic scene of size 13 × 5 km2, representing the city of Tskhinvali is depicted on the top.
Geolocated points indicating the roofless buildings, which were obtained by photo interpretation, are overlaid in
green. A zoom of an area of interest is displayed below. The points are extended to 15m diameter disks to build a
ground truth mask.
The ROC associated to the full scene analysis is depicted in Figure 4. At equal error probability, the fuzzy
membership provides 20% of missed detections and false alarms in comparison to the synthesized mask
. Thus, such automatic product can be used in crisis scenario in order to assess quickly the extent of
damages.
49

missed detection rate
1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
0 0.2 0.4 0.6 0.8 1
false alarm rate
Figure 4. The receiver operational characteristics curve associated to the roofless building detector .
The global probability of error is obtained knowing the prior distribution of damaged locations in the image
. Assuming spatial independence of variables, the pixelwise probability of error is then
obtained by:
(3)
(4)
The threshold giving the minimum global error probability provides the best pixel classification. However,
the minimization of the pixel error can be meaningless depending on the application.
Roofless Building Count Estimation
A method for estimating the number of roofless buildings from a fuzzy membership map was proposed in [8].
In this paragraph, the method is briefly reviewed.
We assume that a subpart of the whole scene is photo-interpreted and the corresponding ground truth is used
to select the best threshold of the roofless membership function . Knowing the average size of a roofless
building and the total destructed area in the scene, the ratio of both quantities indicates the number of
roofless buildings. The total hit area from the ground truth mask is given by:
, where
isthe number of pixels in the image. The average building size is given by the selected disk area (a disk of
diameter m). Then, by construction the true number of roofless buildings is .
To estimate this number, an estimate of the ROC curve and of prior probabilities is required. The ROC curve
can be estimated using the photo-interpreted subregion thanks to the equations (1)-(2). For some threshold ,
the estimated total area is thus given by
. The roofless buildings detection produces
two types of errors: the false alarms and the missed detections. While the false alarms increase the estimated
area, the missed detections decrease it. Therefore, an optimal threshold can be selected such that both types
of error compensate:
(5)
50

where
is the prior probability estimated from the considered subregion. The total roofless area
estimate is thus given by and the number of roofless buildings is obtained by .
Validation of the Count Estimation
Having the ground truth of a subpart of the scene, the number of roofless buildings can be estimated. In this
section, the count estimate is analysed depending on the available ground truth coverage.
To run the first experiment, a coverage size is selected and is expressed in percentage of the scene area. Then,
a square of the chosen size is randomly selected from the scene and is considered as representing the
available ground truth. Finally, an optimal threshold is derived from the knowledge of the approximate
ground truth and the membership function in this subregion thanks to the optimization criterion of (5).
Applying the threshold on the whole membership function enables to estimate the total area , thus the
global number of roofless buildings
. For one chosen size, the estimation is run 50 times from
randomly selected subregions, providing information about the estimator variability. The obtained count
estimates are summarized in Figure 5, where the ground truth coverage is varying. The graph represents the
, and percentiles of the estimate depending on the ground truth coverage. For any
coverage, the median estimate is close to the true number which is . However, the variation of the
estimator decreases when the available ground truth coverage increases. When reaching a ground coverage of
, the estimator variability decreases slowly, such that in half of the cases, the count estimates lies in
.
5000
4500
4000
Count estimate
3500
3000
2500
2000
1500
1000
500
0
3% 6% 9% 13% 17%
Ground truth coverage
Figure 5. Box plot representation of the roofless count estimator variabilities, where each box represents their ,
and percentiles. The horizontal axis represents the percentage of ground truth available with respect to
the whole scene.
The previous analysis does not incorporate the expert knowledge for selecting the subregion to be photointerpreted.
To gain in understanding, a second analysis is conducted with the same data set. Random
subregions of fixed size are selected from the image and their optimal threshold is computed, giving an
estimate of the global number
. Then, each pixel in the region is associated to this estimate, such
that an average of the estimate can be computed per pixel:
(6)
51

Such a map gives insight in the contribution of each pixel to the global number estimate given a fixed
subregion size. In addition, it enables to understand the effects on over or under estimation. The map is
computed for regions occupying 6% of the total area and is displayed in Figure 6.
Figure 6. The map is represented and color coded for the whole scene, presented in Figure 3.
Underestimation is color coded in blue, while overestimation is coded in red, yellow. The ground truth points are
overlaid in black dots.
One can observe that considering subregions which do not contain any roofless buildings lead to an
underestimation of the total number of destroyed building (blue parts). Two subregions producing an accurate
estimate and an overestimation are selected and represented in Figure 7.
a) b)
Figure 7. a) A subregion producing an accurate estimate. c) A subregion producing an overestimate.
The subregions producing overestimations are contaminated by clouds of smoke modifying the illumination
conditions and the automatic process response. The subregions producing accurate estimation contain a
variety of patterns, including roofless buildings, and they are representative of the global scene content.
By rapid visual inspection an expert is able to select a subregion to be photo-interpreted such that it
represents the overall scene content, avoiding problematic subregions containing cloud of low density of
roofless building. Analyzing the histogram of the green part of Figure 7 gives a confidence interval for the
count estimate which is of [745, 1453]. By sampling correctly, the interval is improved in comparison to the
worst-case interval [450, 2000] displayed in Figure 5 for subergions of 6% coverage.
Conclusion
This paper presents the validation of an automatic roofless building count process exploiting photointerpreted
ground truth. The intermediate fuzzy membership map enables to recover the photointerpretation
up to 20% of false alarm and missed detection. Then, the roofless membership map is exploited
52

in order to estimate the number of roofless buildings in the whole scene exploiting partial photo interpreted
ground truth. The validation results show that the number of roofless buildings can be well approximated
considering a sub region representative of the situation.
References
[1] A. S. Belward, H.-J. Stibig, H. Eva, F.Rembold, T. Bucha, A. Hartley, R. Beuchle, D. Al Khudhairy, M.
Michielon, and D. Mollicone, “Mapping severe damage to land cover following the 2004 Indian ocean
tsunami,” International Journal of Remote-Sensing, vol. 28, no. 13, pp. 2977 –2994, 2007.
[2] M. Pesaresi, A. Gerhardinger, and F. Haag, “Rapid damage assessment of built-up structures using VHR
satellite data in tsunami-affected areas,” International Journal of Remote-Sensing, vol. 28, no. 13, pp. 3013–
3036, 2007.
[3] M. Pesaresi and E. Pagot, “Post-conflict reconstruction assessment using image morphological profile and
fuzzy multicriteria approach on 1-m- resolution satellite data; application test on the Koidu village in Sierra
Leone, Africa,” in Proc. Urban Remote Sensing Joint Event, Apr. 11–13, 2007, pp. 1–8.
[4] E. Pagot and M. Pesaresi, “Systematic study of the urban postconflict change classification performance
using spectral and structural features in a support vector machine,” IEEE Journal of Selected Topics in Applied
Earth Observations and Remote Sensing, vol. 1, no. 2, pp. 120–128, Jun. 2008.
[5] M. Matsuoka and F. Yamazaki, “Use of satellite sar intensity imagery for detecting building areas damaged
due to earthquakes,” Earthquake Spectra, vol. 20, no. 3, pp. 975–994, 2004.
[6] P. Gamba, F. Dell’Acqua, and G. Trianni, “Rapid damage detection in the bam area using multitemporal sar
and exploiting ancillary data,” Geoscience and Remote Sensing, IEEE Transactions on, vol. 45, no. 6, pp. 1582 –
1589, june 2007.
[7] L. Gueguen, M. Pesaresi, P. Soille, A. Gerhardinger, “Morphological Descriptors and Spatial Aggregations
for Characterizing Damaged Buildings in Very High Resolution Images,” Proc.of the ESA-EUSC-JRC 2009
conference. Image Information Mining: automation of geospatial intelligence from Earth Observation, Nov.
2009.
[8] L. Gueguen, M. Pesaresi, A. Gerhardinger, P. Soille, “Characterizing and Counting Roofless Buildings in Very
High Resolution Optical Images,” IEEE Geoscience and Remote Sensing Letters, in press.
53

ABSTRACT
Evacuation plans : interest and limits
ROUMAGNAC A. and MOREAU K.
PREDICT Services, France
alix.roumagnac@predictservices.com
Abstract
Facing natural disasters, the elaboration and use of evacuation plans becomes crucial. The
emergency management requires timely, fast and also reliable informations. In this objective the
use of space data has been identified and used to elaborate evacuation plans in short times. The
purpose of this communication is to present through an example, the interest and also the limits
and area for improvement of such initiatives.
As a subsidiary of EADS Geo-information Services, Météo-France and BRL, Predict Services has been
working on elaboration of safety plans and helping in decision for the activation of these plans for 8
years, in France and Haïti. Its crisis management expertise and knowledge has been employed to
evaluate the efficiency of evacuation plans elaborated with short time method and space data in
Haïti.
The communication will present the analysis of the short time and space data maping method used
for the elaboration of emergency plans in Haïti. The analysis will highlight the limits and aspects
that should be improved through two examples : the plan elaborated for Port au Prince, and the
other one elaborated for Léogâne facing a cyclone warning.
The analysis will focused on spatialisation of threats, location of final accomodation, priority
evacuation areas, escape roads, public reception facilities.
55

SHORT PAPER
Outside the Matrix, a review of the interpretation of Error Matrix
results
VEGA EZQUIETA P. 1 , TIEDE D. 2 , JOYANES G. 1 , GORZYNSKA M. 1 , USSORIO A. 1
1
European Union Satellite
2 Z-GIS Research, University of Salzburg
p.vega@eusc.europa.eu, a.jimenez@eusc.europa.eu
1. Introduction
In the context of validation of a geographic information product there are several methodologies that could be
potentially applicable. The Error Matrix is a validation methodology widely accepted. The reasons for that are
many. First of all, is a scientific approach that follows a methodology that is equal for all validation processes,
and the results of different processes could be comparable to each other. Also, the Error Matrix “…is a very
effective way to represent map accuracy in that the individual accuracies of each category are plainly described
along with both the errors of inclusion (commission errors) and errors of exclusion (omission errors) present in
the classification.”(Congalton and Green, 2009).
As defined by Congalton and Green, the Error Matrix is without any doubt an optimal way of measuring the
degree of convergence of two datasets, the one used for the map, and the one used as reference data, also
referred as “ground truth”.
The implementation of this methodology as a valid validation method (if you’ll forgive the repetition) has to
take into consideration the conditions of the input data. The Error Matrix processes classified data. Both Map
Data (or Classified Data) and the Reference Data (or Ground Truth) need to be provided to the Error Matrix
under a classification scheme. The classification scheme must follow certain conditions. In particular the
classes of the scheme must be mutually exclusive and totally exhaustive. To these two conditions we would
add the concept of semiotically balanced that will be explained later on in this article.
The application of the Error Matrix methodologies to products that are not classified following these criteria
produces distortions in the interpretation. The problem starts when some products, commonly accepted by
the user community, cannot match the conditions of Congalton and Green. The lack of any other analytical
methodology alternative to this makes many products being validated with an Error Matrix, even if they don’t
match these conditions.
The distortions of the interpretation of these products come actually from the implementation of a
methodology that, even if optimal in some cases, is not universal. The distortions of the Error Matrix lie,
actually, outside the Matrix.
In this paper we will make an intellectual exercise, a test with several example products, taken from
operational activations of the project G-MOSAIC, and test them against the Error Matrix (at least theoretically).
This exercise will help us understand the limits of application of the Error Matrix.
57

2. Error Matrix in a nutshell
The purpose of this paper is not to explain in detail the functionality of an Error Matrix. This paper is written
under the assumption of a basic knowledge of Error Matrices and its application in validation. Nevertheless, a
very schematic explanation is provided here for the sake of those who are not so familiar with the topic.
As defined by Congalton, “…an error matrix is a square array of numbers set out in rows and columns that
expresses the number of sample units assigned to a particular category in one classification relative to the
number of sample units assigned to a particular category in another classification.”
An example of an Error Matrix could be seen here:
Reference Data
Water Crop Trees Row Total
Water 75 5 12 92
Map Data
Crop 13 50 14 77
Trees 2 3 76 81
Figure 1 Error Matrix Example
Column Total 90 58 102 250
The columns represent one dataset (the Reference Dataset, for example), and the rows represent the Map
Data (or Classified Data). So, in 50 occasions, a sample of Crop was correctly assigned, because both the
Reference and the Map data matched.
Out of 250 samples (the sum of all rows total and columns total), 201 (75 + 50 + 76) were correctly assigned
(and 80% of accuracy). Nevertheless, this figure represents an overall accuracy. The accuracy can also be
calculates per class, and different from the point of view of the producer of the map and the point of view of
the user of the map. For example, the classification of Trees has an accuracy of 74% from the point of view of
the producer (76/102), but it has an accuracy of 93% from the point of view of the user of the map (76/81).
This difference means that too little areas in the map were classified as Trees in the map.
3. Typologies of products to be evaluated
Products to be tested against an Error Matrix would be selected following a classification of products oriented
to the type of data used to compose a map. A map itself, as a final product, is always a raster file where the
values of the pixels are grouped representing meaningful (if possible) information that the reader in the map
can decode in its mind. The problem is that, under the point of view of validation, looking at the final map is in
fact an error. The resampling method used to visualize the map could alter the values of the pixels, some pixels
do not represent geographical information (such as a scale bar) or do not represent ground information (such
as geographical grid).
For this reason, the data to be assessed should be rather the information of the map, not the map itself. And
here, in assessing the information used to compose the map, is when we find different types of information.
There is a commonly accepted difference between raster and vector, as the two basic ways of storing
geographical information. There are other ways to classify the information, and attending to the conditions of
classification schemes exposed by Congalton, an appropriate classification of geographical data could be
according to its continuity or discontinuity. This way, under this classification perspective, the geographical
information used to compose a map could be either Coverage Composed or Feature Composed. What is
meant by each of those two types in now explained:
58

Coverage Composed: Information is coverage composed if all the places of one specific area have a
meaningful value. The first interpretation of this would be believe that “coverage composed” is equal
to “raster”, but this is not necessarily true. The next table provides examples of Continuous Datasets.
This dataset, for example, is continuous
vector coverage. The vector information
represents the urban blocks, and the
color the amount of damage identified.
Transparent has been chosen as the
color for “Not Damaged”, being
meaningful for that reason. This dataset
is continuous and mutually exclusive, so
it could be potentially tested by the Error
Matrix, as long as the process to create
the Reference Data is equivalent to the
one used to create the Map Data.
This dataset is an abstract surface
representing damage derived
automatically. The indicators of damage
are, in this case, the changes in the
shadows of buildings. These indicators
are then interpolated generating a
continuous surface. This product does
meet the conditions exposed by
Congalton and Green, but because there
is a difference between the data and the
representation of the data, this could
lead to distortions if validated with an
Error Matrix.
This dataset is a classified image where
all the information belongs to one of the
classes of the schema. This is the type of
dataset that Congalton and Green uses
as example when explaining the use of
Error Matrix and it matches perfectly all
conditions. This dataset can be perfectly
validated with that method, and it will
provide relevant information.
© Digitalglobe, 2010
Figure 2 Damage Assessment Chile Earthquake.
Concepción. Feb 2010.
Figure 3 Automatic Damage Assessment. Carrefour, Haiti.
Jan 2010.
© Digitalglobe, 2011
Figure 4 Example of a classified image
Feature Composed: Information is Feature Composed if not all places in the area have meaningful
value. The first interpretation of this would be to believe that “vector” is equivalent to “Feature
Composed”, but this is not necessarily true because a raster layer can be composed of a mask-type
59

data, in which certain areas are highlighted with a value and the background is left as values of 0,
meaning areas with no relevant meaning.
This dataset is a situation map. Within a
certain level of complexity, and if it
includes terrain information, these maps
are also known as topographic maps.
This datasets is vector discontinuous,
with different geometries used to
represent different objects. For this map,
29 different elements are extracted and
have a differentiated symbol of the
legend. Some of these symbols are
semantically related (main road and
highway) and some are complete
different classes. If this dataset is
validated using Error Matrix, distortions
of the result will be produced.
This is a crisis map. It is different from
the situation map from the point of view
that it adds a layer of analysis to the
background data. From this point of view
is more and object oriented dataset,
where the quality of the information will
be directly related with the capacity to
properly associate objects related with
the issue that needs to be mapped. A
map of built up areas at risk would be an
example. Another example is the one
present here in the figure on the right.
The information represented here are
changes related with the altering of a
river course. Once again, this dataset is
not continuous and is not mutually
exclusive either (there is only one class).
© Geoeye, 2011
Figure 5 Example of situation Map over Abidjan, Ivory
Coast.
© Geoeye, 2011
Figure 6 Example of criss map, where relevant changes are
highlighted. Costa Rica - Nicaragua, December 2010.
60

This in an analytical map. This map
includes information that cannot be
validated from the ground because it
requires contextual interpretation. In this
case, an evacuation plan representing
the optimal routes to escape a given
location. Each and all objects (the
evacuation routes) must be validated as
a whole, and taking samples along the
route will not provide a real value of how
good the plan is.
© Digitalglobe, 2011
Figure 7 Evacuation Plan Sana'a, Yemen. June 2011.
4. Conditions of the Error Matrix, and how they are not met
The conditions exposed by Congalton and Green are clear and a good filter for the information to be processed
by an Error Matrix. As we have mentioned, not all products accepted today as useful products meet these
conditions.
4.1. Mutually Exclusive:
The classification scheme of the input data can be considered as mutually exclusive when “…each mapped
area fall into one and only one category or class.”
Surface Abstraction: Like a density analysis, for example. The Automatic Damage assessment of
Carrefour, as seen in Figure 3, can be represented as different classes but that is just a representation
method. The number of classes is no more than an option in the process of an algorithm that
calculates the density of damage indicators. Each class cannot be considered as mutually exclusive as,
in fact, there are no classes, but an infinite range of values representing density that are, for
cartographic purposes, represented in classes.
Topographic Maps: Topographic maps do not have the information organized with this principle in
mind. For example, buildings are understood as belonging to a Built Up Area, and a point features,
such as a tower pylon, are decoded in our mind as “on top” of another feature (like a crop field, or a
forest).
Crisis Map: There could be no exclusion because there are no classes. In the example of Figure 6 all the
information is “changes related with the alteration of a river course”. This level of abstraction is not
trying to provide classification tags to particular geometries, but something completely different.
Analytical Map: Some objects of the map are more relevant than other, and in fact they live in
different layers of information. For example, in the case the same road can be a “highway” and an
“evacuation route”. In a Mutually Exclusive environment, the only relationship allowed between
objects in a map is “they touch each other and do not overlap”, which is clearly not true in an
analytical map such as an evacuation plan.
4.2. Totally Exhaustive:
The classification scheme will be totally exhaustive when “…every piece of the landscape will be labeled,”
(Congalton and Green, 2009) This condition can only be met by Coverage Composed data. All the examples
seen before and classified as Feature Composed will not match this.
From the examples seen before, several products would not be totally exhaustive, like the relevant changes
map of Figure 6, topographic maps or evacuation plans like the one in Figure 7.
61

The creation of an “unclassified” or “other” class solves the issue only in a formal way. With such class, the
Error Matrix works, but the meaning, the interpretation of the validation result is altered, as the confusion
between classes means something different if we are talking about two meaningful classes or not. Also, the
creation of an “unclassified” class introduces a distortion between the positional accuracy of an object and its
thematic assignment. How an object is extracted (where is the limit drawn) will influence the result, giving the
idea that a commission of one class (class X) is the omission of another class (unclassified class), when this is
actually not a thematic error, but a positional error.
All this can be easily explained in the next diagram:
Unclassified
Class
Unclassified
Class
Thematic
Agreement
between classes
Figure 8 The inclussion of "Unclassified" class can distort the perception of positional and thematic validation.
4.3. Semiotically Balanced
There is another condition, not expressed by Congalton when enumerating the list of condition for a
classification scheme. It is not specified as a condition, but when exploring the possibilities of fuzzy
classification it can be inferred. For this reason, this condition should be explicitly included at the same level as
the other two. Semiotically balanced means that all classes should have something in common. If they need to
be mutually exclusive and totally exhaustive, they need to be also on a similar semiotic level. For example,
“tower pylon” and “crops” are two non semiotically balanced classes because they have nothing in common.
The existence of one (the commission of one) does not imply the non-existence of the other (the omission of
the other). “Water” and “Built Up Area” are semiotically balanced because they are both land use or land
cover features. That’s what they have in common. Depending on how they are balanced, the semiotic aspect
of the class can be:
- Thematically balanced: For example, if we take the next list: (Main Road, Secondary Road, Local Route,
River), we create an unbalanced selection, because mistakenly tagging an object as Local Route when
it is a Secondary Route is not the same as mistakenly tagging that same object as River.
- Geometrically balanced: Objects of different geometries (points, lines and polygons) are not balanced
because lines do not exclude polygons, points can be inside polygons… topological relationships
among these objects are complex.
5. Distortions in the interpretation derived from a wrong input in the Matrix.
The distortions derived from using datasets that do not match the above mentioned conditions are many. The
following list represents many of these distortions found so far in operational products. This list does not
represent all possible distortions.
5.1. Confusion of Spatial and Thematic Accuracy
As advanced before in Figure 8 the comparison of thematic accuracy between a thematic object of a give class
and the background can be confused with positional inaccuracies. This is particularly true in the case of
situation and topographic maps.
62

In the example in the right, there are several
objects extracted. In particular there is one
place identified as “Checkpoint”. If a sample
for an Error Matrix is taken there, that
sample would return as “Checkpoint”. If this
point was displaced then the ground sample
would probably be classified as “Harbour”. If
the point was gone, missed, then the ground
sample would probably be classified as
“Harbour” as well. A positional inaccuracy
(wrong location of the checkpoint) and a
thematic inaccuracy (omission of a
checkpoint) would be classified by the Error
Matrix as a thematic inaccuracy, an omission
of the “Checkpoint” combined with a
commission of the “Harbour” class.
© Geoeye, 2011
Checkpoi
nt
Figure 9 Situation Map over Ivory Coast. Entrance of a
Harbour facility.
5.2. Equal Value of Errors
In maps with a high complexity of classes, like a topographic map, the errors can have a different operational
meaning. It is true that the Error Matrix allows us to identify the relationship of confusion between classes, but
that does not imply a graduated quality of the errors, so the overall quality is not affected by the different
qualities of errors.
In the example in the right, in very dry areas
streams can look like trails (and in fact they
are used as such). Some classes are very
close in concept, like an Unpaved Road and a
Trail. Confusion one with the other has a low
impact in the usability of the map. Confusing
a dry stream with a trail means a much
higher impact.
Error Matrix can deal with this by applying a
“fuzzy approach”. In this fuzzy approach, a
step of one class of error is accepted as a
correct result, in a certain way “widening”
the central diagonal of the Matrix.
© Geoeye, 2010
Figure 10 This dry stream could be mistaken as a path,
Somalia, June 2010.
Unfortunately, this is only true for classification schemes where all classes cover the same topic and are
gradually different (like types of roads, or levels of damage). In an Abstract Surface like the one presented in
Figure 3 this error would be solved by applying a fuzzy approach.
5.3. Different Results of validation depending on the representation method
Density surface maps (i.e. damage density like the automated Carrefour damage assessment) start from base
data (here: damage indications) and create a surface of information (density map), in a certain way blurring
the information in order to decrease inaccuracies (avoiding selling a high level of accuracy the data does not
63

have, since it is based on damage indications - which were in the case of the automated Carrefour damage
assessment the change of shadows casted by the buildings). Again, the application of standard accuracy
assessment routines for these kinds of maps is, however, limited (Kerle, 2010).
This dataset was validated (see also Tiede et al. 2011) in an exercise post validation with the Haiti Earthquake
2010 “Remote sensing damage assessment: UNITAR/UNOSAT, EC JRC and World Bank” several weeks after the
event. For a comparison of damage intensities between the data sets a kernel density map was also derived
from the reference data set by using damage grades which were likely to have affected the shadows cast by
buildings, which was the indicator of damage identified in the automatic damage assessment of Carrefour
performed by G-MOSAIC.
A rank difference calculation between the different damage density classes was used for the evaluation of
similarities between the two maps. A fuzzy approach, accepting neighboring classes (rank difference between -
1 and +1) as still valid, was carried out. Following Congalton and Green (2009) such an approach is acceptable
if the classes are very similar or even continuous (like in this case) and not discrete.
The validation was then oriented only to the distribution and damage densities, and not to the damage
assessments themselves (since both datasets were performed with a very different focus). This way, the
validation was applied on the representation of both datasets (a kernel density map derived from both
datasets). For the validation the density surfaces where categorized into different damge density classes..
After that, the density surface is then represented is steps. Changes in this process of representation can alter
the final look of the product (not so the essential idea of the map, which is to provide the user with the
knowledge of how the damage intensity is spread in the area). This final look is what will be validated. The fact
that changes in the representation can change the validation result highlight the possible inadequacy of the
methodology used for validation.
Figure 11 Cloud of Points
Figure 12 Kernel Density derived from
the previous points
Figure 13 The same density layer
represented in classes
In the figures above we can see how different the data looks, and it is derived from the same dataset. Each of
these steps is subject to parameters that may alter the final look. These alterations may not be important for
the interpretation of the map by a human being (the human brain will understand which areas are more
damaged and which ones are not and will process the area as a whole). An Error Matrix may return different
values depending on these changes, which is a major flaw in a validation system.
5.4. Limitations in the in the interpretation of Abstraction
Some datasets are not exactly a direct representation of what is in the real world in the same exact location.
Some imply a certain level of abstraction.
- Abstract Surfaces: An Abstract Surface can be perfectly false in 99% of its surface, and still be a very
good product. As in the example below, an Abstract Surface claims that the damage is evenly
distributed. This is not true, damage is discrete, and in case of an earthquake, can be found in
buildings in different levels. If samples are taken in the ground, probably many places can be
considered as “Not Damaged” by the reference data, while will still be represented as “Damaged”
because of its proximity to damaged buildings. The only way to avoid this would be to take a
64

theoretical sample the same shape and size of one class and process it as one single sample, as
explained in the diagram below. Most of the times this is too big and will always be very open to
subjective interpretation.
A sample here does not
take into consideration the
proximity of other values.
A valid sample would be
like this one, but it would
be very open to
subjectivity:
Figure 14 Diagram showing the sampling problems in abstract surfaces
- In case of an evacuation map, some features depend on the context. An evacuation route, for example
is either good or bad for as long as the whole feature is good. Taking a sample in a part of the map
defined as “evacuation route” will only reflect if that part matches the criteria in that point, as it
cannot take into consideration context information.
An evacuation route could have an
unacceptable choke point somewhere,
making the whole feature invalid. Plus, it is
also interesting to assess if it is in fact the
most efficient path between point A and B.
That will not be reflected by an Error
Matrix.
As in the image in the right, taking it as a
sample doesn’t provide us information
about the starting and the end point of the
route.
© Geoeye, 2011
Figure 15 An evacuation route in Yemen, June 2011
5.5. Object Oriented Correlation
Congalton and Green already mentioned this in their book. Non-Site Specific assessments would not account
for the spatial correspondence, while Site Specific assessment would. Although, it is not clear how the
application of an Error Matrix approach can help to reduce this problematic. A stratified random sampling
technique will consider the distribution of classes in a map, but is still not object oriented.
In the next example it can be seen how the lack of an object oriented correlation can produce distortions in its
interpretation:
65

© DigitalGlobe, 2011
© DigitalGlobe, 2011 © DigitalGlobe, 2011
A B C
Figure 16 Ground Truth data for the
Costa Rica / Nicaragua water stream
modifications, March 2011
Figure 17 One possible dataset for
the Costa Rica / Nicaragua water
stream modifications, March 2011
Figure 18 An alternative dataset for
the Costa Rica / Nicaragua water
stream modifications, March 2011
These three figures represent changes that are relevant as indicators of a modification of a river stream in a
border area between Costa Rica and Nicaragua.
- Dataset A: Represents the Reference Data or Ground Truth. This is the data B and C could be validated
against following the Error Matrix method.
- Dataset B: Is a dataset where one of the areas is missing, but the other two are exactly equal to the
one in Dataset A. The correspondence between both datasets could be 65%.
- Dataset C: All three areas are detected, but are roughly depicted, not matching well with dataset A. As
a result, the correspondence between both datasets could be of 65%.
As a conclusion, both datasets are validated as equally good (or wrong). Now, if the idea was to identify
relevant indicators of the modification of a river stream, which of the two datasets represents better the
Reference Data? C and B are not the same in terms of capacity to send information, as B completely misses
one change that is relevant. The result from an Error Matrix point of view, although, could be the same.
6. Conclusions
There is one main conclusion obtained from this paper, and it is the fact that the Error Matrix is a methodology
developed to validate geographical information produced through classification. Trying to extrapolate this
technique to other typologies of datasets introduces distortions that are not so easily detected. They are not
so easy to spot because the Error Matrix system still works and produces results. Only a detail examination
based on experience can spot these distortions that are, in the other hand, impossible to measure.
It is not an easy task to develop a full proposal for new validations systems, and that falls beyond the reach of
this article, but few ideas could be drafted:
- Any evaluation of the quality of a product should take into consideration the purpose of the product
and not only the actual quality of some data parameters. When evaluating the data some features of
the dataset might be easy to measure, but that doesn’t mean they are related with the purpose of the
data. Geographical data can undergo several levels of abstraction, making the information found in the
dataset less connected with the actual features found in the ground, but without decreasing the
overall quality. Examples of very abstract datasets could be found in Analytical Maps and Crisis Maps.
- Identifying different typologies of datasets and applying different methodologies of validation might
seem cumbersome, but could be in fact simpler approach. With ad-hoc solutions applied to different
types of datasets (like the ones proposed in this article, for example) the variability among the
different datasets where validation is applied is drastically reduced and optimized systems can be
developed. Error Matrices could be considered as the optimal way for validating classified imagery
(thematic rasters).
66

References:
Congalton, R.G., and K. Green, 2009. Assessing the Accuracy of Remotely Sensed Data. Principles and Practices,
CRC Press, Boca Raton, London - New York, 183 p.
Kerle, N., 2010. Satellite-based damage mapping following the 2006 Indonesia earthquake - How accurate was
it, International Journal of Applied Earth Observation and Geoinformation, 12(6):466-476.
Tiede, D., Lang, S., Füreder, P., Hölbling, D., Hoffmann, C., Zeil, P., 2011. Automated damage indication for
rapid geospatial reporting. An operational object-based approach to damage density mapping following the
2010 Haiti earthquake. Photogrammetric Engineering & Remote Sensing, 77 (9), 933-942.
Vega Ezquieta, González, Grandoni, Di Federico, 2010. Product Design, the in-line quality control in the context
of rapid geospatial reporting, VALgEO 2010, ISPRA, Italy, 3 p.
67

ABSTRACT
On the complexity of validation in the security domain –
experiences from the G-MOSAIC project and beyond
KEMPER T., WANIA A and BLAES X.
Joint Research Centre of the European Commission, Italy
thomas.kemper@jrcj.ec.europa.eu
Abstract
The geospatial and remote sensing communities have understood that geospatial products in
support of crisis management will be used in decision making only if they are validated properly
and the accuracy is known – this workshop is a manifestation of this. Crisis managers need to know
how accurate the information they receive is. With information on the accuracy of a product, they
can often even accept less accurate information.
Crisis management is not limited to natural or man-made disasters. Earth observation and
geospatial information provide also ‘intelligence’ for humanitarian aid and civil protection
operations. Applications include for example damage assessment during or after conflicts,
identification and enumeration of refugee’s or IDP’s or the monitoring of landuse changes in the
context of a crisis. While validation is very demanding in itself– it is even more complex in the
security domain. In many cases, in particular in violent conflicts, the accessibility to the areas is
limited or impossible and also the availability of trusted, unbiased reference sources is limited. On
top of this, questions of confidentiality may have to be taken into account, where it might be
impossible to disclose information related to such products.
This presentation will give examples of the problems when validating services in the security
domain and aims at providing ideas for validation strategies taking into account the above
mentioned peculiarities. This will be presented based on case studies. An important role in such
strategies is on the one hand the involvement of contacts to people in the field; on the other hand
also ‘remote’ validation based on visual interpretation and cross-comparison are options to
consider.
69

SESSION III
USABILITY OF WEB BASED DISASTER
MANAGEMENT PLATFORMS AND READABILITY
OF CRISIS INFORMATION
Chair: Tom De Groeve
Crisis management is a compelling domain for applying web-based GIS services as a
mean to respond efficiently to demands of time-pressured disaster response. Crisis management
applications require web mapping technologies accessible and customizable to non-specialist end
users. Interoperable and open wed-based geographic information services are being developed
and increasingly used in operational crisis management. What are the characteristics of these
systems that make them suitable for emergency response and crisis management and
differentiate them from traditional web mapping platforms? What are the specific usability issues
that need to be taken into account when designing the interfaces and defining the functionalities
of these systems? How decision makers in emergency management are operationally exploiting
web-based GIS services for monitoring and responding to crisis situations? This session will spot
the light on current state-of-the-art WebGIS implementations developed in support for crisis
management. The focus will be on usability studies as a critical validation checkpoint in the
development of these applications. The end-users feedback on experience using WebGIS services
in everyday operations will also be discussed.
Thanks to the increasing availability and capability of EO sensors, the monitoring of phenomena
occurring on the Earth surface is improving continuously. The effort of public and private
institutions is translated into the implementation of new services providing near-real-time
information about disaster events. These services allow the actors involved in emergency
management and rescue operations to have maps displaying up to date information about the
crisis situation. It is crucial to evaluate how much these crisis products are actually used during
the operations and for which specific purpose. Besides it must be analyzed if map layouts are
optimized with respect to the users’ needs to allow a quick and effective interpretation. In this
session, the map readability is also explored through real cases.
71

SHORT PAPER
Emergency Support System: Spatial Event Processing on Sensor
Networks
SZTURC R., HORÁKOVÁ B. , JANIUREK D. and STANKOVIČ J.
Intergraph CS
roman.szturc@intergraph.com
Abstract:
Command and control systems used in crisis and emergency management are designed for
decision making and to control resources to successfully accomplish missions. An important aspect
of command and control system is therefore situational awareness – information about the
location and status of resources. Emergency Support System (ESS) is a suite of real-time spatial
data centric technologies useful in abnormal events, as well as 7th Framework Programme project
of the European Commission under theme Security.
ESS integrates data from various data collection tools, like cell-phones, unmanned ground stations,
unmanned aerial vehicles, air balloons, etc. Measurements of these data collection tools are based
on the OGC Sensor Web Enablement (SWE) standard series and standards for video transmissions.
Proprietary communication protocols and data formats are replaced by open interfaces. All data
and services are harmonized in the Data Fusion and Mediation System (DFMS), a central
component of ESS and published on the ESS Portal.
Information sources made available through systems such as ESS may continuously generate a
significant amount of data. If not filtered and processed appropriately, this data load may threaten
to overwhelm a decision support system. The DFMS addresses these issues through the Spatial
Event Processing (SEP) mechanism which defines publish/subscribe messaging pattern on the top
of the SWE request/response pattern to facilitate the dissemination of information in a timely
fashion.
This approach in ESS enables command and control systems to get the information they are
interested in as soon as this information becomes available. The mechanism supports the definition
of precise filter and processing rules so that only the relevant information is transmitted. This saves
both, communication and processing resources of systems outside the DFMS, such as the ESS
portal and command and control systems.
Keywords: data fusion, OGC, spatial event processing, sensor, subscribe/notify.
73

Introduction
Complexity of tasks in crisis and emergency management has significantly increased in last decades. One of the
main reasons is the necessity to cooperate between various institutions and domains. More sophisticated
tools are being developed to satisfy the needs for production, processing and displaying information to the
users of command and control systems. Volume of data within these systems is arising. Effective crisis and
emergency management is based on quick decisions. On the other hand, coordinated approach as well
accurate and processed information are needed as well.
Emergency Support System (ESS) has been designed as a suite of real-time spatial data centric technologies
that collects process and provides relevant information about a crisis event. It should be seen as the
supporting system to existing systems in crisis and emergency domain instead of the replacement of existing
command and control systems. At the same time, ESS is an FP7 European research project, under theme
Security, funded between 2009 and 2013. The ESS consortium, consisting of 19 partners, is developing a crisis
communication system that is going to reliably transmit filtered and re-organized information streams to crisis
command systems, which will provide the relevant information that is actually needed to make critical
decisions.
This paper will provide you with the issues of ESS high-level architecture, adoption of the OGC Sensor Web
Enablement (SWE) as well contribution to principles and pilot implementation of the spatial event processing
mechanism.
ESS architecture
ESS architecture is the basis for developing the Emergency Support System. For this reason it was analysed
from several points of view:
• functional and non-functional requirements model,
• components derived from the requirements and the relationship matrix,
• detailed component design model,
• use case and dynamic model,
• detailed description of fundamental architectural aspects that shall be taken into account for the
major ESS layers sensing, service and portal as well as ESS alert system.
Only overall component model architecture will be described further due to the limited extent of this paper.
The ESS component model (see Figure 1 below) provides an overview of the high level architecture of ESS. The
main purpose of this model is to define the organization and dependencies of the system components.
External components are modelled as well in order to improve the understanding of ESS boundaries and
potential interactions with external systems as described above. The architecture depicted in Figure 1 is not a
monolithic package but a system consisting of components and subsystems. Note that not all aspects of the
architecture are covered herein.
ESS consists of three layers – sensor (Data Collection Tools), service (Data Fusion and Mediation System) and
portal (ESS Portal). Sensor layer collects and pre-processes data from various sensors, service layer processes
sensor data and adds different kinds of resources (like background spatial data and information from external
services) and publishes the information via Web services. ESS Portal is a client of ESS services offering
advanced business logic.
74

ESS integrates several existing front end data collection technologies (sensor measurements, cell-phone data
gained from IMSI catcher, etc.) into a unique platform, which is the primary task of Data Collection Tools
(DCT). Besides inputs from DCT, other inputs (such as external Web map services, non-ESS resources and
simulations) are intended as well.
Figure 1. Emergency Support System architecture – main components and subsystems.
Data Fusion and Mediation System (DFMS) is the centralized subsystem working over the ESS database, which
is connected to all front end sensors – through SWE as described in section 3 and other resources activated in
or connected to the system. DFMS oversees communication between sensors and the database, data
harmonization from various sensor products of one type, the fusion of data from various types of sensors,
spatial data localization, and the transmission of data to the ESS Portal subsystem via standardized interfaces.
Transmission is based on open interfaces compliant to the OGC (Open Geospatial Consortium) specifications –
especially Web Map Service (WMS), Web Feature Service (WFS), Filter Encoding (FE) and Catalogue Service for
Web (CSW) as described for crisis and emergency management by Řezník, 2010. Intergraph CS, as one of the
industrial partners, is involved in most of the project tasks including WP5 (DFMS) leadership.
The ESS Portal is the client application of the DFMS within the Emergency Support System. It represents the
user interface, which contains all graphical components, contextual components, log access, etc. and manages
data exchange between underlying layers. It provides functionalities to export data from ESS to other systems.
The ESS Portal can provide any kind of functionality to external systems on the basis of its internal capabilities.
These “applications” are offered in the form of Web services.
75

Sensor Web Enablement adoption
Sensor Web Enablement (SWE) is a set of OGC standards, working group within OGC and hundreds of
implementations around the world. According to Botts et al., 2008, SWE “refers to web accessible sensor
networks and archived sensor data that can be discovered, accessed and, where applicable, controlled using
open standard protocols and interfaces (APIs).” SWE is not a stand-alone initiative since it is harmonized with
other OGC standards dealing with spatial data. Typical applications of the SWE are water sensors (flood
applications), radiological sensors, pollution sensors, Webcams, air- and space-born sensors, mobile heart
sensors and countless other sensors and sensor systems. Only the aspects highly relevant to the ESS will be
described further due to the extent of this paper. More detailed description may be found for instance in Botts
et al., 2008 or Jirka et al., 2009.
ESS uses four basic standards of the OGC SWE portfolio – Observation & Measurements Schema (O&M),
Sensor Model Language (SensorML), Sensor Observation Service (SOS) and Sensor Planning Service (SPS). O&M
is the general conceptual schema where SensorML is the exchange format used within two ESS sensor services
– SOS and SPS. SOS is used for accessing sensor data and metadata, while SPS serves for the parameterization
of a sensor (system) like UAV (Unmanned Aerial Vehicle). SPS is responsible for customization of the
measurements from several points of view – like positioning the sensor, setting the range of sensor
measurements, etc. OGC Sensor Alert Service (SAS) as well as Web Notification Service (WNS) are not intended
in the ESS architecture since their original functionality has been replaced and extended by the spatial event
processing mechanism (as it is described in the following section).
To be compliant with the SWE standards, ESS needs to support the core functionality defined by SOS and SPS.
In addition the SOAP binding of these service operations has to be supported for both services.
Further constraints for SOS are (beyond issues of data format profiling):
• The latest SOS version defines a spatial filtering profile that enables enhanced observation retrieval by
defining additional rules to perform more precise spatial filtering of observations. This profile should
be supported by SOSs accessed by ESS system.
• The GetFeatureOfInterest operation should be supported to enable clients like Portal to prepare the
graphical display.
• Some kind of indicator may be needed in the SOS metadata to distinguish whether a specific sensor is
stationary or not. This would help to display the observation data appropriately. For example, creating
a chart of observations made for one feature of interest is useful only if there are multiple
observations for that feature – which is usually the case for stationary sensors but not for the moving
ones.
• The GetObservationById operation may be needed for auditing purposes, especially if derived
information provides lineage information – such as which basic information ultimately led to the
derived data.
• The transactional operations are not needed in ESS at this stage as DFMS is primarily acting as a client
to existing SWE services. As such, sensors that are deployed in the field should already bring their own
service support with them. However, if that is not the case then for sensor plug-and-play these
operations would be required.
• The DeleteSensor operation is not needed in ESS, as sensor data should be permanently available to
the system (primarily for auditing purposes).
Further constraints and discussions for SPS are:
76

• The definition of tasking profiles. Control of sensors like video cameras stationed on Unmanned
Ground Stations (UGS) or Unmanned Aerial Vehicles (UAV) is an important task in ESS. The definition
of a simple SPS tasking profile would allow using a uniform and open interface to control such sensors.
• Cancelling a task is supported by sensor systems.
• Task reservations are not required in ESS. At this stage of the project, a task prioritization mechanism
is based on assignment of the SPS tasking functionality for certain sensor resources to the user with
the highest priority.
• Computation of feasibility of a specific tasking action is supported, as the service internally have to
support such checks upon a task submission.
• In order to support auditing functionality, the state logger conformance class of SPS, which defines
that a complete status log is stored for a task or tasking request, is supported in ESS.
• The minimum time period how long status information of a task / tasking request is stored by an SPS is
not defined in the standard, only that each SPS instance defines such a time. However, that can be one
month, one hour but also only one second. Thus in ESS defines a minimum time period that needs to
be supported by all SPSs that ESS interacts with.
• Publish/Subscribe functionality is realized by the services to support event processing functionality as
discussed in the following section.
SPS as well SOS support the DescribeSensor operation defined by the SWE. Support of the sensor history
provider and sensor history manager conformance classes (which basically enable the time based retrieval and
modification of sensor metadata) are not required in ESS. A minimal profile of Sensor Model Language
(SensorML) is used within ESS to describe most basic aspects of an ESS sensor.
Profiling the SWE services may continue during ESS implementation phase (i.e. between 2011 and 2012) in
order to be able to adjust to new developments and requirements.
Spatial Event Processing
4.1. Principles
Information sources made available through systems such as ESS may continuously generate a significant
amount of data from heterogeneous information sources. This data load may overwhelm an ESS Portal or a
decision support system if not filtered and processed appropriately. Usually, there are several manually or
semi-manually based processes of filtering the information. Spatial Event Processing (SEP) mechanism offers
significant advantages which are, among others, following:
• automatic filtering and processing mechanism with possibilities of manual inputs,
• based on the location of the event (and thus throwing away information from non-relevant places),
• repeatable mechanism that may be re-used for other processes.
Flood of information within a system such as ESS relies on the number of deployed, active and connected
sensors, frequency with which they gather data, frequency of data received via services and settings of the
video transmissions. ESS or a command and control system operator would be overwhelmed with this
information. Therefore, advanced visualization as well as aggregation techniques are needed inside the ESS. In
other words, the concept of a system like ESS should enable to aggregate low-level data into the higher-level
information that is relevant for the ESS/command and control system operator.
Principles of event processing are in general described for instance in Everding, T. et al., 2009. Spatial event
processing adds location point of view. So-called spatial window defines location relationship between
77

incoming events. As it is depicted in Figure 2, the spatial window is dynamic and computed as a buffer around
the line. In this example, start and end of this line are defined by the vehicle location and destination. Interior
nodes and edges of the line are determined by the route that the vehicle intends to drive. This example is
familiar from car navigation systems. Events that are located within the buffer zone are part of the window's
event set. At T1, the spatial window is shown in grey and events one and two are added to the event set. Let
us assume that these events signal slow the traffic speed. If a cluster with a significant number of such events
is detected in a given time interval, the driver would be notified and an alternative route suggested. This
clustering could be performed as a special select function. The vehicle moves on along its route. At T2, it has
moved about half its way. The window's buffer has been adjusted. Event four has been detected and is now
part of the window. Event three will automatically be rejected as it does not fulfil the window's entry criteria
(as it is not within the buffer zone). In T3, event one and two have been removed from the window's event set
while event seven is added. A number of events happened in the area where events one and two happened.
These events, e.g. indicating a traffic jam, would usually form a cluster. However, as they do not fall within the
spatial window, they are no longer relevant and thus not reported to the driver.
Figure 2. Dynamic spatial window for the Event Processing.
4.2. OGC Event Service
OGC has published the draft discussion paper in mid-2011, in version 0.9, dealing with the Event Service (see
Echterhoff, J. et al., 2011). Work on this paper has been supported by two European research projects – the
Emergency Support System project and the GENESIS project (European Commission, 2008).
OGC Event service is based on the WS-Notification (WS-N). According to Echterhoff, J. et al., 2011 “the Event
Service was developed to satisfy the need for having relevant data available at an OWS pushed to a client as
soon as it is available rather than having the client repeatedly poll the service.” It is obvious, that such service
is not intended as a stand-alone one. Combination with another (OGC) Web services was foreseen from the
beginning with the primary area of interest in the OGC Sensor Web Enablement domain. It is more valuable to
78

have a sensor with the publish-subscribe mechanism rather than use the request-response mechanism. OGC
Event Service consumer receives real-time data matching the filtered criteria of the respective subscriptions.
Publish-subscribe mechanism reduces the amount of needed workflows which results in the reduction of
transmitted data. Therefore, a publish/subscribe mechanism brings significant advantages in applications of
emergency and crisis management.
OGC SWE working group has developed the Sensor Alert Service (SAS) which supported its own
publish/subscribe based operations. Sensor Event Service (SES) was afterwards defined as the successor of the
SAS and tested in OGC Test beds. Since the most of the SWE specific functionality was not present in SES, only
pure WS-Notification based Event Service was finally deployed. SOAP (Simple Object Access Protocol) binding
in versions 1.1 and 1.2 was foreseen during the proposals and tests.
General OGC Event Service uses both patterns: request-response and publish-subscribe. Request-response
pattern has to be used in the first phase of communication to see what are the details of an OGC Sensor Event
Service (e.g. through a GetCapabilities operation). On the other hand, publish-subscribe pattern requires
specific considerations since one-way messages are sent after subscription. Especially, problem of returning a
fault in one-way communication arises. Another problem may arise when a simple client (i.e. not a Web
service) cannot be reached from another service (for various reasons such as firewall or being offline).
The event processing mechanism may contain filter statements. If there is not any, then all available data are
provided. Otherwise, data needs to be matched against the defined filtering criteria. OGC Sensor Event Service
defines three filter levels for such service:
• XPath for filters based on the event structure;
• OGC Filter Encoding for more sophisticated filtering (including spatial and temporal operators);
• OGC Event Pattern Markup Language for Complex Event Processing (CEP) and Event Stream Processing
(ESP) capabilities.
4.3. Implementation
As today’s systems use client-server architecture, data may be processed in general on either client or server
side. Client-side processing has significant disadvantages, since all data have to be passed to the client which
might result in a high traffic load. Server-side, on the other hand, is much more efficient solution. As
mentioned in section 4.2, several standards support server-side approach.
WS-Notification , an OASIS Standard currently in version 1.3, was selected as the ESS event processing system.
WS-Notification is very complex and modular standard containing both mandatory and optional parts.
Fortunately, mandatory parts form very base of the standard since all the rest are optional parts. ESS
implements the most necessary parts supporting required functionality of the system.
Events dispatched by WS-Notification can be categorized by so-called topics, so subscriber can receive only
events related to a given category. Several topics have been defined for ESS. For example,
NewResourceAvailable, MaintenanceEvent, SensorAvailabilityChanged, ExceptionDetected, etc. They indicate,
that a new resource has been introduced to the system, a resource needs a maintenance, availability of a
sensor changed, an unexpected situation occurred, respectively.
Still, categorization by topics is not adequate in some cases and even finer selection of received events is
required. For that purpose topics are able to support expressions specifying further reduction of events sent to
subscriber. In case of ESS topics ConditionSatisfied and GeographicalEvent have been defined. They enable
specifying a condition under which event is sent to subscriber. For instance “/S1/Temperature > 36,”
“/S2/WindSpeed > 12 AND /S2/WindDirection BETWEEN(30, 60)” or “/S3/Position IN BBOX(…).”
79

Note that WS-Notification standard does not define any formal language describing condition form. Instead, it
allows specifying so called dialect, which contains precise definition of used language. Therefore, it is up to the
user and server provider to define such a language.
Acknowledgements
This research has been supported by funding from the EU 7FP project with Grant agreement No. 217951
entitled Emergency Support System.
References
European Commission: CORDIS: FP7 : ICT : Projects : GENESIS : Generic European sustainable information space
for environment. (2008). Retrieved 2011-08-25, from
http://cordis.europa.eu/fetch?CALLER=PROJ_ICT&ACTION=D&CAT=PROJ&RCN=87874.
ESS Project. (2009). Retrieved 2011-08-15, from http://www.ess-project.eu/.
BOTTS, M., PERCIVALL, G., REED, C., DAVIDSON, J. (2008). OGC® Sensor Web Enablement: Overview and High
Level Architecture. In S. Nittel, A. Lambrinidis, & A. Stefanidis, GeoSensor Networks (Vol. 4540, 271 p.).
Berlin Heidelberg: Springer-Verlag.
ECHTERHOFF, J., EVERDING, T. (2011-08-11). OGC Event Service - Review and Current State. Retrieved 2011-08-
22, from Open Geospatial Consortium (OGC):
https://portal.opengeospatial.org/files/?artifact_id=45115.
ESS consortium. (2010-02-15). Deliverable D2.2 Report on High Level Software Architecture. Retrieved 2011-
08-22, from ESS Project: http://www.ess-project.eu/images/stories/Deliverables/ess%20d2.2%20.pdf.
EVERDING, T., ECHTERHOFF, J. (2009). Event Processing in Sensor Webs. Proceedings of the Geoinformatik
2009 - benefits for environment and society conference, 4 p. Osnabrück: Universität Osnabrück (IGF).
JIRKA, S., BRÖRING, A., STASCH, C. (2009). Applying OGC Sensor Web Enablement to Risk Monitoring and
Disaster Management. GSDI 11 World Conference, 13 p. Rotterdam.
OASIS. (2006). OASIS Web Service Notification (WSN) TC. Retrieved 2011-08-24, from http://www.oasisopen.org/committees/wsn/.
ŘEZNÍK, T. (2010). Metainformation in Crisis Management Architecture - Theorethical Approaches, INSPIRE
Solution. In M. KONECNY, S. ZLATANOVA, & T. BANDROVA, Geographic Information and Cartography
for Risk and Crisis Management, 429 p. Berlin: Springer.
80

ABSTRACT
Near‐real‐time monitoring of volcanic emissions using a new
web‐based, satellite‐data‐driven, reporting system: HotVolc
Observing System (HVOS)
GOUHIER M. 1 , LABAZUY P. 1 , HARRIS A. 1 , GUEHENNEUX Y. 1 , CACAULT P. 2 , RIVET S. 2 , BERGES J. 3
1 Laboratoire Magmas et Volcans, CNRS, IRD, Observatoire de Physique du Globe de Clermont-Ferrand,
Université Blaise Pascal, Clermont-Ferrand, France
2 Observatoire de Physique du Globe de Clermont-Ferrand, CNRS, Université Blaise Pascal, Clermont-Ferrand,
France)
3
PRODIG, UMR 8586, CNRS, Université Paris 1, Paris, France
M.Gouhier@opgc.univ-bpclermont.fr
Abstract
We present here a web-based system developed to achieve near-real-time detection and tracking
of volcanic emissions using onsite ingestion of satellite data and output of useable products via
implementation of off-the-shelf algorithms. The system, named HotVolc Observing System (HVOS),
was set up to allowin near-real-time tracking of ash cloud and lava flow emissions, and was first
tested during the April-May 2010 eruption of Eyjafjallajökull eruption and Etna’s January 2011
eruption. HVOS is hosted by the Laboratoire Magmas et Volcans (LMV) which is part of the
Observatoire de Physique du Globe de Clermont-Ferrand (OPGC) based at the Université Blaise
Pascal (Clermont-Ferrand, France). This system is based on the real-time reception and processing
of the full constellation of geostationary satellite data (MSG0, MSG-RSS, GOES-E, GOES-W, MTSAT,
Meteosat-7) allowing worldwide monitoring of volcanic events every 15 minutes. Currently, we
provide open-access, in real-time, to semi-quantitative data (ash index, lava index, SO2 index, RGB
index). This capability was used by French authorities (CMVOA) during both Eyjafjallajökull and
Grimsvötn volcanic crises. Quantitative products (i.e., ash concentration and radius, ash cloud
altitude, SO2 concentration) are also provided in near-real-time either on request or during
volcanic (explosive) crises, as are estimates of lava discharge rates during effusive crises.
81

SHORT PAPER
Image interpreters and interpreted images: an eye tracking study
applied to damage assessment.
CASTOLDI R., BROGLIA M. and PESARESI M.
Joint Research Centre of the European Commission, Italy
Roberta.castoldi@jrc.ec.europa.eu
1. Rationale
JRC is exploring the improvement of the human assessment of building damage by applying image
enhancement processing before photo-interpretation phase. The JRC has designed a set of experiments to
assess the effect of such processing on recognition mechanisms.
In the frame of the Geo-Information and Visual Perception project, we apply a cognitive approach 5 to the
remotely sensed imagery photo-interpretation process, exploring the possibility to improve the assessment of
building damage, traditionally carried out by the time consuming and error prone human interpretation. This
task is often performed following disasters to support the information needs of emergency rescue for
humanitarian relief intervention. Therefore, while on the one hand there is a high pressure to deliver a result
as quickly as possible, on the other hand it is of the highest importance to ensure the quality of the
assessment.
The ISFEREA action (Globesec Unit, IPSC, JRC) has developed several algorithms aimed at promoting the
salience of targets in complex backgrounds, with the purpose of improving semi and fully automatic image
information extraction. As a rich plethora of different processing methods could be at the photo-interpreter’s
disposal, it becomes increasingly useful to test if different processing methods have an effect on the subjective
task performance (quality and speed) of identifying building damage.
There are severe limits on our capacity to process visual information, due to the limits of the brain energy and
the neuronal activity 6 . Stimuli compete, attention filters. The more a stimulus is attractive, the more likely the
information incorporated will be processed 7 .
Therefore a processing method should support the interpreter - involved in a damage assessment - in visually
filtering the huge amount of information at her/his disposal.
5 See the work done by R. Hoffman (Hoffman 1984, 1989, 1990)
6 M.Carrasco, Vision Research 51 (2011) 1484-1525
7 Wolfe, J. M., Vo, M. L.-H., Evans, K. K., & Greene, M. R. (2011). Visual search in scenes involves selective and non-selective pathways.
Trends Cogn Sci, 15(2), 77-84 and Wolfe, J.M., Horowitz, T.S. (2004). What attributes guide the deployment of visual attention and
how do they do it? Nature Reviews Neuroscience, 5 1-7.
83

As study case, we chose the magnitude 7.0 earthquake in Haiti on 12 January 2010, because of the availability
of i) airborne imagery, which resolution allows for visual buildings damage assessment and ii) an official
damage assessment, which can be the starting point to measure the task performance.
The building damage assessment for the affected area has been carried out jointly by the United Nations
Institute for Training and Research (UNITAR) Operational Satellite Applications Programme (UNOSAT), the
European Commission Joint Research Centre (EC JRC), the World Bank Global Facility for Disaster Reduction
and Recovery (GFDRR) and Centre National d’Information Géo-Spatial (CNIGS) representing the Government of
Haiti.
The damage assessment has been conducted through the use of aerial photos provided by the World Bank
(World Bank-ImageCat-RIT Remote Sensing Mission), Google and NOAA, as well as satellite imagery from
GeoEye and Digitalglobe, by comparing pre-earthquake satellite imagery to post-earthquake aerial photos. The
spatial resolution (level of detail) for the satellite imagery used is approximately 50 cm while for the aerial
photos it is approximately 15 to 23 cm.
Image analysts have categorized buildings into different damage classes through manual photo-interpretation.
More teams worked with a coordinated approach at UNITAR/UNOSAT, at the EC JRC, at the World Bank, which
worked with a network of volunteer collaborators, GEO CAN (Global Earth Observation – Catastrophe
Assessment Network), and with ImageCat. The results of the photo-interpretation have been harmonized in a
point dataset. Each point represents a damaged building and the damage grade is classified according to the
European Macroseismic Scale (EMS) 19988. The visual interpretation of aerial ortho-photos allowed only the
proper identification of damage classes 4 (very heavy damage) and 5 (destruction), which in the following are
referred to as t4 and t5 respectively.
a) b)
Figure 1 - images a) and b) represent respectively examples of t4 target AOI and t5 target AOI both
“unprocessed”.
The damage assessment has been provided in rush mode and is affected by a certain degree of error; in
particular, validation using ground collected data showed an overall accuracy between 61 percent and 73
percent9, varying with respect to the damage classes. In order to improve the accuracy of the dataset, two
expert photo interpreters of the ISFEREA team performed a detailed revision of the images included in this
8 G. Grünthal (ed.), “European Macroseismic Scale 1998 (EMS-98)”, CONSEIL DE L’EUROPE, Cahiers du Centre Européen de
Géodynamique et de Séismologie, Volume 15, Luxembourg 1998
9 C. Corbane et al, “A Comprehensive Analysis of Building Damage in the 12 January 2010 Mw7 Haiti Earthquake Using High Resolution
Satellite and Aerial Imagery”, Photogrammetric Engineering & Remote Sensing Vol. 77, No. 10, October 2011
84

experiment. The resulting output has been used as the reference dataset to measure the subjective task
performance in identifying building damage.
The images below (Figure 2) show examples of JRC image processing chains under test in the current
experiment. a) “unprocessed” sub-sample of input image used during post-earthquake damage assessment in
Haiti (2010) in the operational image interpretation tasks, b) “unsharpened” the same image after conditional
local convolution enhancing small details, c) “cc64” the same image after a simplification based on alphaomega
constraint connectivity on connected components and d) “rubble” the same image with injected
knowledge-driven image information extracted by multi-scale differential morphological profiles (DMP).
In these experiments the JRC is examining the human photo-interpreters while performing a target detection
task on a given set of differently processed images in order to achieve a measure of the efficiency of the single
enhancement processing method. During these experiments we record and analyse thinking aloud – semistructured
interview, mouse click responses and eye movements. Figure 3 shows some examples of the output
obtained during the eye tracking sessions: (left) “heat map” representing density of duration and localization
of the fixations during the image analysis and (right) “gaze plot” representing the fixations order.
a) b)
c) d)
Figure 2 – example of image used for damage assessment in Haiti (2010) and some processing under test in
the current experiment. a) “unprocessed” raw image data, b) “unsharpened”, c) “cc64” and d) “rubble”
processing.
85

Figure 3 – example of eye tracking analysis output: (left) “heat map” (right) “gaze plot”.
2. Experiment #1
Before running the full experiment involving a considerable number of participants and a rich set of images,
we stepped into preliminary interviews and a pilot test, as described below:
Thinking aloud – semi-structured interviews: The thinking aloud – semi-structured interview is composed of
4 different image processing methods developed by the JRC. The aim of this preliminary semi-structured
interview was to collect individual opinions deemed to fine-tuning the pilot experiment. Three groups of
participants were involved, each one representative of a particular skilfulness level: none, basic, good
experience. The total number of the participants was 9. The participants were shown simultaneously the same
image tile produced by 4 different processing methods in order to detect destroyed buildings; no time
constraint was given; the participants were asked to think aloud while performing the task and, at the end, to
answer some specific questions about the perception of the images. 4 Dell monitors - 1280x1024 resolution -
were used to display 1 image tile (1024x1024 pixels, 15 cm resolution) each, in 4 different processing methods:
“unprocessed”; “cc64”; “rubble”; “unsharpened”. The interviews were audio-recorded and the results of the
verbal data were assessed to identify processing methods to put under test in the pilot experiment: after
having ranked the processing methods, according to the verbal data analysis, the “unprocessed” tiles were
never ranked the worst; the “cc64” were ranked the worst for 82% of the cases, while the “rubble” were
ranked the best for 40% of cases. Consequently, the “unprocessed” and the “rubble” processing chains were
selected as material of the pilot experiment phase.
The pilot experiment was composed of 2 different processing methods which have been selected according to
the results of the thinking aloud – semi-structured interviews. The test was run on Tobii T120 remote eye
tracker and involved 2 photo-interpreters, 1 skilled and 1 basic-knowledge. A small set of 37 tiles (1024x1024
pixels, 15 cm resolution), 19 unprocessed and 18 rubble, containing a total of 81 targets, was used as stimulus:
the single image tiles were randomly presented and displayed for 5 seconds each. The subjects were fully
instructed about the task and asked to identify the targets – completely destroyed and severely damaged
buildings - by clicking on them. Clicking responses and eye movements were recorded and analysed.
Provisional results showed in both cases (skilled photo-interpreter and basic-knowledge photo-interpreter)
positive influence of the “rubble” processing on the task performance. A mildly significant improvement of
86

accuracy was found with the rubble-processed images over the unprocessed ones (p-value=9.92%). (See Fig.
4).
Figure 4 – example of eye tracking analysis output: (left) “heat map” and clicking responses on AOI in
“rubble” image tile; (right) “heat map” and clicking responses on AOI in “unprocessed” image tile.
The full experiment was designed so as to increment the number of participants, the experience classes and
the image tiles set. The experiment was run on a Tobii T120 remote eye tracker and organized into 2 sub-Tests
(Test1 and Test2).
The Stimulus was composed of 80 image tiles each one present in 2 versions: “unprocessed” and “rubble”; 10
images per numerosity of target (from 0 up to 3 targets). If in Test1 an image tile was presented in the
“unprocessed” method, in Test2, the same image tile was presented in the “rubble” one, and vice versa.
Test1: image_A_un; image_B_rub; image_C_un; ecc…
Test2: image_A_rub; image_B_un; image_C_rub; ecc…
The Participants were totally 30: 10 skilled photo-interpreters, 10 basic-knowledge photo-interpreters, 10
non-experienced subjects divided into 2 Tests:
Test1: 5 skilled photo-interpreters, 5 basic-knowledge photo-interpreters, 5 non-experienced subjects.
Test2: 5 skilled photo-interpreters, 5 basic-knowledge photo-interpreters, 5 non-experienced subjects.
Procedure: Every participant observed 80 image tiles and was asked to seat in front of the monitor and
perform the task of a photo interpreter involved in a damage assessment. Written instructions were displayed
on the screen of the eye tracker at the very beginning of the recording session of each single test. The task was
to explore the displayed image, search for destroyed or severely damaged buildings and click on them. The
single image tiles were displayed randomly for 5 seconds each. The mouse cursor was visible and the subject
could click on those he/she recognized as targets. The clicking response didn’t leave any point on the image:
only in the replay and in the visualisations computed in Tobii Studio the clicking responses were marked on the
image tiles.
87

Analysis: the eye tracker output was analyzed in Tobii Studio and in R. It took into account 4 parameters: the
clicking responses and eye movement metrics – time to first fixation (how long it takes before a participant
fixates on a target AOI) and time from first fixation to next mouse click (how long it takes before a participant
left-mouse clicks on a target AOI) to assess - between-subject - how the processing method and the level of
damage impacted the subjective performance.
AOI have been drawn on the targets identifying t4 and t5.
3. Results:
False positives: a false positive is the clicking response outside the target AOIs. Taking into account the
processing method, the false positives had an increment of 32,9% (see Table 1) in rubble processed images
with respect to the unprocessed ones. Most of the false positives have been generated by a clicking response
set on buildings without roof, like the ones shown in Figure 5.
Table 1
unprocessed
rubble
Rate (%)
FP tot 1281 FP tot 1702 32.9
Unprocessed
a) b)
Rubble
c) d)
88

Figure 5 - examples of eye tracking analysis output: a) and c) show “heat map” and clicking responses on a t4
target AOI respectively in “unprocessed” and “rubble” image tile. b) and d) show, for the same t4 target,
“unprocessed” and “rubble”, clusters of attention automatically generated by the software representing
the percentage of the participant attention.
i) True positives – Clicking response: the clicking responses inside the target t4 and t5 AOIs were considered as
true positives. The general results are shown in Table 2 below:
Table 2
unprocessed rubble Rate %
TP t5+t4 518 606 17
TP t5 377 413 9.5
TP t4 141 193 36.9
Totally the percentage of clicked targets, taking into account the processing method, increased of 17%. Taking
into account the damage level and the processing method, the improvement of the test participants’
performance on t4 was higher than the one on t5.
The impact of the rubble processing on true positives was higher on the skilled participants, as they improved
their performance on t4 and t5 respectively of 54,1% and 20,3% in rubble processed image tiles. The nonexperienced
participants improved as well their performance on t4 significantly on rubble processed image
tiles. The impact of the processing method, as far as the clicking response on t4 and t5, was lower for the basic
participants (See Table 3 and Chart 1).
Table 3
Unprocessed → Rubble
% increment rate
t4
t5
skilled 54,1 20,3
basic 18 6
no exp 42,6 3,14
Chart 1
ii) True positives – Observation count and time to first fixation:
89

The “observation count” - how many observations have been spent on t4 and t5 in the uprocessed and rubble
processed images – spots an increment of 10,4% in the t4 observations in the rubble processed image tiles
with respect to the unprocessed ones. As far as the observations on t5, the increment, was of the 7% (see
Chart 2)
Chart 2
The “time to first fixation” decreases, in average for all the participants classes, from t4 in unprocessed image
tiles (2,5 seconds) to t5 in rubble processed (2 seconds): it means that the damage level and the processing
methods impact the time to fixate a target (Chart 3). The main trend is mostly reflected by the skilled
participants’ ocular behaviour.
Chart 3
iii) True positives – time from first fixation to next mouse click: this metric spots that the “reaction time” to
the target given by the clicking response is faster when the damage level is higher. In the case of t4 the
processing method helps in reducing interestingly the distance, in time, from the fixation on the target AOI and
the clicking response. (See Chart 4)
90

1,25
1,2
1,15
1,1
1,05
1
0,95
1,2
time from first fix to click
1
1,1 1,1
0,9
t4 un t4 rub t5 un t5 rub
Chart 4
4. Conclusion:
The experiment shows that the level of damage and the processing methods of the image tiles impact all the
participant groups performance and ocular behaviour, as far as the metrics took into account. In particular, the
lower damage level targets, the t4s, more difficult to be found, if enhanced by the rubble processing method,
are more likely detected and recognized. Given the time pressure and the huge amount of information (visual
and cognitive) to process during a damage assessment, the enhancement given by the rubble processing
method offers a valid support decreasing the detection time and the clicking response. (See chart 5)
Summarizing chart
600
3
500
400
300
2,5
335
2,3
370
512
2,2
377
548
413
2
2,5
2
1,5
observ count
click count
time to first fix
200
1,2
1
193
1,1 1,1
1
time from first
fix to click
100
141
0,5
0
t4 un t4 rub t5 un t5 rub
Chart 5
0
5. Further steps:
But still it is necessary to decrease the rate of false positives, mostly present in the rubble processed image
tiles. To deepen the issue we designed a tool, based on eye tracking technology, combining eye gaze data with
Computer-Assisted Detection algorithms to improve detection rates 10 of true positives.
10 G. D. Tourassi, M. A. Mazurowski, B. P. Harrawood and E. A. Krupinski, "Exploring the potential of context-sensitive CADe in
screening mammography," Medical Physics 37, 5728-5736
91

ABSTRACT
Crisis maps readability: first results of an experiment using the
eye-tracker
CASTOLDI R., CARRION D., CORBANE C., BROGLIA M. and PESARESI M.
Joint Research Centre of the European Commission, Italy
Roberta.castoldi@jrc.ec.europa.eu
Abstract
An eye tracker is a device for measuring eye positions and eye movement. Eye trackers are used in
research in the most different fields: from ophthalmology to visual perception, linguistics to visual
design, psychology to human computer interface. At JRC we are applying eye-tracking to damage
assessment analysis based on satellite images and aerial photos. According to the eye-mind
hypothesis, eye movements can be a window on the cognitive processes and are able to reveal
reasoning strategies involved in tasks execution.
An empirical study has been designed in the specific application of crisis maps in support of first
responders, field operators and decision-makers involved in emergency events management.
The purpose was to run an experiment on a group of crisis events management actors to explore
the way they interact with emergency products (e.g. digital maps) to come out with
recommendations regarding the best practices for making crisis maps more efficient,
comprehensible and usable for the end-users. To analyse the user/map interaction we relied on
eye movements data and cognitive task analysis methods (e.g. retrospective thinking aloud and
questionnaires). The first qualititative results of this experiment will be presented.
93

SESSION IV
TOWARDS ROUTINE VALIDATION AND QUALITY
CONTROL OF CRISIS MAPS
Chair: Michael Judex
Crisis maps are gradually moving from the research domain to the production domain
and if on the one hand standardization is still a long way off, on the other hand the awareness of
the importance of reliable, consistent and usable products is steadily raising. Several actors are
actually involved in some kind of check, quality control or validation of the maps during their
ordinary workflow: users may not formally accept or refuse a product but, de facto, they have to
decide if they will use it or not and they have to decide if they ingest it or not into their GIS, in
particular if the product is a vector layer; providers verify their output against requirements,
specifications, or against a formal checklist before releasing them; sometimes validation is
assigned to an independent party. This session will present and discuss practical cases and
experiences of maps validation and quality control integration in the ordinary workflow, involving
different points of view.
95

SHORT PAPER
A methodological framework for qualifying new thematic services
for an implementation into SAFER emergency response and
support services
RÖMER H., ZWENZNER H., GÄHLER M. and VOIGT S.
German Aerospace Center (DLR), German Remote Sensing Data Center (DFD), Oberpfaffenhofen, Germany
hannes.roemer@dlr.de
Abstract:
In the FP-7 GMES SAFER project a pre-operational service for emergency response and emergency
support products was implemented to reinforce the European capacity to respond to emergency
situations. SAFER was not only focusing on “rapid mapping” and validated products during the crisis
phase but also on the enrichment of the service with a wider set of thematic services. For the
selection of new thematic services not only a high accuracy of products was of interest. Moreover,
e.g. service maturity, user interest and compliance to the SAFER operational model are important
issues to guarantee a validated service.
The aim of this contribution is to present a methodological framework that was developed and
applied for the evaluation and qualification of selected thematic services into the SAFER portfolio
Version 2 (V2). The concept is characterized by strong user involvement including European Civil
Protection Organisations and Humanitarian Aid Organisations. The framework consists of several
steps comprising – among others – the definition of assessment criteria (here termed as Service
Evolution Criteria), the Service Maturity Analysis (SMA), a ranking of interest/relevance by involved
users and an operational performance check (operational check = OC). In total 19 Service Evolution
Criteria were defined in collaboration with the users and were applied for both, the SMA and the
OC. The criteria cover aspects of software and data sustainability, service producing time, user
support and user availability, service transferability, metadata compliance and the reliability of the
map contents. The SMA was designed to assess whether the services are mature and sustainable
whereas during the OC the services were tested under operational conditions. The OC was
conducted in collaboration with several project partners, e.g., the JRC conducting a scientific and
technical validation of the delivered products.
The qualification process led to a substantiated suggestion of thematic services to be implemented
into SAFER V2 and thus served as an important decision support for the project stakeholders.
Finally, with the selected approach it was ensured that the thematic variety of the existing “rapid
mapping” services have been substantially increased.
97

Keywords: rapid mapping, qualification, emergency response, thematic services
1. Introduction
With the aim to strengthen the European capacity to respond to emergency situations, a pre-operational
service for emergency response and emergency support products was implemented in the FP-7 GMES SAFER
project. Two major aims of SAFER are (a) the improvement, consolidation and validation of information
services focussing on rapid mapping during the response phase and (b) the enrichment of existing preoperational
services with a wider set of information products covering more widely the response cycle, from
the prevention phase to the post-crisis phase. This second priority implies a longer-term qualification process
which has started in the beginning of the project on 1 January 2009 and finished in July 2011 and is termed in
the following as Service Evolution.
The focus of this contribution is to present the methodological framework that was designed and developed
within the context of Service Evolution of SAFER. Furthermore, as the framework was already successfully
applied and implemented within SAFER, the developed methodology is not only of theoretical but also of great
practical value.
A crucial element of the multi-stage concept includes a strong involvement of European users, such as
European Civil Protection Organisations and Humanitarian Aid Organisations represented by the UN. Their
main role in the framework encompasses particularly the identification of qualification criteria (Service
Evolution Criteria) and their contribution to the evaluation of the added-value of the new services in
comparison to the existing Emergency response and support services (Core Services = CS).
In general the framework was developed to evaluate the maturity, operability of the new thematic services as
well as their added-value in relation to the CS. In addition to the user community, the Service Evolution
process was supported by the JRC, CNES, e-GEOS, Infoterra (UK and Germany) and EUSC.
The following chapter 2.1 provides a comprehensive overview of all steps of the qualification framework,
whereas chapters 2.2 – 2.5 will pick up some of these steps in more detail.
2. Methodological framework
2.1. General approach
As illustrated on figure 1, the Service Evolution describes the process of qualifying new thematic services for
an implementation into the existing pre-operational model of the SAFER project. Service Evolution explicitly
focuses on the evaluation and qualification of the services themselves, rather than on an in-depth analysis and
validation of the provided products. The latter was conducted by JRC in parallel and includes a technical and
scientifically validation where also external experts from different research domains were involved. As
indicated by the red dashed frames on fig. 1, the involvement of users played a fundamental role throughout
the qualification process. In the first step, the identification of Service Evolution Criteria, users (i.e. the Italian
Civil Protection Authority, DPC), service providers (i.e. DLR, EUSC, ITUK) and other project partners agreed on
the definition of 19 Service Evolution Criteria to be used for further qualification steps, in particular the Service
Maturity Analysis (SMA). During the SMA, all thematic service providers gave a detailed inventory of the
maturity of their services.
98

The provided information was then checked against the predefined criteria. A further qualification stage
includes a ranking of interests of involved users. In total 13 National civil protection organisations and five
humanitarian aid organisations were asked to rank those thematic services that passes the SMA to a level of
interest or relevance. In the operational check (OC) a realistic test scenario was created, where the service
providers had to show the operational performance of their services.
A prioritisation of the thematic services was made on the basis of the SMA, the ranking statistics as well as the
OC. In addition, the required amount of budget and separation from the CS were additionally taken into
account during this pre-selection. This prioritisation served as an essential basis for the final decision on the
qualification of the services which was taken by the SAFER executive committee (EXCOM). It needs to be
emphasized that the users were also involved in the decision phase of the framework and where represented
in the committee by the Project User Board (PUB). The qualification process leads over to the implementation
phase where the existing product portfolio versions 0 and 1 and also many other elements of the preoperational
model of SAFER had to be updated.
The following sections give a more comprehensive overview of the four major qualification steps, the criteria
selection, the SMA, user ranking and the OC.
Service Portfolio (V0, V1)
Thematic services
S E R V I C E E V O L U T I O N
Identification of Safer Service
Evolution Criteria
Service Maturity Analysis (SMA)
Ranking of interest of users
Operational Performance Check
(OC)
Prioritization of thematic services
Decision by Project Executive
Committee (EXCOM)
Scientific & technical validation
Updated Service Portfolio (V2)
Qualified thematic services
Figure 1. The concept and workflow of Service Evolution.
User
involvement
2.2. Selection of Service Evolution Criteria
As illustrated on fig. 1, the Safer Service Evolution Criteria were firstly defined in a collaborative work during a
workshop on 9 June 2009 in cooperation with the user forum, represented by the Italian Civil Protection
Authority (DPC), the rapid mapping service provider community (DLR, ITUK) and other project partner
99

organisations that are responsible for the product dissemination and geo-data infrastructure (e-GEOS), the
product validation (JRC) and the quality control (CNES).
In a first step all contributing partners have defined their own Service Evolution Criteria from their point of
view and expertises. The second step included the synthesis of these criteria which was done by DLR. At this
stage, the consortium agreed on the definition of 19 Service Evolution Criteria to be used for the further
service qualification process. The criteria can in general be divided into the following four criteria groups:
Service performance, Product quality, Dissemination and Usability/Additional value.
Service performance criteria were mainly defined by the well established service providers and the user
community. A major criterion here is the sustainability with regard to the required EO and non-EO data
sources and the support of additional software/tools used for the product generation. Furthermore, service
performance refers to the time requirements for different activation modes, the 24 hours / 7 day availability in
case of Emergency Response services, the required costs and the technical support provided for the users.
Even though a scientific and technical validation of the products was carried out by JRC on a sample basis, each
new service provider should be familiar with the validation scheme. Product quality criteria include map and
layout criteria, such as consistency between map and legend symbols, compatibility between geographic
projections of the different entities or geographic information layers included in the same product.
The criteria dealing with the product dissemination cover the type of delivered data sets, the metadata
compliance to ISO 19115 standards and, in case of data publication as remote services, the compliance to OGC
reference standards (WMS, WFS etc.).
The Usability/Additional value refers to the innovative and additional value of the service compared to existing
European services and the CS as well as to the service transferability. The latter targets on the question
whether the service is limited and applicable to specific areas/regions or whether there are dependencies on
specific data availabilities. A further criterion is the User feedback from previous GMES projects.
The selected Service Evolution Criteria were only slightly modified and updated after the first workshop and
played a fundamental role for the next qualification, the SMA that is presented in the next chapter 2.3.
100

Technical compliance / Gateway 1
eGEOS
Users
DPC
Emergency (support) mapping
DLR, ITUK
Validation
JRC
Workshop on the
definition of
Service Evolution Criteria
Quality control
CNES
Service Evolution Criteria
Service Maturity Analysis
1
The Safer Gateway is a web application sustaining the GMES Emergency Response Service
and providing several interfaces according to the user profile.
Figure 2. Safer service criteria identification.
2.3. Service Maturity Analysis
During the Service Maturity Analysis (SMA) all thematic SP gave a detailed inventory of the maturity of their
services by filling in a dedicated Service Maturity Questionnaire (SMQ). In the SMQ the questions were closely
oriented to the predefined Service Evolution Criteria as described in section 2.2.
Regarding the evaluation of the SMQ, two of the questions in the SMQ were considered as mandatory criteria:
firstly the thematic SP had to state if their product/service can be considered as being mature and that the SP
wants to have it implemented in the next SAFER version. Secondly, the SP must guarantee that a sustainable
supply of EO and non-EO data can be assured. Only if these criteria were fulfilled, the service/product was
checked against the other remaining questions. A quantitative evaluation scheme comprising three different
levels of importance respectively weighting factors was applied for the other questions. For example, the
knowledge and usage of the SAFER template was considered less important than the general transferability of
the product to other areas or a support that can be provided in English language. In order to achieve a
maximum of transparency in the evaluation, each SP was provided with an evaluation sheet in addition to the
SMQ. This contains the information on the evaluation points and the weighting factors assigned to each
question, respectively those questions considered as mandatory criteria.
The number of evaluation points (respectively the percentage values) achieved by each SP served as an
important quantitative basis in the Service Evolution in general. Furthermore, the results were related to the
quantitative results derived from the ranking of interest of involved users (cp. section 2.4).
101

SAFER Service Evolution
SMQ /
Evaluation Sheet
SPs
Evaluation
Excluded from
further consideration
Meet mandatory criteria?
No Yes
Quantitative evaluation
Filled in SMQ
SERVICE MATURITY ANALYSIS
Figure 3. The Service Maturity Analaysis (SMA).
2.4. Ranking of interest of users
Generally, the user community in SAFER is represented by the Project User Board (PUB). In order to get a
wider feedback than from the five PUB members only, the members of the External User Advisory Committee
(EUAC) were addressed during an EUAC conference. The participants comprised the five humanitarian aid
organisations WFP, UNOSAT, UNHCR, UNICEF and IFFRC as well as 13 National Focal Points from Germany, UK,
Hungary, Austria, Bulgaria, the Netherlands, Portugal, Croatia, France, Bosnia & Herzegovina, Italy, Greece and
Sweden. They were asked to rank the thematic services which passed the Service Maturity Analyses for SAFER
version 1 and 2 according to a level of importance or interest. The ranking scheme applied ranges from 1 for
very low interest, to 5 for very high interest (2=low interest, 3=medium interest and 4 high interest,
respectively).
Since the SAFER project is a strongly user-driven project, the user ranking was a major component of the
general qualification process of the thematic services. In order to account for unequal interest and impact of
the different disaster types (e.g. flood is of much more interest that earthquake for European users), the user
interest was categorized for each disaster type, because the aim of SAFER was to enrich the Service with a
wide variety of thematic services, and not only flood services, for example.
2.5. Operational performance check
The operational performance check (OC) is a key step of the Service Evolution process. It aims at assessing
whether the new thematic services can be offered under operational conditions. Similar to the SMQ, the
assessment criteria were closely related to the predefined evolution criteria (cp. section 2.2). In contrast to the
SMA, the focus was on those criteria that were related to the operational performance, in particular the user
support, time requirements, service transferability, technical compliance and service sustainability. Thus, the
OC comprises different sub-exercises for the respective criteria group that were carried out in collaboration
with other project partner organisations, such as eGEOS, ITUK, EUSC and the JRC (cp. fig 4). As already
mentioned in section 2.1, the product validation was indeed carried out in parallel to the Service evolution
process, respectively to the OC, but was not a component of the Service Evolution in a narrower sense.
At the beginning of the exercise a time window during which the OC had to be carried out was given to each
SP to have them on alert. During the first week of this period, each SP was provided with the Service Request
Form (SRF) which contains general information about the test scenario, such as the area of interest (AOI), the
deadline for product delivery and the information required for product dissemination. The SRF represents the
102

official SAFER document used to specify and standardise the service request of the user. Figure 4 illustrates
that most of the sub-exercises were carried out after the product generation; however some tests were also
carried out right after triggering of the service.
Reliability of
information content
JRC
User Support
DLR
Sustainability
Non EO-data, EO-data, Software
ITUK, DLR
Scenario
definition
SP(s)
Product
generation
Product
Product
Dissemination
General
evaluation
DLR/JRC
Time requirements
DLR/JRC
User Support
DLR
Technical compliance
eGEOS
SAFER Service
Evolution
OPERATIONAL CHECK
Figure 4. The operational performance check (OC).
The service transferability was assessed by choosing test scenarios outside of the SP’s working area and was
carried out by JRC and DLR. User support was evaluated via phone interviews in order to check the SP’s
availability, its ability to provide user support in English language and its flexibility to deal with potential user
requirements, such as to make small adjustments to their product (e.g. change the projection from UTM to a
local projection) even after product finalisation. The time requirement was checked by comparing the
deadlines for the product delivery as indicated in the SRF, with the actual time of product delivery (upload of
the data). The time-requirements are closely oriented to the time requirements that apply to the CS and are
related to the respective activation mode (rush mode or emergency support mode). The technical compliance
check was conducted after product dissemination in order to evaluate the metadata quality, i.e., the
conformity to ISO19115/19139 and INSPIRE standards. The service sustainability check encompass the check
of the sustainability of software applied (e.g., technical support, license model) and the EO- and non EO-data
sets that were either required for processing or for product improvement (i.e., data sources, time required for
data acquisition, etc.).
As the OC was the only practical test within the Service Evolution process the results served as an essential
basis for the general evaluation of the service performance. The products that were delivered in the frame of
the OC exercise provided a good basis for the users to assess what they can expect from the thematic services
under operational conditions.
3. Concluding remarks and outlook
The objective of this contribution was to present a methodological framework that has already demonstrated
its practical value within the GMES Safer project. In summary, the framework holds two major strengths: a) the
close cooperation with involved users throughout the evaluation process and 2) the integration and
103

consideration of many different assessment criteria within the evaluation process which was realized by close
cooperation with different partner organisations and experts. Therefore, it can be concluded, that the selected
framework provided a comprehensive and reliable basis for a fair and transparent qualification process of the
thematic services that were implemented into the SAFER operational model. Based on the practical
experiences with the application of the framework, it can be assumed that the general structure of the
methodology is also transferable and useful in comparable application cases.
Even though the independent scientific product validation was carried out in parallel to the Service Evolution,
the authors agree that a closer link between both parts would have simplified the qualification process in
general. However, the absolute certainty on how the users will benefit from the new thematic services will
turn out in the future. Here the most important indicators are the number of user requests per time period
(activations) and the degree of user satisfaction in case of an activation of a new thematic service.
References
GAEHLER, M.; FOERSTER, A.; ZOSSEDER, K.; ZWENZNER, H. (2010): Report of defining the selection criteria for transfer
of thematic services to the pre-operational emergency response services. Project report SAFER-
D21000.1-SDD-DLR-01.03, 32p.
ROEMER, H.; ZWENZNER, H. (2011): Service Maturity Analysis (V2). Project report SAFER-D21000.4-SDD-DLR-
01.00, 310p.
ROEMER, H.; ZWENZNER, H. (2011): Portfolio of thematic and technologic innovation services within the project
and impact on operational architecture - Version 2. Project report SAFER-D21000.5-SDD-DLR-01.00,
171p.
104

EXTENDED ABSTRACT
A methodology for a user oriented validation of satellite based
crisis maps
JUDEX M. 1 , SARTORI G. 2 , SANTINI M. 3 , GUZMANN R. 3 , SENEGAS O. 4 , SCHMITT T. 5
1 Federal Office of Civil Protection and Disaster Assistance, Germany
2 World Food Programme, Italy
3
Dipartimento della Protezione Civile, Italy
4
United Nations Institute for Training and Research – Operational Satellite Applications Programme (UNOSAT),
Switzerland
5
Ministère de l’intérieur, Direction de la Sécurité Civile, France
michael.judex@bbk.bund.de
Abstract
The European Commission together with the European Space Agency has established a civilian geospatial
initiative unparalleled by any other civilian minded project today, under which, no other
similar constellation has come together. The initiative “Global Monitoring for Environment and
Security” (GMES) has the objective to provide access to geo-information being it from earth
observation recourses or in-situ measurements. Expected benefits range from political decision
making to security of citizens and hence the development is first and foremost user driven. One of
the ongoing projects is the Services and Applications For Emergency Response (SAFER) developing
the procurement of satellite based cartographic products and analyses in case of natural or man
made disasters to responsible authorities.
Being a user driven project, SAFER encompasses a component designed to validate project’s
products by the users. Unlike ‘traditional’ Scientific and Technical validation, whereby processes
and products are tested against pre defined protocols using a representative and significant
sample, the main challenge for the Project User Board (PUB) lied in establishing a methodology
that struck the right balance between validating objective, and subjective criteria, from the users
perspective. Users have differing requirements, multiple expectations, and different levels of
technological sophistication; how to reconcile these multiplicities is at the core of establishing an
appropriate validation methodology.
For the Validation process, the PUB compares the product requested by the user – based on the
Service Request Form (SRF) form and the emergency activation Data Sheets – with the User
Feedback (UFF) form, and then juxtaposes these with the products Portfolio to analyze if the
105

product was delivered as promised. The PUB evaluates both documents – the SRF and UFF forms –
using two distinct indexes; the ‘Coherence index’ – how closely the product reflected the promised
product, in terms of technical content, delivery time etc – and the ‘Satisfaction Index’ – how useful
and valuable was the product in planning and supporting the Emergency Response. The Coherence
index focuses on objective criteria extracted from the forms, measured contextually within a
specific disaster type; in short it is ‘content’ centred. The Satisfaction index on the other hand,
measures subjective criteria; and thus is ‘experience’ centred.
Therefore, PUB compares the product ‘Requested’ with the product ‘Delivered,’ and analyzes the
answers with weighted indicators derived from prioritized needs. The PUB has worked in assessing
the results and to fine-tune the models by adjusting the indicators’ weights on the basis of the
quality of products, and to establish ‘thresholds’ that determine if the product is acceptable or not.
To help address this fine-tuning the concept of Macro-indicators to simplify the process has been
introduced. Roughly speaking, the PUB members have a ‘sense’ of the weight allocation, based on
the experience of specialists and on the realities in the field – the questions: ‘is the product good,
useful, coherent?’ become the locus of the equation. Macro indicators (non quantitative) are the
general criteria used to translate the process into logic and to realign the micro indicators
(indicators of the coherence and satisfaction indexes) with these macro indicators.
The analysis was applied to the activations performed between the periods of December 2010 and
beginning of May 2011. The case study is related to 15 activations. The validation methodology and
mechanism developed by the PUB has been successfully applied and has proven to be successfully
functioning.
The results of the validation show that products did not reach the maximum coherence score
because what is delivered doesn’t completely reflect what has been requested by users. However,
in most cases the satisfaction index gives high scores and indicates that the users were happy with
the results. The reason behind this phenomenon could be that the user is a) still satisfied with the
product or b) the user changed his request after submitting the SRF without proper documentation
in the SOR or c) parts of the products are delivered without mentioning in the operational
documents.
106

EXTENDED ABSTRACT
Quality policy implementation: ISO certification of a Rapid
Mapping production chain
ALLENBACH B. 1 , FONTANNAZ D. 2 , RAPP JF. 1 , PEDRON JL. 2 , CHAUBET JY. 3
1 SERTIT, France
2 CNES, France
3
APAVE, France
Bernard.allenbach@sertit1.u-strasbg.fr
Abstract
One of the challenges initially pursued by the SAFER project was to qualify and validate an
Emergency Response Service to increase its acceptance by the users. The initial view of quality as
set up by the writers of the project has split "Quality" into several work packages or tasks handled
by different partners. Additionally a qualification-validation was also expected from the users. A
quality management system was implemented aiming at coordinating quality issues through the
following means: a quality assurance plan, a service control plan, specific service performance
indicators computation and a set of associated regular quality reports. Close to the end of the
project and concerning quality management a main lesson learnt is the difficulty to obtain
application of such quality assurance plan when contractual links between partners are weak. Short
duration of the project and the number of expected service portfolio versions are also major
handicaps for the integration of quality requirements into operational procedures. Nevertheless
positive results from this experience are numerous and the first is of course the quality awareness
of all the actors of the project and the global improvement in the understanding of what quality
should or could be in our realm. As a conclusion, a draft attempt to consider roles and
responsibilities, from the quality view point, of the different actors implied in the Emergency
Response is proposed. This effort relies strongly on the statement that quality control is closely
linked with the core production knowhow of each actor; by the way is logically and usually a
corporate and mainly private undertaking in the industry. Overall quality (service quality) is
expected through the addition of the internal quality of all the segments – actors - procedures)
chained to create the service. Hence, "quality can be published" at all scales at the interfaces
between processes by comparing results with internal process references and/or external
references like the portfolio specifications eventually normalized. It should be noticed that,
unfortunately, it is not always possible to quantify everything we would like to measure to ensure
quality conformity. Then, another and more comprehensive way to setup quality management is to
use generic standardized quality methodology like ISO9001:2008. SERTIT has done this choice.
107

More, the will was to impact strongly production; thus the domain of certification has been
focused on the Rapid Mapping production chain integrating the fundamental timely constraint for
rush production. Certification has proved to be a tough task requiring a lot of resources over a one
year process. Despite the focused perimeter, quality management has truly, often deeply,
impacted all major management processes and resources of the service. But the story got a happy
end as SERTIT has obtained the certification of its Rapid Mapping Service under the following
denomination and scope: “Production and publication within 6 hours after reception of the first
satellite data of crisis geo-information for civil protection services.” This successful endeavor,
consistent with SAFER targets and industry methodology, materialize for the user the jump done
from best effort to certify production means, methodologies and resources.
108

AUTHORS INDEX
A
AL-KHUDHAIRY D................................................................ 8
ALLENBACH B. ................................................................. 107
B
BERGES J...........................................................................81
BLAES X. ...........................................................................69
BROGLIA M. ................................................................ 83, 93
C
CACAULT P........................................................................81
CARRION D. ......................................................................93
CASTOLDI R................................................................. 83, 93
CHAUBET JY. ................................................................... 107
CHAVENT N.......................................................................13
CORBANE C. ......................................................................93
D
DILOLLI A. .........................................................................17
E
EDWARDS S. .....................................................................15
EHRLICH D. .......................................................................37
F
FONTANNAZ D. ............................................................... 107
G
GÄHLER M. .......................................................................97
GALLEGO J. .......................................................................37
GORZYNSKA M. .................................................................57
GOUHIER M. .....................................................................81
GUEGUEN L.......................................................................47
GUEHENNEUX Y. ...............................................................81
GUZMANN R. .................................................................. 105
H
HARRIS A. .........................................................................81
HORÁKOVÁ B. ...................................................................73
HUBBARD D. .....................................................................11
J
JANIUREK D. ..................................................................... 73
JOYANES G. ...................................................................... 57
JUDEX M. ....................................................................... 105
K
KEMPER T....................................................................37, 69
KOETTL C. ......................................................................... 15
L
LABAZUY P. ...................................................................... 81
LANFRANCO M. ................................................................ 17
LOMBARDO D. .................................................................. 17
M
MOREAU K. ...................................................................... 55
O
OSTERMANN F. ................................................................. 29
P
PEDRON JL...................................................................... 107
PESARESI M. .......................................................... 47, 83, 93
R
RAPISARDI E. .................................................................... 17
RAPP JF. ......................................................................... 107
RIVET S............................................................................. 81
RÖMER H. ........................................................................ 97
ROUMAGNAC A. ............................................................... 55
S
SANTINI M...................................................................... 105
SARTORI G. ..................................................................... 105
SCHMITT T...................................................................... 105
SENEGAS O. .................................................................... 105
SOILLE P. .....................................................................37, 47
SPINSANTI L...................................................................... 29
STANKOVIČ J..................................................................... 73
109

T
TIEDE D. ...........................................................................57
W
WANIA A. ....................................................................37, 69
U
USSORIO A. .......................................................................57
Z
ZWENZNER H. ................................................................... 97
V
VEGA EZQUIETA P..............................................................57
VOIGT S. ...........................................................................97
110

EUROPEAN COMMISSION
EUR 24948 EN – Joint Research Centre – Institute for the Protection and Security of the Citizen
Title: Conference Proceedings: VALgEO 2011 - 3rd International workshop on Validation of geo-information
products for crisis management.
Authors: Christina Corbane, Daniela Carrion, Marco Broglia, Martino Pesaresi
Luxembourg: Publications Office of the European Union
2011 – 111 pp. – 21 x 30 cm
EUR – Scientific and Technical Research series – ISSN 1018-5593 (print), ISSN 1831-9424 (online)
ISBN 978-92-79-21379-3 (print)
ISBN 978-92-79-21380-9 (PDF)
doi:10.2788/73045
Abstract
This report is a collection of contributions presented in the 3 rd International Workshop for Validation of Geoinformation
Products for Crisis Management- VALgEO 2011- organized by the JRC on October 18-19, 2011.
The annual VALgEO workshop sets out to act as an integrative agent between the needs of practitioners in
situation centers and in the field guiding the Research and Development community, with a special focus on
the quality of information.
The conference proceedings reflect the work presented and discussed during the workshop. The chapters are
organized following the four thematic sessions:
• The role of validation in Information and Communication Technologies (ICT) for crisis management
• Validation of Remote Sensing derived emergency support products
• Usability of Web based disaster management platforms and readability of crisis information
• Towards routine validation and quality control of crisis maps

How to obtain EU publications
Our priced publications are available from EU Bookshop (http://bookshop.europa.eu), where you can place
an order with the sales agent of your choice.
The Publications Office has a worldwide network of sales agents. You can obtain their contact details by
sending a fax to (352) 29 29-42758.

The mission of the JRC is to provide customer-driven scientific and technical support
for the conception, development, implementation and monitoring of EU policies. As a
service of the European Commission, the JRC functions as a reference centre of
science and technology for the Union. Close to the policy-making process, it serves
the common interest of the Member States, while being independent of special
interests, whether private or national.
LB-NA-24948-EN-C