Community Key Performance Indicators (Community KPIs) for the IoT and Smart Cities

futureeverything

In 2016, the CityVerve project set out with one goal to increase meaningful citizen participation in the Smart City and Internet of Things. To help realise this vision, FutureEverything introduced Community Key Performance Indicators (or Community KPIs) as a framework to involve communities and residents in the design process of the Smart City, and to engage them as stakeholders and contributors to define and measure success criteria for the CityVerve project.

Community Key

Performance Indicators

(Community KPIs) for

the IoT and Smart Cities

A Collaborative Framework for Project Assessment

Drew Hemment, Mel Woods, Vimla Appadoo, Lily Bui


Community Key

Performance Indicators

(Community KPIs) for

the IoT and Smart Cities

A Collaborative Framework for Project Assessment

Authors

Drew Hemment

University of Dundee & FutureEverything

Vimla Appadoo

FutureEverything

Mel Woods

University of Dundee

Lily Bui

MIT

FutureEverything Report

August 2016

1


Contents

1. Introduction

1.1 About CityVerve

1.2 Introducing a People-Centred Approach to IoT and Smart City Development

1.3 A Need for Community Key Performance Indicators (Community KPIs)

1.4 Scope

2. Community Key Performance Indicators

2.1 What are Community KPIs?

2.1.1 Foundations of Community KPIs

2.1.2 Example of a Community KPI

2.2 Why Evaluate IoT & Smart Cities Projects?

2.3 When to Evaluate

2.3.1 Before: Establishing a baseline and goals

2.3.2 During: Tracking progress

2.3.3 After: Measuring changes and outcomes

2.4 Who is Involved

2.4.1 Who is “the Community,” and How Representative is the Sample

2.4.2 More on Community Champions

2.5 What Should be Evaluated

2.6 Limitations

2.7 Replicability

3. Toolkit: Creating and Evaluating Community KPIs

3.1 A Collaborative Framework for Assessment

3.2 Workshop: Envisage

3.3 Mini-Workshop: Iterate

3.4 Mini-Workshop: Assess

3.5 Workshop: Reflect

4. Reporting tools

4.1 Reporting Template for Goals, Indicators, and Outcomes

4.2 Digital Tools for Sustained Community Engagement

5. References 36

List of Figures

Fig. 1. Process for collaborative development of Community KPIs

Fig. 2. Constellation of community stakeholders

Fig. 3. Framework for Development of Community KPIs in IoT & Smart City projects

Fig. 4. Creating observational systems for assessing Community KPIs

4

4

6

8

9

10

10

10

11

12

13

13

13

13

14

15

16

17

17

18

19

19

21

25

27

30

33

33

35

7

14

20

28

2


About FutureEverything

Publications and

Reports

Each year FutureEverything proposes, develops and

responds to particular themes. These themes are

provocations, designed to open up a space for debate and

practice, made tangible through art and design projects.

FutureEverything Publications seek to contribute to an

international dialogue around these themes.

FutureEverything Reports seek to contribute to best

practice by disseminating frameworks, methods, results

and findings.

Dr. Drew Hemment

Founder & Creative Director, FutureEverything

3


1. Introduction

Community KPIs are proposed and introduced in CityVerve

as one way to build meaningful participation by local

residents and communities in the governance and design of

a major IoT and Smart Cities project.

The proliferation of sensor and data analysis technologies offers new ways to configure cities

– we have seen this applied in streetlamps, vehicles, heating equipment, bus stops, air quality

monitoring, and so on.

These emerging technologies have given rise to the ‘Internet of Things’ (IoT), networks of devices

and sensors that can communicate with each other (Anderson, 2014). By extension, the IoT

provides enabling technology forSmart Cities’; cities in which ubiquitous sensors and devices

allow for more efficient processes of city management, smoother flow of systems, and optimized

use of infrastructure (Hollands, 2008; Komninos, 2009).

However, a city is never merely its buildings, open spaces, streets, or vehicles. Nor is technology

there solely for engineers to optimise processes. IoT and Smart City development can make lasting

changes to the places in which people live, work and play. It can shape neighbourhoods and

the lives of residents, in the same way as the development of urban road systems did in the 20 th

Century.

Therefore residents and communities have a stake, and need a voice, in deciding which

technologies are most relevant to them, as well as defining the processes that measure the

success of these technologies.

1.1 About CityVerve

CityVerve is the UK’s Internet of Things Smart City Demonstrator, a large scale project funded by

InnovateUK (£10M) that aims to make Manchester a world leader in ‘Smart City’ and ‘Internet of

Things’ technology. It is led by the City of Manchester and is made of up 18 partners, including

FutureEverything, the project lead for Culture and Public Realm, with specific deliverables in Human

Centred and Participatory Design, Citizen Engagement, and Art Commissions.

Starting with Manchester, CityVerve hopes to create projects, tools, and a way of doing things with

Internet of Things Smart City technologies that can be replicated in cities around the world.

In CityVerve, there are a number of individual projects (or ‘use cases’) that implement and

demonstrate IoT technologies and design solutions for Manchester, within four themes: Transport,

Culture, Health, Energy. The project ideas, developed during the process of writing the bid, include:

4


- CityVerve will convert ‘flag and pole’ bus stops into safe places with

location-based services, sensors/beacons, mobile apps and intelligent digital signage.

People will check-into their bus stop and let bus operators know they are waiting for their

service.

Management of chronic respiratory conditions - CityVerve will set up a ‘biometric sensor

network’ which will help improve responses to patients’ conditions and improve how local

healthcare services work.

Community wellness - a network of sensors positioned in parks, along commuter and

school routes will track the progress of individuals and teams competing against each

other for physical activity and fun. Examples include the “Great Space Race Challenge” for

Manchester residents to walk the distance to the moon.

- Manchester, like many cities, is seeing a growth of traffic and congestion.

To reduce car use, alternative forms of transport need to be attractive and safe. Smart

lighting, in addition to connected street lighting, will help address this.

Bike sharing - The Manchester Corridor through-route will soon become bus and bike only.

Bike sharing schemes can be expensive to install and maintain, and so an alternative is to

use Internet of Things enabled bikes in a crowd-sourced and maintained, secure bike

sharing service. It will also include ‘e-cargo’ bikes to make ‘last-mile’ deliveries on the

Corridor.

- Street furniture and connectivity infrastructure such as lamp

posts and street cabinets on the Manchester Corridor will be used to monitor air quality at

different heights and locations. Information will be passed to those with health conditions

and made generally available to support walking options and routes.

- An online social network access through local wi-fi hotspots, giving

access to cultural events as well as community and commercial services.

Community KPIs, and a wider human centred and participatory design approach, are proposed

and piloted in CityVerve. (1) The aim is to involve communities and residents in the design process,

and to engage them as stakeholders and contributors to define and measure success criteria for

the project.

In an ideal scenario, citizen input would be incorporated at the outset of a project during design and

ideation phases so that communities can have a say in the conceptualisation of the technological

solutions themselves. The methods and principles described in this report are designed to support

this approach. In CityVerve, citizens were not involved in early phases of design and ideation

of project ideas due to the nature of the project structure and timeline. Presented in this report,

therefore, is not a full human centred design journey but rather the incorporation of human centred

design and citizen engagement principles and methods into project ideas that have already been

proposed. Nevertheless, citizens can still play a significant role in project design and assessment

using the methods described below. These constraints are discussed further in Section 2.6

(‘Limitations’).

(1) The framework proposed in this report is built on prior work in a citizen sensing project, Making Sense (http://making-sense.

eu) where Community Level Indicators are co-created by citizens not only to define, track and measure progress towards their

objectives and outcomes, but additionally, to annotate sensor data for extended analysis. Community Level Indicators were

conceived and applied in the context of IoT and citizen sensing for environmental monitoring (Woods, Hemment, Bui, 2016).

5


In the past, IoT and Smart City projects have tended to privilege a top-down approach, over

human-centred design, and with little or no engagement of citizens in design or assessment.

Effectively, this elides the point of view of the users these emerging technologies are designed to

serve, be that of the citizen, or specialist users in industry or government. As a result, many projects

have failed to build trust among consumers, accommodate concerns on issues such as privacy, or

deliver services that people really need or want.

Research has shown that Smart City and IoT projects of the past have been unsuccessful due

to lack of citizens’ adoption and sustainable use of smart technologies (Nam & Pardo, 2011;

Greenfield, 2013; Saunders & Baeck, 2015). In early use cases such as Korea’s New Songdo, the

United Arab Emirates’ Masdar City, and Portugal’s PlanIT Valley, Greenfield (2013) points out that

this techno-centric view of the smart city fails to have taken into account the social needs that the

technology should address.

As a counterweight to these top-down precedents, a number of IoT and Smart Cities initiatives

have emerged that go some way to demonstrate the value of a people-centric approach. On the

one hand, human centred design is deployed to put the focus on real problems people have,

and solutions that people will want to use. Additionally, the practice of participatory design in IoT

and Smart Cities projects can directly involve citizens themselves in key phases of design and

decision making for technological projects, very importantly adding a layer of social value to new

technologies (Hemment & Townsend, 2013).

In CityVerve, Community KPIs are introduced as one component in a human centred and

participatory design framework that is tailored for IoT and Smart City projects. (2) The framework

proposed is open prototyping, which involves opening the design and development process to

many contributors, and delivering interventions that are open and accessible to various publics

(Hemment, 2015). In CityVerve, this is achieved by introducing Community KPIs, support for project

teams in human centred design, Community Champions, who will support user research, and

artworks to engage various publics in envisioning futures.

CityVerve project teams, lead by design experts and facilitators, are introduced to human centred

design principles and methods, and then supported to introduce them during the design,

deployment and analysis of the projects. This can help develop understanding of users, tasks and

environments to inform technology and use case development, through ongoing user engagement,

requirements gathering, a focus on user experience, and through user-centred evaluation.

Distinctive participatory design methods will also be used to propose solutions for cities and the

citizens who inhabit them. Community members will be invited into a workshop setting in order to

help co-design key goals and indicators for success. This phase will also involve CityVerve team

members as well as various stakeholders to collaboratively propose and prototype ideas that focus

on known pain points.

Following these design and prototype phases, public trials and demonstrations of solutions will

then enable testing with a wide range of participants, and can help build visibility and engagement

(2) Building on past bottom-up Smart Cities projects such as Smart Citizens Manchester (http://futureeverything.org/projects/

smart-citizen) and Making Sense EU (http://making-sense.eu)

6


around a theme. The public, users and stakeholders can be engaged in shaping questions,

collecting data and adding new interpretation.

This report proposes a four-step process for collaborative design of Community KPIs, expressed

in four community-oriented workshops, described in full detail in Chapter 3. Project and use case

stakeholders (which include citizens/community members/users) will be involved in stages of

envisioning broader goals and indicators, iterating specific goals and indicators for individual

use cases, assessing finalised goals and indicators with mutually agreed-upon observational

mechanisms, and reflecting on the progression of the goals and indicators.

As a disclaimer, this is merely a proposed process for the development of Community KPIs and

may be different in practice, as each stage will involve different participants who may add new

perspectives and dimensions to the work.


ITERATE


REFLECT

broad goals and

indicators

specific goals

and indicators

for use cases

finalized goals

and indicators

on progression

of goals and

indicators

Fig. 1. Process for collaborative development of Community KPIs.

Using this framework, citizens, designers, project leads, and researchers can collaboratively

contribute input for projects by identifying benefits and threats of IoT and Smart City technologies;

set goals for how they are used and assessed; as well as propose services that the technologies

can bring to different users. Opening the design process in this way can create points of contact for

various contributors and users at different stages in the development process, it can entail multiple

points of openness and synthesis.

The key consideration throughout this work is the concerns of the citizens. Due to project

constraints, the design interventions and work in CityVerve – novel for an IoT and Smart Cities

project of this scale – will not be fully citizen-driven. Nonetheless, instead of relying solely on topdown

input from governments and institutions, incorporating human centred and participatory

design elements into the conceptualisation of IoT and Smart City innovations offers a layer of input

that represents citizens, who are essentially the core ‘users’ of every city.

Participatory approaches face significant challenges due to disenfranchisement and barriers to

social inclusiveness. Strategies need to include people in deprived as well as economically thriving

neighbourhoods to ensure that technologies and services respond to the needs, experiences and

aspirations of the wider population.

Manchester is not alone in attempting to implement more community-oriented, human centred

design methods in IoT and Smart City projects. Other cities are also addressing this need by

gathering input from people and communities. We see this in the Bristol Approach, which leans

7


on human centred design methods for citizen sensing deployments (Balestrini, Creus, Masfarre,

Canigueral, & Arguedas, 2016). We also see this in Barcelona, where citizens are helping cocreate

participatory processes for urban environmental sensing (Balestrini, Diez Ladera, Polvora, &

Nascimiento, 2016). In Bologna, the city collaborated with citizens to create legislation to maintain,

repair, and produce common resources, and services (Ibid).

CityVerve in Manchester is one project in a wider movement that aspires to redefine the traditional

smart city narrative, putting people - instead of technology - first.

1.3 A Need for Community Key Performance Indicators

(Community KPIs)

CityVerve strives to address a common problem shared among other smart city programmes in the

world: a lack of evidence sharing across cities.

While sharing of findings, best practices, and recommendations can contribute to a more effective

and efficient ecosystem of smart city projects, this is not always done (Saunders & Baeck, 2015).

As a result, IoT and Smart City projects attempt to solve similar issues but end up making the same

mistakes with little to no exchange of common problems and best practices among them.

Thus, there is a tremendous opportunity to normalise the practice of assessment for smart city

projects in a way that can be transferable across different projects, and potentially different city

initiatives altogether.

In CityVerve, we propose the use of Community Key Performance Indicators (Community KPIs) to

serve as a common language for sharing best practices and lessons learned across IoT and Smart

Cities projects. Community KPIs are proposed to augment and to complement other extant forms

of assessment and evaluation.

As opposed to the use of KPIs in the traditional business context, Community KPIs are presented

as a novelty to CityVerve and are therefore meant to be piloted across use cases to evaluate factors

that are crucial to user acceptance of technologies and services.

Discussed in much further detail in Chapter 2, Community KPIs refer to social metrics designed to

evaluate projects and are co-designed with community members. By implementing Community

KPIs, a broader range of stakeholders have a chance to consider what objective measures are

most relevant to them, enabling participants to become more informed and invested partners in use

case development.

Potentially, we envisage that the use of Community KPIs will give local residents and citizens

a voice on emerging IoT technologies and services developed in their city or neighbourhood;

contribute to user research and requirements gathering in early stages of design; and demonstrate

a collaborative framework for assessment that might be replicated in other IoT and Smart Cities

projects globally.

8


This report aims to propose a practical framework for developing, implementing, and assessing

Community KPIs for the CityVerve project in Manchester, with scope to be replicated in other

related IoT and Smart Cities projects around the world.

In CityVerve, the Community KPIs so developed will form one portion of the KPIs for the project.

They will therefore contribute to the success and assessment criteria for CityVerve. As these

methods are novel, they are not implemented universally in all ‘use cases’. Importantly, however,

they are presented to introduce best practices for incorporating citizen engagement in key design

phases of the projects before, during, and after their respective deployments.

introduces an overview of how Community KPIs can be used as a metric for evaluation

and provides some examples.

proposes a series of workshops and tools that can be implemented throughout the

course of CityVerve projects to create a baseline, track progress, and measure change.

provides a proposed reporting template for capturing and evaluating Community KPIs

as well as recommendations for tools for engaging citizens.

Key Terms

Community Key Performance Indicators

(Community KPIs): social metrics to evaluate

factors that are crucial to user acceptance of

technologies and services.

Internet of Things (IoT): A network of physical

devices, vehicles, buildings and other items—

embedded with electronics, software, sensors,

actuators, and network connectivity that enable

these objects to collect and exchange data.

A city that integrates information

and communication technology (ICT) and

Internet of Things (IoT) solutions to manage a

city’s assets

Barriers to user acceptance: Factors that

prevent users from adopting a programme,

behaviour, or technology

A participatory design

framework to develop and test a concept or

process through input of external contributors,

and through interventions that are open and

accessible to various publics or audiences.

Community KPIs are one element within an

open prototyping approach.

Human centred design: An approach to

problem-solving that begins with the audience

for whom one is designing and ends with new

solutions that are tailored toward their needs

Participatory design (also known as

An approach

to design attempting to actively involve all

stakeholders (e.g. employees, partners,

customers, citizens, end users) in the design

process to help ensure the result meets their

needs and is usable.

Project refers to either the wider CityVerve

initiative, which encompasses all use cases,

or the term can refer to general IoT and Smart

Cities projects in other contexts.

Use case refers to one of the sixteen individual

interventions within CityVerve that implement

and demonstrate IoT and Smart Cities

technologies.

9


2. Community Key

Performance

Indicators

2.1 What are Community KPIs?

Community Key Performance Indicators (Community KPIs) engage citizens as stakeholders in

defining and measuring the success of smart city and IoT projects. In CityVerve, they refer to social

metrics for evaluating factors that are crucial to user acceptance of technologies and services.

We borrow the concept of KPIs from management science and contend to re-interpret it with a

community approach. Distinct from common business metrics, which often focus on economic

parameters, we mean for Community KPIs to address social parameters, orienting on who the

‘users’ are and their needs -- in this case, the citizens.

CityVerve envisages that the use of Community KPIs will give local residents and citizens a voice on

how to assess emerging IoT technologies and services developed in their city or neighbourhood;

contribute to user research and requirements gathering in early stages of design; and demonstrate

a framework that might be replicated in other IoT and Smart Cities projects globally.

We conjecture that the Community KPIs will align with some of the key challenges concerning

user acceptance, such as privacy, trust, and data sovereignty. However, given the nature of the

participatory process and grounded approach these conjectures will remain unknown until the end

of the community input.

2.1.1 Foundations of Community KPIs

Connections to business metrics

Traditional Key Performance Indicators (KPIs) often appear as economic metrics tied to an

evaluation framework to measure the performance of a project. KPIs, in other words, evaluate

the success of an organisation or an activity in which it engages. They have been used widely in

management science to assess the the performance of an established business process such

as marketing and sales, manufacturing, supply chain management, and IT project execution and

operation.

Distinct from common business metrics that address economic parameters, Community

KPIs additionally address social parameters, particularly because they are developed using a

participatory design methodology. They offer a framework for evaluation that is not only based

on commercial factors but is also based on how citizens see their city and its technological

interventions.

10


In the same way that businesses might use traditional KPIs to evaluate their success at reaching

targets, Community KPIs aim to evaluate the success of technological interventions in a community

or city. For example, whereas a traditional business KPI may look at the number of people who

have purchased a product or service, a Community KPI may look instead at how a community

member experiences that product or service, and whether their use of it is sustained over time.

Community KPIs can, of course, also include metrics that look at factors that overlap with

commercial factors and interests, for instance the number of times an app was downloaded, or

how many people constitute a potential market for a certain digital product. However, the focus of

Community KPIs will most often be based on how citizens perceive their city and its technological

interventions.

Connections to social indicators

Additionally, Community KPIs borrow from the concept of social indicators, or community-level

indicators (CLIs). Social indicators appeared as early as the 1830s in Belgium, France, England, and

the United States when they were applied to social and health reform issues (Rethoret 2013; Cobb

& Rixford 1998). Physicians and statisticians hoping to understand urban disease epidemics began

to look at the social components of Census data, which eventually led them to formulate models

linking disease with poverty and other social conditions.

Today, Community-Level Indicators (CLIs) are widely understood and used in a variety of projects to

track progress and success rates of community-driven projects. Their use works toward rendering

the invisible visible, in the sense that the more abstract and immaterial outcomes of socially oriented

projects (e.g. awareness, loyalty, and so on) tend to be difficult to capture on their own.

For example, in Making Sense EU, a project that involves the deployment of environmental sensing

initiatives across three European cities, CLIs will be used to track how communities form around

technologies, among other factors (Woods, Hemment, Bui, 2016).

2.1.2 Example of a Community KPI

Community KPIs can be designed to measure objective, observable proxies for more abstract

outcomes and changes. The following example looks at one of the CityVerve use cases and

proposes some hypothetical Community KPIs that might be relevant to the initiative in measuring

changes and outcomes.

Example: Air Quality Community KPIs

One of the CityVerve use cases involves installing sensors in street furniture (e.g. lamp posts, street cabinets) on the

Manchester Corridor to monitor air quality at different heights and locations. Readings from these installations will be

communicated to those with health conditions related to poor air quality, as well as made generally available for the public

to support walking route options.

Some possible Community KPIs to track for a project such as this one might include the following:





11


Communicating value

The ability to evaluate a project not only allows for tracking progress and outcomes, but it also

allows for determining the overall value of the work completed. Using indicators to measure

outcomes helps to determine whether the intervention made a difference in the community for

which it was designed.

Furthermore, having a common set of indicators to evaluate initiatives allows for comparing results

from one use case to the next. In CityVerve, community members will be asked to help design

project-wide goals and indicators as well as goals and indicators for individual use cases. Ideally,

project leads will eventually be able to look at CityVerve’s use cases and see, through the lens of

project-level Community KPIs, how one intervention fared in comparison to another.

Including community priorities

Instead of taking into consideration technology alone, Community KPIs offer a dimension of user

feedback that focuses on how well IoT and Smart City technologies were received by the very

people for whom the technologies were designed.

Involving citizens in designing the process for use case evaluation allows for the representation of

community priorities and concerns in the resulting evaluation framework. Community members

consequently bear a stake in measuring a project or use case’s success.

In this way, Community KPIs can help project managers gain insight on the main barriers to

user acceptance, in other words why some technologies are not successful or are not widely

adopted by communities. They may also inspire ideas for new services that might be derived from

commonly identified threats and benefits of IoT and Smart Cities technology.

Tracking change over time

Whereas it is entirely possible to collect direct feedback from users of IoT and Smart City

technologies in a more immediate, short-term setting, the challenge of measuring long-term change

lies in navigating the nebulous, complex factors that indicate meaningful social change.

Not only does using Community KPIs support evaluation before and after a project, but it also

involves evaluation in medias res while the project is taking place, enabling project managers to

collect feedback about their work, draw conclusions from it, and choose (if necessary) whether to

change course or to continue in the same direction as before.

At the very least, Community KPIs provide a common framework from which to evaluate what

kind of change CityVerve projects seek to affect, and to what extent the projects are successful

in doing so.

12


2.3 When to Evaluate

Project timelines can often be broken down into a before, during, and after period during which

evaluation is crucial to understand how the project is progressing, and whether the intervention is

effective or not.

2.3.1 Before: Establishing a Baseline and Goals

The ‘before’ period of a project’s assessment focuses mostly on planning and coordination among

partners, project leads, and community members. This is when those involved in the project will

likely be setting and prioritising short- and long-term goals. Here, it is crucial to establish a baseline

for project or use case assessment so that in the ‘during’ and ‘after’ phases (discussed below), one

can understand what changed, and how this change is different from when the project began.

At the outset of a project or use case’s implementation, project leads and Community Champions,

who comprise the main research team, should work to capture project goals as defined by

community members and other involved stakeholders. Goals can be a starting point for developing

Community KPIs in order to track progress.

For example, if a group of citizens in Manchester identify a commonly shared goal to better

understand air pollution and its public health risks, a correlating Community KPI might be the

number of publicly available resources about air quality, pollution, and actionable steps for

prevention or mitigation.

To be clear, this is not to say that new goals cannot also be set after the project has begun.

However, much heavy lifting happens during this before stage, and social changes can occur early

on in these early days (among the stakeholders) as a project begins to form around initial shared

goals.

Documentation might include (but is by no means limited to) materials from brainstorm sessions,

co-designed roadmaps and action plans, promotional materials to garner participation, contextual

research for the environmental matter of concern, meeting minutes, number of community

members in attendance, communication channels, and more.

2.3.2 During: Tracking Progress

Measurements taken during the project’s deployment are meant to capture feedback for everything

else in between, with a sharp focus on Community KPIs that signal whether or not a project has

been effective up until that point.

Possible categories for measurement of participation might be the number of members who

attend events at each stage, their frequency of attendance, and the turnover rate. Planning

products such as written objectives, or committees that form can be documented as well. Here

too media coverage (e.g. television, radio, print media, social media), financial resources (e.g.

grants, donations, in-kind services), services provided (e.g. classes, lectures, workshops held), and

community actions (e.g. petitions, protests, rallies) can be recorded.


‘After’ measurements reflect the period after which projects have ended, or they may reflect the

13


esults of certain sub-stages within a project. They may be measured by observing changes in

programmes, such as a new or modified service or programmes; changes in policies, such as a

new or modified policy; changes in practices, such as a new or modified practice; or changes in

behaviour.

2.4 Who is Involved

Smart City and IoT projects often engage a diversity of stakeholders across different sectors in the

city. Successful projects seek to engage different stakeholders in important design and decision

making processes so as to incorporate input from different perspectives.

For CityVerve projects, stakeholder groups include project team members, design experts,

Community Champions, local residents, representatives of voluntary and service user groups,

artists and researchers. The same mix of stakeholders can be scaled to other IoT and Smart City

project as well.

Project Leaders

Community

Champions

Design Experts

Citizens,

Community

Members, Users

CityVerve


Researchers

Artists

Representatives

of voluntary and

service user groups

Fig. 2. Constellation of community stakeholders for CityVerve, and potentially other IoT and Smart City projects.

Community champions are local people trained in role play, facilitation, theatre, filmmaking

who will support user research and support citizen engagement as part of participatory design

processes. Given their key roles and long-term involvement with CityVerve projects, they are best

poised to facilitate the process of Community KPI definition and measurement as members of the

CityVerve design and research team.

14


Project team members work specifically on any of the 16 CityVerve use case proposals in

a management or facilitation capacity. In CityVerve, they come from a range of public and

governmental agencies, technology corporations, and SMEs.

Design experts support use case project teams to implement a human centred and participatory

design process. The use cases are currently ideas we will validate through the user research, prior

to development and deployment. In CityVerve, the design team is supplied by FutureEverything.

Researchers from universities or related institutions aid in the design and assessment of

Community KPIs throughout the course of CityVerve projects and can also consider other specific

aspects of IoT and Smart City projects to develop high-level findings from the work being done in

practice. In CityVerve, research contributors on Community KPIs are supplied by FutureEverything.

Project lead on Evaluation in CityVerve is University of Manchester.

Citizens, community members, or users are residents in Manchester, who currently have

no exposure to the CityVerve project. In CityVerve, participating citizens are drawn from the

relevant neighbourhood (surrounding the ‘Manchester Corridor’) and service user groups. The

role of citizens is described in further detail in Section 2.4.1 (‘Who is ‘the community’ and how

representative is the sample?’).

Representatives of voluntary and service user groups are intermediaries of community groups

within the city who can speak on behalf of the interests and concerns of others. In CityVerve,

relevant voluntary, civil society and service user groups are identified by the project team. The

groups may be those whom an organisation is already engaged with, for example ‘Friends of

Alexandra Park’, or a community sports trust.

Artists will participate in some project stages and be commissioned to develop two large scale IoT

public realm installations, with scope for these installations to interplay with existing street furniture,

such as bins, lamp posts and bus stops. The artworks will enable the public, customers and

stakeholders to explore and/or interact with IoT technology and use cases. The public, users and

stakeholders can be engaged in shaping questions, collecting data and adding new interpretations.


The goal in CityVerve is to develop a blueprint for IoT and Smart Cities globally. This creates a

challenge in defining a representative sample, as the population of interest would be defined as all

residents of all cities.

Community members will be recruited directly by CityVerve and FutureEverything team members

through outreach to community groups that may be most impacted by the use cases. Some

other citizens will be “users” identified by Use Case Project Leads, who can best speak to certain

problems that proposed use cases aim to address.

Our team anticipates that the group of citizen participants in CityVerve workshops many not

necessarily consist of the same individuals throughout all the workshops and their various

stages, and we fully recognise that the sampling will not be completely representative of the City

of Manchester.

15


However, the strategy to ensure that selection of community participants does reflect people

concerned is to invite representatives of community and user groups most likely to be directly

impacted in Manchester. We therefore will work with representatives of community and user groups

in place of a statistically true sample frame.

The Community KPI workshops proposed in Chapter 3 aim to establish citizens as intermediaries

and gatekeepers of community groups within the city who can speak on behalf of the interests

of others. This structure follows a “liquid democracy” model in which delegates of a community

represent the views and intentions of that community (Paulin, 2014). We also anticipate that the

Community Champions will fill a gap in keeping community members informed and engaged

throughout the project, and in addition serve as critical points of contact for use cases.

It is also worth noting that IoT and Smart Cities projects elsewhere have dealt with similar

constraints for recruiting citizen participation (i.e. representativeness, inclusivity). In some cases,

proposed projects scaffold work that has already been done by pre-existing communities and

do not necessarily start with work that involves building a community from the ground up, based

on a single project or call to action. In other cases, socially-oriented IoT and Smart Cities projects

can benefit from relying on existing social networks and “champions” or representatives of larger

community groups (Polvora, Nascimiento, Sanders, & Graell, 2016; Balestrini, Diez Ladera, Polvora,

& Nascimiento, 2016).

CityVerve aims to be as strategic as possible in working with community representatives in order

to be as representative and inclusive of the views of people likely to be impacted by Use Case

interventions.

2.4.2 More on Community Champions

Over the two years of the CityVerve project, there needs to be a constant, as citizen participants

may vary throughout the implementation of each use case. The constant in CityVerve will be

Community Champions, individuals trained locally in role play, facilitation, theatre, filmmaking

who will support user research and support citizen engagement as part of participatory design

processes.

They are paid members of the project team and contribute as researchers, rather than

contributing their own insights as community members. Community Champions are the people

most consistently contributing to Community KPI definition and measurement. In CityVerve, the

Community Champions structure is designed by FutureEverything, and draws on young people

from the Creative Experts scheme by Contact, one of the leading youth theatres in the UK.

During workshops, one of the main roles of Community Champions is to present more technical

concepts in IoT such as privacy, interoperability of data ownership, and so on, in a way that can be

understood by the general public.

It is worth noting that their role is essential not only in facilitating workshops for the co-development

of goals and indicators, but their consistent involvement throughout the project will help to deliver

community outreach and engage communities in key design phases.

In order for community members to contribute, they need a structured way to stay informed and

engaged. Community Champions, by helping to deliver community outreach and engagement,

16


serve as a social network through which citizens can remain involved and become partners in use

case development.

Given their key roles and long-term involvement with CityVerve projects, they are also best poised

to consistently contribute to Community KPI definition and measurement. They will use creative

methods, such as role play, within participatory workshops to communicate IoT design challenges.


Ideally, stakeholder groups (including members from all sectors mentioned above) will be

responsible for co-designing Community KPIs.

Goals and indicators identified by community members may or may not be technologically driven.

While Use Case Leads may be interested in issues such as privacy preservation, interoperability of

data across platforms, data ownership, and so on, these should be supplemented with communitydriven

goals and indicators, which may revolve around concerns over pressing social issues, policy,

or otherwise.

Furthermore, it is crucial that community members have an opportunity to learn the basics on

more technical aspects of projects and use cases in order to make informed decisions about their

importance and implementation.

By relying on a process of co-design to develop Community KPIs, all stakeholders can potentially

weigh in on what objective measures are most relevant to them, enabling participants to become

more informed and invested partners in use case development.

2.6 Limitations

Because the use of Community KPIs is a novel approach within CityVerve, it is still subject to

validation and is not without its limitations.

1. Our team acknowledges that the sample of citizen participants involved in creating

Community KPIs will not necessarily be a representative one. However, we intend to, to

the extent that is possible, represent concerns of community members by working with

intermediaries of community groups within the city who can speak on behalf of the

interests of others.

2. The human centred design and citizen engagement processes that FutureEverything will

lead were introduced after ideas for use cases had already been formed. In CityVerve, the

project ideas were developed when the proposal was written, by different project partners.

This involved widely varying types and degrees of user research, between, say, a health

use case and an energy use case. As a rule, citizens were not involved as co-creators of

the project ideas. In the implementation of these methods in CityVerve, therefore, the focus

is on validation of ideas, not on developing new ideas around emergent needs.

3. The methods and principles here described will not be implemented universally and

uniformly across all use cases or project themes. This is due to the constraints of

17


introducing a novel approach in a project of this scale, and in a sector accustomed to very

different project management methodologies.

4. The human centred design and citizen engagement processes is led by FutureEverything.

But FutureEverything does not ‘own’ the use case ideas, and implementation is dependent

on the use case teams. The risk is high that the commitment of use case teams wanes, or

does not materialise in the first instance, due to lack of alignment, or simple fatigue in such

a long and complex project.

Consequent to these limitations, our team is neither proposing a full human centred design journey,

nor a full framework for citizen engagement, formed from the outset of ideation of use cases.

Nonetheless, citizens can still play a significant role in monitoring the progress of use cases through

the assessment process, during which citizen feedback can be crucial to shaping how the project

and its use cases progress and how they are evaluated from the project team’s end.

It is critical to manage expectations of participants, and to document outcomes, so that learnings

can inform future implementation.

2.7 Replicability

One ambition for a Community KPI approach is that it will be flexible enough to implement in other

IoT and Smart Cities projects elsewhere. While its current design does contain some features

that are distinct to CityVerve (i.e. constraints and resources as stated above in ‘2.6 Limitations’),

our team’s intent is that the framework can be applied to other cities, sectors, technologies and

communities.

Any new implementation, however, would require equivalent local strategies to be developed in

order to account for differences in audience, technologies, context, resources, constraints, and

so on.

18


3. Toolkit: Creating and

Evaluating Community

KPIs

The following section describes a framework for codesigning

Community KPIs with multiple stakeholders.

It comprises a series of workshops and follow-up miniworkshops

that can be held with stakeholder groups

throughout the timeline of various IoT and Smart Cities

projects and can be replicated in projects outside of

CityVerve.

That said, however, this framework is initially designed for a CityVerve workshop taking place on

7 th September 2016 in Manchester, during which CityVerve team members will present the results

of initial user research alongside a “provocation” on best practices and barriers to user acceptance

of former IoT and Smart City projects.

The series of workshops all focus on development, refinement, deployment, and evaluation of

Community KPIs over time.

3.1 A Collaborative Framework for Assessment

Unlike traditional top-down methods for assessment and evaluation, this report describes a

collaborative methodology for IoT and Smart City assessment, piloted in CityVerve. This is within

an open prototyping framework, a novel participatory design framework that involves citizens as

stakeholders in developing a concept or process. (3)

Community KPIs are one aspect of an overall open prototyping approach. To involve citizens as

stakeholders in the process of designing and evaluation framework for projects, CityVerve aims

to gather input from citizens and multiple external contributors, and through interventions that are

open and accessible to various publics or audiences. This starkly contrasts with a more top-down

approach in which a centralised body determines the evaluative framework for a project.

(3) Open prototyping is a framework to develop and test a concept or process through input of external contributors, and through

interventions that are open and accessible to various publics or audiences. Opening the process up can create points of contact

to various contributors and users at different stages in the development process, it can entail multiple points of openness and

synthesis.

19


The Community KPIs created by citizen stakeholders can be categorised in two main ways: (1)

Project-level Community KPIs based on core values and goals of CityVerve as a whole and (2) more

granular Community KPIs tailored toward individual use cases, within the different themes.

This approach allows for capturing progress over time at different scales. The more granular

Community KPIs will provide insight on outcomes of individual projects in relation to the higher-level

goals and values of the wider CityVerve agenda.

All workshops are envisaged to be led principally by project leads and Community Champions,

who will be engaged with CityVerve initiatives in the long-term, unlike other stakeholder groups

which may fluctuate or change throughout the course of the wider, two-year CityVerve project.


ITERATE REFLECT

1. Engage

stakeholders in

human centered,

participatory

workshops to

establish common

goals and indicators

for projects.

2. Further refine

goals and indicators

for specific use

cases based on

broader, projectlevel

goals and

indicators.

3. Develop

observational

systems for

tracking indicators

during project

implementation.

Assess progress in

goals and indicators.

4. Review and

evaluate outcomes.

Reflect on best

practices for

assessment.

Fig. 3. Collaborative framework for development of Community KPIs in IoT and Smart City projects.

The preliminary Envision Workshop will focus on mobilising stakeholders (which includes citizens/

community members/users) and discovery of core goals and the Community KPIs that encompass

them. The results from this workshop will constitute the higher-level Community KPIs from which all

other KPIs for individual projects will be derived.

Following the initial visioning exercise are two mini-workshops: an focusing

on further refining and iterating Community KPIs based on the higher-level goals of CityVerve,

and an focusing on how to deploy the assessment of Community KPIs

throughout each individual project. These workshops are meant to be follow-up exercises for the

main Visioning Workshop and are designed for Community Champions to carry out the work with

the more specific project teams.

Then, at the conclusion of each individual CityVerve initiative, a final Reflect Workshop focused on

evaluation and reflection provides a participatory method for understanding key findings from the

assessments.

In between workshops, we will explore different ways of keeping community members engaged

with the conversation. This will mostly be done through the Community Champions, who will liaise

between CityVerve project leads and community members. We will also explore the use of digital

20


tools, discussed further in Section 4.2, to sustain relationships and discussion about the project as

it progresses.

3.2 Workshop: Envision


Purpose:

To build the first iteration of Community KPIs for CityVerve.

When to run this workshop:

This workshop will be run at a point where CityVerve Use Case Leads are building their use cases by gathering user

research through internal conversations or research undertaken during the human centred design process. Use Case Leads

should also have been introduced to key concepts of Human Centred Design.

Who should be involved:








to 2 users.

What you’ll need:




in two)








Community KPIs to measure the success of CityVerve that will be used over the next two years for CityVerve.

Total duration: 2 hours, 30 minutes

Preparation

In advance of the workshop, the use case representative will be given preparation as homework that includes compiling

and/or summarising the following items:

What existing user research can be used in this workshop?

: What is the problem you are solving?

What are some key ideas for innovating solutions for these problems?

21


Workshop Plan

Introduction (30 minutes total)

Welcome and introduction: (10 minutes)

Introduction to CityVerve and smart cities:




The introduction opens the door for the community to understand what will be happening in Manchester over the next two

years and for all attendees to gain a deeper understanding of why a community voice is important to the project.

(20 minutes)

Project leads, Community Champions, and facilitators break down the communication barriers between Project Leads

and the community. This will focus on addressing anxieties and fears from all attendees and facilitators to create an open

environment.

A presentation that follows the culture setting will focus on best practice and barriers to user acceptance. What does

success look like for IoT and Smart Cities projects? Examples of benefits and threats in the IoT are presented, grouped

under thematic groupings, represented by a matrix on the wall or a canvas (e.g. Usefulness, Convenience, Sense of Self,

Wellbeing). The matrix will be used throughout the workshop to ground the eventual KPIs in reality and to make sure that

both attendees and use cases have thought through potential barriers and best practices.

The matrix will be a key component of the Project Lead presentations (see below) and the basis of the community

understanding of CityVerve. Each Use Case group will have the matrix available to them at all times throughout the

workshop.

Group Work (60 minutes total)

(10 minutes) Community Champions split between the two groups

Group 1: Project Leads

Project Leads are split into their own group. A facilitator will give project leads a template to work with, that defines how they

should present their use case, research and ideate goals. The matrix presented in the provocation piece will be a key part of

the template. Project Leads will be asked to consider:

1. To define the problem they are solving in one sentence

2. Their technology solution

3. How barriers will affect their solution and impact their ability to solve their problem

Group 2: Citizens and users

Community members will be in the second group. A facilitator will give them a more in-depth understanding of CityVerve

and their role in defining the Community KPIs. Citizens and users should have an opportunity to share their responses to the

questions about the provocation content.

They will be asked 2 questions based on the provocation content:

1. Are there any barriers that you think are missing?

2. How would you describe your ideal world with CityVerve?

22


Regroup: (50 minutes, spending about 5 minutes for each use case)

Each use case lead will begin the regroup session by presenting the information that they have put together using the

template provided to them. This will share contextual information and give a brief introduction about the project idea, the

problem they are solving, the results of user research to the whole workshop.

This will be done for every use case. Community attendees will be given the opportunity to ask Project Leads questions

regarding their Use Cases.

Group into Use Cases

Attendees of the workshop will be given writing materials (i.e. post-its, paper, markers, etc.) to respond to questions

prompts. Each group will also have a variety of templates to help structure their conversation. This will include the matrix

used to present the barriers and acceptance of IoT.

For each use case, the Community Champions should lead a discussion on the following questions, including any other

relevant ones to the community or use case at hand. Using the Use Case Leads as experts the aim of the discussion is for

the community to understand the relevance of the Use Case and the technology that will be implemented in Manchester.

The Use Case Leads need to be listening to the community perspective to break down any assumptions that they have

brought to the table.



friends. In what ways can it positively impact on the lives of residents?



Participants should document their responses on post-its and place them in the middle of the table or on a wall where

responses are visible by everyone, using the matrix template as a guideline for clustering information.

Facilitators and participants should then draw out key themes from the responses using templates (including the IoT barriers

and a KPI tick list) as inspiration.

Break (5 minutes)

Goals & Indicators (60 minutes total)

Identify goals: (20 minutes)

After citizens and users have submitted their responses, the discussion should then move toward what community goals

participants want to establish for the use cases.

Participants should submit at least 1 goal for each theme that defines what success looks like for the use case in their

communities and Manchester as a whole.

Identify indicators: (20 minutes)

For every community goal that is identified, participants must now also identify a related indicator that can be used to

measure progress.

For example, if a community goal is to “improve the physical wellbeing of local people,” a relevant indicator might be “the

number of people jogging in the park.” If a community goal is to “increase use of public transport,” an indicator might be

the number of passenger journeys over a given route before, during, and after the pilot.” An indicator for “people having

control over their own data” might be “access to and use of tools to set permissions.” And for “fewer people exposed to air

pollution in the neighbourhood” it might be “people using a clean route app on a daily basis.”

This exercise can be done one goal at a time, or if there is enough wall or table space to host responses to all goals at once,

organisers can choose to proceed as they see most suitable.

23


The resulting goal-indicator pairs will reflect the high-level Community Key Performance Indicators of CityVerve initiatives

that can be generalisable across projects and use cases.

These indicators will be the basis for the Community Champions to develop their workshop and understanding of what the

community needs. Key things to consider are:



Project leads leave room for feedback discussion

Project Leads will have the opportunity to reflect on the workshop so far and think about any further research they need to

do. They will be asked:





This will be mapped out onto the matrix.

Vote: (20 minutes)

Once all community goals and indicators have been submitted, organisers should prompt attendees to appraise the goals

by level of importance to them and/or their community.

The following questions should be discussed regarding the existing goals and Community KPIs, and responses should be

recorded on post-its or somewhere visible to everyone:





The community should then narrow down goals to a list of key goals and indicators that they would like to achieve. These

will then be categorised into different levels of for the project (CityVerve, Use Case and Community).

These will be written onto a final Community KPI template defining:




Project leads back and close workshop

Document: (time varies)

Make sure to record the results of each stage of the workshop so that the process is documented for future reference.

These results should be shared with participants of the workshop.

Community Champions will also be documenting the workshop conversations in order to capture infprmation to inform their

continued community engagement.

24


3.3 Mini Workshop: Iterate


Purpose:

In a smaller setting, to refine goals and Community KPIs for individual CityVerve projects based on overarching goals and

indicators identified in the Envision Workshop. These more granular Community KPIs will be project-specific and serve as a

guide for assessment of each individual initiative in relation to the “big picture” goals and indicators of CityVerve.

When to run this workshop:

After the Envision Workshop and preferably before major implementation of an individual CityVerve project.

Who should be involved:







What you’ll need:









Total duration: 45 minutes

Preparation

Community champions, project leads, and facilitators should work together to summarise findings from the Visioning

Workshop to be presented to mini-workshop participants in the form of a PowerPoint, brief oral presentation, printed

pamphlets, etc.

Workshop Plan

Introduction (20 minutes total)

Welcome and introduction: (5 mins)

All participants should have an opportunity to introduce themselves to each other.

Review: (15 minutes)

Community Champions, project leads, and facilitators should work collaboratively to present the materials from the

Preparation phase of this workshop in order to review results from the Envision Workshop.

Goals and Community KPIs identified during the Envision Workshop should be placed somewhere visible during this

workshop, either projected on a screen, placed on the a wall, or printed out for participants.

25


This review step is particularly important because participating citizen stakeholders may vary from one workshop to the

next. In order to make decisions on what is important going forth, citizen participants (as well as all other participants) must

have a mechanism by which they understand what the concerns and values were of citizen groups that preceded them.

Group Work (25 minutes total)

Discuss: (10 minutes)

Community champions, project leads, and facilitators may divide all participants into smaller working groups or decide

to work together as one whole group. The following questions should be discussed regarding the existing goals and

Community KPIs, and responses should be recorded on post-its or somewhere visible to everyone:




Rate: (15 minutes)

Organisers should prompt attendees to appraise the goals by level of importance to them and/or their community. There are

several ways to order the goals, one of which is to categorise them by feasibility and relevance. Organisers may use colored

dots or other methods to categorise each goal/indicator as such:

These are goals the community thinks are important and feasible. You will want to pursue these

right away.

These are goals the community thinks are important, but will be difficult to accomplish. You

should try to do these, but keep in mind these goals will require extra effort to accomplish.

These are goals that the community thinks are easy to accomplish, but are also are not all that

important. You should do these if you need to increase your group’s credibility.

These goals are of low importance to your community, and are difficult to do. The only time you

would want to do these is if you know something the community doesn’t; because you won’t get much support

and even if you do succeed it may not matter.

The group should then narrow down goals to a list of key goals and indicators that the group would like to achieve based on

these features.

Both organisers and participants should actively discuss whether goals and indicators identified seem appropriately related.

Document: (time varies)

Make sure to record the results of each stage of the workshop so that the process is documented for future reference.

These results should be shared with participants of the workshop.

26


3.4 Mini Workshop: Assess


Purpose:

To collaboratively determine observational systems to monitor identified Community KPIs for measuring the progress of

individual projects.

When to run this workshop:

After the Iterate Workshop.

Who should be involved*:






of project-level and use case-level goals/indicators and availability to monitor their progress. If this is not the case, an

orientation should be provided in the Review section of this mini-workshop.

What you’ll need:









Total duration: 30 - 45 minutes

Preparation

Community champions, project leads, and facilitators should work together to summarise findings from the Envision and

Iterate Workshops to be presented to mini-workshop participants in the form of a PowerPoint, brief oral presentation,

printed pamphlets, etc.

Reporting Template

See section 4.1 for a proposed reporting template to capture data on project-level and use case-level goals and indicators.

This template can be used as a starting point during the mini-workshop to visualize a possible format for capturing

information. Community members should be encouraged to come up with additions and revisions to the template to suit

use case needs.

27


Workshop Plan

Introduction (20 minutes total)

Welcome and introduction: (5 mins)

All participants should have an opportunity to introduce themselves to each other.

Review: (15 minutes)

Organisers should work collaboratively to present the materials from the Preparation phase of this workshop in order to

review results from the Visioning and Refine Workshops.

Goals and Community KPIs identified during previous workshops should be placed somewhere visible during this

workshop, either projected on a screen, placed on the a wall, or printed out for participants.

Tools (25 minutes total)

Decide and delegate: (25 minutes)

Attendees should decide how observations of Community KPIs would be conducted. Is it direct observation, participant

observation, self-reporting, electronic/mechanical observation, public records, or otherwise?

Decide on which tools are most relevant and useful for observing the Community KPIs. The tools used for monitoring

progress should be agreed upon by all present so that information can be shared in a coordinated and efficient manner

across individual projects.

Participants should also decide how often the Community KPI should be measured. Is it at intervals? Once? Before and

after?

Delegate tasks to individuals who would like to be accountable for making these observations. Will it be a Community

Champion? Will it be a project team member? Will it be an independent researcher? Include the “who” next to the other

factors.

Community KPI

How will it be measured?

(Tools)

How often should it be

measured?

(Timing)

Who will be responsible for

collecting measurements?

(Team)

Fig. 4. Creating observational systems for assessing Community KPIs.

This information should be crystallised into a working plan that designates specific tasks to specific people who assume

responsibility for tracking Community KPIs.

28


Document: (time varies)

Make sure to record the results of each stage of the workshop so that the process is documented for future reference.

These results should be shared with participants of the workshop.

29


3.5 Workshop: Reflect


Purpose:

To review results from observation of Community KPIs and articulate the long-term changes that a community wants to see

beyond the timeline of the project.

When to run this workshop:

About a week or two after individual projects have concluded.

Who should be involved:





projects after their commencement




What you’ll need:









Total duration: 1.5 hours

Preparation

Community champions, project leads, and facilitators should work collaboratively to compile the following materials before

the workshop:





Summarised materials should focus on answering the following questions:




30


Workshop Plan

Introduction (30 minutes total)

Welcome and introductions: (10 minutes)

All participants should have an opportunity to introduce themselves to each other.

Review and reflect: (20 minutes)

Organisers should spend time reviewing goals and indicators for the project under evaluation, supported by any relevant

materials from the Preparation phase of the workshop.

This review step is particularly important because participating citizen stakeholders may vary from one workshop to the

next. In order to make decisions on what is important going forth, citizen participants (as well as all other participants) must

have a mechanism by which they understand what the concerns and values were of citizen groups that preceded them.

Assess (60 minutes total)

Evaluate: (30 minutes)

1. Identify which goals have been completed, and which indicators show they have been completed.

After writing down goals and indicators, identify which goals have been reached by the use case’s end. Note any important

indicators that point to a change since the project’s launch.

2. Determine which goals have not been completed and identify reasons why this might be the case.

Constructively look at which goals have not yet been reached. Discuss and document why this may be the case, making

note of which indicators point to lack of change since the project’s launch.

3. Discuss what new goals have arisen since the start of the work.

Asking questions almost inevitably leads to asking more questions. Allot time toward discussing what new questions or

goals have arisen since the beginning of the use case’s implementation. Document these too.

4. (Optional) If available, compare results from this project with other CityVerve project results from other Reflect

Workshops.

How were observations conducted in other CityVerve initiatives? Did other projects encounter similar or different

challenges? What best practices can we glean from the work in other projects?

Looking ahead: (30 minutes)

1. Decide how to communicate results

Discuss and identify what potential audiences for this information might be. Is it policymakers? The community itself? The

media? Other organisations?

This data can provide valuable insight into the work the community has done. It will also point to work that still needs to be

done and opportunities for further investigation in the future.

If the report highlights weaknesses, that is certainly understandable. In fact, that is the main purpose of this step of the

evaluation process. If, after completing the report, your group draws some conclusions that will help strengthen your action

plan, then it is worthwhile.

In other words, low numbers should inspire positive change, and hopefully not just dissatisfaction.

2. Divide the work

Will multiple people be responsible for putting together content for the report, a designated individual, the organiser, a third

party, or otherwise? Finally, decide who is responsible for what aspect of reporting out this information and divide the work

appropriately.

31


Document: (time varies)

Make sure to record the results of each stage of the workshop so that the process is documented for future reference.

These results should be shared with participants of the workshop.

Community Champions will also be documenting the workshop conversations in order to capture infprmation to inform their

continued community engagement.

32


4. Reporting Tools

4.1 Reporting template for goals, indicators, and outcomes

The reporting template below may be used as a reporting tool for capturing data about goals,

indicators, and outcomes. The template is designed to document these variables in a general

sense, reflecting CityVerve’s wider project-level objectives. At the same time, it leaves room to

capture data about use case-level goals, indicators, and outcomes.

In order to use the template, community members should, in the top section, write down projectlevel

(CityVerve) goals, their related indicators, and the outcomes of those indicators during

measurement periods. Then, the same should be done in the bottom section for use caselevel

goals, indicators, and outcomes. Community members should also note the theme (i.e.

transportation, community wellbeing, etc.) for each use case in the designated space for this

information.

The template should be used as a starting point for discussion during the ‘Assess’ mini-workshop

to visualize a possible format for capturing information. Community members are encouraged to

be critical of this template and to adapt it as needed for respective use cases. They should also

be encouraged to come up with additions and revisions to the template to suit the observational

systems they design to measure specific Community KPIs.

Additionally, the template format can be adapted for the ‘Reflect’ workshop to organize feedback

about what was accomplished or not during the course of a use case, and how these goals,

indicators, and outcomes relate to wider project goals.

33


Project Community KPIs (CityVerve)

Theme:

Goal

Indicator


Goal

Indicator


Goal

Indicator


Use Case Community KPIs

Goal

Indicator


Goal

Indicator


Goal

Indicator


34


4.2 Digital tools for sustained community engagement

It is important to consider how to maintain a relationship with community members and

representatives between and beyond workshops. In CityVerve, the Community Champions will

remain the constants during the development and implementation of use cases, and their role is to

continue engaging community participants, albeit participants that may come and go over time.

However, it is worth exploring, through the course of the workshops and mini-workshops, different

communication channels that may enable further discussion or the collection of follow-up input

from community members after workshops have been completed. These channels can also

become meaningful when sharing results from use cases, keeping community members informed

about assessment periods, and conveying other news from the wider CityVerve project.

Above all, they could prove to be a way of sustaining engagement among community members

themselves, project leads, Community Champions, designers, software developers, and so on.

Below are a handful of recommendations for different platforms that may be used to achieve this.

formed using Google Groups or similar services can be a way to engage community

members who are inclined to communicate with each other mostly through e-mail.

like Facebook Groups and Twitter lists can be used independently from or

in addition to other communication channels for audiences who regularly check social media feeds.

Chat clients such as Slack, GroupMe (SMS-compatible), WhatsApp, and Telegram may be used

for those who are more likely to check their mobile phones for messages and updates.

like DemocracyOS, Textizen, Groundsource, and Google Forms allow for project

leads to design surveys to solicit additional feedback from community members and can be

deployed between workshop periods or after workshops have been completed.

35


5. References

Anderson, J. (2014). The Internet of things will thrive by 2025. Pew Research Center. Available at:

http://www.elon.edu/docs/e-web/imagining/surveys/2014_survey/Elon%20Pew%20Future%20

of%20the%20Internet%20of%20Things%20Report%205-14-14.pdf (Accessed: 26 August 2016).

Balestrini, M., Diez Ladera, T., Polvora, A., & Nascimiento, S. (2016). Mapping Participatory Sensing

and Community-led Environmental Monitoring initiatives. Making Sense EU Deliverable D.6.2+D.4.1.

Balestrini, M., Creus, J., Masfarre, G., Canigueral, A., & Arguedas, D. (2016). A Future in Common:

Understanding and Framing Common Strategies for Bristol. Bristol Citizen Sensing Programme,

Phase 2, Deliverable 1. Ideas for Change.

Castro, M., Jara, A. J., & Skarmeta, A. F. (2013). Smart lighting solutions for smart cities. In

Advanced information networking and applications workshops (WAINA), IEEE, pp. 1374-1379.

Cobb, C.W., & C. Rixford. (1998). Lessons Learned from the History of Social Indicators. San

Francisco, CA: Redefining Progress. Retrieved from http://rprogress.org/publications/1998/

SocIndHist.pdf

Greenfield, Adam. (2013). Against the Smart City. Self-published pamphlet.

Hemment, D. & Townsend, A. (eds.) (2013). Smart Citizens. FutureEverything Publications. Available

at: http://futureeverything.org/wp-content/uploads/2014/03/smartcitizens1.pdf

Hemment, D. (2015) Open Prototyping. Available at: http://futureeverything.org/news/openprototyping-alpha/

(Accessed: 29 August 2016).

Hollands, R.G. (2008). Will the real smart city please stand up? City, 12(3), pp. 303–320. doi:

10.1080/13604810802479126.

Komninos, N. (2009) ‘Intelligent cities: towards interactive and global innovation environments’,

International Journal of Innovation and Regional Development (IJRD), 1(4). doi: 10.1504/

IJIRD.2009.022726

Nam, T., & Pardo, T. A. (2011). Smart city as urban innovation: Focusing on management, policy,

and context. In Proceedings of the 5th International Conference on Theory and Practice of

Electronic Governance(pp. 185-194). ACM.

Neirotti, P., De Marco, A., Cagliano, A. C., Mangano, G., & Scorrano, F. (2014). Current trends in

Smart City initiatives: Some stylised facts. Cities,38, 25-36.

Paulin, A. (2014, May). “Through liquid democracy to sustainable non-bureaucratic government.” In

Proc. Int. Conf. for E-Democracy and Open Government, pp. 205-217.

36


Polvora, A., Nascimiento, S., Sanders, E., & Graell, M. (2016). Co-Designing Participatory

Approaches for Communities. Making Sense EU Deliverable 4.2.

Rethoret, L. (2013) Social indicators literature review. In Developing the Columbia Basin Rural

Development Institute’s Social Research Pillar. Castlegar, British Columbia: Columbia Basin RDI.

Available at: http://www.cbrdi.ca/wp-content/uploads/Social-Literature-Review.pdf (Accessed: 11

July 2016).

Saunders, T. and Baeck, P. (2015) Rethinking Smart Cities From The Ground Up. Available at:

http://www.nesta.org.uk/publications/rethinking-smart-cities-ground (Accessed: 15 August 2016).

Woods, M., Hemment, D., & Bui, L. (2016). Community-Level Indicators. Making Sense EU

Deliverable 5.4.

Zanella, A., Bui, N., Castellani, A., Vangelista, L., & Zorzi, M. (2014). Internet of things for smart cities.

IEEE Internet of Things Journal, 1(1), 22-32.

Links

CityVerve

http://www.cityverve.org.uk

FutureEverything

http://www.futureeverything.org

InnovateUK

https://www.gov.uk/government/organisations/innovate-uk

37


Colophon


Dr. Drew Hemment

Published by:

FutureEverything

BASE

Manchester Science Park

Greenheys Lane

Manchester

M15 6LR

UK

futureeverything.org

2016

Design:

Supanaught

The work is licensed under a Creative

Commons Attribution-NonCommercial

3.0 Unported License

British Library Cataloguing-in-Publication

Data: A British Library CIP Record is

available

38

More magazines by this user