Information and Knowledge Management using ArcGIS ModelBuilder

Multi-Scope Evaluation of Public Administration Initiatives
in Process Automation
Luigi Lavazza
Università degli Studi dell’Insubria, Varese, Italy
luigi.lavazza@uninsubria.it
Abstract: Public administrations are among the most active organizations in promoting process automation
based on software systems. For evident reasons, public administrations are interested in evaluating the
effectiveness of their process automation initiatives. However, when performing these evaluations, public
administrations have to deal with a specific characteristic of their environment: the scope of the assessment can
be more or less wide, i.e., it can extend beyond the boundaries of the public administration, to include the
companies, organizations, citizens, suppliers, etc. that directly or indirectly interact with the processes. In fact, the
effectiveness of a process initiative does not concern only the public administration that is promoting it, but
involves all the stakeholders that, to various extents, participate in the process. Although traditional methods for
measurement planning and execution – like the GQM (Goal/Question/Metrics) – can be used, evaluators have to
define specific means to deal with the multiplicity of scope definitions that can be chosen in order to define the
boundaries of the assessment. In this paper we illustrate the usage of the GQM to define measurement plans and
how to update them in order to deal with extended scopes. The proposed technique is illustrated through two
case studies, taken from actual evaluations in which the author was involved. The presented evaluation approach
is more effective than traditional econometric methodologies. On the one hand, it lets the evaluator choose the
set of indicators that are more suitable for the purpose; on the other hand it goes beyond the traditional economic
and financial indicators, extending the analysis to factors that allow a more comprehensive understanding of the
effects of process automation.
Keywords: software process measurement, process evaluation, evaluation scope, Goal/Question/Metrics (GQM)
1. Introduction
Public administrations (PAs) are interested in process automation because of several reasons: to
increase internal efficiency; to favor citizens in accessing the services provided by the PA; to enhance
or make more efficient business activities, not necessarily involving the PA (in this case the process
automation services are seen as infrastructures). Whatever the type of process automation initiatives
carried out by a PA, several actors and stakeholders are generally involved, because of the very
nature of PAs. In fact, a PA is generally involved in providing services to the citizens, to suppliers, to
companies, to other PA entities, to no-profit organizations, etc. While automating its services, the PA
affects not only its internal processes, but also the ones of the citizens, suppliers and external entities
that use (sometimes indirectly) these services.
In general, a PA needs to evaluate the efficiency of its services. This is of course due to the fact that
the provided services are paid with the citizens’ money (via taxes): the PA has to show citizens (and
electors) that their money is well spent. The need for evaluation is even more pressing when new
initiatives are undertaken: in fact the success of the initiative has to be proved, and weak points and
problems have to be identified in order to be tackled and corrected. When evaluations of process
automation initiatives are decided, the multiplicity of scopes emerges as a variable to be duly taken
into account. In fact, by definition, an evaluation makes reference to a scope. In our case, the effects
of the process automation initiatives are perceived at various levels: the PA itself, the companies and
citizens that directly interact with the PA, the citizens that are the targets of the final effects of the
process.
For instance, in a process for procuring educational material, the PA is the procurer, a company is the
supplier, a set of schools are the immediate beneficiary, and the pupils are the final beneficiaries. As
a consequence, the evaluation can be performed at various levels: one can take into account the
effects for the PA, the effects for the company, the effects for the school, the final consequences for
citizens (pupils and their families).
It is quite clear that the sets of issues that are of interest for the various actors and stakeholders are
disjoint. For instance, the citizen will be interested in the cost and quality of the supply, while the
supplier will be interested also in the duration and the profitability of the supply process.
294