A Technical History of the SEI

Measurement and Analysis
The Challenge: Measuring Software Development Capabilities and
Products
Measurement and analysis in software engineering has long been a topic of interest. Without it,
there is no clear, quantitative picture of software development capabilities or a basis for predicting
or comparing products and processes. Although early software projects used several systems for
predicting software costs, measurement was done in different ways and was often based on definitions
that were inconsistent. The need for a standard and reliable set of measures that would help
acquisition program managers and software development contractors alike was an early priority
for the SEI.
A Solution: Approaches for Collecting and Analyzing Data
In response to the DoD’s 1991 Software Technology Strategy, the SEI agreed to lead the development
of a set of core measures to “help the DoD plan, monitor, and manage its internal and contracted
software development projects” [Carleton 1992]. In collaboration with measurement experts,
including those who developed prediction systems, the SEI developed definition
frameworks for a set of core measures. The measures focus on size, defects, effort, and schedule.
These definition frameworks make it possible for organizations to use the measures that best
match their processes and infrastructure while benefiting from a standard way of describing the
operational definitions in detail.
Once the definition problem for measures was resolved, the SEI turned its attention to helping organizations
decide what to measure. Experience from management information systems shows
that many reports could be generated that have little to no impact on decision making within the
organization. The SEI began to investigate the goal-question-metric method for aligning measurement
with information needs in the organization [Basili 1984]. The SEI modified this approach to
include explicit consideration of the output of the measurement activity—that is, the indicator to
be used by decision makers. The method was dubbed goal-question-indicator-measure (GQ(I)M).
Using the GQ(I)M approach helps mitigate the risk of measuring and reporting information that
provides little or no value to the organization.
As approaches for conducting measurement effectively matured and were disseminated, the next
significant challenge became data analysis [Paulk 2000]. The notion of analysis, especially analysis
related to process improvement, was often equated with the high-maturity practices of the SW
CMM and CMMI. More focus on the “analysis” part of measurement and analysis was also
spurred by the establishment of the Measurement and Analysis process area in the CMMI [CPT
2002].
An early and foundational work in this area was Measuring the Software Process, which showed
how statistical process control techniques could be fruitfully applied to software data [Florac
1999]. This was followed by substantial work on the application of Six Sigma analytical techniques
to software engineering [Penn 2007]. The Six Sigma connection provided a rich set of
tools as well as a “brand” that already had roots in many organizations, facilitating its adoption.
The current work on developing estimates early in the DoD acquisition lifecycle incorporates
CMU/SEI-2016-SR-027 | SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY 149
Distribution Statement A: Approved for Public Release; Distribution is Unlimited.