A Technical History of the SEI

Appraisal Methods
The Challenge: Predicting Software Engineering Performance
As the importance of software grew substantially in DoD procurements, so did the need for evaluating
software contractors’ abilities to competently perform on software engineering contracts.
Meanwhile, a team at IBM was also investigating its own software engineering performance and
noticed that different IBM sites varied in their levels of predictable performance. To find out why,
people began to collect the factors contributing to success and compare sites on this basis, which
enabled them to identify issues to tackle and best practices to emulate. A structured approach began
to emerge from this work that allowed for site visits for gathering relevant information, identifying
opportunities, and establishing priorities for improvement. As problems with software contractors
came to a head in the mid-1980s, the DoD turned to the SEI.
A Solution: Assessing the Capability of Contractors
The DoD asked the SEI to use its “objective broker” status to help figure out how to fairly and effectively
determine a contractor’s likely performance based on that contractor’s capability when
producing software. In response, the SEI began to increase its role in software process assessments,
drawing on the work at IBM and other organizations.
In 1986, the SEI work had begun in earnest towards establishing a formal process for evaluating
potential software performance, when the U.S. Air Force and MITRE Corp. asked it to develop a
site assessment method and related model that could be used to assess commercial and government
software organizations. SEI experts began concentrating on a process that would facilitate
objective and consistent assessments of the ability of potential contractors to develop software in
accordance with up-to-date software engineering methods.
In 1987, A Method for Assessing the Software Engineering Capability of Contractors was published
[Humphrey 1988]. The primary goal was to provide a standardized, publicly available
method that could be periodically reviewed and modified. The method was a structured assessment
approach intended to augment contractor evaluation methods in use at the time. The method
document included a Maturity Questionnaire, a five-level Process Maturity Framework, and a
brief set of guidelines for conducting an assessment and evaluating the results.
Even before the Maturity Questionnaire was formally published, the SEI technical staff recognized
the need for a more detailed description of the assessment process and, in July 1987, published
a preliminary report on conducting assessments [Humphrey 1987]. It is interesting to note
that this report envisioned internal assessments for process improvement and evaluation assessments
conducted by DoD procurement personnel. This integration of process assessment methods
soon dissolved, resulting in two similar but separate methods and constituencies: Software Process
Assessment and Software Capability Evaluation. Each of the method descriptions had requirements
for team selection and training, planning, conducting on-site activities, and reporting
results.
Software Process Assessments (SPAs), originally called SEI-Assisted Assessments, were first described
in 1989. The SPA was used to identify an organization’s major problems and engage
CMU/SEI-2016-SR-027 | SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY 114
Distribution Statement A: Approved for Public Release; Distribution is Unlimited.