A Technical History of the SEI

An in-progress study of 214 TSP projects provides additional evidence of the benefits of disciplined
practice. The average CPI (Cost Performance Index) for these projects was 0.93, the average SPI
(Schedule Performance Index) was 0.88, and the average System Test Defect Density was 1.32 defects
per KSLOC (1000 lines of code, or LOC).
Specific examples of improvements include these:
Hill Air Force Base, near Salt Lake City, Utah, is the first U.S. government organization to be
rated at CMM Level 5. The first TSP project at Hill found that team productivity improved
123 percent and test time was reduced from an organizational average of 22 percent to 2.7
percent of the project schedule.
Boeing, on a large avionics project, had a 94 percent reduction in system test time, resulting
in a substantial improvement in the project schedule and allowing Boeing to deliver a highquality
product ahead of schedule [Davis 2003].
Teradyne found that, prior to the TSP, defect levels in integration test, system test, field testing,
and customer use averaged about 20 defects per KLOC. The first TSP project reduced
these levels to 1 defect per KLOC. Since it cost an average of 12 engineering hours to find
and fix each defect, Teradyne saved 229 engineering hours for every 1000 LOC of program
developed.
Advanced Information Services reported in 2012 that its use of the TSP continues to result in
systems with very predictable schedule and quality. The company is currently averaging
0.3765 defects per KSLOC during user acceptance testing. In fact, quality and schedule are so
predictable with TSP that the company is able to support fixed price contracts that include a
warranty against defects after user acceptance test. In 2011, Advanced Information Services
delivered a large, 570 KSLOC software application to the Selective Service System with a delivered
defect density of 0.097 defects/KSLOC [Sheshagari 2012, Ratnaraj 2012].
Beckman Coulter reported in 2012 that first-time use of the TSP resulted in 5 to 100 times
improvement in fielded software on six different medical devices [Van Eps 2012].
A major contributor to the success of TSP teams, besides data, is the commitment and ownership
generated during the launch and sustained throughout the life of the project. It is the synergy that
is created when a team has a common goal and each and every person on that team understands
how his or her work and everyone else’s work contributes to the achievement of that goal.
The SEI Contribution
If even the smallest programs are not of the highest quality, they will be hard to test, take time to
integrate into larger systems, and be cumbersome to use. The SEI was an early contributor to the
idea that software could be significantly improved, from the bottom up, by bringing discipline to
the performance of individual engineers and engineering teams. Only a statistically managed software
engineering discipline can support the growing size and complexity of today’s systems. PSP
and TSP provide the framework and data required.
CMU/SEI-2016-SR-027 | SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY 146
Distribution Statement A: Approved for Public Release; Distribution is Unlimited.