A Technical History of the SEI

Federal Aviation Administration Study
The Challenge: Evaluate a Problematic FAA System Under Development
By 1995, the FAA had spent $2 billion on a project to upgrade the in-flight air traffic control
(ATC) system. The attempt to upgrade the system had gone through many well-documented and
costly delays [Glass 1997]. At that time, it continued to fail some of its system tests; as a result, it
still had not been put into service. The existing system, which had been launched a number of
years earlier, had obsolete hardware and software and was becoming very difficult to maintain. In
addition, because of rapidly expanding hardware advances, there were not enough replacement
hardware parts available (mostly transistors), and the hardware in use was slow.
A Solution: SEI Architecture Evaluation Methods
The SEI was contacted by senior officials at the U.S. Department of Transportation (DoT) and the
DoD to conduct an independent technical assessment to determine if the system could be salvaged.
This work by the SEI, in a joint project with Lincoln Laboratory, represented one of the
first times that the SEI had done work on behalf of a non-DoD federal agency. Lincoln Laboratory
personnel had domain knowledge of ATC systems and knew the FAA well, and the joint
SEI/Lincoln Laboratory team worked together very successfully. The FAA also provided the team
with three members to contribute insights from the FAA perspective and provide unfettered access
to personnel and documents. The team was asked to provide input to FAA management on
the technical question of whether the upgraded system was salvageable and, if so, what would
need to be done to feel confident that the system would perform at an acceptable level. If the system
was not salvageable, the FAA was prepared to walk away from the two billion dollar investment
that it had made.
The SEI/Lincoln Laboratory/FAA team split into subgroups to investigate the areas of modifiability,
availability, scheduling, code quality, documentation, software tools, maintenance environment,
testing, change management, and software processes.
To address the question of modifiability, the SEI developed a set of likely change scenarios. At
that time, the SEI was just beginning to develop scenario-based methods for evaluating software
architectures. The prime contractor was asked to demonstrate the amount of change that would be
required for each of the likely changes to the system. As a result, the subgroup determined that
with regard to modifiability, significant changes could be put in place with relatively little disruption
to the system. This was the first practical application of a method for assessing the quality attribute
of modifiability, and this method formed part of the foundation for the SEI Software Architecture
Analysis Method. In turn, after much future analysis, some of the ideas for SAAM later
evolved into the Architecture Tradeoff Analysis Method and the Quality Attribute Workshop.
The other subgroups focused on their respective topics. One group analyzed the quality attribute
of availability because the system had been developed to have extensive hardware and software
redundancy, and many of the decisions on the structure of the system had been made to ensure
availability. The availability assumptions were examined; while many different structures for
availability could have been developed, the one in use was judged to be adequate.
CMU/SEI-2016-SR-027 | SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY 245
Distribution Statement A: Approved for Public Release; Distribution is Unlimited.