A Technical History of the SEI

e-estimation as a means to assess progress. Risk-reduction practices such as prototyping
or modeling and simulation may lead, for example, to both reduced “cost to complete”
and also narrowed variances (i.e., greater confidence that the estimates are accurate).
c. Incentives and acquisition practices. How can acquisition practices be developed to
build on advances in measurement and process models that feature a structuring of incentives
that enables government and contractors to collaborate effectively in development
of architectures and in iteration at scale?
d. Software sustainment and modernization. The reality of many sustainment efforts is
that they are really supporting a continual evolution and modernization of systems.
Planning for continuous evolution, for example in the form of identifying and separating
dimensions of variability, is a necessary feature of architectural design. Continuous
evolution, importantly, can also involve discontinuous change as infrastructures and
subsystems evolve and new choices emerge.
2. Evidence-Based Software Assurance and Certification. The goal of this second element of
vision is a dramatic reduction in the cost and difficulty of making assurance judgments related to
quality and security attributes. Achieving this goal is particularly important as systems become
more complex and evolve more rapidly. Current approaches for certification and accreditation are
largely based on an after-the-fact evaluation of a snapshot of a system.
While after-the-fact approaches are effective for certain well-defined categories of components
and systems, they tend to break down as systems increase in complexity, scale, and dynamism.
They also tend to hinder ongoing evolution, rapid reconfiguration, dynamic loading of components,
autonomy, and composition and interlinking of systems of systems. Put simply, these established
techniques do not scale up, and they do not work well for the emerging software framework-based
systems now prevalent in commercial and infrastructural applications.
The industry folklore has long asserted that quality-related activities, including security-related
assurance, can consume half of total development costs for larger systems. For example, the IBM
Systems Journal states that in a typical commercial development organization, “the cost of providing
[the assurance that the program will perform satisfactorily in terms of its functional and nonfunctional
specifications within the expected deployment environments] via appropriate debugging,
testing, and verification activities can easily range from 50 to 75 percent of the total
development cost.” [Hailpern 2002]. Additionally, after-the-fact evaluation practices can add a
year or more to the elapsed time required to develop and deploy software-reliant systems. Commercial
systems, including products and software as a service and cloud-based systems, tend to
undergo a relatively rapid and continual evolution. For many of our DoD and infrastructural systems,
we similarly need to support a continuous evolution.
Some areas of particular technical emphasis include
a. Architecture and composition principles. These enable separate evaluation of individual
components, with the possibility of combining results to achieve aggregate assurance
CMU/SEI-2016-SR-027 | SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY 289
Distribution Statement A: Approved for Public Release; Distribution is Unlimited.