A Technical History of the SEI

enchmark. Results of testbed experiments were documented in technical reports and papers and
presented at DoD and industry workshops.
Both Hartstone and the INS simulator contributed important artifacts and results to two other SEI
efforts: the Real-Time Scheduling in Ada (RTSIA) effort and the Distributed Ada Real-time Kernel
(DARK). Both benefited from early testbed experimentation with a programmable real-time
clock device driver and the design and test of approaches to periodic task scheduling. Members of
the testbed team also collaborated with the Advanced Real-Time (ART) project at Carnegie
Mellon University. Several DoD programs based language and processor decisions on the runtime
evaluations provided through this testbed.
The SEI Contribution
The testbed validated the earlier benchmarking contributions of organizations such as PIWG and
the University of Michigan. It also contributed a new benchmark, Hartstone, to address a gap in
the area of measuring deadline-driven computing. Hartstone provides a highly parameterized
benchmark capable of stress testing Ada runtime systems by varying the workload, priority, frequency,
and number of concurrent tasks to be executed. At the time the REST project concluded,
organizations other than the SEI were proposing to create a distributed version of Hartstone and to
implement it in programming languages other than Ada. The lessons learned from the testbed experiments
were incorporated into a comprehensive guide to the selection and evaluation of Ada
compilers as a companion to the Ada Adoption Handbook [Weiderman 1989a].
References
[Altman 1987a] Altman, Neal & Weiderman, Nelson. Timing Variation in Dual Loop Benchmarks
(CMU/SEI-87-TR-021). Software Engineering Institute, Carnegie Mellon University,
1987. http://resources.sei.cmu.edu/library/asset-view.cfm?assetID=10335
[Altman 1987b] Altman, Neal. Factors Causing Unexpected Variations in Ada Benchmarks
(CMU/SEI-87-TR-022). Software Engineering Institute, Carnegie Mellon University, 1987.
http://resources.sei.cmu.edu/library/asset-view.cfm?assetid=10341
[Clapp 1986] Clapp, Russell M.; Duchesneau, Louis; Vols, Richard A.; Mudge, Trevor N.; &
Schultze, T. “Toward Real-Time Performance Benchmarks for Ada.” Communications of the
ACM 29, 8 (August 1986): 760-778.
[Curnow 1976] Curnow, H. J. & Wichmann, B. A. “A Synthetic Benchmark.” The Computer
Journal 19, 1 (February 1976): 43-49.
[Donohoe 1987] Donohoe, Patrick. A Survey of Real-Time Performance Benchmarks for the Ada
Programming Language (CMU/SEI-87-TR-028). Software Engineering Institute, Carnegie
Mellon University, 1987. http://resources.sei.cmu.edu/library/asset-view.cfm?assetID=10381
[Donohoe 1990] Donohoe, Patrick. Hartstone Benchmark Results and Analysis (CMU/SEI-90-
TR-007). Software Engineering Institute, Carnegie Mellon University, 1990. http://resources.sei.cmu.edu/library/asset-view.cfm?assetID=11177
CMU/SEI-2016-SR-027 | SOFTWARE ENGINEERING INSTITUTE | CARNEGIE MELLON UNIVERSITY 29
Distribution Statement A: Approved for Public Release; Distribution is Unlimited