U. Glaeser

8.2 Performance Evaluation: Techniques, Tools,
and Benchmarks
Lizy Kurian John
Introduction
State-of-the-art high performance microprocessors contain tens of millions of transistors and operate at
frequencies close to 2 GHz. These processors perform several tasks in overlap, employ significant amounts
of speculation and out-of-order execution, and other microarchitectural techniques, and are true marvels
of engineering. Designing and evaluating these microprocessors is a major challenge, especially considering
the fact that one second of program execution on these processors involves several billion instructions
and analyzing one second of execution may involve dealing with tens of billion pieces of information.
In general, design of microprocessors and computer systems involves several steps (i) understanding
applications and workloads that the systems will be running, (ii) innovating potential designs, (iii) evaluating
performance of the candidate designs, and (iv) selecting the best design. The large number of
potential designs and the constantly evolving nature of workloads have resulted in designs being largely
adhoc. In this article, we investigate major techniques used in the performance evaluation process.
It should be noted that performance evaluation is needed at several stages of the design. In early stages,
when the design is being conceived, performance evaluation is used to make early design tradeoffs.
Usually, this is accomplished by simulation models, because building prototypes of state-of-the-art
microprocessors is expensive and time consuming. Several design decisions are made before any prototyping
is done. Once the design is finalized and is being implemented, simulation is used to evaluate
functionality and performance of subsystems. Later, performance measurement is done after the product
is available in order to understand the performance of the actual system to various real world workloads
and to identify modifications to incorporate in future designs.
Performance evaluation can be classified into performance modeling and performance measurement,
as illustrated in Table 8.2. Performance measurement is possible only if the system of interest is available
for measurement and only if one has access to the parameters of interest. Performance measurement
may further be classified into on-chip hardware monitoring, off-chip hardware monitoring, software
monitoring, and microcoded instrumentation. Performance modeling is typically used when actual
systems are not available for measurement or, if the actual systems do not have test points to measure
every detail of interest. Performance modeling may further be classified into simulation modeling and
analytical modeling. Simulation models may further be classified into numerous categories depending
© 2002 by CRC Press LLC
TABLE 8.2 A Classification of Performance Evaluation Techniques
Performance measurement Microprocessor on-chip performance
monitoring counters
Off-chip hardware monitoring
Software monitoring
Micro-coded instrumentation
Performance modeling Simulation
Trace driven simulation
Execution driven simulation
Complete system simulation
Event driven simulation
Software profiling
Analytical modeling
Probabilistic models
Queuing models
Markov models
Petri net models