U. Glaeser

is generally the most important measure of performance. Execution time is the product of the number
of instructions, cycles per instruction (CPI), and the clock period. Throughput of an application is a
more important metric, especially in server systems. In servers that serve the banking industry, airline
industry, or other similar businesses, what is important is the number of transactions that could be
completed in unit time. Such servers, typically called transaction processing systems use transactions per
minute (tpm) as a performance metric. Millions of instructions per second (MIPS) and million of floating
point operations per second (MFLOPS) were very popular measures of performance in the past. Both
of these are very simple and straightforward to understand and hence have been used often, however,
they do not contain all three components of program execution time and hence are incomplete measures
of performance. Several low level metrics are of interest to microprocessor designers, in order to help
them identify performance bottlenecks and tune their designs. Cache hit ratios, branch misprediction
ratios, number of off-chip memory accesses, etc. are examples of such measures.
Another major problem is the issue of reporting performance with a single number. A single number
is easy to understand and easy to be used by the trade press. Use of several benchmarks also makes it
necessary to find some kind of a mean. Arithmetic mean, geometric mean, and harmonic mean are three
ways of finding the central tendency of a group of numbers; however, it should be noted that each of
these means should be used in appropriate conditions depending on the nature of the numbers which
need to be averaged. Simple arithmetic mean can be used to find average execution time from a set of
execution times. Geometric mean can be used to find the central tendency of metrics that are in the form
of ratios (e.g., speedup) and harmonic mean can be used to find the central tendency of measures that
are in the form of a rate (e.g., throughput). Cragon [2] and Smith [3] discuss the use of the appropriate
mean for a given set of data. Cragon [2] and Patterson and Hennessy [4] illustrate several mistakes one
could possibly make while finding a single performance number.
The rest of this chapter section is organized as follows. “Performance Measurement” describes performance
measurement techniques including hardware on-chip performance monitoring counters on microprocessors.
“Performance Modeling” describes simulation and analytical modeling of microprocessors and
computer systems. “Workloads and Benchmarks” presents several state-of-the-art benchmark suites for a
variety of workloads. Due to space limitations, we describe some typical examples of tools and techniques
and provide the reader with pointers for more information.
Performance Measurement
Performance measurement is used for understanding systems that are already built or prototyped.
Performance measurement can serve two major purposes: tune this system or systems to be built and
tune the application if source code and algorithms can still be changed. Essentially, the process involves
(i) understanding the bottlenecks in the system that has been built, (ii) understanding the applications
that are running on the system and the match between the features of the system and the characteristics
of the workload, and (iii) innovating design features that will exploit the workload features. Performance
measurement can be done via the following means:
• Microprocessor on-chip performance monitoring counters
• Off-chip hardware monitoring
• Software monitoring
• Microcoded instrumentation
On-Chip Performance Monitoring Counters
All state-of-the-art high performance microprocessors including Intel’s Pentium III and Pentium IV, IBM’s
POWER 3 and POWER 4 processors, AMD’s Athlon, Compaq’s Alpha, and Sun’s UltraSPARC processors
incorporate on-chip performance monitoring counters, which can be used to understand performance of
these microprocessors, while they run complex, real-world workloads. This ability has overcome a
serious limitation of simulators, that they often could not execute complex workloads. Now, complex
run-time systems involving multiple software applications can be evaluated and monitored very closely.
© 2002 by CRC Press LLC