Performance and Evaluation of Lisp Systems - Dreamsongs

2
to be a plethora of facts, only those aspects of a Lisp implementation that are
the most important for evaluation will be discussed. Throughout, the impact of
these issues and trade-offs on benchmarks and benchmarking methodologies will
be explored.
The Lisp implementations that will be used for most examples are:
INTERLISP-10 [Teitelman 1978], INTERLISP-D [Burton 1981], INTERLISP-Vax
[Masinter 1981a] [Bates 1982], Vax NIL [White 1979], S-1 Lisp [Brooks 1982b],
FRANZ Lisp [Foderaro 1982], and PDP-10 MacLisp [Moon 1974],
1.1 Levels of Lisp System Architecture
The performance of a Lisp system can be viewed from the lowest level of the
hardware implementation to the highest level of user program functionality. Un-
derstanding and predicting Lisp system performance depends upon understanding
the mechanisms at each of these levels. The following levels are important for char-
acterizing Lisp systems: basic hardware, Lisp ‘instructions,’ simple Lisp functions,
and major Lisp facilities.
There is a range of methodologies for determining the speed of an implemen-
tation. The most basic methodology is to examine the machine instructions that
are used to implement constructs in the language, to look up in the hardware
manual the timings for these instructions, and then to add up the times needed.
Another methodology is to propose a sequence of relatively small benchmarks
and to time each one under the conditions that are important to the investigator
(under typical load average, with expected working-set sizes, etc). Finally, real
(naturally occurring) code can be used for the benchmarks.
Unfortunately, each of these representative methodologies has problems.
The simple instruction-counting methodology does not adequately take into ac-
count the effects of cache memories, system services (such as disk service), and
other interactions within the machine and operating system. The middle, small-
benchmark methodology is susceptible to ‘edge’ effects: that is, the small size of
the benchmark may cause it to straddle a boundary of some sort and this leads
to unrepresentative results. For instance, a small benchmark may be partly on
one page and partly on another, which may cause many page faults. Finally, the
real-code methodology, while accurately measuring a particular implementation, 1
1 Namely, the implementation on which the program was developed.