Agile Performance Testing - Testing Experience

to create high, spiking, random loads. We would sustain these
loads for up to 48 hours. If none of the systems crashed, none
of the network bridges lost connectivity to one or both networks,
no data were lost, and if no applications terminated abnormally,
then that was a passing test result.
Figure 5: Maximum network configuration for the operating system
Lesson 3: Test the Tests - and the Models
Okay, so now it’s clear that realism of the environment and the
load are important. I mentioned earlier that performance, load,
and reliability testing are complex endeavors, and that means
that it’s easy to screw them up. These two areas of realism are
especially easy to screw up.
So, if realism of environment and load is important, how can we
be sure we got it right, before we go live or ship? The key here is
to build a model or simulation of your system’s performance and
use the model or simulation both to evaluate the performance of
the system during the design phase and to evaluate the tests during
test execution. In turn, since the model and simulation are
equally subject to error, the tests also can evaluate the models.
Furthermore, a model or simulation which is validated through
testing is useful for predicting future performance and for doing
various kinds of performance “what-iffing” without having to resort
to expensive performance tests as frequently.
There are a couple of key tips to keep in mind regarding modeling
and simulating. First, spreadsheets are good for initial performance,
load, and reliability modeling and design. Second, once
these spreadsheets are in place, you can use them as design documents
both for the actual system and for a dynamic simulation.
The dynamic simulations allow for more “what-if” scenarios and
are useful during design, implementation, and testing.
Here’s a case study of using models and simulations properly during
the system test execution period. On an Internet appliance
project, we needed to test server-side performance and reliability
at projected loads of 25,000, 40,000, and 50,000 devices in the
field. In terms of realism, we used a production environment and
a realistic load generator, which we built using the development
team’s unit testing harnesses. The test environment, including
the load generators and functional and beta test devices, are
shown in Figure 6. Allowing some amount of functional and beta
testing of the devices during certain carefully selected periods of
the performance, load, and reliability tests gave the test team and
the beta users a realistic sense of user experience under load.
Figure 6: Internet appliance test environment
So far, this is all pretty standard performance, load, and reliability
testing procedure. However, here’s where it gets interesting.
Because a dynamic simulation had been built for design and
implementation work, we were able to compare our test results
against the simulation results, at both coarse-grained and finegrained
levels of detail.
Let’s look first at the course-grained level of detail, using an extract
from our performance, load, and reliability test results presentation.
We had simulation data for the mail servers, which
were named IMAP1 and IMAP2 in Figure 6. The simulation predicted
55% server CPU idle at 25,000 devices. Table 1 shows the
CPU idle statistics under worst-case load profiles, with 25,000
devices. We can see that the test results support the simulation
results at 25,000 devices.
Server CPU Idle
IMAP1 59%
IMAP2 55%
Table 1: Server CPU idle figures during performance tests
Table 2 shows our more fine-grained analysis for the various
transactions the IMAP servers had to handle, their frequency per
connection between the device and the IMAP server, the average
time the simulation predicted for the transaction to complete,
and the average time that was observed during testing. The
times are given in milliseconds.
Transaction Frequency Simulation Testing
Connect 1 per connect 0.74 0.77
Banner 1 per connect 2.57 13.19
Authorize 1 per connect 19.68 19.08
Select 1 per connect 139.87 163.91
Fetch 1 per connect 125.44 5,885.04
Table 2: Server transactions and times, simulated and observed during
testing
70 The Magazine for Professional Testers www.testingexperience.com