Agile Performance Testing - Testing Experience

Let me start with a simple, yet realistic example. I have a webbased
application which allows a student to register for a course.
Identifying a load test for this application is straightforward. Say
I have 400 students on the rolls and a student, on the average registers
up to 5 courses each semester. With a little investigation,
I may identify a maximum load of 200 students simultaneously
trying to register for a single course. I may also find that an average
registration takes 10 minutes and the general trend is that
students register for their courses in one shot. Hence peak loads
may stretch up to 50 minutes. To briefly outline how I would test
this scenario, my load test would invoke 200 students (randomly
chosen from a data sheet listing 400 students and the courses
they need to register for) and have them simultaneously register
for their courses, one course after the other. I would run this
test for an hour (expecting registrations to be complete by then).
Amongst many parameters that I will be monitoring, I am particularly
interested in transaction response time. This is a key parameter
that translates to how much time end users wait for responses
from the application during specified transactions. I will
stick to this parameter for the purpose of my discussion, though
it will apply to any parameter of interest. At the end of my test I
generate a report that will display response time plotted for the
duration of the test (see figure 1). I will be happy to see that the
transaction response time has not significantly crossed 2000 milliseconds
at any point. The result indicates that during peak activity
and maximum load my application will not take more than 2
seconds to respond. Assuming my acceptable response time limit
is more, there is reason for cheering.
Is Your Application’s Performance
Predictable?
by Suri Chitti
© Martinan - Fotolia.com
Yet, there is more work to do. I have established acceptable performance
under peak conditions. However, I have not established
predictability. I now need to try and understand the range of
loads my application will be subject to. Assuming I investigate
and find that loads range between 25 to 200 simultaneous registrations,
I wish to see response times beginning at 25 registrations
and ending at 225 with stepped increases of 25 at every six
minute intervals in my run. In my result, I wish to see a responsive
trend in transaction response time with load, similar to the green
graph in figure 2. The joyful aspect with a responsive trend is that
there is a linear correlation between load and response time in
my operational load range. The correlation is a strong indicator
of predictability. I will be unhappy if I cannot see the correlation,
or even worse, if I see an erratic (red graph in figure 2) trend in
response time with load.
If my application architect uses a web engine that can take loads
very much above the range I have identified, I may notice a flat
trend (purple graph in figure2) indicating there is no significant
increase in response time with load in my range. However, my
architect is, for performance purposes, using an elephant gun for
bird hunting, and that will have its own problems, least of which
is the money wasted on needless performance characteristics.
An application that shows a good correlation between transaction
response time and load in the operational load range during
testing is likely to also be predictable during operation. On
the other hand, applications that erratically respond to load, or
have patches of unpredictabilty in their operational load range,
are also likely to have unpredictable problems during operation.
78 The Magazine for Professional Testers www.testingexperience.com