Agile Performance Testing - Testing Experience

URLs or form data change dynamically, the logic that builds them
needs to be reproduced in the test scripts.
Besides this, it can be necessary to model periodic AJAX calls, for
example to automatically refresh the content of a ticker that
shows the latest news or stock quotes. For a realistic load simulation,
this also needs to be simulated by the load test scripts.
Challenge 4: Client-Side Web Browser Behavior
For correct and realistic load simulations, the load test tool needs
to implement further web browser features. Here are a few examples:
• caching
• CSS handling
• HTTP redirect handling
• parallel and configurable image loading
• cookie handling
Many of these features are supported by load test tools, even if
the tools act on the HTTP level, but not necessarily all of them are
supported adequately. If, for example, the simulated think time
between requests of a certain test case is varied, a low-level test
script might always load the cacheable content in the same way –
either it was recorded with an empty cache and the requests are
fired, or the requests were not recorded and will never be issued.
DOM-Level Scripting
What is the difference between HTTP-level scripting tools and
DOM-level scripting tools? The basic distinction between the levels
is the degree to which the client application is simulated during
the load test. This also affects the following characteristics:
• Representation of data: DOM-level tools use a DOM tree
instead of simple data structures.
• Scripting API: The scripting API of DOM-level tools works on
DOM elements instead of strings.
• Amount and quality of recorded or hard-coded data: There is
much less URL and form data stored with the scripts. Most of
this data is handled dynamically.
DOM-level tools add another layer of functionality on top. Besides
the handling of HTTP, these tools also parse the contained HTML
and CSS responses to build a DOM tree from this information,
similar to a real web browser. The higher-level API enables the
script creator to access elements in the DOM tree using XPath expressions,
or to perform actions or validations on certain DOM
elements. Some tools even incorporate a JavaScript engine that is
able to execute JavaScript code during the load test.
Advantages
DOM-level scripting has a number of advantages:
• Load test scripts become much more stable against changes
in the web application. Instead of storing hard-coded URLs
or data, they operate dynamically on DOM elements like
“the first URL below the element xyz” or “hit the button with
id=xyz”. This is especially important as long as application
development is still ongoing. As a consequence, you can start
scripting earlier.
• Scripting is easier and faster, in particular if advanced script
functionality is desired.
• Validation of result pages is also easier on the DOM level
compared to low-level mechanisms like regular expressions.
For example, checking a certain HTML structure or the content
of an element, like “the third element in the list below
the second H2” can be easily achieved by using an appropriate
XPath to address the desired element.
• Application changes like changed form parameter names
normally do not break the scripts, if the form parameters are
not touched by the script. But, if such a change does break
the script because the script uses the parameter explicitly,
the error is immediately visible since accessing the DOM tree
element will fail. The same is true for almost all kinds of application
changes described above. Results are more reliable,
because there are fewer hidden differences between the
scripts and the real application.
• CSS is applied. Assume there is a CSS change such that a formerly
visible UI element that can submit a URL becomes invisible
now. A low-level script would not notice this change. It
would still fire the old request and might also get a valid response
from the server, in which case the mismatch between
the script and the changed application could easily remain
unnoticed. In contrast, a DOM-level script that tries to use
this UI element would run into an error that is immediately
visible to the tester.
• If the tool supports it, JavaScript can be executed. This avoids
complicated and error-prone re-modeling of JavaScript
behavior in the test scripts. JavaScript support has become
more and more important in recent years with the evolution
of Web 2.0/AJAX applications.
Disadvantages
There is one disadvantage of DOM-level scripting. The additional
functionality needs more CPU and main memory, for instance to
create and handle the DOM tree. Resource usage increases even
more if JavaScript support is activated.
Detailed numbers vary considerably with the specific application
and structure of the load test scripts. Therefore, the following
numbers should be treated with caution. Table 1 shows a rough
comparison, derived from different load testing projects for largescale
web applications. The simulated client think times between
a received response and the next request were relatively short.
Otherwise, significantly more users might have been simulated
per CPU.
Scripting Level Virtual Users per CPU
HTTP Level 100..200
DOM Level 10..30
DOM Level + JavaScript execution 2..10
If you evaluate these numbers, please keep in mind that machines
are becoming ever more powerful and that there are many flexible
and easy-to-use on-demand cloud computing services today,
so that resource usage should not prevent DOM-level scripting.
Conclusion
Avoid hard-coded or recorded URLs, parameter names and parameter
values as far as possible. Handle everything dynamically.
This is what we have learned. One good solution to achieve this
is to do your scripting on the DOM level, not on the HTTP level.
If working on the DOM level and/or JavaScript execution are not
possible for some reason, you always have to make compromises
and accept a number of disadvantages.
We have created and executed web-application load tests for
many years now, in a considerable number of different projects.
Since 2005, we have mainly used our own tool, Xceptance LoadTest
(XLT), which is capable of working on different levels and supports
fine-grained control over options like JavaScript execution.
In our experience, the advantages of working on the DOM level, in
many cases even with JavaScript execution enabled, generally by
far outweigh the disadvantages. Working on the DOM level makes
scripting much easier and faster, the scripts handle many of the
dynamically changing data automatically, and the scripts become
much more stable against typical changes in the web application.
22 The Magazine for Professional Testers www.testingexperience.com