Agile Performance Testing - Testing Experience

To illustrate this, let’s take a look at the table below. A company
has 6 devices/resolutions and each sanity test is estimated to
take 2 hours. Now, the company currently works with 20 different
versions. If you multiply the number of devices by the number of
versions, this gives a total of 120 deliverables which in turn translates
to 240 hours of testing!
However, by externalizing the versions’ parameters, so that these
can be thoroughly tested once, we reduced the initial testing duration
to ~72 hours.
Now, after verifying that this new process works correctly and
with no failures, future tests will be estimated at only ~24 hours
– which amounts to 10% of the regular testing time. This significantly
lower percentage is mainly due to the ease of testing the
externalized parameters and the need to sanity-test only a few of
the deliverables for validity in the next versions, rather than testing
all deliverables over and over again.
Deliverables Regular Test Externalized
Parameters-
Initial Test
Number of
Devices/Resolutions
Number of
Versions
Total Number
of Deliverables
Testing Time
(hours)
Externalized
Parameters-
Recurring
Tests
6 6 20
20 20 6
120 120 120
240 72 24
Testing big chunks of new content. To test new content, it is always
faster to use desktop PC simulators and regular web browser
to test content included in mobile apps and mobile web apps
vis-à-vis databases. The content doesn’t change and while we do
not care about UI and on device functionality, we can definitely
trust the PC tools for pure data testing, e.g. calculations done by a
mobile/mobile web calculator.
The ‘Obvious’ – Can’t do without methodology
Every combination of application and mobile platform has its
own special common problem. A mobile testing team should
map the known and recurring issues and create threshold test
cases for them. These should be run on every version delivered by
the developers. Embedding these test cases will result in virtually
no major issues with devices’ compatibility and flawlessly passing
criteria tests (AppStore testing, Symbian Signed, etc). It will
also put more focus on testing new functionality and delivering
high quality instead of wallowing in the same bugs.
Things you should look out for:
• Known issues with your application per platform
• Known issues on the tested platform (application independent)
• Critical issues in regards to platform style guides and device
interoperability
Do not re-invent the wheel. Use manufacturers/OS test criteria as
a base for creating a threshold acceptance test that is briefly run
on the application before any major testing cycle. Combine this
with the previous point (your application-specific issues) and you
have a winning approach..
Test texts and graphics only in a second round after they were approved
by stakeholders. Mobile applications often change graphic
layout and strings (because of localization or marketing needs).
The need to test on various devices and resolutions significantly
slows down the testing phase and subsequently the quality of
mobile testing. In order to reduce the number of iterations, and
thus saving time and improving quality, the relevant ‘owners’
should approve each change before testing. This also helps bringing
external functions into the testing process and makes them
partners in the testing teams’ quest for quality, deliverability and
time to market.
The previous point leads us to the section required when approaching
an application for testing graphics, strings and localizations.
Test planning and execution should be separated from
logic and functionality testing.
Only after logic and functionality
have passed the initial set of tests,
should the focus move towards
the visual aspects of the application.
Otherwise, we run the risk
of finding basic functional flaws
at the last moment. As ‘visual’
testing requires a longer testing
time and usually results in higher
modification rates, it should be
tested following these two preconditions:
1. Specifically approved by
stakeholders (see previous point)
2. Logic and functionality have
reached an acceptable level of
maturity.
Our experience shows that failing
to adhere to these rules results in
quality and timelines being sacrificed
on the altar of testing mere
graphic aspects.
Another good practice for localization testing, which proved to
decrease our testing times, is the ‘Dual-Language’/’Dual-Resolution’
testing. Our experience shows that it allowed for early identification
of root problems which is critical when testing multiple
languages towards a release.
1. Pick the two most ‘problematic’ languages for your application.
These are commonly the one with longest strings and
the one more technically problematic (e.g. double-byte, diacritic
punctuation or middle eastern).
2. Run the test cases on these languages on two identical de-
14 The Magazine for Professional Testers www.testingexperience.com