Model-Based Testing - Testing Experience

different protocol simulators, system configuration interfaces
and automatic log analyzers. This convinced us of the possible
benefits and the decision was made to continue with the next
goal to gather enough information for a business case calculation.
To calculate return on investment, we started to model an existing
and already tested functionality in the selected domain.
The aim was to calculate the earnings both for an initial model
creation and for a model update when some new functionality is
added to the existing model. We were also interested in the fault
finding capability of this new technology. As a result of this exercise
we proved that model-based testing is less time consuming.
We achieved 15% (650 hours to 550 hours) improvement during
initial modelling (creating reusable assets) and 38% (278 hours
to 172 hours) improvement during incremental add-on built on
top of the existing assets. We also found three additional minor
faults compared to previous manual tests of the same functionality.
Hours compared cover all testing-related activities like documentation,
execution and test result reporting.
Based on these results we were able to prove a positive business
case with break-even in 1.5 years, and we received an approval
to deploy the technology on a smaller scale to enable additional
studies.
Deployment scope and goals
Based on our experiences in deploying new technology, we decided
to deploy MBT in a smaller project. Three different areas
were selected for the deployment: Functional Testing, Call State
Machine Component Testing and Database Component Testing.
The goals of the deployment were to confirm the pilot results in
a real project environment, to confirm the business case and ROI
calculation, and to show that the technology can be adapted to
other development projects. Furthermore, we also wanted to create
a new mode of operation and collect engineer feedback.
Results
During Call State Machine Component Testing, manual and model-based
testing activities were organized to go in parallel, in order
to be able to get comparable results with both technologies.
The results showed that in the first phase component testing
with MBT was 25% faster compared to the corresponding manual
test, while in the second phase of the implementation an additional
5%-point gain was measured in terms of work hours. In
code coverage analysis MBT again outperformed manual testing,
resulting in 87% coverage compared to the 80.7% measured for
the manual test design in the first phase. In the second phase of
implementation, this was further increased to an excellent coverage
level of 92%. These higher coverage levels resulted in an improvement
also in fault finding: on top of the 12 faults found with
manual design, MBT discovered 9 additional software defects. According
to our defect escape analysis, three of these would have
been probably caught only during a very late testing phase.
During Database Component Testing, the same measures resulted
in similarly attractive numbers; Component testing was
performed in 22% less time than previously planned with outstanding
code coverage of 93% for the database and 97% for the
database interface controller.
www.testingexperience.com
In the Functional Testing area, our focus was on how to handle
the complexity of the system under test. This was successfully resolved
by our modellers. To have comparable results, part of the
testing team performed manual test planning and design, while
the modellers worked in parallel on their system models. The final
result was in accordance with the gain in other deployment areas:
20% gain in the speed of test planning and design phase. Analysis
showed that 80% of the faults have been found by both methods,
but the remaining 20% of the faults have been discovered only
by one of the test design methods (10% only with MBT, 10% only
with manual design). This also proves that the scope of manual
and model-based test design is different in nature; so combining
these two approaches is highly recommended for optimal results.
Effect on every-day work
We had to focus also on integrating modelling activities into our
existing software lifecycle model, and processes.
Figure 2. Feedback loops defined for all stages of modelling
To achieve this, continuous communication sessions were established
in all areas and two review levels ensured feedback loops
inside a sprint: a model review with modellers, specification
experts, lead designers; and a test case review with all the engineers
in the affected domain. Also new roles were introduced in
the scrum teams: modeller and back-end scripter. An automatic
documentation interface was taken into use towards our test
management system to gain further efficiency.
In addition to the positive results and the changes in our daily
work, the reaction of engineers was very important, so we executed
a satisfaction survey to learn about the opinions of the team
members. Their comments were positive: “A nicely built model
can be understandable for anyone. There is no need to check
several other documents, only the model.” “Modelling is fast and
comfortable after having some experience.” “It is also motivating
that serious code defects were found with MBT (even after executing
manually designed test cases).”
Lessons learned
In the nearly 3 years we’ve put in for introducing this new technology,
we have learnt that MBT can be successfully adopted on
complex projects and can introduce significant improvements
to testing. However, to achieve this, a step-by-step pilot and deployment
approach is needed with clear goals, Go/No-Go decision
criteria, and business sponsorship. Strict reporting practices
The Magazine for Professional Testers
35