Framework Driven Test Automation Position Paper - Collabera

Position Paper
Framework Driven Test
Automation
Realizing Cost Benefits through
Framework Driven Test Automation

The old age testing maxim was “Automation Reaps Cost Benefits”. Thrilled with the prospects of
achieving cost benefits and better quality, organizations adopted test automation methodologies
and tools only to find themselves grappling with increased costs, higher timelines and lower tool
utilization. Studies indicate that 63% of test automation projects fail due to multitude of factors –
• Wrong choice and underutilization of testing tools
• Lack of reusability of test cases and components,
• Need for script development effort each time automation tool is used,
and
• Need for technical experts along with subject matter experts.
The new age testing maxim is “Intelligent Automation Reaps Cost Benefits”. This gives rise to
the question - How to do Intelligent Automation? The answer – Framework driven Test
Automation.
Framework driven test automation enables
80
organizations to better their test automation
70
approach and realize cost benefits. Our
experience in executing testing projects for
fortune 500 clients across the globe have
Percentage
60
50
40
30
shown that framework driven test
20
automation can enable organizations realize
close to 70% savings in effort across
10
0
Savings in Effort
Savings in Time
design, development, scripting and
execution stages of testing.
Design & Development
Test Suite Maintenance
Test Case Scripting
Overview
The need for reducing manual effort and timelines in carrying out testing by deploying tool sets for
applying test cases repetitively led to the adoption of test automation. However, the dynamic
business landscape has led to a scenario wherein the need for reduced costs and better quality is
all the more aggravating – leading to the creation of gaps in effective execution of testing.
Framework based test automation aims to bridge these gaps through a set of assumptions,
concepts, and practices that provide support for automated testing. Creating a base platform or
support for automated testing, these frameworks enable test cases to be abstracted to logical
levels, stored and executed in a consistent manner. Abstraction of test cases helps subject matter
experts with minimal technical knowledge to use automation tools, reducing dependency on
technical experts.
Value. Accelerated.

Evolution
Framework based test automation has evolved from capture replay approach (one of the earlier
modes) to script based, data driven and keyword driven approaches.
Manual
Testing
Capture
& Replay
Script
Based
Data
Driven
Keyword
Driven
Hybrid
Approach
Capture & Replay: An approach wherein the various phases of a test case on execution are
captured and maintained in a repository. The test case can then be replayed with changes in test
data to obtain results. This approach suffers from the disadvantage of being time consuming and
expensive when running complex test cases due to the inability to record such cases and
maintain them.
Script Based: A technique in which a technical expert generates scripts specific to the
automation tool used along with the test data. This approach suffers from the disability of having
to hard code test data into test scripts each time the test script gets executed.
Data Driven: A technique where a single test script is used repeatedly with varying input and
response data that comes from a predefined data set (data files such as data pools, ODBC
sources, cvs files, Excel files, DAO objects, ADO objects etc). This approach suffers from the
disability of taking more time to build, track and maintain test scripts.
Keyword Driven: This framework requires the development of keywords, each keyword
independent of the test automation tool used to execute them and the test script code that
"drives" the application-under-test and the data. In this approach, discrete functional business
events that make up any application are described using a short text description (Keyword) and
associated parameter value pairs (arguments).
Hybrid Approach: A framework that can be both keyword driven while also providing enhanced
functionality for data driven scripts. This allows data driven scripts to take advantage of the
powerful libraries and utilities that usually accompany a keyword driven architecture. On the other
hand it allows the usage of scripts to perform some tasks that might be too difficult to reimplement
in a pure keyword driven approach, or where the keyword driven capabilities are not
yet in place.
Value. Accelerated.

Collabera Approach
Collabera enhances the test automation process through its proprietary framework – automate,
a hybrid approach, which enables organizations to attain the multiple benefits of keyword and
data driven approaches. This approach is a combination of all approaches described above,
pulling from their strengths and trying to mitigate their weaknesses. On one hand this approach
enables data driven scripts to be more compact and less prone to failure than they otherwise
would have been; and on the other, they facilitate the gradual and manageable conversion of
existing scripts to keyword driven equivalents when and where that appears desirable.
This framework is “easy to use” and institutes of division of labor – enabling teams to involve
members from diverse backgrounds and take advantage of their respective strengths. It
encapsulates complexity of test automation scripting language from test specialists and business
/ domain subject matter experts by allowing them to script test cases in simple English Language
thus obviating dependency on technical experts.
The table below provides a comparison of automate with other test automation approaches.
Feature
Capture
Playback
Keyword
driven
Automate
Complexity of Initial Implementation
Simplicity of use
Division of Labour (Between SME/Domain/Technical
Experts)
Reuse of Common Business Functionalities
Complexity of Maintenance
Ease in Incorporating Changes
Support for Iterative and Agile development model
Provision of Exception / Error Handling
Capability of generating English Language Documentation
of Automated Test Cases
Key benefits in using automate are
• Reusability of automation scripts enabling client to realize better ROI
over longer periods
Value. Accelerated.

• Reusability of common business functionalities to achieve Reduction in
Cost-to-Market and Time-to-Market along with Quality Improvements
• Capability of generating English Language Documentation of
Automated Test Cases
Cost Benefit Using automate
Automate based test automation takes a good deal of effort in the initial stages of set up and
may indicate that the costs during initial stages are much more than that involved in traditional
testing methods. However based on our experience in executing test automation based on
automate for one of our key clients, the benefits were realized after 4 test cycles. The same
has been represented in the graph below:
25
20
Total Cost
15
10
5
Manual Test ing
Framework based Aut omat ed Test ing
0
1 2 3 4 5 6 7 8 9 10 11 12
Test Cycle
* Refer Appendix 1 for detailed break down of cost structure.
Appendix 1: Cost Benefits Using automate
These figures have been computed based on our experience in carrying out test automation
projects for clients across the globe. Actual results vary on project to project basis and depend on
various factors like complexity of application, number of test cases, feasibility of application etc.
Item Manual Automate
Time Cost Time Cost
Unit Cost $70 per day $70 per day
Total no of test cases 632 632
Time to design/automate
test case
4 per hour 30 per hour
Value. Accelerated.

Time to design/automate
all test cases
Time to execute and
analyze each test case
and results
Time to execute and
analyze all test cases and
results
Total run through the
code release for one
cycle
158 hrs $1610 22 hrs
4 per hour or
32 per day
158 hrs $ 1610
30 per hour or
240 per day
(depending upon
the speed of the
processor and
the speed of the
application)
3 days of
execution+7 days
of manual +2
days script
rework=12 days
$3390
$ 840
$ 1610 $4230
Manual Automate Tool Cost Total
Time to Design and Execute 1610 3390 1000 4390
Subsequent cycle execution times 1610 840
Cycle Manual (in 1000's) Automate (in 1000's)
1 1.61 4.39
2 3.22 5.23
3 4.83 6.07
4 6.44 6.91
5 8.05 7.75
6 9.66 8.59
7 11.27 9.43
8 12.88 10.27
9 14.49 11.11
10 16.1 11.95
11 17.71 12.79
12 19.32 13.63
Total 125.58 108.12
Assumptions and computational values:
• Daily builds and test runs (5 times a week)
• Manual tests take 5 days to design, 2 hours to run
• Only half the manual tests would be run on any given day (1 hour) with
the other half run the following day
• Automated tests take 15 days to design and implement, automatically
run (zero cost)
Value. Accelerated.

• Automation is done with batch scripts and integrated into the build
process, requiring $1,000 in added hardware, with a useful life of 3
years
• Automated tests need to be maintained every 25 runs, one day of work
required
• People cost $100,000 per year = $400 per day = $50 per hour
About Collabera
Collabera is one of the fastest growing end-to-end information technology services and solutions
firms worldwide. We work with leading Global 2000 firms from the Financial Services,
Manufacturing & Retail, Technology, Communications & Media domains. Collabera delivers
highly responsive and innovative solutions that bridge our client's Execution Gaps through our
proprietary methodologies - Momentum workshops, ABS (Asset Based Services) and global
delivery model - helping them experience accelerated value.
With revenues of over USD 300 million, Collabera employs over 4000 professionals across 22
locations and seven world-class delivery centers in the US, Europe, India and China regions. Our
Global Delivery model leverages a best-in-class Quality system (including SEI CMMI Level 5 and
ISO 27001 security certifications) to deliver a full portfolio of services that include Application
Development and Management, Independent Testing, Outsourced Product Development,
Enterprise Software Solutions, Business Intelligence & Datawarehousing.
Value. Accelerated.

www.collabera.com
solutionsinfo@collabera.com
Value. Accelerated.