i i i i - erni-consultants.com

erni.consultants.com

i i i i - erni-consultants.com

TesT managemenT 1

eRnI essenTIaLs

TEST MANAGEMENT


enables & delivers

maRCeL sTOOP

marcel.stoop@erni.ch

senior Consultant and

Certified Testmanager

bei eRnI schweiz ag

ForEword

Software is one of the most complex

products of human activity. Its complexity

will further increase as a result of

system networking. Nevertheless, testing

plays a secondary role in many development

departments, even though there

are many good reasons to test software

intensively and systematically.

A key element of testing is verifying the

actual behaviour of the software relative

to the defined requirements. In addition,

testing reduces the risk of high follow-up

costs, because the earlier an error is found,

the less it costs to correct it. In the first

instance, testing is a means to eliminate

errors. If its potential is properly utilised,

testing is a power control instrument that

provides comprehensible quality control

during the entire development process.

My experience shows that nowadays highquality

software can only be produced

using an interleaved development and

testing process.

eRnI - Innovation in Process and Technology

TesT managemenT 3

1. ForEword 2

2. INTroducTIoN 5

3. SoFTwArE TESTS 7

4. ThE 10-PoINT chEcklIST 11

5. ThE TEST PlAN 15

6. ThE V ModEl 19

7. METhodIcAl TEST cASE GENErATIoN 23

8. AVoId coMbINATIoNAl ExPloSIoNS 27

9. TEST cASE cATAloGuE 31

10. TEST AuToMATIoN 35

11. ThE TEST ProcESS 39

12. ProFESSIoNAl TESTErS 43

12.1 Test manager 43

12.2 Test architect 45

12.3 Test designer 46

12.4 Test automator 47

12.5 Tester 48

13. docuMENT TEMPlATES 51

13.1 Test plan 51

13.2 Test case specification 52

13.3 Test report 53

14. TEST ToolS 55

15. FurThEr INForMATIoN 57

15.1 Reference documents 57

15.2 standards 59

15.3 Web links 59


2. INTroducTIoN

TesT managemenT 5

Testing software is extremely costly and difficult. Why?

• Anyone who tries to test software systematically for the first

time soon gives up in the face of the astronomical number

of test cases. They result from the «combinational explosion»

of input data and internal software states.

• In many cases, there is no time left for testing because

development took longer than planned and the delivery

date is looming.

• Testing is regarded as unnecessary and unattractive work.

• Software testing is a young discipline of software engineering,

and it was ignored by the academic world for a long

time.

• Management in many places sees testing exclusively as

a cost factor, even though it is an element of the production

chain and control process that creates added value and

increases quality.

«Inadequately trained testers, a compulsive desire to test eve-

rything completely, and the tight constraints of project environments

lead to inadequate testing of software.»


TesT managemenT 7

3. SoFTwArE TESTS: rulES,

rEGulATIoNS ANd PrINcIPlES

It is helpful to recall the following empirical observations before

you start testing:

• Testing costs comprise 40% to 60% of total project expenditures

(Harry M. Sneed and Stefan Jungmayr, Produkt- und Prozessmetriken

für Softwaretest, Informatik-Spektrum, Vol. 29, Issue

1, Feb. 2006, Springer Verlag, pp 23–38).

The higher the required level of quality, the higher the cost

of testing.

• Prior to systematic testing, every software item contains an

average of 10 to 30 errors for every 1000 lines of code

(based on a survey of software developers).

• The required level of software quality is between 0.5 and

1.0 error per 1000 lines of code, depending on the application

area.

• Tests must reveal 9.0 to 29.0 errors for every 1000 lines of

code to be tested.

• The cost of finding additional errors in well-tested software

increases rapidly.

• The error density in software modules varies widely.

On average, 30% of the software modules of a system

contain 70% of the errors.

• It is less expensive to correct errors at an early stage of

development than at a later stage.


Non-conformance costs are a decisive factor in R&D costs. The

cost of an error increases exponentially with the time between

when it arises and when it is discovered or corrected.

The following empirical factors clarify this relationship:

Req. specification

analysis

& design

1 1 1 6 12 20

= relative error cost

Implementation

Integra-

tion test

system

test

Field

operation

• Effective testing does not start at the end, but instead at the

beginning when the requirements are generated and

reviewed. The costs of correcting errors are the lowest at

the early stage of the production process, and the benefits

are the highest. All of the subsequent production stages

benefit from high-quality specifications.

• There is no «best» test method. The most effective approach

is a combination of reviews, developer tests and tests performed

by an independent testing group.

TesT managemenT 9

«a small, specialised team that fulfils the roles of test manager,

test designer and tester in a professional manner is more efficient

and effective than large, ad-hoc test teams.»


4. ThE 10-PoINT chEcklIST

TesT managemenT 11

The following checklist is an aid for test managers who take on

a project lacking an explicit test process. It will help them quickly

achieve objectively measurable progress.

1. Obtain an overview of the system to be tested. Generate

a large, well-organised system block diagram.

2. Consult the major stakeholders to determine the business

objective of the tests.

3. Where are the requirements and specifications? A highlevel,

sequentially numbered list of «items to be tested» is

sufficient (see Section 9, «Test Case Catalogue: How to prioritise

tests based on risks»).

4. Prioritise the specification items, business processes, data

imports and system components according to a comprehensible

scheme (see Section 9, «Test Case Catalogue: How

to prioritise tests based on risks»).

5. Define the test stages (see Section 6, «The V Model: Execution

sequence of the test stages») and assign each stage to a responsible

test team.

6. Generate the test plan.

7. Ensure that each team has at least one member who is fully

dedicated to the tests. Coach this member to achieve success.

For example, show him how test cases and test data can be

derived purposefully and efficiently from the requirements.

8. Measure the test results. Show them as trend charts and communicate

them regularly to everyone. In that way, testing will

be regarded by everyone as an effective control process.


9. Arrange for suitable sharing of information. Daily «stand-

up meetings» with the test teams and weekly, moderated

test coordination meetings with the overall project manager

and all test team leaders are proven methods.

10. In your role as test manager, consult with the project manager

to agree on the plan and assist the change control

board in prioritising changes and incorporating them into

the plan.

TesT managemenT 13


TesT managemenT 15

5. ThE TEST PlAN: PlANNING TESTS

ITErATIVEly ANd INcrEMENTAlly

Requirements spec.

Development

Test design

Test execution

Iteration number

BesT PRaCTICe

Incr. 1

n-2

Incr. 2 Incr. 3

Incr. 1 Incr. 2 Incr. 3

Incr. 1

n-1

Incr. 2 Incr. 3

Incr. 1

Tests are performed in parallel with other activities, not after

them. The iterative incremental procedure is especially suitable

for this. In each iteration, tests are prepared and executed for the

additional increment:

• The tests are formulated after the requirements package has

been generated.

• The tests are executed immediately after the requirements

have been implemented in the software.

RuLes

In the iterative incremental procedure, the activities of requirements

definition, development, test design and text execution

take place in parallel.

n

Incr. 2

n+1

Incr. 3

n+2


In each iteration («n»)

• The requirements team generates the requirements for the

next increment («n + 1»).

• The development team generates the software for the current

increment («n»). The test design team generates the

corresponding test cases.

• The test team executes the test cases for the previous increment

(«n – 1») and reports the errors.

suCCess FaCTORs

• Analyse the lists of functional and non-functional requirements.

• Obtain an overview of the architecture and the dependencies

of the components.

• Prioritise the requirements and components from the test

perspective, based on the risks (see Section 9, «Test Case

Catalogue: How to prioritise tests based on risks»).

• Define the scope of the increments such that each one can

be tested independently.

BeneFITs

• Iterative testing creates confidence in the quality of the system

at an early stage.

• Continuous, objective assessment of project progress and

software quality.

• Detecting errors at an early stage leads to cost savings.

• Avoiding test backlogs. Based on experience, there is never

enough time at the end of a project to handle a test backlog.

TesT managemenT 17


TesT managemenT 19

6. ThE V ModEl: ThE ordEr

oF ExEcuTIoN oF ThE TEST STAGES

Development

system requirements

and specifications

architecture

(component & subsystem)

BesT PRaCTICe

unit or module unit test

Code & implementation

Tests

system test

Integration test

• Testing is necessary at every stage of the V model. Generating

test cases starts top-down as soon as the requirements or

specifications are available. Test generation starts with the

system tests. The integration tests come next, and finally the

unit tests.

• By contrast, the tests are executed bottom-up. Unit testing

starts as soon as a unit or component is complete. It is followed

by the integration tests and system tests.

RuLes

• Plan the test cases top-down. This means the system tests

must be drafted first.

• Execute the test cases bottom-up, starting with the unit tests.


suCCess FaCTORs

• Defined criteria for handover between the test stages. The

usual practice is to define a set of test cases that must be completed

without any errors before the test object is handed

over to the next level of testing. These test cases are called

«smoke tests», «basic sanity tests», «basic operation tests» or

«quick tests» in the trade jargon.

BeneFITs

• Generating test cases at the system level can be started at an

early stage of the project. This ensures that all of the tests

that have been executed are available and fully documented

at the time-critical final testing stage.

• Well-defined handover criteria increase the reliability of the

subsequent test stages. As a result, the course of the test object

is stable and the majority of the tests yield meaningful

results.

TesT managemenT 21


TesT managemenT 23

7. METhodIcAl TEST cASE GENErA-

TIoN: how To dErIVE TEST cASES

FroM rEquIrEMENTS

Functional requirement

(use case)

BesT PRaCTICe

1..n

3

Test case

Derive one or more test cases for each functional requirement

and each non-functional requirement (NFR).


non-functional requirement

(NFR)

1..n

Throughput,

Response time,

availability, Reliability,

etc.

• Derive 1 to n test cases (TCs) for each use case (UC):

1 UC -> n TCs

• Derive 1 to n test cases (TCs) for each non-functional requirement:

1 NFR -> n TCs

RuLes

Rules for deriving test cases for functional requirements:

• For each functional requirement, formulate at least one

test case that checks the normal case.


• Depending on the complexity, additional test cases are

necessary to check correct behaviour for alternative process

sequences.

Rules for deriving test cases for non-functional requirements:

• For each non-functional requirement, formulate at least

one test case that ensures compliance with the required

performance or quality specifications.

• Depending on the complexity, additional test cases are necessary

to check correct behaviour under various conditions (data

volume, access rate, environmental conditions, configuration).

suCCess FaCTORs

Better estimates can be obtained by selecting a number of sample

test cases for a representative group of requirements according to

the complexity. For example:

• 1 test case for simple requirements

• 3–5 test cases for moderately complex requirements

• 10–20 test cases for highly complex requirements

BeneFITs

• A uniform procedure for all types of requirements

• The rough scope of testing can be estimated easily from

the list of functional and non-functional requirements.

TesT managemenT 25


TesT managemenT 27

8. AVoId coMbINATIoNAl

ExPloSIoNS: how MANy TEST cASES

ArE NEcESSAry For EAch uSE cASE?

1. Insert card invalid

2. enter PIn code

3. enter amount

number of test cases

BesT PRaCTICe

incorrect PIn

Is it necessary to define a test case for every conceivable com-

bination of alternatives? No, because in practice many alternatives

are mutually exclusive. In the illustrated example of a bank

machine, «card expired» and «incorrect PIN code» are mutually

exclusive because an expired card will either be retained by the

machine or rejected. There are only 9 possible test cases, not 24.

RuLes

• First priority: test the normal sequence of events.

• Second priority: test the alternative sequences.


blocked

4 + 2 + 3 = 9 4 * 2 * 3 = 24

expired


• Third priority: test the combinations of alternative

sequences that are possible in practice.

suCCess FaCTORs

• First test the normal sequence of events of the use case and

verify that no «show stoppers» occur.

• After that, systematically test the alternative sequences and

their possible combinations.

BeneFITs

• Using this rule, the number of test cases is directly proportio-

nal to the number of alternative sequences of events instead

of exponentially proportional.

TesT managemenT 29


TesT managemenT 31

9. TEST cASE cATAloGuE:

how To PrIorITISE TESTS bASEd

oN rISkS

BesT PRaCTICe

Business

relevance

(B)

Complexity

(C)

From a quality perspective, it makes sense to match the test in-

tensity to the risk of faulty behaviour.

non-detectability

(N)

Risk

(R=B x C x N)

Item under test 1–5 1–5 1–5 1–125

Customer requirements

• Generate contract 5 4 2 40

• Post premium 5 2 4 40

• Open claim file 5 3 4 60

Interfaces

• FTP 2 3 2 12

• Web service 5 5 5 125

Potential error

sources

• Mixing up

customers

5 4 5 100


You should test intensively in cases where errors can cause major

loss or damage and have a high probability of occurrence.

A proven formula for estimating the risk is:

R = B x C x N

where:

R = Risk

B = Business relevance

C = Complexity, or in other words the probability of an error

N = Non-detectability of an error (without specific test precautions)

RuLes

The test case catalogue consists of a weighted list of «test

objects», and it specifies how many tests are necessary and the

required level of detail for specifying the tests.

1. Generate a structured list of:

• Customer requirements (functional requirements, performance

requirements and quality requirements)

• Laws, regulations and standards

• Technical specifications (functional and non-functional)

• Interfaces

• Dates

• Potential error sources

• Hazards from the operational, enterprise (financial)

and end-user perspectives

TesT managemenT 33

2. Have the individual risk factors evaluated and reviewed

by representatives of the various departments:

• Business relevance from the business perspective

• Complexity from the development perspective

• Non-detectability from the test management perspective

suCCess FaCTORs

• If no test management tool is available, an Excel list with a

separate worksheet for each group is sufficient.

• Have the list reviewed by the various departments.

• Have the items evaluated by the persons with the best professional

expertise.

• Manage the test cases and associated test case specifications

in Excel or by using a test management tool (see Section 14,

«Test Tools: Which tool is suitable for what»).

BeneFITs

• A test case catalogue makes the scope of the tests tangible

and manageable.

• The term «test coverage» is defined uniformly: «100% test coverage»

means that all items that can be tested are tested. The

cost per test object is transparent, plannable and controllable.

• All departments are taken into account. They state which

tests are important and meaningful for them.


10. TEST AuToMATIoN:

uSING rEGrESSIoN TESTING

To booST EFFIcIENcy

entering automated

test scripts

Hardware

BesT PRaCTICe

Test automation:

TesT managemenT 35

• Dramatically increases the number of tests conducted per

iteration or release. This results in increased quality with

reduced test costs. Testing efficiency increases.

• Is important when numerous changes (hot fixes, patches or re-

leases) are expected and they must be tested in rapid

sequence.

• Is used especially for regression tests. Such tests ensure

the continuity of system stability and system functionality

after changes have been made.

(Web)client

server

Database

«gateway» to

peripheral systems

subsystem


RuLes

The principal steps of test automation are:

• Generating test case specifications and running them

manually with a «capture and replay» tool that automatically

generates test scripts.

• Using programming methods to augment the test scripts

with input and configuration data for various test environments.

• Generating test metrics: number of tests performed, number

of pass & fail results per level of difficulty, and recommendations.

suCCess FaCTORs

• Test automation presupposes an understanding of programming

and must be handled the same as a software development

project.

• The key to success is continual, economical maintenance of

the automated test cases during the entire life cycle of the

software to be tested.

• Using test scripts in a version management tool and continual

adaptation to changing requirements and specifications

are ongoing activities.

BeneFITs

• Automated tests help enhance test efficiency and ensure uniform

quality control.

• Test automation requires a major investment in test tools

and automated test scripts. This expenditure is only justified

for the most risky tests (see Section 9, «Test Case Cata-

TesT managemenT 37

logue: How to prioritise tests based on risks»), which must be

conducted frequently and quickly.


TesT managemenT 39

11. ThE TEST ProcESS: how rolES,

PhASES, AcTIVITIES ANd

docuMENTS work ToGEThEr

BesT PRaCTICe

The test process consists of four phases, which are the same

at each stage of testing:

• Test planning

• Test design

• Test execution

• Test evaluation

The following documents must be generated, at least at the

system test stage:

• Test plan

• Test case specification

• Test log and error messages

• Test report

RuLes

The four phases encompass the following activities:

• Test planning:

The test manager generates the test plan and defines the test

procedure (test stages, test objectives, and automation).

• Test design:

The test designers formulate the test case specifications for

the requirements or identified test cases.

• Test execution:

The testers conduct the tests according to the test case speci-


fications and compare the actual results with the expected

results as formulated in the test cases. They record the

circumstances under which the tests are conducted and the

results in a test log. They generate error reports in case of any

non-conformances.

• Test evaluation:

The test manager evaluates the test logs and error reports

and summarises them in a test report. The test report also

includes his recommendation as to whether the test object

should be released.

suCCess FaCTORs

• Fill the roles with test specialists (see Section 12, «Professional

Testers: A specific requirements profile for each test role»).

• Formulate a test plan for each stage. These plans can take the

form of individual documents, or they can be combined into

a single master test plan for all of the stages.

• Provide templates for the test documents.

• Define the required contents of the error reports so the

developers can reproduce the errors and find the causes.

BeneFITs

Compliance with this process leads to four main benefits:

• Comprehensible test results.

• The tests are performed with reference to the quality standard

of the requirements, rather than intuitively based on

the knowledge of the testers.

• Developers can reproduce and correct the errors without taking

up the time of the testers.

TesT managemenT 41

• Interdepartmental change management meetings allow the

errors to be assessed individually. The team decides whether

each error must be corrected or retested.

Refer to the graphic presentation of the test process

on the rear cover.


TesT managemenT 43

12. ProFESSIoNAl TESTErS: A SPEcIFIc

rEquIrEMENTS ProFIlE For EAch

TEST rolE

A professional test team encompasses the roles of test manager,

test designers and testers. The tasks and requirements of these

roles are described in this section in the form of check- lists.

These lists will help you find suitable candidates and familiarise

them with their roles.

12.1 TEST MANAGEr

«Leads, organises and decides.»

The test manager leads the test team, is a strong decision maker

and has experience in testing.

Tasks

• Adapts the test process to the requirements of the project.

Plans test preparation and execution, agrees on the plan

with the overall project manager and the release manager,

and budgets the test expenditures.

• Organises and manages the team of test designers and

testers.

• Coordinates planning with the requirements team and

ensures that the requirements are reviewed.

• Tasks the test designers with generating the test cases and

ensures their quality by means of reviews.


• Manages the test cases using a test management tool and

establishes their relationship to the requirements.

Participates in change management meetings. Assesses

the effects of errors, changes to requirements and software

modifications in terms of the cost and effort needed to

generate new test cases or modify existing test cases

and repeat tests or conduct new tests.

• Tasks the testers with performing the tests and

supervises their work.

• Organises the test infrastructure (test tools, test systems

and test setup).

• Rejects instable or insufficiently pre-tested software.

• Monitors and reports test progress, quality and accrued

costs.

• Summarises the test results in the test log, interprets

the results and generates the test report with

a recommendation regarding release or

rework of the software.

RequIRemenTs PROFILe

• Ideally, certified at the ISTQB/ASQF Certified Tester

Advanced Level (Test Manager).

• A strong decision maker and competent in conducting

negotiations.

• Has experience as a test designer, and optionally as a

requirements specialist or developer.

• Communicates well and is accepted at the level of SW/IT

development manager, project manager, project management

and quality management.

12.2 TEST ArchITEcT

TesT managemenT 45

«Optimises test costs for multiple products.»

The test architect provides technical advice to the test manager.

He analyses the requirements, is familiar with the software

architectures of the systems under development, and gives technical

instructions to the test designers and test automators.

Tasks

• Considers commercial aspects with regard to test synergies

between various product lines.

• Performs a cost/benefit analysis for various test strategies.

Defines the technical portion of the test plans (test concept).

• Establishes a triage on an economic basis to determine which

tests will be performed in house, which ones will be performed

by software partners, and which ones will be performed

in specialised test centres.

• Ensures that the test designers reuse proven test patterns.

RequIRemenTs PROFILe

• Ideally, certified at the ISTQB/ASQF Certified Tester

Advanced Level (Test Manager).

• Has an analytical and entrepreneurial way of thinking.

• Has an overview of the product range of the company.

• Has experience as a test designer, requirements specialist and

software architect.

• Communicates well and is accepted by the software architect,

the manager of the requirements specialists, the SW/IT

development manager and company management.


12.3 TEST dESIGNEr

«Develops effective test cases.»

Test designers can grasp requirements quickly, generate methodically

structured test cases, and communicate non-conformances

in a professional and comprehensible manner.

Tasks

• Analyses the product requirements, technical specifications

and system architecture.

• Derives test cases from the functional and non-functional

requirements and describes them such that they can be

understood and executed easily by the testers and developers.

• Develops automated integration tests and system tests

(optionally assigned to a separate test automator role).

• Generates or acquires the necessary test data.

• Communicates well with the requirements specialists and

software developers.

• Analyses discovered errors and helps assess the consequences

and causes of errors.

RequIRemenTs PROFILe

• Ideally, certified at the ISTQB/ASQF Certified Tester

Foundation Level.

• Holds a college or university degree in informatics or

engineering.

• Comfortable dealing with complex systems and requirements.

• Has experience in requirements analysis.

• Has experience in software development.

TesT managemenT 47

• Able to express himself precisely in writing.

• Accepted by the software developers and requirements

specialists.

12.4 TEST AuToMATor

«Automates recurrent test cases.»

Test automators can rapidly capture the most important operating

processes of a system and its peripheral system control elements

(test drivers and stubs) in test automation tools, automate

recurrent test cases based on these processes, analyse faulty processes

and communicate errors professionally and comprehensibly.

Tasks

• Analyses requirements for automated tests, plans the use of

test automation tools and applies such tools.

• Generates a framework of the most important operating processes

of a system and its peripheral system control elements.

• Automates and maintains test cases and executes them.

• Communicates well with the requirements specialists and

software developers.

• Analyses discovered errors and helps assess the consequences

and causes of errors.

RequIRemenTs PROFILe

• Ideally, certified at the ISTQB/ASQF Certified Tester

Foundation Level.


• Has experience in software development.

• Comfortable dealing with complex systems and requirements.

• Has an understanding of requirements analysis.

• Able to express himself precisely in writing.

• Accepted by the software developers.

12.5 TESTEr

«Works reliably and precisely.»

Testers perform tests according to test case specifications conscientiously

and reliably and report non-conformances using a

suitable tool.

Tasks

• Uses test case specifications (test cases and test data)

to learn about the tests to be performed.

• Prepares the system for conducting the test.

• Provides the necessary test data.

• Carries out the individual test steps according to the test

case specifications and observes the behaviour of the system.

• Records execution of the test and the observed results.

• Evaluates each test result as to whether it is satisfactory or

not satisfactory.

• Enters errors in the error database.

• Supports developers in reproducing errors.

RequIRemenTs PROFILe

• Has experience in dealing with the system to be tested,

such as in the role of user.

• Proceeds conscientiously.

TesT managemenT 49

• Is reliable in logging tests and test results and entering errors

in the error database.

• Accepted by the software developers and requirements

specialists.


TesT managemenT 51

13. docuMENT TEMPlATES:

TrIEd ANd TruE STrucTurES

Every enterprise, every development process, every quality management

system and every reference process has its own document

templates. The following suggested tables of contents illustrate

how the multitude of information can be organised

systematically so it can subsequently be retrieved.

13.1 TEST PlAN

1. Introduction and general information

2. Test objectives (from the business perspective)

3. Strategy for achieving the objectives & test strategy

3.1 Test stages and their test objectives

3.2 Test tools

4. Identification of the test objects or the requirements

to be tested

5. Test execution and test instructions

5.1 Composition of the test tasks

5.1.1 General requirements

5.1.2 Test activities (and optionally specific test objectives)

5.1.3 Other important activities

5.2 Test infrastructure investments

5.3 Test organisation

5.4 Assessment of test cost and effort

5.5 Coordination and agreements

5.6 Time dependencies and milestones


6. Reporting and test documentation

6.1 Registration of errors

6.2 Test reports

6.3 Test documentation

7. Key figures

8. Risk analysis

13.2 TEST cASE SPEcIFIcATIoN

1. Introduction and general information

2. Test overview

2.1 Test strategy for the unit to be tested

2.2 Special properties of the unit to be tested

2.3 Criticality of the unit to be tested

3. Test environment

3.1 Tools

3.2 Test data

3.3 Test configuration

4. General procedure for performing the test

4.1 Preparation

4.2 Execution and registering errors

4.3 Evaluation and recording results

5. Test cases

5.1 Name of the requirement, function, use case, etc. to be tested

5.2 Prerequisites

5.3 Description of the individual test steps in tabular form

5.4 Post-test activities (actions necessary to restore the system

to its initial state, if necessary)

no. Test stepdescription/

Input

1

2

...

13.3 TEST rEPorT

1. Identification of the test object

TesT managemenT 53

2. Summary assessment of the test end criteria and release

recommendation

3. Reasons for the release recommendation

3.1 Key figures and their interpretation

3.2 Test cycles, test results and test cost & effort

3.3 Estimated residual error rate

4. Assessment of the attainment of the objective

4.1 Test objectives

expected

result

4.2 Costs and deadlines

4.3 Deviation from the test plan

actual

result

5. Not tested, open items and measures

assessment

(satisfactory/

not satisfactory)


14. TEST ToolS: whIch Tool

IS SuITAblE For whAT

TesT managemenT 55

The following list provides an overview of some relatively wellknown

tools. It is by no means exhaustive. Information and

comments about the current compositions of other tools can be

found on the Internet.

application area manufacturer: Product name

Test management • IBM Rational: TestManager

• Mercury: TestDirector

• Microsoft: Excel, Word

system test automation • IBM Rational: Robot,

FunctionalTester (.Net)

• Mercury: Winrunner,

quicktest (.Net)

Integration test automation • IBM Rational: Test RealTime

unit test tools • IPL: Cantata, Cantata++

• JUnit, CppUnit, NUnit


15. FurThEr INForMATIoN:

uSEFul rEFErENcE SourcES

15.1 rEFErENcE docuMENTS

TesT managemenT 57

«Basiswissen Softwaretest: Aus- und Weiterbildung zum Certified

Tester, Foundation Level nach ASQF- und ISTQB-Standard»,

Andreas Spillner and Tilo Linz, dpunkt.verlag, 3rd and revised

edition, 2005, ISBN 3-89864-358-1

• Ideal for preparing for the Certified Tester Foundation Level

examination.

• Combines the theory and practice of software testing.

Aimed at software test managers and test designers. Does not

contain any document templates or tables of contents.

«Lehrbuch der Software-Technik, Band 2: Softwarware-Management,

Software-Qualitätssicherung, Unternehmensmodellierung»,

Helmut Balzert, Verlag Spektrum der Wissenschaft,

2000, 2nd edition, ISBN 3-82740-301-4

«Vademekum des Software-Testens, SAQ-Leitfaden für die

Planung der Software-Qualitätssicherung», Daniel Baumann,

Thomas Hofmann, Donat Hutter, Ruedi Jäggin, Doron Moritz

and Markus Pfister; SAQ, Schweizerische Arbeitsgemeinschaft

für Qualitätssicherung, 2000

• A thin A4 booklet written by process and quality engineers

in the Swiss industrial sector. Discusses all aspects of testing,

including the test process, test roles, test metrics and docu-


ment templates. Includes a summary of important books and

standards.

«Methodisches Testen von Programmen», G. J. Myers, Verlag

R. Oldenbourg, 6th edition, 1999, ISBN 3-486-25056-6

• The most frequently cited book on testing. Even today, it is

pleasurable reading for every interested tester, test designer,

test manager or test process designer.

«Management und Optimierung des Testprozesses (mit TPI

und TMap)», Martin Pol, Tim Koomen and Andreas Spillner,

dpunkt.verlag, 2000, ISBN 3-932588-65-7

• An aid to systematic optimisation of test processes in a

business environment.

«Testing Embedded Software», Bart Broekman and Edwin Notenboom,

Addison-Wesley, 2003, ISBN 0-321-15986-1

• Good procedural practices, such as for safety analyses.

Many tools in the form of tables.

«Software Testing: Tests, Verfahren, Werkzeuge; Die Praxis

des Rapid Application Testings; Agiles Qualitätsmanagement»,

Manfred Rätzmann, Galileo Press, 2002, ISBN 3-89842-

271-2

«Lessons Learned in Software Testing: A Context-Driven Approach»,

Cem Kaner, James Bach and Bret Pettichord, Wiley,

2002, ISBN 0-471-08112-4

• Primarily aimed at testers. Shows the advantages and disad-

TesT managemenT 59

vantages of the standard IEEE template for software test plans.

«Software automatisch testen», Elfriede Dustin, Jeff Raschka

and John Paul, Springer-Verlag, 2001, ISBN 3-540-67639-2

• Primarily aimed at testers. Shows the advantages and disadvantages

of the standard IEEE template for software test plans.

15.2 STANdArdS

• IEEE Std 829-1998, IEEE Standard for Software Test

Documentation

• IEEE Std 1008-1987, IEEE Standard for Software Unit Testing

• IEEE SWEBOK, Guide to the Software Engineering Body of

Knowledge, Chapter 5, «Software Testing»

• FDA, General Principles of Software Validation; Final

Guidance for Industry and FDA Staff, January 11, 2002

15.3 wEb lINkS

• Swiss Testing Board

http://www.swiss-testing-board.ch

• Website for the book «Basiswissen Softwaretest»

http://www.dpunkt.de/certified-tester

• «The Economic Impacts of Inadequate Infrastructure for

Software Testing», National Institute of Standards & Technology,

USA, May 2002

http://www.nist.gov/director/prog-ofc/report02-3.pdf


ThE TEST ProcESS: how rolES,

PhASES, AcTIVITIES ANd docuMENTS

work ToGEThEr

Project

plan



List of

requirements

sysTem

TesT

InTegRa-

TIOn TesT

unIT/

mODuLe TesT

TesT PLannIng TesT DesIgn TesT exeCuTIOn TesT evaLuaTIOn

i i

i i

Test manager

Test case

catalogue


Requirements

Test case

Test designer specification Tester

Test

protocol


error reports

TesT managemenT 61

Test manager

Test

report


Planning Development execution evaluation

The test process consists of the same four activities at each stage of testing:

planning, developing, executing and evaluating tests. The test manager is

responsible for planning and evaluating the tests. The test designer develops

the test cases, while the testers are responsible for executing the tests.


enables & delivers

www.erni-consultants.com

More magazines by this user
Similar magazines