Pareon Verify

dblhelix

1KpdxWX

Pareon Verify

Test suite for dynamic

software verification

Find the hard-to-find errors

other tools don’t find

“We are going to definitely incorporate

the Vector Fabrics toolset as an important

part of our quality offering“

Copyright © 2015 Vector Fabrics B.V.

Jan Asselman

Quality team leader / Altran Intelligent Systems


Avoid software errors and high repair

costs with Pareon Verify

Test suite for dynamic software verification finds difficult-to-detect software

errors and makes lengthy debugging sessions a thing of the past.

The amount of software in embedded

systems, cell phones, or transportation

vehicles such as cars and airplanes is

exploding in size and importance. This

development has led to a exponential

growth in the number of potential

software errors or “bugs”.

Cost of an error

A study by NASA [1] shows the cost of fixing a software

error found in the field is 100x more expensive than

the same error found already during development.

It is more important than ever to effectively and

thoroughly test software before it is shipped. With the

common use of C++ nowadays, it is also common to

have more dynamic behavior and multithreading. This

leads to a new category of errors that is hard to detect

using the traditional testing tools and methods.

Errors are expensive

Having to repair software errors post-release in production

or in the field is a costly process. Expensive,

even without taking into account any damage to the

brand and the public image of the manufacturer.

Development Testing In the field

The mounting costs of a software defect.

Thus it is crucial that as many errors in the code as

possible are detected and corrected before the product

enters the market.

2 Copyright © 2015 Vector Fabrics B.V.


24

14

24

100

14

Car Sofware

Average high-end car

F-35 Fighter Jet

2013

Boeing 787

Total flight software

100

*million lines

of code

19,4

21,4

8,5

30,1

19,4

8,5

20,6 21,4

30,1

20,6

% Single

processor

% Multi

processor

% Multi core

% Multi core &

Multi processor

% Don’t know

Growing size and complexity of code increases the number of errors in code.

100 million lines of

complex code

It is not uncommon for a car or an airplane to have

over 100 million lines of code running. As the amount

and the of complexity of code increases, so does the

number of errors in the code.

Another component to drive up the complexity and

the number of errors in code is multicore usage. More

and more processors are already multicore, which

brings new challenges to software correctness and

verification.

Increasing dynamic behavior

Modern software testers are challenged to cope with

the ever increasing complexity and the ever increasing

dynamic behavior of the software. Dynamic behavior -

for example dynamic memory use and multithreading

- is hard to test for. Bugs in dynamic behavior tend to

be hard to reproduce. Examples of these errors include

buffer overflows, memory leaks, and data races.

Heisenbugs

A whole new category of software errors is nowadays

labelled as a “Heisenbug”: named after the uncertainty

principle elaborated by the German theoretical

physicist Werner Heisenberg. Heisenbugs are hard to

diagnose because they tend to disappear when under

scrutiny: debuggers or so-called “printf-debugging”

tend to adjust the run-time circumstances enough to

make the bug disappear.

For instance, a Heisenbug could relate to a timing

problem: in a debugger, timings change and the

bug does not appear. This makes the subsequent

detection and resolution of these errors an extremely

complex and thus costly process.

For the reasons mentioned above, validating software

without dynamic analysis is no longer an effective

strategy. Software projects are growing in size, the

software itself is becoming more dynamic and the

hardware used to operate the software is becoming

increasingly sophisticated. Without employing dynamic

analysis, the complex software errors will remain

undetected in the product to cause problems later.

Automated dynamic analysis by Pareon Verify

This trend led Vector Fabrics to develop Pareon Verify

- a tool for dynamic software verification. Pareon

Verify allows for automated dynamic analysis of coding

errors in large volumes of source code. This process is

able to detect software errors that cannot be identified

using other tools or methods.

Increasing size and complexity of applications

cause more errors to occur in software.

Copyright © 2015 Vector Fabrics B.V.

3


The advantages of

dynamic code analysis

Dynamic analysis - or runtime analysis, as it’s sometimes

called - can be seen as an MRI scanner for

software. By looking inside the application while it

executes, it is possible to detect errors in the

internal behavior.

Dynamic analysis observes at the application’s internal

structures and behavior and observes where errors

originate. In doing so, it pinpoints the primary cause of

the error. This way, dynamic analysis identifies errors

that remain undetected during other tests.

Limits of existing methods

Errors in dynamic behavior are notoriously difficult

to detect with traditional testing tools and methods.

Errors in the dynamic behavior don’t trigger when the

application is run in its normal testing environment,

but cause problems when environment, timing or

circumstances change.

Static analysis is limited in its capability to predict the

behavior of dynamic structures and cannot foresee all

possible combinations of inputs, code paths, dynamic

memory and thread timing in the full application scope.

Unit testing is limited in scope to a smaller part of the

application, and cannot detect errors that are caused

by the global behavior or the environment of the

application.

Dynamic analysis looks at the entire application and is

not limited to a local unit scope in its error detection.

Additionally, as it looks inside the application to see

errors in individual operations in a white-box fashion,

it reports errors in an application even if that error

does not cause the application to fail its black-box test

on a particular test case.

Cost of an error

Therefore, dynamic code analysis is a part to a

comprehensive software testing process. It complements

static code analysis and unit testing. Dynamic

analysis offers the key advantage: it reveals

the most expensive errors that would otherwise

remain undetected.

Dynamic analysis

Unit & system tests

Static analysis

Coding style checks

Compilation

The role of the dynamic analysis in the software

verification process.

Heisenbugs

Functional errors

Coding errors

Coding style violations

Syntax errors

Dynamic analysis to help

Dynamic analysis is not hindered by these limitations:

as it observes the application’s actual operation while

being executed, it can see all the combinations of

dynamic and global behavior that are triggered during

the execution, regardless of complexity.

4

Copyright © 2015 Vector Fabrics B.V.


How it works?

Dynamic analysis works by instrumenting the target

application to see the behavior inside the application.

The instrumentation in the application binary creates

a stream of events, or a trace, that is captured by the

analyzer and studied for errors in the behavior. The

analyzer then reports the found errors in a clear and

concise way with detailed references to source code

for easy correction of the error.

Pareon Verify detects a range of software errors, including

Heisenbugs.. Pareon Verify can be completely

integrated into the development process so that new

software changes can be continuously screened for

new errors.

The workflow of Pareon Verify is as follows:

1. Compile

Use Pareon’s instrumenting C and C++ compiler

as a drop-in replacement for your (cross)compiler.

Set-up your build in minutes.

2. Execute

Execute the instrumented program on a host PC

or target embedded board. Run with representative

input data or test suite.

3. Analyze

Conveniently analyze the resulting execution trace

on your host PC. Minimal memory and performance

impact from the analysis on the target!

4. Resolve bugs

Get detailed messages for each error. Instantly see

how to fix the problem. Variable names, full call

stacks. No false positives!

Invalid memory access

Array out-of-bounds

Uninitialized memory reads

Use-after-free

Race conditions

Memory leaks

Invalid pointer usage and buffer overflows e.g. through bad pointer arithmetic

Access outside a buffer or using wrong indexes into a multidimensional array

Using variables that have not been written yet, generally leading to data corruption

or crashes

Use of dangling pointers, pointing to memory that no longer exists

Multiple threads accessing the same data without proper synchronization,

generally leads to data corruption

Used memory that is not released will cause system to slow down or run out of memory

Hard-to-detect errors Pareon Verify finds.

Copyright © 2015 Vector Fabrics B.V. 5


?

Example Multicore errors

?

Write

10

10

Read

Write

10

Computer hardware increasingly uses multicore

processors. Not just PCs, but also cell phones have up

to 8 cores inside nowadays. This causes problems on

the software side as multicore software development

is harder. So-called “data races” already occurred on

single core CPUs when interacting with interrupts, but

are now much Readmore widespread due to the multithreaded

programs actually running threads in parallel.

A data race arises when two threads access the same

10

variable without proper synchronization. As a result,

threads could read stale or just wrong data - data

corruption will happen.

Time

Thread A

Shared

memory

Read and write from different threads to

shared memory.

?

Thread B

Time

Thread A

Shared

memory In general, Thread multi-threaded B programs will have more

dynamic behavior and thus will be more susceptible to

dynamic software errors.

Developers today run their tests on the target hardware

to ensure they test the exact same behavior as

when the software is deployed in the field. Pareon

Verify detects data races even when the race does

not trigger during the test. This allows the developer

to run tests on a much more accessible host platform

and still prevent timing-related issues in the field.

ad

Write

10

Read

“We are glad we are now able to

catch some really difficult bugs

much earlier“

10

Daniele Lacamera

Lead developer / PicoTCP

ead B

Time

Thread A

Shared

memory

Thread B

Data race, read of uninitialized data because

the read happens before the write is done.

6 Copyright © 2015 Vector Fabrics B.V.


Solving errors early through

continuous integration

Many professional software development companies

deploy a development process called “continuous

integration”. Upon on check-in of a new piece

of code or at least every night, a central server will

take all source code, build it and run a set of tests

on it. If any of the tests break, the newly submitted

code is rejected and the developer has to fix

it. This allows to find a lot of errors earlier in the

process: whilst the developer is still working. As the

code is still fresh on his mind, fixes will be faster.

Using Pareon Verify, it is now also possible to

deploy dynamic error finding during continuous integration.

Instead of just receiving a “pass” or “fail”

from the automated tests, Pareon Verify will also

report if the code flow leading to the verdict was

correct. Pareon Verify reports if the test result is

only correct by accident. Pareon Verify also reports

if there were unwanted side effects. The benefit of

this early detection is that developer can correct

the error immediately - even before it triggers a

test case to fail and a crash to happen, saving a lot

of time and costs in the future.

To reduce maintenance costs, it is essential

that errors be removed from a software design

as early as possible. A well-known Japanese car

manufacturer notably had to issue recalls for

problems that were linked to software errors in

a number of its models. It is estimated that the

recalls cost the company two billion dollars, not

to mention the damage to its corporate image.

Benefits of Pareon Verify

Less errors in the field

By capturing the errors before the hit the field,

companies can reduce maintenance costs of

software. The customer satisfaction and the quality

image of the brand will improve when errors are

no longer triggered in the field. An error found

during testing is 5x more expensive than one

found during development, and it can be up to

100x more expensive when found in the field - not

to mention impacts to customer satisfaction and

brand image.

Improved time to market

Often, during the integration and system testing

phase’ the number of bugs found by the test team

only grows. Developers cannot tell when they are

done fixing all the issues as one issue might be

hiding the next. As a result, project and release

schedules tend to slip, delaying the release or the

end of the project. With Pareon Verify, companies

will be able to cut significantly back on the time

spent on debugging and correcting errors, which

in turn reduces the risk of project delays. In practice,

this can mean weeks’ worth of time-saving

and a more predictable test and integration phase.

Summary

By using Pareon Verify, software can be brought to

market more quickly, cheaper and with fewer error

reports from customers. Software developed with

Pareon Verify is of a higher quality and is completed

sooner. This allows companies to improve the

quality image of their software.

http://ntrs.nasa.gov/archive/nasa/casi.ntrs.nasa.gov/20100036670.pdf

For more information about Pareon Verify,

please visit: www.pareonverify.com

Copyright © 2015 Vector Fabrics B.V.

7


About Vector Fabrics

Software is an essential part of virtually every modern

product or service. Vector Fabrics specializes in development

and testing tools for complex software that is

used in embedded systems, including automotive, IoT

devices, network equipment, consumer electronics,

aerospace, medical and industrial systems, and more.

The Pareon family of dynamic analysis tools provides

customers with deep insight into how their software will

actually behave when running, ensuring a higher level

of reliability, safety and performance. Using these tools,

businesses are able to prevent costly software bugs and

improve the time-to-market for new products.

Hogeweg 39A

5301 LJ Zaltbommel

The Netherlands

Phone +31 40 8200 960

Email info@pareonverify.com

www.pareonverify.com

Copyright © 2015 Vector Fabrics B.V. All Rights Reserved.

Similar magazines