Test suite for dynamic
Find the hard-to-find errors
other tools don’t find
“We are going to definitely incorporate
the Vector Fabrics toolset as an important
part of our quality offering“
Copyright © 2015 Vector Fabrics B.V.
Quality team leader / Altran Intelligent Systems
Avoid software errors and high repair
costs with Pareon Verify
Test suite for dynamic software verification finds difficult-to-detect software
errors and makes lengthy debugging sessions a thing of the past.
The amount of software in embedded
systems, cell phones, or transportation
vehicles such as cars and airplanes is
exploding in size and importance. This
development has led to a exponential
growth in the number of potential
software errors or “bugs”.
Cost of an error
A study by NASA  shows the cost of fixing a software
error found in the field is 100x more expensive than
the same error found already during development.
It is more important than ever to effectively and
thoroughly test software before it is shipped. With the
common use of C++ nowadays, it is also common to
have more dynamic behavior and multithreading. This
leads to a new category of errors that is hard to detect
using the traditional testing tools and methods.
Errors are expensive
Having to repair software errors post-release in production
or in the field is a costly process. Expensive,
even without taking into account any damage to the
brand and the public image of the manufacturer.
Development Testing In the field
The mounting costs of a software defect.
Thus it is crucial that as many errors in the code as
possible are detected and corrected before the product
enters the market.
2 Copyright © 2015 Vector Fabrics B.V.
Average high-end car
F-35 Fighter Jet
Total flight software
% Multi core
% Multi core &
% Don’t know
Growing size and complexity of code increases the number of errors in code.
100 million lines of
It is not uncommon for a car or an airplane to have
over 100 million lines of code running. As the amount
and the of complexity of code increases, so does the
number of errors in the code.
Another component to drive up the complexity and
the number of errors in code is multicore usage. More
and more processors are already multicore, which
brings new challenges to software correctness and
Increasing dynamic behavior
Modern software testers are challenged to cope with
the ever increasing complexity and the ever increasing
dynamic behavior of the software. Dynamic behavior -
for example dynamic memory use and multithreading
- is hard to test for. Bugs in dynamic behavior tend to
be hard to reproduce. Examples of these errors include
buffer overflows, memory leaks, and data races.
A whole new category of software errors is nowadays
labelled as a “Heisenbug”: named after the uncertainty
principle elaborated by the German theoretical
physicist Werner Heisenberg. Heisenbugs are hard to
diagnose because they tend to disappear when under
scrutiny: debuggers or so-called “printf-debugging”
tend to adjust the run-time circumstances enough to
make the bug disappear.
For instance, a Heisenbug could relate to a timing
problem: in a debugger, timings change and the
bug does not appear. This makes the subsequent
detection and resolution of these errors an extremely
complex and thus costly process.
For the reasons mentioned above, validating software
without dynamic analysis is no longer an effective
strategy. Software projects are growing in size, the
software itself is becoming more dynamic and the
hardware used to operate the software is becoming
increasingly sophisticated. Without employing dynamic
analysis, the complex software errors will remain
undetected in the product to cause problems later.
Automated dynamic analysis by Pareon Verify
This trend led Vector Fabrics to develop Pareon Verify
- a tool for dynamic software verification. Pareon
Verify allows for automated dynamic analysis of coding
errors in large volumes of source code. This process is
able to detect software errors that cannot be identified
using other tools or methods.
Increasing size and complexity of applications
cause more errors to occur in software.
Copyright © 2015 Vector Fabrics B.V.
The advantages of
dynamic code analysis
Dynamic analysis - or runtime analysis, as it’s sometimes
called - can be seen as an MRI scanner for
software. By looking inside the application while it
executes, it is possible to detect errors in the
Dynamic analysis observes at the application’s internal
structures and behavior and observes where errors
originate. In doing so, it pinpoints the primary cause of
the error. This way, dynamic analysis identifies errors
that remain undetected during other tests.
Limits of existing methods
Errors in dynamic behavior are notoriously difficult
to detect with traditional testing tools and methods.
Errors in the dynamic behavior don’t trigger when the
application is run in its normal testing environment,
but cause problems when environment, timing or
Static analysis is limited in its capability to predict the
behavior of dynamic structures and cannot foresee all
possible combinations of inputs, code paths, dynamic
memory and thread timing in the full application scope.
Unit testing is limited in scope to a smaller part of the
application, and cannot detect errors that are caused
by the global behavior or the environment of the
Dynamic analysis looks at the entire application and is
not limited to a local unit scope in its error detection.
Additionally, as it looks inside the application to see
errors in individual operations in a white-box fashion,
it reports errors in an application even if that error
does not cause the application to fail its black-box test
on a particular test case.
Cost of an error
Therefore, dynamic code analysis is a part to a
comprehensive software testing process. It complements
static code analysis and unit testing. Dynamic
analysis offers the key advantage: it reveals
the most expensive errors that would otherwise
Unit & system tests
Coding style checks
The role of the dynamic analysis in the software
Coding style violations
Dynamic analysis to help
Dynamic analysis is not hindered by these limitations:
as it observes the application’s actual operation while
being executed, it can see all the combinations of
dynamic and global behavior that are triggered during
the execution, regardless of complexity.
Copyright © 2015 Vector Fabrics B.V.
How it works?
Dynamic analysis works by instrumenting the target
application to see the behavior inside the application.
The instrumentation in the application binary creates
a stream of events, or a trace, that is captured by the
analyzer and studied for errors in the behavior. The
analyzer then reports the found errors in a clear and
concise way with detailed references to source code
for easy correction of the error.
Pareon Verify detects a range of software errors, including
Heisenbugs.. Pareon Verify can be completely
integrated into the development process so that new
software changes can be continuously screened for
The workflow of Pareon Verify is as follows:
Use Pareon’s instrumenting C and C++ compiler
as a drop-in replacement for your (cross)compiler.
Set-up your build in minutes.
Execute the instrumented program on a host PC
or target embedded board. Run with representative
input data or test suite.
Conveniently analyze the resulting execution trace
on your host PC. Minimal memory and performance
impact from the analysis on the target!
4. Resolve bugs
Get detailed messages for each error. Instantly see
how to fix the problem. Variable names, full call
stacks. No false positives!
Invalid memory access
Uninitialized memory reads
Invalid pointer usage and buffer overflows e.g. through bad pointer arithmetic
Access outside a buffer or using wrong indexes into a multidimensional array
Using variables that have not been written yet, generally leading to data corruption
Use of dangling pointers, pointing to memory that no longer exists
Multiple threads accessing the same data without proper synchronization,
generally leads to data corruption
Used memory that is not released will cause system to slow down or run out of memory
Hard-to-detect errors Pareon Verify finds.
Copyright © 2015 Vector Fabrics B.V. 5
Example Multicore errors
Computer hardware increasingly uses multicore
processors. Not just PCs, but also cell phones have up
to 8 cores inside nowadays. This causes problems on
the software side as multicore software development
is harder. So-called “data races” already occurred on
single core CPUs when interacting with interrupts, but
are now much Readmore widespread due to the multithreaded
programs actually running threads in parallel.
A data race arises when two threads access the same
variable without proper synchronization. As a result,
threads could read stale or just wrong data - data
corruption will happen.
Read and write from different threads to
memory In general, Thread multi-threaded B programs will have more
dynamic behavior and thus will be more susceptible to
dynamic software errors.
Developers today run their tests on the target hardware
to ensure they test the exact same behavior as
when the software is deployed in the field. Pareon
Verify detects data races even when the race does
not trigger during the test. This allows the developer
to run tests on a much more accessible host platform
and still prevent timing-related issues in the field.
“We are glad we are now able to
catch some really difficult bugs
Lead developer / PicoTCP
Data race, read of uninitialized data because
the read happens before the write is done.
6 Copyright © 2015 Vector Fabrics B.V.
Solving errors early through
Many professional software development companies
deploy a development process called “continuous
integration”. Upon on check-in of a new piece
of code or at least every night, a central server will
take all source code, build it and run a set of tests
on it. If any of the tests break, the newly submitted
code is rejected and the developer has to fix
it. This allows to find a lot of errors earlier in the
process: whilst the developer is still working. As the
code is still fresh on his mind, fixes will be faster.
Using Pareon Verify, it is now also possible to
deploy dynamic error finding during continuous integration.
Instead of just receiving a “pass” or “fail”
from the automated tests, Pareon Verify will also
report if the code flow leading to the verdict was
correct. Pareon Verify reports if the test result is
only correct by accident. Pareon Verify also reports
if there were unwanted side effects. The benefit of
this early detection is that developer can correct
the error immediately - even before it triggers a
test case to fail and a crash to happen, saving a lot
of time and costs in the future.
To reduce maintenance costs, it is essential
that errors be removed from a software design
as early as possible. A well-known Japanese car
manufacturer notably had to issue recalls for
problems that were linked to software errors in
a number of its models. It is estimated that the
recalls cost the company two billion dollars, not
to mention the damage to its corporate image.
Benefits of Pareon Verify
Less errors in the field
By capturing the errors before the hit the field,
companies can reduce maintenance costs of
software. The customer satisfaction and the quality
image of the brand will improve when errors are
no longer triggered in the field. An error found
during testing is 5x more expensive than one
found during development, and it can be up to
100x more expensive when found in the field - not
to mention impacts to customer satisfaction and
Improved time to market
Often, during the integration and system testing
phase’ the number of bugs found by the test team
only grows. Developers cannot tell when they are
done fixing all the issues as one issue might be
hiding the next. As a result, project and release
schedules tend to slip, delaying the release or the
end of the project. With Pareon Verify, companies
will be able to cut significantly back on the time
spent on debugging and correcting errors, which
in turn reduces the risk of project delays. In practice,
this can mean weeks’ worth of time-saving
and a more predictable test and integration phase.
By using Pareon Verify, software can be brought to
market more quickly, cheaper and with fewer error
reports from customers. Software developed with
Pareon Verify is of a higher quality and is completed
sooner. This allows companies to improve the
quality image of their software.
For more information about Pareon Verify,
please visit: www.pareonverify.com
Copyright © 2015 Vector Fabrics B.V.
About Vector Fabrics
Software is an essential part of virtually every modern
product or service. Vector Fabrics specializes in development
and testing tools for complex software that is
used in embedded systems, including automotive, IoT
devices, network equipment, consumer electronics,
aerospace, medical and industrial systems, and more.
The Pareon family of dynamic analysis tools provides
customers with deep insight into how their software will
actually behave when running, ensuring a higher level
of reliability, safety and performance. Using these tools,
businesses are able to prevent costly software bugs and
improve the time-to-market for new products.
5301 LJ Zaltbommel
Phone +31 40 8200 960
Copyright © 2015 Vector Fabrics B.V. All Rights Reserved.