A Comparison of Open Source and Commercial Static Analysis ...

ANTLR (Java Code) - Fixed Defects 2013• 40K LOCCategoryTotalCoverity: Exceptional resource leaks 4FindBugs: Bad practice 3FindBugs: Dodgy code 6Coverity: Incorrect expression 1Coverity: Null pointer dereferences 5Coverity: Resource leaks 1Total 208Confidential: For Coverity and Partner use only. Copyright Coverity, Inc., 2013

Linux- Defect Example Resource Leakalloc_fn: Calling allocation function"kzalloc". bss_cfg is assigned9

Linux- Defect Example Resource Leakbss_cfg is not freed10

Linux- Defect Example Resource Leakbss_cfg out of scope and leaksThe Fix:http://marc.info/?l=linuxwireless&m=134135643727424&w=211

Python Defect - Memory CorruptionMemory was allocated for variable "buffer".line 10123, memory was de-allocated for variable "buffer".12

Numerous Options ExistConsiderations• Does it find critical defects?• What is the false positiverate?• Is it actionable?• Is it accurate?• Is it integrated to myworkflow?• How do I managepersistency

Jenkins Analysis• Analyzed Jenkins version 1.496 core code using up-to-dateCoverity and FindBugs (as of Dec 2012)18 Copyright 2013, Coverity, Inc.

Different Things are Found19628FindBugs627Only 28 issues shared between Coverity and FindBugs19Confidential: For Coverity and Partner use only. Copyright Coverity, Inc., 2013

Comparison by Defect TypeCoverityType Coverity FindBugs Shared DefectsUnhandled exceptions(incl. NULL deref)7979 7 5Resource leaks 86 12 13Concurrency problems 22 10 9Critical DefectSubtotalCoding Standards, BestPractices, Other8622187188 29 279 598 1Total Bugs 196 627 2820Confidential: For Coverity and Partner use only. Copyright Coverity, Inc., 2013

Freeradius Analysis• Freeradius version: 2.1.12 (released 30th Sep, 2011)• Clang Analyzer version: checker-275 (23 rd May, 2013)• Coverity version: 6.6.1 (July, 2013)21 Copyright 2013, Coverity, Inc.

Different Things are Found1213Clang97Only 3 issues shared between Coverity and Clang22Confidential: For Coverity and Partner use only. Copyright Coverity, Inc., 2013

Comparison by Defect TypeType Coverity Clang Shared DefectsMemory 79 5 0Resource leaks 86 3 0Control Flow, Concurrentaccess, Other22 30 1High + Medium Defects 188 38 1Coding Standards, BestPractices, OtherCoverity119831039 59 2Total Bugs 121 97 323Confidential: For Coverity and Partner use only. Copyright Coverity, Inc., 2013

Freeradius: 2.2.1 (released 17 th Sep, 2013)Security Vulnerability: “We scanned therlm_eap_tls.c file with the LLVM checker-267,taken from http://clang-analyzer.llvm.org/. It didnot find this issue. However, a Coverity scan diddiscover it.”http://freeradius.org/security.htmlConfidential: For Coverity and Partner use only. Copyright Coverity, Inc., 2013

Two years later …Freeradius Analysis• Freeradius version: freeradius 2.2 (released 2013)• Clang Analyzer version: checker-275 (23 rd May, 2013)• Coverity version: 6.6.1 (July, 2013)25 Copyright 2013, Coverity, Inc.

Two years later ….Fixed in FREERADIUS 2.2.Coverityfound in2.1.12Clangfound in2.1.12Fixed inFixed inImpact Category2.2.1 Shared2.2.1 Clang CategoryHigh Memory - corruptions 7 3High Memory - illegal accesses 4 3 5 3 Use-after-freeHigh Resource leaks 9 8 3 Memory leakHigh Uninitialized variables 5 2 2 2 Assigned value is garbageMedium API usage errors 6MediumConcurrent data accessviolations 1Medium Control flow issues 18 5Medium Error handling issues 19Dead incremeant/Dead initialization/Unix APIMedium Incorrect expression 14 7 9 3Medium Insecure data handling 5 5Medium Integer handling issues 1Medium Null pointer dereferences 13 6 1 19 1 Dereference of null pointerMedium Program hangs 1Low Code maintainability issues 4 2 59 1 Dead AssignmentLow Parse warnings 1Low Security best practices violations 13 3Total 121 42 3 97 1042 0f Coveritydefects were fixed10 0f Clang defectswere fixedConfidential: For Coverity and Partner use only. Copyright Coverity, Inc., 2013

FreeRADIUS Quality practicesFrom Alan Dekok• Use APIs which make it harder for issues to arise (explicit lengths, etc.)• On 3.0 branch, build with *no* C compiler warnings• Use autobuilds (https://travis-ci.org/FreeRADIUS/freeradius-server/) builds withclang && gcc, and builds debian packages• Coverity:• New Coverity builds every day, Coverity are emailed to the core team. Many can be fixeddirectly from the summary in the email• This practice ensures basic code sanity. What it *can't* do is ensure logical correctness. We'vehad a few bugs slip in where the code passes all checks, but is logically incorrect. i.e. it doesn'timplement part of a protocol correctly.• Finding those issues automatically is harder. Doing a protocol test suite for a complex daemonis very difficult. With the 3.0 branch, we're now running more unit tests, for basicfunctionality. That helps, but more tests are needed.• For us, Coverity is an indispensable part of our daily development routine. It's helped to makeFreeRADIUS better software. And it's helped to make us better programmers.

How Does Your Code Compare?The bar has been raised on what is considered good quality software.69 defect density vs. 1.0Defect Density by Project Size: Open Source vs. ProprietaryLines of code Open Source Proprietary1M .75 .66Average .69 .6828Confidential: For Coverity and Partner use only. Copyright Coverity, Inc., 2013

SCAN.Coverity.COM – Free for the OpenSource CommunitySign Up Today29

Q&AConfidential: 30 For Coverity and Partner use only. Copyright Coverity, Inc., 2011