Certitude™ Functional Qualification with Formal Verification

Certitude 

Functional Qualification with Formal Verification 

Jean-Marc Forey 

November 2012 

Springsoft Proprietary

Topics 

Case study presentation 

Why Verification 

Verification efficiency 

Formal verification 

Introduction to Certitude Functional 

Qualification 

Results obtained on the studied case 

Conclusion 

Q&A 

2 Springsoft Proprietary

Case study presentation 

Synchronous fifo 

•Clock, Reset_ 

•ReadEn, WriteEn, DataIn 

•FifoEmpty, FifoFull, DataOut 

• All outputs are clocked 

Formal verification environment 

•Focused on flags 

•Leveraged from an other design 

•7 assertions 

Dynamic verification environment 

•Focused on data path 

•Directed test 


Why verification? 

Designs are made by humans 

Humans make mistakes 

Hence designs have bugs 

However customers / consumers want 

• Bug free designs 

• Safer, more reliable products 

• More features, more powerful products 

How can providers satisfy the above constraints? 

=> Have computers make designs 

=> Wait for bugs to be reported by the users 

=> Audit the design to squeeze out the bugs 


Verification 

Set of techniques to find bugs in designs 

• Human contribution is primordial 

• Specs, Architecture, Methodology ... 

• Implementation aspects assisted by computers 

• For test generation 

• To check simulation results 

• To check which properties hold 

• And provide counter examples, for those that don’t 

But there are some little hurdles… 

• Size of the problem and available time 

• How to measure verification? 

• What is the meaning of the measure? 

• How to combine measurements? 



Verification is partial 

• Art: what to verify, at which level, methods 

• Risk management: what to skip, when to stop 

• Combine techniques 

Verification too is subject to bugs 

• Missing properties 

• Missing / broken checkers 

• Missing tests 

• Missing scenarios 

Bugs in the VE hide bugs in the design 

How to gauge the efficiency of the verification? 


Verification efficiency 

Metrics to estimate what was done 

• Bug rate 

• Functional coverage 

• Dynamic verification only 

• Structural coverage(s) of the design 

• Line, block, condition, expression, toggle, 

fsm, branch, path, MC/DC, ??? 

• Which one(s) make(s) sense with Formal 

verification? 


Metrics meaning 

Metrics to estimate what was done 

• Really??? 

• not covered => not verified 

• But Covered doesn’t mean Verified 

not (not covered => not verified) (verified => 

covered) 

How to combine metrics? 

• Ex: line, toggle, fsm, functional 

• How to combine data from dynamic / formal VE? 

What are the achievable scores for a given 

design? 

How much is a metric (score) telling about the 

VE effectiveness? 



Some key questions about formal 

verification in the following slides 



Properties 

• How many are needed? 

• Which ones? 

• Only asked questions are answered... 

• Have you got all the right questions? 



Properties 

No (explicit) stimuli, but there are implicit ones 

• All stimuli combinations are considered as being 

valid 

• Unless restricted by constraints 

• Doesn’t mean they are all valid 

• What if some combinations are illegal and no 

constraint was set 

• All the properties are passing 

• Isn’t it strange if invalid inputs don’t lead to output 

misbehaviours 

• How many properties are missing? Which ones? 

• Are the reached cover points valid??? 

• Some may be reachable only through forbidden input 

combinations 



Properties 

No (explicit) stimuli 

Execution traces / waveforms 

• For counter examples only 

• One has to believe the formal tool when it 

reports a pass 



Properties 

No (explicit) stimuli 

Traces 

How much of the design is used to get the 

proofs? 

Which aspects of the design are “verified”? 

White box verification remains a misleading 

friend 

IU 

ID 

. 

B1 

P1 B2 P2 

? 

13 



not (FifoFull && FifoEmpty) 

ReadEn 

WriteEn 

DataIn 

Cone Of Influence 

Reduced cone 

FifoEmpty 

FifoFull 

DataOut 

In the above example 

• Inputs ReadEn/WriteEn can influence the FifoEmpty 

output 

• Only 1 gate is needed to prove the property 

• The property is necessary but not sufficient 

Cone of influence != What is verified ≅ Reduced cone 

14 


How do you find the reduced cone? 

Inject (artificial) bugs! 

At least one property fail 

• Good 

All property passes 

• Bad 

• The same bug would be missed 

• Can be very shocking 

• Bugs with similar effect would be missed too 

• Ex: not detected SA0 on FifoFull 

• Missing properties? Over-constraints? 

• More verification is needed 

• Dynamic and/or formal 


Certitude principle 

Inject faults (artificial bugs) 

ReadEn 

WriteEn 

DataIn 

FifoEmpty 

FifoFull 

DataOut 

Run the verification 

• At least 1 Fail => fault is detected (good) 

• All Pass => fault is not detected (bad) 

16 


Certitude principle 

Inject faults (artificial bugs) 

ReadEn 

WriteEn 

DataIn 

FifoEmpty 

FifoFull 

DataOut 

Run the verification 

• At least 1 Fail => fault is detected (good) 

• All Pass => fault is not detected (bad) 

17 


How it Works 

Modifies RTL code to insert faults 

out1 = f(i1) out1 = 1’b0 // disconnect output and tie it to constant 

if (a) if (TRUE) // remove “else” branch 

f1(); 

f1(); 

else 

f2(); 

else 

f2(); 

a = b | c a = b & c // change operator 

Same principle and definition is applicable to both 

dynamic and formal verification 

• Combining data from dynamic and formal is immediate 


Case study results 

Fifo presented previously 


• Focused on flags 

• Properties: 

P_correct_flag_interval: not(F1 ##1 !F2[*0:14] ##1 F2) 

• With F1=FifoEmpty and F2=FifoFull 

• With F1=FifoFull and F2=FifoEmpty 

P_never_FE_FF: not (FifoEmpty && FifoFull) 

P_deassert_FE: WriteEn && !ReadEn |=> !FifoEmpty 

P_deassert_FF: !WriteEn && ReadEn |=> !FifoFull 

P_FF_Stable: !ReadEn && FifoFull |=> FifoFull 

P_FE_Stable: !WriteEn && FifoEmpty |=> FifoEmpty 


Case study 

Qualification of the formal verification 

• 28 NA and 14 ND out of 83 faults 

• NA: not in the cone of influence of any 

property 

• ND: all properties are proven on the 

modified design 

• SA0 on FifoEmpty and FifoFull are ND 

• A pair of properties don’t seem effective 

p_correct_flag_interval: not(F1 ##1 

!F2[*0:14] ##1 F2) 


Case study 


Case study 

Added 2 properties 

p_getFull: !ReadEn throughout (WriteEn[->4]) |=> 

FifoFull 

p_getEmpty: !WriteEn throught (ReadEn[->4]) |=> 

FifoEmpty 

Got 2 failures (due to 2 bugs) for specific 

Read/Write pointer values 

• Due to expressions like 

if (… && (ReadPtr+1==WritePtr) && …) 

• Was replaced by 

`define ONE {{ADDR_WIDTH{1’b0}},1’b1} 

if (… && (ReadPtr+`ONE==WritePtr) && …) 

Qualification of the formal verification after fixes 

• 34 NA and 2 ND out of 93 faults 


Case study 


Case study 

Dynamic verification focused on data side 

Qualification of the dynamic verification 

• 5 NA, 9 NP, 11 ND out of 83 faults 

• NA: non activated 

• NP: non propagated; no influence at the boundary of 

the design and test passed 

• ND: output behaviour is different, and tests pass 

Improved VE and found 2 bugs on the data side 

• Related to Read / Write in the memory when the fifo is 

empty/full 

Qualification of the dynamic verification after the 

fixes 

• 2 NA, 2 NP, 2 ND faults out of 95 faults 

• ND are detected by the static VE 

• NP are in redundant code 


Conclusion 

Certitude helps understand what parts of the design are 

verified 

• Even more important with formal verification 

Addressing verification issues pointed out by Certitude 

allows verifier to discover design bugs 

• The bugs discovered with formal would probably not have 

been found with dynamic verification – too corner case 

Easy to merge metrics from dynamic and formal 

verification environments 

• Get a global picture of the verification strengths and 

weaknesses 

• Address the issues where best suited 

• Optimize the verification effort 

Certitude provides a strong unified metric for both 

dynamic and formal verification 


Q&A 


Impact of Poor Verification Environment 

Wasted CPU cycles 

• Missing / buggy checkers and assertions 

mean RTL bugs go undetected even with 

more tests and CPU time 

Inefficient resource allocation 

• Which blocks are poorly verified? 

• Do I need better stimulus / scenario? 

Late-stage identification of problems 

• ECOs required close to tape-out 

Silicon re-spins 

• Bug escapes Significant $$ 


Effective Verification 

It’s all about activation, propagation, and detection 

Activation 

Bug 

Propagation 

Detection 

Stimulus 

Design under 


Compare 

Reference Model 

Verification Environment 

To detect a bug… 

• The stimulus must activate the buggy logic 

• An effect of the bug must propagate to an observation point 

• The environment must detect the behavior difference due to the bug 


Existing Tools are Insufficient 

Code coverage measures activation, but says 

nothing about propagation or detection 

Activation 

Bug 

Propagation 

Detection 

Stimulus 

Design under 


Compare 



Functional coverage checks “important” functional 

points, but is subjective and incomplete 


Introducing Functional Qualification 

Functional Qualification 

Activation 

Fault 

Propagation 

Detection 

Stimulus 

Design under 


Compare 



• Inserts “artificial bugs” called faults into the design 

• Measures the ability of the verification environment to 

activate, propagate, and detect the faults 

• “Qualification” of the verification environment against many 

inserted faults provides objective measure of overall quality 

and identifies holes and weaknesses 


Typical Problems Found by Certitude 

Missing or incomplete 

checker/assertion 

• Hole in test plan 

• Test plan item not implemented 

• Test plan item misinterpreted 

• Checker disabled and forgotten 

Missing or incomplete test 

scenario 

• Hole in test plan 

• Test plan item not implemented 

• Environment over-constrained 

• Thought test wasn’t important 

Process problem 

• Script error gives false positive “pass” 

• Checker disabled and forgotten 

• Areas that need closer review 

Redundant logic or dead code 

• Unreachable due to RTL bug 

• Useless logic from re-use 

Areas of low coverage 

• Some logic not activated by any test 

• Logic activated but poorly propagated 

• Test subset not as good as thought 


Certitude Functional Qualification System 

The industry’s first and only Functional Qualification 

solution 

Production-proven 

• Extensive worldwide customer list 

• Processor, communication, automotive, network, storage, 

wireless, multimedia, … 

Supports all common RTL languages and simulators 

• Verilog, VHDL, SystemVerilog 

• VCS, Questa, IUS/NC 

Supports all verification environments 

• Verilog, VHDL, SystemVerilog, C/C++, Specman/e, Vera, … 

Integrates quickly and easily with your simulation 

environment 

• Based on infrastructure already available in your regression 

flow

Certitude™ Functional Qualification with Formal Verification

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?