Benchmark 5 Example

ICCAD-2012 CAD Contest in 

Fuzzy Pattern Matching for 

Physical Verification and 

Benchmark Suite 

J. Andres Torres 

ICCAD 2012 (November 5-8 th )

2 

Outline 

� The evolution of physical verification 

� State of the art results 

� Contest evaluation criteria 

ICCAD – November 2012 

© 2010 Mentor Graphics Corp. Company Confidential 

www.mentor.com

3 

Evolution in Physical Verification 

Percent of Total PV Activity 

100% 

90% 

80% 

70% 

60% 

50% 

40% 

30% 

20% 

10% 

0% 


Migration of Mfg Requirements Into Design 

90nm 65nm 45nm 28nm 20nm 

Technology Node 

Double Patterning (DP) 

Litho Simulation (LFD) 

Smart Fill 

CMP Simulation (CMPA) 

Critical Area Analysis (CAA) 

Recommended Rules/Scoring 

Mfg Required 

Mfg Recommended 

Traditional DRC 


www.mentor.com

4 

Requirements on physical verification 

� Be aware of anything that can go wrong in the fab and 

prevent certain structures to appear in the layout to 

maximize product yield. 

� Many types of errors come from the FAB: Rapid thermal 

anneal, etch and litho to name a few. 

� Geometric checks have been the workhorse in physical 

verification, and because they are intrinsically range 

constraints they are able to cover (and eliminate) a large 

number of difficult patterns. 

� Models have been used when traditional rules would need 

to be overly aggressive and thus open up more valid 

topologies that otherwise would need to be constrained. 



www.mentor.com

5 

Runtime Vs accuracy requirements 

� Geometric based rules are the fastest but may limit the 

design space too aggressively. 

� Model based verification is most accurate but it is limited 

to model availability and runtime. 

� Is there a way to reach a compromise to breach the gap 

between the two? 



www.mentor.com

6 

State-of-the-art results 

Drmanac et 

al, DAC09 

Torres et 

al, SPIE09 

Ding et al, 

ICICDT09 

MLK – ANN 

GD GD+LR 

� Desired target performance for a 28nm layout 

— Low false alarm: 80% 

— Portability: General calibration strategy for different types of 

patterns. 


www.mentor.com 

MLK – SVM 

GD GD+LR 

Avg Hhit 88% 80% 87% 88% 82% 89% 83% 

Avg Nhit 94.523% 99.985% 99.809% 99.847% 99.994% 99.895% 99.998% 

False alarm / mm 2 300K 1.1K 13.5K 10K 0.45K 7.5K 0.13K 

Avg CPU hour / mm 2 356 30 10 1.5 1.5 2.0 2.0 

Avg real hour / mm 2 100 8.50 2.80 0.40 0.40 0.52 0.52 

ICCAD – November 2012

7 

The importance of accuracy and portability 

Left: simulating “Ding et al, ICICDT09” ; 

Middle: reported from “Torres et al, SPIE09”; 

Right: current state of the art. 

� Open issues 

› Calibration is tedious and not general enough 

› Performance about twice as slow as minimum requirement. 

› Still large number of badly classified errors for other testcases 


1mm 


www.mentor.com

8 

Calibration test case description 

� For IP reasons only clips and areas around the hotspots 

and selected non-hotspot locations were selected and 

provided for calibrating the process. 


� The clip input layout as well 

as the frames around the 

“core” and frame context of 

the hotspot were provided. 

� The core was generated by a 

one ambit box around the 

hotspot. 

� The frame provided one and 

half ambits of context. 


www.mentor.com

9 

Test case description 

� Five test cases were provided for various metal layers 

which have a high content of 2D structures. Ranging from 

32 to 28nm processes. 

Testcase Calibration/ 

Blind size* 

[MB] 

Technology 

Type 1 

count** 


www.mentor.com 

Type 2 

count** 

Benchmark1 0.1/6.4 Interconnect 32nm 60/133 39/90 





*Blind size cases are in OASIS format in flat format 

** The count corresponds to the calibration and blind set 

ICCAD – November 2012

10 

Expected challenges 

� Find an efficient way to sample the layout to indentify the 

potential locations that cause problems 

� Calibration of locations where because of the hotspot 

shapes are not small, the underlying calibration pattern is 

wider (non square). 

� There a hundreds of millions of structures in the layout 

and yet the problem is that the amount of “bad” data in 

real production is really small. However is significant 

enough and the variants in layout so different that the 

contest is a test to evaluate the potential of pushing the 

state of the art to a region where methods can be useful. 



www.mentor.com

11 

Scoring: Ideal area 

False count 

12 

Scoring: Typical signatures in blind testcases 

False count 

Exact 

Pattern 

Matching 

Time 


Hit count 

Simulations 

� Exact pattern matching is very 

fast < 0.1 CPU-HRS/mm 2 with 

low false counts but for a 

complete blind testcase the hit 

count can be zero 

� Model based simulation is most 

accurate (golden reference 

which means hit count = 100% 

and false count zero), but with 

performance of the order of 

100 CPU-HRS/mm 2 

� The contest’s challenge is to 

move from the exact pattern 

matching range towards the 

ideal area. 


www.mentor.com

13 

Contest results 


� Some entries met the 

requirement of runtime less 

than 1 CPU-hr/mm 2 

� None of the contest entries met 

the requirement of greater than 

80% accuracy. 

� Most entries matched the Extra 

requirement of < 100 Counts 

per mm 2 but at the expense of 

heavy accuracy loss. 

� For that reason the following 

criterion was selected: 

— Best average accuracy (highest) 

— Followed by best extra (lowest) 

— The highlighted region shows the 

ideal zone which remains empty 

� Second and third place teams 

are defined by their average 

accuracy 


www.mentor.com

Team 

14 

Average method performance 

Runtime 

[CPU-hrs/mm 2 ] 

Hit Ratio 

[Hit/Total]*100% 

Extra 

[Count/mm 2 ] [Hit/Extra] 

21 0.74 29.05 562.95 0.052 

40 8.10 21.39 214.20 0.100 

31 0.56 19.80 362.30 0.055 

4 0.31 15.52 47.11 0.329 

17 1.48 10.87 0 N/A 

6 0.18 10.65 147.92 0.072 

32 25.60 0.41 186.08 0.002 

2 0.15 0.09 0.27 0.333 

� While team 40 has a better average [Hit/Extra] ratio, Team 21’s average accuracy and 

average runtime was better. 

� Team 17 decided to go for what appears an exact pattern matching approach. Meaning 

that they had difficulties identifying hotspots in the blind sections of the benchmarks. 



www.mentor.com

15 

Benchmark 5 Example: Team 21 (1 st place) 

Section containing 

calibration 


Full blind set 

� The primary goal of 

the contest was to 

investigate the 

possibility to predict 

patterns outside of 

the calibration set. 

� Team 21 did the 

best job in that 

regard. 


www.mentor.com

16 

Benchmark 5 Example: Team 40 (2 nd place) 


calibration 



� Assuming the same 

hit rate the 

reduction of extras 

is of paramount 

importance when 

using this method in 

real production. 

� Team 40 provided 

the best average 

hit/extra ratio of a 

blind dataset. 

� Unfortunately the 

accuracy was low. 


www.mentor.com

17 

Benchmark 5 Example: Team 31 (3 rd place) 


calibration 



� Team 31 was the 

next best accuracy 

with a ratio 

comparable to team 

21. 


www.mentor.com

18 

Benchmark 5 Example: Team 17 (best extra) 


calibration 



� Special mention to 

team 17 Whose 

method provided 

zero extras but at 

the expense of not 

predicting any 

configurations in 

the full blind areas 

of the benchmarks. 


www.mentor.com

19 

Acknowledgements 

� To all the teams for your hard work and dedication. Do 

not feel discouraged, this is an important problem looking 

for a solution, and I will be more than happy to test your 

code again. 

� To the Contest organizers for giving me this opportunity, 

it was a lot of work, but it was fun and worth doing. 



www.mentor.com


20 

THANK YOU 


www.mentor.com

3 rd Place Winner 

cada031: Extinguisher 

Chi-Yuan Liu, Sheng-Yen Chen, 

Iou-Jen Liu, and Prof. Yao-Wen Chang 

National Taiwan University

2 nd Place Winner 

cada040: UTDetector 

Bei Yu, Jhih-Rong Gao, 

and Prof. David Z. Pan 

The University of Texas at Austin

1 st Place Winner 

cada021: Iris’ Excalibur 

Geng-He Lin and Yen-Ting Yu 

National Chiao Tung University

Benchmark 5 Example

Create successful ePaper yourself

Delete template?

Save as template?