Project Management Embedded Failures (2) Jack Ganssle
Project Management Embedded Failures (2) Jack Ganssle
Project Management Embedded Failures (2) Jack Ganssle
You also want an ePaper? Increase the reach of your titles
YUMPU automatically turns print PDFs into web optimized ePapers that Google loves.
<strong>Project</strong> <strong>Management</strong><br />
<strong>Embedded</strong> <strong>Failures</strong> (2)<br />
<strong>Jack</strong> <strong>Ganssle</strong><br />
jack@ganssle.com
The Tacoma Narrows Bridge<br />
The Tacoma Narrows Bridge<br />
4 months after opening, Nov 7, 1940<br />
jack@ganssle.com
Costs<br />
George Golden Bronx- Tacoma<br />
Washington Gate Whitestone Narrows<br />
Completed 1935 1937 1939 1940<br />
Span 3500 ft 4200 ft 2300 ft 2800 ft<br />
Cost $59.5m $35m $19.7m $6.4m<br />
jack@ganssle.com
Titan IVb Centaur<br />
$500m<br />
jack@ganssle.com
Ariane 5<br />
$500m<br />
jack@ganssle.com
Chinook<br />
27 dead<br />
jack@ganssle.com
Patriot Missile<br />
28 dead, 100 injured<br />
jack@ganssle.com
Therac 25<br />
3 dead<br />
jack@ganssle.com
Radiation Deaths in Panama<br />
21 dead<br />
jack@ganssle.com
Our Criminal Behavior<br />
No Code Inspections<br />
Implicated in the Chinook helicopter, Multidata<br />
Radiotherapy device, Therac 25.<br />
Average uninspected code contains 50-100 bugs<br />
per 1000 LOC. Inspections find most of these.<br />
Cheaply.<br />
jack@ganssle.com
80<br />
70<br />
60<br />
% bugs<br />
found<br />
50<br />
40<br />
30<br />
20<br />
10<br />
0<br />
0 100 200 300 400 500 600 700 800<br />
Lines of code inspected per hour<br />
jack@ganssle.com
Our Criminal Behavior<br />
Inadequate testing<br />
Implicated in the Clementine, NEAR, Mars<br />
Polar Lander, Pathfinder, Mars Expedition<br />
Rover, Titan IVb, Ariane, Sea Launch, Chinook,<br />
Therac 25, Multidata, pacemakers, Los Alamos<br />
incident, huge digital thermometer.<br />
jack@ganssle.com
Our Criminal Behavior<br />
Too expensive to change (dead product)!<br />
Cost<br />
to<br />
update<br />
the<br />
software<br />
Changes slow the<br />
business down!<br />
Cleaning up:<br />
cost = r (“refactoring to clean code”)<br />
cumulative cost = 1+r+r+r+...<br />
= 1 + n*r<br />
Not cleaning up:<br />
cost = m (“dealing with the mess”)<br />
cumulative cost = (1)(1+m)(1+m)(1+m)... = (1 + m) n<br />
Time (actually, number of changes made)<br />
jack@ganssle.com
Our Criminal Behavior<br />
Lousy exception handlers<br />
Implicated in the Ariane, Los Alamos incident,<br />
Clementine, Yorktown, Mars Expedition Rover,<br />
and many others<br />
This means adopting a culture of anticipating<br />
and planning for failures!<br />
jack@ganssle.com
Our Criminal Behavior<br />
Ignoring or cheating the VCS<br />
Implicated in the NEAR, Pathfinder, Titan IVb,<br />
EFF, and FAA incidents.<br />
Find a list of VCSes at:<br />
www.codeorganizer.com/version_control/tools.htm<br />
jack@ganssle.com
Our Criminal Behavior<br />
The use of C!<br />
jack@ganssle.com
jack@ganssle.com
Bug Rates<br />
C (worst)<br />
500 bugs/KLOC<br />
C (average)<br />
50-100 bugs/KLOC<br />
C (auto generated) 12.5<br />
Ada (worst) 50<br />
Ada (average) 25<br />
Ada (auto generated) 4.8<br />
SPARK (average) 4<br />
jack@ganssle.com
Our Criminal Behavior<br />
“The major objection from experienced<br />
programmers was that the compiler could not<br />
possibly turn out object code as good as theirs.”<br />
The real issue: is the code good enough<br />
jack@ganssle.com
Our Criminal Behavior<br />
Fact: You can write crappy code in any<br />
language.<br />
jack@ganssle.com
Maintainability: C vs C++<br />
40% of C bugs took more than 2 hrs to fix;<br />
70% of C++ bugs took more than 2 hours.<br />
jack@ganssle.com
Primary Language Used<br />
jack@ganssle.com
Our Criminal Behavior<br />
Not using the tools we already have to tame C<br />
and C++<br />
MISRA - www.misra.org.uk<br />
Lint - www.gimpel.com, or<br />
www.splint.org<br />
Safer C - www.leshatton.org/index_SA.html<br />
Standards – www.ganssle.com<br />
Static test- complexity and quality analyzers<br />
jack@ganssle.com
Our Criminal Behavior<br />
Using lousy watchdogs<br />
state = 0x5555<br />
Call WDT_A<br />
•<br />
•<br />
•<br />
•<br />
state+= 0x2222<br />
Call WDT_B<br />
WDT_A:<br />
WDT_B:<br />
halt if state!= 0x5555<br />
state+=0x1111<br />
return<br />
halt if state!= 0x8888<br />
state=0<br />
kick dog via two I/Os or MMU<br />
return<br />
jack@ganssle.com
Better Watchdogs<br />
Maxim MAX6323, TI TPS3813<br />
jack@ganssle.com
The Boss’s Criminal Behavior<br />
Reuse is not a panacea<br />
Implicated in the Ariane, Uwatec and many<br />
others.<br />
Reuse is extremely difficult.<br />
See “Confessions of a Used Program Salesman”<br />
by Will Tracz<br />
jack@ganssle.com
The Boss’s Criminal Behavior<br />
Reuse is poison!<br />
jack@ganssle.com
We don’t practice reuse<br />
jack@ganssle.com
What Does Reuse Mean<br />
Reuse: building projects one component at a<br />
time, instead of one line at a time.<br />
Software Salvaging: Using code not designed<br />
to ever be reused<br />
Carrying-over Code – Porting code from an<br />
old project to a new one<br />
jack@ganssle.com
Reuse Realities<br />
Reuse requires:<br />
• A will to reuse<br />
• A well-documented set of code<br />
• Code that has no dependencies<br />
• An available set of code<br />
jack@ganssle.com
No Free Lunch<br />
There’s No Such Thing As A Free Lunch when<br />
it comes to software reuse.<br />
1. Before you can develop code for reuse you<br />
must have developed it at least 3 times.<br />
2. Before you can reap the benefits of reuse<br />
you must have reused it 3 times.<br />
jack@ganssle.com
Griss’s Levels of Software Reuse<br />
Maturity Level<br />
Initial<br />
Salvaging<br />
Planned reuse<br />
Systemic reuse<br />
Level of Reuse<br />
-20% to 20%<br />
10% to 50%<br />
30% to 40%<br />
50% to 70%<br />
Requirements<br />
Biz as usual<br />
Smart people<br />
Depends on luck<br />
Smart people<br />
Maintenance probs<br />
Reuse library<br />
<strong>Management</strong> support<br />
Incentives<br />
Reuse library<br />
Reuse process<br />
Reuse metrics<br />
Education<br />
Domain-oriented reuse<br />
80% to 90%<br />
Domain analysis<br />
Application generators<br />
jack@ganssle.com
Reuse Realities<br />
I/O<br />
Application<br />
Algorithms<br />
jack@ganssle.com
The Boss’s Criminal Behavior<br />
140<br />
120<br />
Schedules can’t rule:<br />
100<br />
80<br />
60<br />
40<br />
20<br />
Corollary: Tired people make mistakes<br />
Implicated in the Clementine, NEAR, Mars<br />
Polar Lander and many others<br />
0<br />
0 0.2 0.4 0.6 0.8 1 1.2<br />
jack@ganssle.com
Not tracking schedules<br />
jack@ganssle.com
Not controlling schedules<br />
1986 2001<br />
Cost $12.6b $28.7b<br />
Schedule 9.4 yrs 19.2 yrs<br />
jack@ganssle.com
The Boss’s Criminal Behavior<br />
Be wary of financial shortcuts!<br />
Implicated in the Takoma Narrows Bridge,<br />
Ariane, MGM fire, and many others<br />
jack@ganssle.com
Are we criminals<br />
Or are we still in the dark ages<br />
But there’s a lot we do know, so we’re<br />
negligent – and will be culpable – if<br />
we don’t consistently use best<br />
practices.<br />
jack@ganssle.com
Remember!<br />
Enter the evaluation form and be a part of making Øredev even better.<br />
You will automatically be part of the evening lottery<br />
jack@ganssle.com
Change language