Presentation

Improving subjective estimationsusing the paired comparisonsmethodInternational Forum on COCOMOandSoftware Cost EstimationEduardo MirandaOctober 24-27, 2000Ericsson Research Canada October 2000 E. Miranda, COCOMO/SCM 15

Agenda• Subjective estimates• Estimating person-hours• Can we do better? - The paired comparison method• The process• Example• Evaluation• Why does it work?• Summary & further researchEricsson Research Canada October 2000 E. Miranda, COCOMO/SCM 15

Subjective estimations are a way of life formany activities and organizations• Lack of necessary information at the beginning of theproject• Vocabulary foreign to the stakeholders• Some techniques are domain specific• Require significant time and effortEricsson Research Canada October 2000 E. Miranda, COCOMO/SCM 15

Estimating person-hours2.521.510.50-0.5-1LMC-DurationLMC-MhrsLMC-FPTEI-DurationTEI-MhrsTEI-FP-1.5Ericsson Research Canada October 2000 E. Miranda, COCOMO/SCM 15

Can we do better? - The paired comparisonmethod500450400350300LOC250200150100500stack queue binarytreeActuallinkedlist (a)Paired comparisons (numeric scale)reference(string)moduleslinkedlist (b)balancedtreehashtableFinger in the wind (Ad-hoc)Paired comparisons (verbal scale)Ericsson Research Canada October 2000 E. Miranda, COCOMO/SCM 15

The paired comparisons processArtifacts tobe sizedRank the artifactsfrom largest tosmallerOrderedListCompare theartifacts pairwiseestablishing theirrelative sizeJudgementMatrixReview internalinconsistenciesDefinition Explanation Relative ReciprocalValueEqual size Ei / E ≤ j 1.251 1(0~25%)Slightly bigger 1.25< Ei / ≤ Ej 1.751.15 .87(smaller)(25 ∼ 75%)Bigger (smaller) 1.75 < Ei / Ej ≤ 2.275(75 ∼ 1.5 .66275%)Much Bigger 2.275 < E i / E j ≤ 5.753 .33(smaller)(275 ∼ 575%)Extremely bigger 5.75 < Ei / ≤ Ej 106 .16(smaller)(575 ∼ 1000%)Verbal Scale (optional)Calculate ratioscale &InconsistencyIndexRatioScaleReferenceArtifact(s)Calculateabsolute sizesSizedArtifactsEricsson Research Canada October 2000 E. Miranda, COCOMO/SCM 15

The math - judgement matrixArtifacts g a b c d I j h e fg 2 2.2 3 3 4 4.5 6 7 9a 1 1 2.2 3 3 3 5 6b 1.1 1.2 1.3 5 6 6 7c 2 2 2 2.3 4.5 5d 1 1.5 2 3.4 4I 3 3 3.3 4j 4 4 4.5h 1 2e 2fAnxn=[ a ]ij⎧⎪a⎪⎪⎪⎪= ⎨a⎪⎪⎪⎪⎪a⎩ijiiji==ss= 1ij1aij, How much bigger (smaller) is E, Every element has the same size as itself, If Eiis a(bigger) than Eijwithrespect to Etimes bigger (smaller) than E , then Eiijjjis1/aijtimes smallerEricsson Research Canada October 2000 E. Miranda, COCOMO/SCM 15

The math - ratio and size calculations(a)(c)νi⎡⎢s⎢⎢s⎢⎢⎢⎢⎢⎢s⎣=r*rr*rkk12skr*rknkknkn∏j=1⎤⎥⎥⎥⎥⎥⎥⎥⎥⎥⎦aij(d)InconsistencyIndex=(b)nn∑∑i= 1 j>iri=⎛⎜lna⎝( n − 1)( n − 2)2n∑l=1ijνiνlν− lnνij⎞⎟⎠2Crawford and Williams’ procedure. (a) Row’s geometric mean, (b) Ratio scale,(c) Size vector, (d) Inconsistency IndexEricsson Research Canada October 2000 E. Miranda, COCOMO/SCM 15

Verbal scalesDefinitionSoftware RelativeValueAHP RelativeValuesEqual size 1 1Slightly bigger (smaller) 1.25 3Bigger (smaller) 1.75 5Much Bigger (smaller) 4 7Extremely bigger (smaller) 7.5 9Ericsson Research Canada October 2000 E. Miranda, COCOMO/SCM 15

MinimumTime - automating the processArtifacts tobe sizedRank the artifactsfrom largest tosmallerCompare theartifacts pairwiseestablishing theirrelative sizeReview internalinconsistenciesReferenceValue(s)SizedArtifactsEricsson Research Canada October 2000 E. Miranda, COCOMO/SCM 15

ExampleBGFCDEIJEricsson Research Canada October 2000 E. Miranda, COCOMO/SCM 15

Evaluation - data structures500450400350300LOC250200150100500stack queue binarytreeActuallinkedlist (a)Paired comparisons (numeric scale)reference(string)moduleslinkedlist (b)balancedtreehashtableFinger in the wind (Ad-hoc)Paired comparisons (verbal scale)Ericsson Research Canada October 2000 E. Miranda, COCOMO/SCM 15

Evaluation - geometric figures120100806040200Actual Ad-hoc paired comparison verbal scaleEricsson Research Canada October 2000 E. Miranda, COCOMO/SCM 15

Evaluation - geometric figures9080706050403020100reference Ad-hoc paired comparison verbal scaleEricsson Research Canada October 2000 E. Miranda, COCOMO/SCM 15

Evaluation - accuracyAre all the methods the same ?SUMMARYGroups Count Sum Average Variancefiw 10 370 37 157.64pc 9 151.7 16.855556 168.11278vs 7 168.3 24.042857 20.599524ANOVASource of Variation SS df MS F P-value F critBetween Groups 1981.862 2 990.93109 7.8937886 0.002454 3.42213Within Groups 2887.259 23 125.53302Are there any differences between PC and VS?vs pcMean 24.04286 16.85556Variance 20.59952 168.1128Observations 7 9Hypothesized Mean Difference 0df 10t Stat 1.545675P(T

Why does it work?• Differences rather than absolute values• Explicit paired comparisons• Redundant values• The result is an average of many individual observationsEricsson Research Canada October 2000 E. Miranda, COCOMO/SCM 15

Summary and further research• The paired comparisons method provides an structuredapproach for doing subjective estimations which is:– more precise– more accuratethan ad-hoc approaches.• Future research:– Corroboration of results– Validating software verbal scale– Application into domains others than software sizing (effort, leadtime)Ericsson Research Canada October 2000 E. Miranda, COCOMO/SCM 15

References• G. Bozoki, An expert judgement based software sizingmodel, Lockheed Missiles & Space Company and TargetSoftware• J. Karlsson & K. Ryan, A Cost-Value Approach forPrioritizing Requirements, IEEE Software,September/October 1997• T. Saaty, Multicriteria Decision Making: The AnalyticHierarchy Process, RWS Publications, 1996• G. Crawford and C. Williams, The Analysis of subjectivejudgement matrices, Rand Corporation, 1985Ericsson Research Canada October 2000 E. Miranda, COCOMO/SCM 15

The EndEricsson Research Canada October 2000 E. Miranda, COCOMO/SCM 15