MSA Example: Attribute or Categorical Data - Juran Institute

MSA for Attribute or Categorical DataMSA Example:Attribute orCategorical DataAll Rights Reserved, Juran Institute, Inc.1#### XXXX.#### vx

MSA for Attribute or Categorical DataDefinitionsAccuracy: Overall agreement of the measured value with the true value (which may be an“expert” value). Bias plus precision.Attribute Data: Discrete qualitative data.Attribute Measurement System: Compares parts to a specific set of criteria and accepts theitem if the criteria are satisfied.Bias: A systematic difference from the true value. Revealed in the differences in averages fromthe true value.Precision: Variation in the measurement process.R&R: Repeatability and Reproducibility. Two elements of precision.Repeatability: The variation observed when the same operator measures the same itemrepeatedly with the same device.Reproducibility: The variation observed when different operators measure the same parts usingthe same device, sometimes it can be the same operator using different devices.The list provides a quick reference for key terms used in Measurement System Analysis.All Rights Reserved, Juran Institute, Inc.2#### XXXX.#### vx

MSA for Attribute or Categorical DataThe Fundamental MSA Question“Is the variation (spread) of the measurement systemtoo large to study the current level of processvariation?”Product VariabilityProcess VariabilityVariationin themeasurementprocess(Observed Variability)Total Variability+ =MSA for Continuous Processes 3 .PPTAll Rights Reserved, Juran Institute, Inc.Quantifying VariationLike all processes, the measurement process has CTQs. The graph above listssome of the most common CTQs used for the measurement process. MSAquantifies the amount of variation for:• Accuracy• Repeatability• Reproducibility• Stability (this is typically covered in the Black Belt workshop)• Linearity (covered in Black Belt workshops)All Rights Reserved, Juran Institute, Inc.3#### XXXX.#### vx

MSA for Attribute or Categorical DataBiasTrueValue orStandardBiasObservedAveragePossible Causes of Bias• Sensor not properlycalibrated• Improper use of sensor• Unclear procedures• Human limitationsMSA for Continuous Processes 4 .PPTAll Rights Reserved, Juran Institute, Inc.BiasBias is the difference between the observed average of measurements and the trueaverage. Validating accuracy is the process of quantifying the amount of bias in themeasurement process. Experience has shown that bias and linearity are typicallynot major sources of measurement error for continuous data, but they can be.In service and transaction applications, evaluating bias most often involves testingthe judgment of people carrying out the measurements.‣ExampleA team wants to establish the accuracy of its process to measure defects ininvoices. First, they gather a “standard” group of invoices and have an “expert”panel establish the type and number of defects in the group. Next, they have thestandard group of invoices measured by the “normal” measurement process.Differences between averages the measurement process came up with, and whatthe known defect level was from the expert panel represented the bias of themeasurement process.All Rights Reserved, Juran Institute, Inc.4#### XXXX.#### vx

MSA for Attribute or Categorical DataRepeatabilityPossible Causes of PoorRepeatabilityEquipment• Gage instrument needsmaintenance• The gage needs to be morerigidRepeatabilityPeople• Environmental conditions(lighting, noise)• Physical conditions (eyesight)MSA for Continuous Processes 5 .PPTAll Rights Reserved, Juran Institute, Inc.RepeatabilityRepeatability is the variation in measurements obtained when one operator usesthe same measurement process for measuring the identical characteristics of thesame parts or items.Repeatability is determined by taking one person, or one measurement device, andmeasuring the same units or items repeatedly. Differences between the repeatedmeasurements represent the ability of the person or measurement device to beconsistent.Possible causes of the lack of repeatability are listed on the slide.All Rights Reserved, Juran Institute, Inc.5#### XXXX.#### vx

MSA for Attribute or Categorical DataReproducibilityMean ofthe measurementsof Operator BMean ofthe measurementsof Operator APossible Causes of PoorReproducibility• Measurement procedure is notclear• Operator is not properly trainedin using and reading gageReproducibility• Operational Definitions notestablishedMSA for Continuous Processes 6 .PPTAll Rights Reserved, Juran Institute, Inc.ReproducibilityReproducibility is very similar to repeatability. The only difference is that instead oflooking at the consistency of one person, you are looking at the consistencybetween people.Reproducibility is the variation in the average of measurements made by differentoperators using the same measurement process when measuring identicalcharacteristics of the same parts or items.Possible causes of poor reproducibility include: measurement process is not clear,operator not properly trained in using the measurement system, and operationaldefinitions are not clear nor well established.All Rights Reserved, Juran Institute, Inc.6#### XXXX.#### vx

MSA for Attribute or Categorical DataAttribute Measurement Systems Study• Discrete qualitative data• Go/no-go basis; or limited data categories• Compares parts to specific criteria for accept/not accept orto be placed in category• Must screen for effectiveness to discern good parts frombad• At least two appraisers and two trials each• If available, have Quality Master rate parts firstMSA for Continuous Processes 7 .PPTAll Rights Reserved, Juran Institute, Inc.Listed here are the key highlights of conducting an MSA for attribute or categoricaldata. The “parts” can be invoices, parts or reason codes for customer returns, forexample.All Rights Reserved, Juran Institute, Inc.7#### XXXX.#### vx

MSA for Attribute or Categorical DataAttribute MSA StudyAttribute MSA ExampleAppraiser A Appraiser BMaster 1 2 1 21 G G G G G2 G G G G G3 G NG G G G4 NG NG NG NG NG5 G G G G G6 G G G NG G7 NG NG NG G NG8 NG NG NG G G9 G G G G G10 G G G G G11 G G G G G12 G G G G G13 G G NG G G14 G G G G G15 NG G G G G16 G G G G G17 G G G G G18 G G G G G19 G G G G G20 NG G G G GG = GoodNG = Not GoodMSA for Continuous Processes 8 .PPTAll Rights Reserved, Juran Institute, Inc.This shows the results of 2 rounds using 2 appraisers, assessing the same 20items.All Rights Reserved, Juran Institute, Inc.8#### XXXX.#### vx

MSA for Attribute or Categorical DataChallenges of Continuous Process MSA• MSA study is an experiment• Requires two or more trials for calculating Repeatability• Needs a way to present the inspection units to the appraisermultiple times• Is not possible within the continuous processMSA for Continuous Processes 9 .PPTAll Rights Reserved, Juran Institute, Inc.When conducting an MSA for a continuously running process, parts should betaken off-line to conduct the MSA study.All Rights Reserved, Juran Institute, Inc.9#### XXXX.#### vx

MSA for Attribute or Categorical DataCase Example: Visual Inspection of GlassPackersCutterCatwalkInspectorGlassMSA for Continuous Processes 10 .PPTAll Rights Reserved, Juran Institute, Inc.The example given here is that of visual inspection of glass.All Rights Reserved, Juran Institute, Inc.10#### XXXX.#### vx

MSA for Attribute or Categorical DataCase Example: Challenges to Overcome• Bias to the standard could be evaluated on-line.• Repeatability and Reproducibility (R & R) could not beevaluated on-line.• A method had to be devised to allow the inspectors to viewthe same pieces of glass repeatedly.• The solution was an off-line conveyor which simulated theon-line condition as closely as possible.MSA for Continuous Processes 11 .PPTAll Rights Reserved, Juran Institute, Inc.All Rights Reserved, Juran Institute, Inc.11#### XXXX.#### vx

MSA for Attribute or Categorical DataCase Example: Attribute MSA Method Employed• 20 pieces of glass from the process that included both goodand bad samples were selected.• A team of people well versed in the quality standardclassified each piece of glass as either “pass” or “fail.”• All regular inspectors independently evaluated each piecetwice (in random order).• The inspectors used a log sheet to record the data.• Minitab ® was used to analyze the data.MSA for Continuous Processes 12 .PPTAll Rights Reserved, Juran Institute, Inc.There were two outcomes in this inspection or measurement process: pass or fail.Twenty pieces, a team of inspectors, and two rounds (or trials) were used in theMSA.All Rights Reserved, Juran Institute, Inc.12#### XXXX.#### vx

MSA for Attribute or Categorical DataCase Example: Attribute MSA Study DataExcerpt offull datafor 20inspectorsMSA for Continuous Processes 13 .PPTAll Rights Reserved, Juran Institute, Inc.This slide shows the data in MINITAB ® . The “Standard” column documents thecorrect or expert answer for each piece of glass.All Rights Reserved, Juran Institute, Inc.13#### XXXX.#### vx

MSA for Attribute or Categorical DataCase Example: Attribute MSA Study ResultsAssessment AgreementDate of study : 9/13 and 9/19/07Reported by: Bill Sunderland and Bob PanatieriName of product: SolarMisc:Mainline MarkingWithin AppraisersAppraiser vs Standard1009095.0% C IPercent1009095.0% C IPercent8080Percent7060Percent7060505040TonyM ollyTimEarlStev eJeannieAppraiserJohnRonnieC lintLarry40TonyM ollyTimEarlStev eJeannieAppraiserJohnRonnieC lintLarryMSA for Continuous Processes 14 .PPTAll Rights Reserved, Juran Institute, Inc.The graph on the left shows the agreement (or repeatability) of each appraiserbetween Trial 1 and Trial 2.The graph on the right shows the agreement of each appraiser with the Standard.As shown on both graphs, the blue dots show the percent agreement and theredlines are the 95% confidence intervals.All Rights Reserved, Juran Institute, Inc.14#### XXXX.#### vx

MSA for Attribute or Categorical DataCase Example: Attribute MSA Study Results (continued)MSA for Continuous Processes 15 .PPTAll Rights Reserved, Juran Institute, Inc.This slide shows the Within Appraiser agreement.For example, Larry scored 100% - Trial 1 and Trial 2 are in full agreement.On the other hand, Allen scored only 50% agreement – there is only a 50%agreement between his Trial 1 and Trial 2 measurements. Or he disagrees withhimself half the time!All Rights Reserved, Juran Institute, Inc.15#### XXXX.#### vx

MSA for Attribute or Categorical DataCase Example: Attribute MSA Study Results (continued)MSA for Continuous Processes 16 .PPTAll Rights Reserved, Juran Institute, Inc.This slide shows the agreement of each Appraiser (across both trials) with theStandard.For example, Larry has 89% agreement with the Standard but, Allen only has 39%agreement with the Standard.All Rights Reserved, Juran Institute, Inc.16#### XXXX.#### vx

MSA for Attribute or Categorical DataCase Example: Attribute MSA Study Results (continued)MSA for Continuous Processes 17 .PPTAll Rights Reserved, Juran Institute, Inc.This shows the level of agreement across all Appraisers. In this case, only 5.56%agreement!All Rights Reserved, Juran Institute, Inc.17#### XXXX.#### vx