How to Verify Performance Specifications - COLA

LabGuide 13How to VerifyPerformance SpecificationsVERIFICATION OF PERFORMANCE SPECIFICATIONSIn 2003, the Centers for Medicare and Medicaid Services(CMS) updated the CLIA ’88 regulations. As a result ofthe updated CLIA regulations, laboratories need to verifythe performance specifications of all nonwaived, FDAapproved,unmodified tests added to the laboratorymenu on or after April 24, 2003. Before April 24, 2003,this requirement only applied to new high complexitytests. Now, before reporting patient results, you mustperform this verification study:• Whenever you add a new instrument or method,including multiple versions of the same instrumentor method• When you add a new analyte to your menu thatwill be performed on an existing instrument• When switching methods or reagent manufacturerfor an analyte already on your menu• Before putting a “loaner” instrument in use*• After relocating an existing instrument to a newenvironment**When any of the above situations occur, CLIA requiresyou to verify the manufacturer’s stated performancespecifications (in the package insert) for accuracy, precision,reportable range, and patient reference range forthat test before you report patient results. The verificationprocess confirms that the instrument and/or testmethod performs as the manufacturer intended whenutilized in your laboratory environment, with your testingpersonnel, for your patient population.CMS did not set specific requirement details or offer proceduresthat tell you how to perform this verification. Itis up to the laboratory director to establish appropriateprocedures. The manufacturer can assist you by providingmaterials, procedures, and statistical analysis, butthey may not perform the actual testing of the samplesused in the verification process for you.Note: This LabGuide only addresses FDA-approved,unmodified tests. For tests that are not FDA-approved, orfor tests that are FDA-approved but modified by the laboratory,you must establish performance specificationsfor the test to validate the method before reportingpatients. Establishing performance specificationsinvolves additional steps and will be covered in a separateLabGuide.WHAT PERFORMANCE SPECIFICATIONSNEED TO BE VERIFIED?For unmodified, FDA-approved tests, you must verifyaccuracy, precision, reportable range, and referenceranges for quantitative tests (tests that give a numericalresult). If the test is qualitative (a test that gives a negativeor positive result), only verification of accuracy isapplicable.Definitions:Accuracy is how close a test result is to the true value.You must verify that the test gives correct results in your* If the loaner instrument is the exact same make, model, and test method, and will be used to test the same analytes, then this requirementcan be reduced. You may verify comparable performance by testing and obtaining acceptable results on at least 2 levels of controlsAND either previously tested PT samples or previously tested patient samples.**Only relocate an instrument to a new location that meets the manufacturer’s requirements for environmental conditions. Consider thepossibility that the new location has environmental conditions that are different enough to impact instrument performance. The newlocation may have different electrical circuitry, potential interference, and changes in temperature, humidity, and airflow. If after relocationcomparable results are obtained on previously tested specimens, QC results are in range, and the laboratory determines that theyhave adequately demonstrated that performance specifications are not affected, then a full new performance verification study wouldnot be required.13-1© COLA--7/07. COLA LabGuide® is a registered trademark of COLA.

LabGuide 13Verification ProcedureBefore beginning, the instrument must be properlyinstalled in its permanent location and calibrated.Definitions, formulas, example calculations, and a worksheetare provided at the end of this LabGuide.Step 1Start with precision. Test at least five repetitionsof at least two samples with different values. Donot use a zero value sample when verifying precisionor accuracy. Test the samples:• In the same run (2 of the 5 repetitions)• In a different run on the same day (3rd repetition),and• On different days (4th and 5th repetitions performedon different days)If possible, have different staff perform the 3rd,4th and 5th repetitions to check for operator variance.(This may not be critical if the test uses afully automated test system that is not operatordependent.)To assess precision, calculate the mean, standarddeviation (SD), and coefficient of variation (CV) foreach set of results. Typically, you will find a differentCV at different ranges for the analyte. Comparethe CVs obtained to that stated in manufacturer’sliterature regarding performance specifications.The smaller the CV, the more precise the test.Step 2The results obtained in the precision study can beused to assess accuracy.• Compare the mean of the five repetitions foreach sample to the known value of the sample• Determine the difference between the two values,and compare to the limits of acceptabilitystated in the package insert for known samples,or use allowable Proficiency Testingscoring limits (found in the PT SummaryReport).Another simple way to evaluate the results is tocalculate a ratio of means for comparison. Dividethe mean you obtained by the known value andmultiply by 100. The closer the result is to 100, themore accurate the test.Step 3It is important to remember that reportable rangedefines the span of values for which the laboratorycan report results. If you used samples with a verylow value and a very high value for Step 1 (precision)and Step 2 (accuracy) and the results wereacceptable, then you have verified the reportablerange. If however, the samples used do not coverthe full range of values for which you expect toreport results (for instance, you might have usedQC or calibrators for Step 1 & 2), you will need totest at least one repetition of two additional sampleswith known values - one at the low end andone at the high end of the stated reportable range.If a zero calibrator is available for the analyte, thenyou may extend the low end of your reportablerange down to zero, provided you actually obtain azero result when you test that sample. Use thesame process for evaluating accuracy to evaluateyour results for reportable range.If you are using the same samples to verify accuracy,precision, and reportable range, COLA recommendsthat you include a third sample with a valuein the middle of the range (where your normalpatients are expected to fall.) By doing this andobtaining acceptable results, you have also performedan initial calibration verification.Step 4You may use the manufacturer’s suggested referencerange for the test when you first beginpatient testing. The laboratory is required to monitorthe applicability of this normal range andmake adjustments as necessary. The goal is to usea reference range within which the majority ofhealthy individuals without the disease will fall,without including individuals with the disease.Perform this simple test to compare the manufacturer’sreference range to data obtained when testingnormal patients:• Collect specimens from a minimum of 10 normalpatients.• Test each specimen only once, and spread thetesting over a minimum of three days.• Add up the values obtained for each sampleand divide by 10 to calculate the normalpatient “mean”. Calculate the standard devia-13-4© COLA--7/07. COLA LabGuide® is a registered trademark of COLA.

LabGuide 13tion (SD). Establish the range of plus andminus two SD from the mean.• Compare this range to the manufacturer’srange.Interpretation: If the lab’s range falls within themanufacturer’s range, the reference range is comparable.Perform the test again when you haveresults for 20 or 30 normal patients to see if thelab’s range still falls within the manufacturer’s referencerange when you have a larger pool of data.Continue to perform this check periodically as partof the lab’s quality assessment program.In the event the ranges obtained above are notcomparable, your lab will need to establish anappropriate reference range for your patient population.Document all activities involved in verifying performancespecifications. Retain this documentation for as long asthe test method is in use and for 2 years after discontinuingthe method.For COLA laboratories, the new COLA criteria, whichreflect the revised CLIA requirements for verificationof performance specifications, go into effect June 1,2007, not April 24, 2003.Beginning with the implementation of the new COLAcriteria on June 1, 2007, COLA labs must perform verification(or establishment) of performance specificationsfor all new nonwaived (moderate and high complexity)tests introduced on or after June 1, 2007.COLA labs should complete the following sections ofthe COLA Self-Assessment to ensure compliance withall the requirements associated with Verification ofPerformance Specifications: VER 1-4, and VER12-14Document all activitiesinvolved in verifyingperformancespecifications. Retainthis documentation foras long as the testmethod is in use and for2 years afterdiscontinuing themethod.RESOURCES• If you are adding testing in a new specialty or subspecialty,see LabGuide 12 for additional importantinformation.• See CLIA Facts 2 and 16B for details on the CLIAregulations for verification of performance specifications.• CMS has published Brochure # 2 called Verificationof Performance Specifications that can be viewedon-line at:www.cms.hhs.gov/CLIA/05_CLIA_Brochures.asp13-5© COLA--7/07. COLA LabGuide® is a registered trademark of COLA.

LabGuide 13DEFINITIONS, FORMULAS,AND EXAMPLE CALCULATIONSDefinitions, formulas, and example calculations are providedbelow to assist you in completing the verificationof performance specifications process.Mean (x): The mean is the average of the results. It is thesum of all results for the sample divided by the number(n) of results:x = (x 1 + x 2 + …..x n ) ÷ nExample: You have 5 results for a sample: 5.0, 4.7, 5.1,4.9, and 4.8(5.0 + 4.7 + 5.1 + 4.9 + 4.8) ÷ 524.5 ÷ 5 = 4.9The mean is 4.9Standard deviation (SD): Standard deviation (SD) is apredictable measure of variation around the mean. Theless variation around the mean, the closer the SD is tothe ideal of zero. Traditionally, 2 standard deviationsfrom the mean is usually considered acceptable variation.SD is the square root of the sum of the squared differencesof each result from the mean, divided by n – 1:Σ (X – X) 2n – 10.10 ÷ 4 = 0.25Complete the calculation by determining the square rootof .025:0.25 = 0.158The SD for these values is 0.158To determine a 2 SD range, subtract the determined SDtwice from the mean to get the lower limit and add thedetermined SD twice to the mean to get the upper limit:4.9 – 0.158 – 0.158 = 4.5844.9 + 0.158 + 0.158 = 5.216Since the test is reported to 1 decimal place, we roundoff our calculations to arrive at a 2SD range of:4.6 – 5.2Coefficient of variation (CV): The coefficient of variationis the standard deviation of a set of results divided bythe mean result. The smaller the CV, the better, withresults below 15% generally considered good target valuesfor most test methods (check the manufacturer’s literature).CV is expressed as a percentage or as a decimalfraction less than 1. Multiply the decimal fraction by 100to get the CV in percent.For our example, .158 ÷ 4.9 = 0.0322 x 100 = 3.2%Here, we continue with our example. To calculate the SD,first we need to calculate x – x for each result (withoutregard to positive or negative), then square those resultsand add them up:(5.0–4.9) 2 + (4.7–4.9) 2 + (5.1–4.9) 2 + (4.9–4.9) 2 + (4.8 – 4.9) 25 – 15.0 – 4.9 = .1 .1 2 = 0.014.7 – 4.9 = .2 .2 2 = 0.045.1 – 4.9 = .2 .2 2 = 0.044.9 – 4.9 = 0 0 2 = 04.8 – 4.9 = .1 .1 2 = 0.01.01 + .04 + .04 + 0 + .014Then divide the total of 0.10 by 4 (n – 1):0.10413-6© COLA--7/07. COLA LabGuide® is a registered trademark of COLA.

LabGuide 13VERFICATION OF PEFORMANCE SPECIFICATIONS WORKSHEETNote: Modify this worksheet to test more samples or additional repetitions, as determined by your laboratory.Date ________________ Analyte/Test Method __________________________________ Initials ______Qualitative Test:Day 1- Known Negative samples: #1 ________ ________ #2 ________ ________Known Positive samples: #1 ________ ________ #2 ________ ________Day 2- Known Negative samples: #1 ________ ________ #2 ________ ________Known Positive samples: #1 ________ ________ #2 ________ ________Conclusion ____________________________________________________________________________________________________________________________________________________________________Comments ____________________________________________________________________________________________________________________________________________________________________Quantitative Test:Samples tested ______________________________ Known values #1 ________ #2 ________PrecisionDay 1:2 repetitions of each sample in same run 1 repetition in different run#1 _____/_____ #2 _____/_____ By ____________ #1 _____ #2 _____ By ____________Day 2: 1 repetition of each sampleDay 3: 1 repetition of each sample#1 _____ #2 _____ By ____________ #1 _____ #2 _____ By ____________CalculationsMean (x) = (x 1 +x 2 +….x n ) ÷ n (for this worksheet, n = 5)Sample #1 Sample #2x 1 _________ _________x 2 _________ _________x 3 _________ _________x 4 _________ _________x 5 _________ _________Sum of results _________ _________÷ n = (x)_________ _________13-7© COLA--7/07. COLA LabGuide® is a registered trademark of COLA.

LabGuide 13Standard Deviation (SD)(x 1 – x) 2 + (x 2 –x) 2 + (x 3 –x) 2 + (x 4 –x) 2 + (x 5 – x) 2n – 1Sample #1 Sample #2x 1 - x = ______ squared = _______ x 1 - x = ______ squared = _______x 2 - x =_______ squared = _______ x 2 - x = _______ squared = _______x 3 - x =_______ squared = _______ x 3 - x = _______ squared = _______x4 - x =_______ squared = _______ x4 - x = _______ squared = _______x5 - x =_______ squared = _______ x5 - x = _______ squared = _______Total = _______Total = _______÷ n – 1 = _______ ÷ n – 1 = _______= _______ = _______Coefficient of Variation (CV)Sample 1 Sample 2SD _____ ÷ x _____ = CV _____ SD _____ ÷ x _____ = CV _____x 100 = ________CV in % x 100 = ________CV in %CV compared to manufacturer limits ____________________________________________________________________Acceptability of results ________________________________________________________________________________Comments ____________________________________________________________________________________________________________________________________________________________________________________AccuracyCompare the mean ( x) for each sample (calculated above), to the known value.Sample #1 x = ___________ Known value = ___________ Difference ___________Sample #2 x = ___________ Known value = ___________ Difference ___________Compared to ❏ manufacturer limits ❏ PT limits ❏ other, specify ______________________________Or, use ratio of means ( x ÷ known value x 100)Sample #1 ________________________ Sample #2 ________________________Acceptability of results ________________________________________________________________________________Comments ____________________________________________________________________________________________________________________________________________________________________________________13-8© COLA--7/07. COLA LabGuide® is a registered trademark of COLA.