The Point of Care Quality Control Debate - Whitehat

The Point of Care Quality Control DebateThomas Koshy, Ph.D., Sr. Director, Scientific AffairsAlere, Inc

Disclaimers• I work for Biosite/Inverness/Alere• Alere produces testing devices for use at the POC…so Ihave more data on the testing areas my company covers;with special attention to ED (i.e. cardiac) POCT• I think POCT is pretty cool• This is usually a 90-120 minute talk!!• So I’m going to cover some of this way too fast…..

A Show of HandsEQCEP23IQCPs

What is QC?Where did itcome from?How doesQC assurequality data?

ARCHITECT ® ImmunoassayProcess Path

What about QC for these guys?

Alere Triage ® Assay ProcedureWhat the User SeesStep 1Step 2Step 3Add whole blood to deviceInsert device into MeterRead Results

Alere Triage Test DeviceSample PortSample enters hereReaction ChamberA small fraction of theplasma sample mixeswith the dried reagentsThree Internal ControlsIndependent high controlzones and a zero controlconfirm that the test hasbeen completed correctlyWaste ReservoirThe majority of thesample acts as a washand collects in theperimeter of the deviceBlood FilterCells are separated fromplasma, eliminating theneed for centrifugationTime GateA hydrophobic surfaceacts as a time barrier andensures an appropriatereaction timeAssay ZonesThe assay analytes andthe fluorescent-taggedantibodies are capturedon separate zones of thedevice

Triage Device ControlsUp to two immunoreactive positive controlsOne negative or non-specific binding control• Helps in detection of interfering substancesThese controls mimic the captures zones ofanalytes and verify:• Sufficient sample added to the device• The device performed the immunoassay correctly• Lack of gross interferences• Proper device manufacture• Proper insertion of device into meter

Alere Triage QC DeviceQC Device is runevery day.Mated to a meterCartridge identical topatient test cartridgeTrue optical systemcheck, notelectronicallysimulatedSix fluorescentzones of varyingintensityFluorescent readsare compared to theperformance atinstrumentmanufactureChecks InstrumentCalibration,Laser/CartridgeAlignment, LaserFunctionality

QC LockoutsUser ID - Only validusers can operate thesystemLot Expiration Date -Expired reagents cannotbe runExternal Controls -Controls must be run onnew reagent lots andaccording to thefrequency set by the LabsupervisorQC Device Not Run -The QC Device must runaccording to thefrequency set by the LabSupervisorQC Device Failure - AllQC Device tests mustmeet specifications

Four key CMS regulations formoderately complex tests493.1253• Test method verificationaccuracy, precision, reportable rangeand reference ranges493.1254• Maintenance and function checks493.1255• Calibration and calibration verification493.1256• QC proceduresCLIACMS 2004 brochure on how to complete the initial “performance verification.” p.2

493.1256 – QC proceduresFor each test system, the laboratorymust test, at a minimum, two levels ofexternal QC materials each day itperforms a nonwaived test.However, the regulations now allowthe laboratory to reduce thefrequency of testing external QCmaterials (equivalent QCprocedure) for certain test systems.CLIACMS: Equivalent Quality Control Procedures Brochure #4

So I Have a Device With EQCFeatures. What Would CMS LikeMe to Do?Follow the manufacturer’s package insert.Evaluate the system’s capability to monitor allanalytical elements of the testing procedure• Operator, analysis, environment, sample addition,sample/reagent interactions, test completion time.Evaluate the system’s Equivalent QC by oneof three processesCMS: Equivalent Quality Control Procedures Brochure #4

EQC EvaluationEvaluation ProcessExternal QCchecksOption 1System monitors allanalytic componentsDaily testing withinternal monitoringsystems10 consecutivedays of passingexternal QCAt least onceper monthOption 2System monitors someanalytic componentsDaily testing withinternal monitoringsystems30 consecutivedays of passingexternal QCAt least onceper weekOption 3System monitors noanalytic componentsNA60 consecutivedays of passingexternal QCAt least onceper weekCMS: Equivalent Quality Control Procedures Brochure #4

EQC EvaluationEvaluation ProcessExternal QCchecksOption 1System monitors allanalytic componentsDaily testing withinternal monitoringsystems10 consecutivedays of passingexternal QCAt least onceper monthOption 2System monitors someanalytic componentsDaily testing withinternal monitoringsystems30 consecutivedays of passingexternal QCAt least onceper weekThere is NO evidence that 10 or 30 consecutive days of passing QC providesassurance that subsequent monthly testing is sufficient!

Normal Population Distributions±2 SD captures 95% of data(0.95).ρ passing 10 reps = 0.95 10 =0.598.0.95 30 =0.215ρ passing 10/11 reps = 0.897± 1 SD captures 68% of data±2 SD captures 95% of data±3 SD captures 99.7% of data

EQC EvaluationEvaluation ProcessExternal QCchecksHow does 60 consecutive days of passing QC provide assurancethat subsequent monthly testing is sufficient for a test thatmonitors NO analytic components?Option 3System monitors noanalytic componentsNA60 consecutivedays of passingexternal QCAt least onceper weekCMS: Equivalent Quality Control Procedures Brochure #4

The Pen is Mightier than QC

So I Have a Device That ClaimsEQC: What Do I Have to Do?Follow themanufacturer’spackage insert.Section 493.1256–QC procedures isNOT being enforced.CMSCAPCOLAJCInspectors willcontinue toissueEducationalLetters.POC.07300:20 days.QC 24 R:10daysQSA.02.04.01:10/30 days

The QC Regulations• Who accredits your lab?•CMS?•CAP?•Joint Commission?•COLA?• What do they require?

The QC Regulations• Who accredits your lab?•CMS?•CAP?•Joint Commission?•COLA?• What do they require?

CLSI to the Rescue!!!EP23User Defined QCProtocols for In VitroDiagnostic DevicesBased onManufacturer’s RiskMitigation Informationand the User’sEnvironmentEP18Risk ManagementTechniques toIdentify and ControlLaboratory ErrorSources

This is the future for QC of POCTRisk Management approach to Quality Control• Evaluate the QCfeatures of thedevice1• Analyze otherelements ofvariability thatmust be controlled2 • Assess theseverity of failuresin each step of thetesting process34• Device QC testingto monitor andcatch said failuresThis represents a shift from mere QCcompliance to true Quality Control

3/9/12 CMS Official MemorandumKey concepts from EP-23 will be an acceptable alternativeQC policy. The New CLIA QC policy will be entitledIndividualized Quality Control Plan (IQCP)IQCPs are a formal representation and compilation ofmany things laboratories currently do for quality.IQCPs permits the laboratory to customize its QC planaccording to environment, reagents, testing personnel,specimens, and test system.IQCP will be voluntary: Laboratories will have two choicesfor QC compliance: 1) Two levels of QC per day or, 2)IQCP. Package insert requirements must be met.Education andtransition datesTBDEQC will be phased out at the end of the education andtransition period

CAP? JC? COLA?CMS will solicitaccreditingorganizations(AOs) todetermine theirinterest in IQCP.Accredited laboratoriesshould continue to meettheir accreditingorganizations’ QCstandards until theyreceive notice from theirAOs.CMS presentation at CLSI EP23 workshop, May 2012

Where to Obtain Information• CMS/CLIA Website:http://www.cms.hhs.gov/clia/• CMS CLIA Central Office:410.786.3531• IQCP Link:IQCP@cms.hhs.gov• EP23 WorkbookCMS presentation at CLSI EP23 workshop, May 2012

Instead of This……Evaluation ProcessExternal QCchecksOption 1System monitors allanalytic componentsDaily testing withinternal monitoringsystems10 consecutivedays of passingexternal QCAt least onceper monthOption 2System monitors someanalytic componentsDaily testing withinternal monitoringsystems30 consecutivedays of passingexternal QCAt least onceper weekOption 3System monitors noanalytic componentsNA60 consecutivedays of passingexternal QCAt least onceper weekCMS: Equivalent Quality Control Procedures Brochure #4

Do This!!!Life-Cycle Risk Management Process

Risk/Hazard AssessmentsGruesomelyComplexIneffectiveHand-waving

StatisticalDataExisting QCprocedures

Risk Management approach to QCFirstunderstand thedifferencebetween hazardand risk

Risk Assessment ToolsArrival to /EDLab marker TATTime to treatment• BrainstormingPresentation of symptomsIntervention• 5 Whys• Fishbone diagrams• Process mappingTime to evaluation/test orderingTime from markerreceipt to diagnosisIntervention Prep Time(i.e. time fromdiagnosis to cath labnotification andtransportation)ProcessBeginProcess Process DataDecisionProcessG. Cooper, BioRad. 2007 AACC QC Webinar

Testing Process Fishbone DiagramTransportCentrifugation/aliquottingRelease/reportPost-analyticalMD orders testMD receivesresultPhlebotomy/LabelingPre-analyticalAccessioningInstrument analysisAnalytical

Testing Process Fishbone DiagramTransportCentrifugation/aliquottingRelease/reportMD orders testMD receivesresultPhlebotomy/LabelingAccessioningInstrument analysisAnalytical

Instrument Error Fishbone DiagramSpecimenOperatorPotential FailuresIncorrectResultLab EnvironmentReagentsAnalyzer

FMEA BasicsFunction FailureModesEffectsofFailureSeverityCause ofFailureProbability

Let’s Learn with an Example

Function FailureModesEffectsofFailureSeverityCause ofFailureProbabilityDesignControlParachuteChutedoesn’topenInjury,AbrasionsFailure tounfurl

Function FailureModesEffectsofFailureSeverityCause ofFailureProbabilityDesignControlParachuteChutedoesn’topenInjury,AbrasionsFailure tounfurlChutetears

Function FailureModesEffectsofFailureSeverityCause ofFailureProbabilityDesignControlParachuteChutedoesn’topenInjury,AbrasionsFailure tounfurlChutetearsFall anddieAgeBirds,planes

Function FailureModesEffectsofFailureSeverityCause ofFailureProbabilityDesignControlParachuteChutedoesn’topenInjury,AbrasionsFailure tounfurlChutetearsFall anddieAgeBirds,planesRopeRopebreaksAge

Unacceptable orAcceptablewith Risk BenefitRecommend, Implement &Verify Risk Control ActionsEvaluate Residual RiskAcceptableRequiresRisk BenefitUnacceptableProceed with UsePerform RiskBenefitAnalysisDiscontinue Use47

Risk Benefit48

Variables to ConsiderEnvironmentalconditions:Temperature,humidityIntended medicaluse of test result:HIV vs triglycerideClinical setting: Mainlab, POC,Outpatient, ER, ICU,Ambulance, NontraditionalsettingTime lapse: Areresult acted onimmediately or not?Testing frequency,testing personneland turnoverCondition of ancillaryequipment:Centrifuges,refrigerators, heatbathsPower requirements/fluctuationsRadio andelectromagneticwavesAge of the deviceG. Cooper, BioRad. 2007 AACC QC Webinar

Develop an FMEAThink in terms of the five elements of aprocess.People:Training,Experience,AttitudeMaterials(Reagentsandconsumables):Integrity,Storage,Reconstitution, Preparation(mixing), UseEquipment(Hardwareand Software):Use,Maintenance,ReliabilityMethods:Calibration,Capability,Sensitivity,Specificity,Accuracy,PrecisionEnvironment:Temperature,Humidity, Airflow, Powersupply, WaterqualityBTW: This is committee work!G. Cooper, BioRad. 2007 AACC QC Webinar

PackageInsertVendorOutputRegulatoryRequirementsRisk AnalysisClinicalRequirementsLabInformation

Ranking Severity of Failureand Probability of HarmNegligible• Inconvenience or temporarydiscomfortMinor• Temporary injury or impairment notrequiring professional medicalinterventionSerious• Injury or impairment requiringprofessional medical interventionCritical• Permanent impairment or lifethreateninginjuryCatastrophic• Results in patient deathFrequent• Once per weekProbable• Once per monthOccasional• Once per yearRemote• Once every few yearsImprobable• Once in the life of the test systemISO 14971

Risk Acceptability MatrixSeverity of HarmProbability Negligible Minor Serious Critical Catastrophicof harmFrequent X X X X XProbable OK X X X XOccasional OK OK OK X XRemote OK OK OK OK XImprobable OK OK OK OK OKISO 14971

Process Severity EvaluationCriteriaEffect Severity of effect RankingHazardous, without warningHazardous, with warningVery HighHighModerateLowVery LowMinorVery MinorNoneMay endanger patient. Involves non-compliancewith gov’t. regulation without warning.Same as above only with warningMajor injury to patient requiring emergencyinterventionMinor injury to patient; patient dissatisfiedResults acceptable; not cosmetically satisfactory100% of results may have to be retested; somepatient dissatisfactionTiming/efficiency defects noticed by most usersSame as above, but, defect noticed by averageuserSame as above, but, defect noticed only by thediscriminating userNo effect10987654321Adapted from Quality Support Group, Inc

Process Occurrence EvaluationCriteriaProbability of Possible Failure C pk RankingsFailureRatesVery high, failure is > 1 in 2< 0.33 10almost inevitable 1 in 3> 0.33 9High, repeatedfailuresModerate, occasionalfailuresLow, relatively fewfailuresRemote, unlikely1 in 81 in 201 in 801 in 4001 in 20001 in 15,0001 in 150,000< 1 in 1,500,000> 0.51> 0.67> 0.83> 1.00> 1.17> 1.33> 1.50> 1.6787654321Adapted from Quality Support Group, Inc

Process Detection EvaluationCriteriaQualitative probabilityRemote likelihood that erroneous resultswould be undetected• detection reliability at least 99.99%• detection reliability at least 99.80%Low likelihood that erroneous resultswould be undetected• detection reliability at least 99.5%• detection reliability at least 99%Moderate likelihood of detection• detection reliability at least 98%• detection reliability at least 95%• detection reliability at least 90%High likelihood that that erroneousresults would be undetected• detection reliability at least 85%• detection reliability at least 80%Extreme likelihood that erroneousresults would be undetectedQuantitative probability Rankingof not detecting1/10,0001/5,0001/2,0001/1,0001/5001/2001/1001/501/201/10 +12345678910Adapted from Quality Support Group, Inc

Now….What Needs Fixing?• Identify those conditions that lead to unacceptable levels oferror severity and frequency.• Determine operating processes or tests (quality control) todetect those conditions1st2nd3rd4th• Eliminate causes of failure so that it does not OCCUR• Reduce probability of OCCURRENCE• Reduce SEVERITY of the failure• Improve DETECTION of the failureQuality Support Group, Inc

As a Result You WillIdentify each componentand its functionIdentify the potentialfailure mode for eachcomponentIdentify the potentialeffect(s) of the failureand rate the severity ofeachIdentify the potentialcauses of the failure andrate the occurrence ofeachIdentify the currentcontrol for eachoccurrence and rate thedetection capabilityCalculate the RPN foreach itemRank RPNs from highestto lowestTake action on thoseitems designated ashigh riskG. Cooper, BioRad. 2007 AACC QC Webinar

Do This!!!Life-Cycle Risk Management Process

You’re Gonna Need HelpDevice manufacturers need to provide LOTS moreinformation about their QC featuresDetailed descriptions of device risk mitigation featuresIdentify the targeted failure mode for each mitigationDescriptions of how the risk mitigation feature or recommended actionperforms its intended functionKnown limitations of the risk mitigation feature or recommended actionStudies performed to verify the feature or recommended action achieves theintended purposeG. Cooper, BioRad. 2007 AACC QC Webinar

EP22-Presentation of Manufacturer'sRisk Mitigation Information for Usersof in vitro Diagnostic Devices• Guidance to Vendors• Document design features that detectand/or control test system variabilityand/or failures.• Describe failure modes, risk reductionfeatures and data to support theeffectiveness of those features.

EP22 Items Vendors WouldHave AddressedReagentdeterioration• During shipment• Over timeExpiredreagentsQC sampledegredationCalibratordegredationSample dataentry errorOperatorcertificationsLow/HighsamplevolumeClots/particulates/bubblesSamplecarryoverWear & tearonreplaceablepartsEnvironmental limitationsSamplelimitationsQCmaintenance

This is the future for QC of POCTA Risk Management approach to Quality ControlNo more onesize-fits-allformulasEvaluate theQC features ofthe deviceAnalyze otherelements ofvariability thatmust becontrolledAssess theseverity offailures in eachstep of thetesting processDevice QCtesting tomonitor andcatch saidfailuresThis represents a shift from “QualityCompliance” to true Quality Control

A Show of HandsEQCEP23IQCPs

Questions?Thank You!Today is the youngest you’ll be for the rest of your life. Act like it.