Immunogenicity and Setting Cut Points

Immunogenicity and SettingCut PointsGetting Everybody on the Same PageandCapturing Questions for Discussion during “Round Table”Jeffrey SailstadPresident and Sr. ConsultantSailstad and Associates Inc.and B 2 S1

Basis of Presentation• Analogies will be used to get everybody on thesame page• A simplified data sets will be used to illustrate theassignment of screening and confirmation cut points.• As we go through the illustration, feel free to posequestions we will capture those and address thoseduring the second presentation or during the 2 ndhour.2

Screening Cut Point• Definition: Assay‟s response level at andabove which a sample is classified to be„potential‟ positive & below which it isdefined to be negative („not detected‟)• Determined statistically from theresponses in a subset of samples that arepresumed to be ADA negative.3

Anti-Drug Antibody Controls and CutPoints are like aTheater4

But 5% of the time we hit something above ourcut point, is it the floor or something elseIf we hit something else it’s likely not the floor95% of time the floor will be at this level or below8

Upper BalconiesHigh PCMiddle BalconiesMedium PCFront RowLow PC9

Upper BalconiesHigh PCMiddle BalconiesMedium PCFront RowLow PC11

Upper BalconiesHigh PCMiddle BalconiesMedium PCFront RowLow PC12

Upper BalconiesHigh PCMiddle BalconiesMedium PCFront RowLow PC13

Upper BalconiesHigh PCMiddle BalconiesMedium PCFront RowLow PC14

Assay Background has 2 VariationComponents• Analytical• BiologicalAssay derived factorsTest sample factorsLet’s go back to the theater15

Biological vs. Analytical96 96 94 9695 96 93 9696 96 94 9685 86 96 94 96 94 96 96 96 94 96 95 96 94 96 96 94 96 97 96 94 9684 86 95 95 96 95 96 96 97 93 95 95 96 93 97 96 95 96 96 96 95 9785 86 96 94 96 94 96 96 96 94 96 95 83 82 96 96 94 96 97 96 94 9616

Balanced Design for ValidationExperiments51 drug naïve ADA negative sample6 runs, 2 analysts (3 runs per analyst)Each sample tested in DuplicateReportable result:• Average of the duplicate* samplesNegative sample, Low VC, High VC; > 2results/plate, each in duplicate*Three plates used per Run for testing thesesamples.Include drug-spiked samples as well for confirmation cut point.Precision, Confirmation CP, can also be evaluated from thisexperiment.17

Balanced DesignValidation Serum SamplesAnalyst Assay Run Assay PlateP1R1P2P3P1A1R2P2P3P1R3P2P3P1R4P2P3P1A2R5P2P3P1R6P2P3S1-S17 S18-S34 S35-S51XXXXXXXXXXXXXXXXXX18

Balanced Design – includesinhibitorValidation Serum SamplesAnalystAssayRunAssayPlateS1-S17S1-S17plusinhibitorS18-S34S18-S34plusinhibitorS35-S51S35-S51plusinhibitorP1 X XR1P2 X XP3 X XP1XA1R2P2XP3XP1XR3P2XP3XP1 X XR4P2 X XP3 X XP1XA2R5P2XP3XP1XR6P2P3XX19

Balanced Design – includescontrolsControl SetAnalystAssayRunAssayPlateFrontFront +InhibitorBackBack +InhibitorP1 X XR1P2 X XP3 X XP1 X X X XA1R2P2 X X X XP3 X X X XP1 X XR3P2 X XP3 X XP1 X XR4P2 X XP3 X XP1 X X X XA2R5P2 X X X XP3 X X X XP1 X XR6P2 X XP3 X X20

Types of Screening Cut Points• Fixed cut point –• is determined in pre-study validationand the same value is used for in-studyphase• It is used for test samples until there isa need for revalidation or a change incut point• Important – for this approach to work,one must demonstrate similar mean andvariances across runs during pre-study21validations.

Types of Screening Cut Points• Floating cut point: –• Is calculated by multiplying a specificnormalization factor, determined fromthe pre-study validation data, to thebiological background obtained duringthe in-study phase.• Alternatively, biological backgroundvalues can be added to thenormalization factor if logtransformation was not necessary in theanalysis.22

Types of Screening Cut Points• Dynamic Cut Point: –• a cut point that changes between the plates ina run, between runs in a study, or betweenstudies, and does not use the variationestimates from pre-study validation.• The latter characteristic differentiates it from afloating cut point.• This approach is necessary only when thevariability between sample results from eachrun is significantly different between assayruns.• A practically limiting factor is that a significantnumber of samples are needed in eachplate/run to compute this cut point.23

Types of Screening Cut Points• Dynamic Cut Point: –• a cut point that changes between the plates ina run, between runs in a study, or betweenstudies, and does not use the variationestimates from pre-study validation.• The latter characteristic differentiates it from afloating cut point.• This Not approach advisable necessary - only only if needed when thevariability between sample results from eachrun is significantly different between assayruns.• A practically limiting factor is that a significantnumber of samples are needed in eachplate/run to compute this cut point.24

Screening CP Evaluation Flow-SchemeData: ~50 samples > 3runs, 2 analyst (ifneeded)Compare means and variancesBetween runs/instruments/analystsOutlierEvaluationInvestigateDistributionMeans similarMeans differentNon-normalNormalVariancesimilarVariancedifferentVariancesimilarVariancedifferentTransform DataLn or LogNon-normalOutlierEvaluationConfirmDistributionNormalFixed cutpointFixed cutpoint perinstrumentUse NC.ISto determinecut pointDynamiccut pointFloatingcut pointInstrumentor Analystspecificfloating CPCalculateCP.V andCF perinstrument oranalystDynamiccut pointUse NC.ISto determinecut point95 thpercentileMean + 1.645 X SD orRobust AlternativeNC.IS X CF, if transformedNC.IS + CF, if not transformedValidation Cut Point(CP.V)V. Devanarayan, PhD, Abbott LabsCorrection Factor (CF)=CP.V/NC.V, if transformed=CP.V-NC.V, if not transformedScreening Cut Point25

Cut Point: Distribution & OutliersNeed the distribution to be adequately normal for:• Comparing run mean and variances between runs• Using parametric approach for calculating cut point.Need to remove/down-weigh outliers to ensure thatthe false negative rate applies to the expectedrange of ADA negative response (also, due to riskbased strategy).First, evaluate the distribution of original and logtransformeddata, averaged across runs.• Compare Skewness & Shapiro-Wilks p-values.• Rest of the analysis should then be performed onwhichever scale is most symmetric26

Distribution & OutliersOriginal Data (OD)Log-transformed0 0.2 0.4 0.6 0.8 1 1.2-2 -1.5 -1 -0.5 0Shapiro-Wilk W TestW0.474479Prob

Distribution and OutliersRun 1 Log ODRun 2 Log ODRun 3 Log ODRun 4 Log ODRun 5 Log ODRun 6 Log OD000-0.5-1-1.5-2-2.50-0.5-1-0.5-1-1.50-0.5-1-1.50-0.5-1-0.5-1-1.5-3-1.5-2-2-1.5-3.5-2Shapiro-Wilk W TestProb

Distribution and OutliersRun 1 Log ODRun 2 Log ODRun 3 Log OD-0.4-0.25-0.4-0.6-0.8-1-1.2-1.4-1.6-1.8-0.5-0.75-1-1.25-1.5-0.6-0.8-1-1.2-1.4-1.6With outliers excludedall runs normally distributed-2-1.75-1.8MeanStd DevStd Err Meanupper 95% Meanlow er 95% MeanN-1.3212850.29610640.0441409-1.232325-1.41024645MeanStd DevStd Err Meanupper 95% Meanlow er 95% MeanN-1.1287590.282750.0421499-1.043811-1.21370645MeanStd DevStd Err Meanupper 95% Meanlow er 95% MeanN-1.1243970.26583710.0396287-1.044531-1.20426345Shapiro-Wilk W TestW0.963326Prob

Distribution & OutliersAfter excluding all outliersMeanStd DevStd Err Meanupper 95% Meanlow er 95% MeanN-1.2055780.2744780.0409168-1.123115-1.2880445-2 -1.75 -1.5 -1.25 -1 -0.75 -0.5Prob

How the SD should be calculate?• Devan will point out how this shouldbe done.31

Fixed or Floating Cut Point?Compare run means using mixed effects ANOVA.• On untransformed or log-transformed data, depending onwhichever is more symmetric.• With Runs as fixed effect, Subject as random effectsCompare run variances using Bartlett‟s or Levene‟s test (orsimilar method).If run means and variances are similar, use Fixed cut point.If run means are different, use Floating cut point.If run variances are different, compare the runs from eachanalyst or instrument used in the design, and investigateother potential analytical issues with the assay.32

Fixed or Floating?Log OD-1-21 2 3 4 5 6runAnalyst AAnalyst BMean significantly different (p

Floating Cut PointAdditive or Multiplicative Correction Factor?• When floating CP is used, need to determine acorrection factor (CF)• Let• CP.V= Cut point from validation data• NC.V= Negative Control from validation experiment• NC.IS=Negative Control for in-study run• If untransformed i.e. normal and & CP analysiswas done on untransformed data,• CP.V-NC.V=CF• Floating CP= NC.IS+CFRemember Untransformed use Additive34

Floating Cut PointAdditive or Multiplicative Correction Factor?If only the log transformed data i.e. normal and CPanalysis was done on a log scale,log (CF) = log(CP.V) – log (NC.V)CF = CP.V / NC.VFloating CP – NC.IS * CFRemember if transformed CV use Multiplication.35

Immuno-Depletion usingTherapeutic• Specific binding – Antibody totherapeutic product on solid phase(bases of screening assay) is typicallyinhibited by addition of therapeuticproduct in solution.• Non-specific binding – typically isnot altered the same or to the sameextent with addition of therapeuticproduct in solution.36

Switches and Balls• The specific inhibition of positivecontrols is much like a light switch.• The non-specific (inhibition orenhancement) of antibody negativesamples is much like pool balls.37

Specific inhibition as representedby positive controlsUninhibitedInhibitedThe degree of inhibition is reproducibleThe travel of the switch is predictable.38

Non-specific inhibition asrepresented by antibody negativesamplesENCIBreak 1Break 2The response (enhancement or inhibition) is typically random39

Setting LimitsLowPositiveControlN= 2X over 6 assaysADA Neg.samplesspikedunspikeN=50 samples or 2 or more assays40

Setting LimitsLowPositiveControlLarger, consistentResponse smallvariability –lowerlimit setADA Neg.samplesspikedunspikeMuch smallerresponses, highvariability –upperlimit set41

Confirmatory Cut PointExperiment, Data & CalculationsData:~ 50 drug naïve samples spiked with drug (preferably with the same samples &in the same Screening CP experiment)~ 4-6 runs total, >= 2 instruments/analysts when appropriateCompute % inhibition for each sample. Take log if needed.• Confirmation CP = Mean (% Inhibition) + 2.33*SD (% Inhib.)• Distribution, outliers, variability, etc., should be investigated.• 2.33 1% false positive rate (use 3.09, if 0.1% is desired)• FAQ: How to handle samples with negative inhibition?(Devan will get into this)• Devan will also address why 2-3 runs are adequate.42

Confirmation Cut PointexampleIndividual OD 450-650 (Analyst A)OD 450-650 (Analyst B)Human Serum NS-1 %CV w/ inhib %CV %Inh b NS-1 %CV w/ inhib %CV %Inh b mean % inhib SD % inhib CV1 0.331 7.9 0.271 3.7 18.1 0.382 6.9 0.336 2.1 12.0 15.084 4.3027 28.52 0.144 5.4 0.126 0.6 12.5 0.174 1.2 0.153 3.2 12.1 12.284 0.3048 2.53 0.031 16.2 0.026 0.0 16.1 0.035 2.0 0.033 2.2 5.7 10.922 7.3643 67.44 0.381 1.5 0.340 5.2 10.8 0.458 0.6 0.428 0.7 6.6 8.656 2.9776 34.45 0.259 3.0 0.225 4.4 13.1 0.317 3.4 0.284 0.5 10.4 11.769 1.9214 16.36 0.040 1.8 0.030 4.7 25.0 0.048 0.0 0.039 5.5 18.8 21.875 4.4194 20.27 0.172 0.8 0.156 11.4 9.3 0.201 0.4 0.208 2.7 -3.5 2.910 9.0403 310.78 0.190 4.5 0.189 1.9 0.5 0.233 1.5 0.241 1.8 -3.4 -1.454 2.8000 -192.69 0.038 1.9 0.040 5.4 -5.3 0.051 1.4 0.052 1.4 -2.0 -3.612 2.3351 -64.610 0.668 2.6 0.478 0.7 28.4 1.002 0.1 0.769 3.6 23.3 25.848 3.6696 14.211 0.020 3.6 0.026 36.0 -30.0 0.028 7.7 0.027 0.0 3.6 -13.214 23.7386 -179.612 0.022 3.3 0.017 21.4 22.7 0.024 5.9 0.024 3.0 0.0 11.364 16.0706 141.413 0.037 19.1 0.028 0.0 24.3 0.045 0.0 0.043 3.3 4.4 14.384 14.0572 97.714 0.951 0.3 0.659 2.6 30.7 1.403 0.5 1.180 0.4 15.9 23.300 10.4723 44.915 0.558 2.2 0.317 2.5 43.2 0.812 1.7 0.482 5.3 40.6 41.915 1.8028 4.316 0.281 2.0 0.230 3.1 18.1 0.338 2.5 0.283 1.3 16.3 17.211 1.3274 7.717 0.036 0.0 0.026 8.3 27.8 0.048 0.0 0.034 0.0 29.2 28.472 0.9821 3.418 0.049 4.4 0.036 2.0 26.5 0.189 27.4 0.042 3.4 77.8 52.154 36.2372 69.519 0.576 0.9 0.506 0.0 12.2 0.721 1.7 0.596 0.2 17.3 14.745 3.6658 24.920 0.020 3.6 0.017 0.0 15.0 0.030 2.4 0.029 27.3 3.3 9.167 8.2496 90.021 0.020 0.0 0.021 10.3 -5.0 0.025 5.7 0.025 2.9 0.0 -2.500 3.5355 -141.422 0.104 6.1 0.079 0.9 24.0 0.117 1.8 0.094 4.5 19.7 21.848 3.0974 14.223 0.021 3.4 0.016 0.0 23.8 0.027 15.7 0.026 16.3 3.7 13.757 14.2170 103.324 0.016 8.8 0.015 4.9 6.3 0.024 9.0 0.022 23.0 8.3 7.292 1.4731 20.225 0.317 11.8 0.279 2.3 12.0 0.365 1.0 0.332 4.5 9.0 10.514 2.0833 19.826 0.023 6.1 0.023 9.4 0.0 0.041 13.8 0.052 67.3 -26.8 -13.415 18.9712 -141.427 0.079 3.6 0.060 0.0 24.1 0.124 1.1 0.078 0.0 37.1 30.574 9.2250 30.228 0.054 1.3 0.026 2.8 51.9 0.090 4.7 0.040 1.8 55.6 53.704 2.6189 4.929 0.017 8.3 0.013 0.0 23.5 0.020 0.0 0.019 3.8 5.0 14.265 13.1023 91.943

Confirmatory Cut Point%Inhibition100*1Signal of sample with Inhibitor (s)Signal of unspiked sample (us)504403.5%Inhibition30after log tranf.32.5202101.5100.5• Subjective thresholds (e.g., 50% inhib.) have been proposedbefore.• Shankar et al (2008) proposed an objective thresholdbased on biological & analytical variability.44

Confirmation Cut PointexampleAll Data Exclude sample 15log of ratiolog of ratio-0.25 -0.2 -0.15 -0.1 -0.05 0 0.05 0.1-0.5 -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4Normal(-0.0548,0.05604)Shapiro-Wilk W TestW Prob

Confirmation Cut PointexampleWith five sample excluded as outlierslog of ratioQuantilesCP-0.15 -0.1 -0.05 0 0.05Normal(-0.0509,0.0498)Shapiro-Wilk W TestW0.98904799(%inhibiti100*Prob

Acknowledgements• Wendell Smith, Ph.D. from B 2 S• Ron Bowsher, Ph.D. from B 2 S• Viswanath Devanarayan, Ph.D. from Abbot• The LBABFG of AAPS:Which included the immunogenicity sub-groupwhich authored the current “validation” whitepaper.Gopi Shankar, Viswanath Devanarayan, LakshmiAmaravadi, Yu Chen Barrett, Ronald Bowsher, DeborahFinco-Kent, Michele Fiscella, Boris Gorovits, SusanKirschner, Michael Moxness, Thomas Parish, ValerieQuarmby, Holly Smith, Wendell Smith, Linda A.Zuckerman and Eugen Koren47

Contact InformationJeff SailstadSailstad and Associates Inc.Email: sailstad@aol.comPhone: (919) 225-634648