TEST METHOD VALIDATION Randall J. Troutman CPP Senior ...

TEST METHOD
VALIDATION
Randall J. Troutman CPP
Senior Engineering Manager
Oliver Medical

Presentation Disclaimer
Oliver Medical is devoted to building awareness while
enhancing educational value among medical packaging
professionals through seminars such as this.
The views, topics, and suggestions discussed today are
for educational purposes only and therefore should not
be considered the last word on these issues.
It is the sole responsibility of the attendee and his or her
company to determine the suitability of information
presented today. As well as to investigate all other
sources of information in regards to application of the
content.

• Background
Presentation Overview
• ASTM Guidance / Definitions
• ASTM F88-05 (overview)
• ASTM F1929-98 (overview)
• Getting Started in your lab
• References

• FDA “hot button” issue -
What is all the buzz?
– Recent release of revised 11607 documents have caused
confusion in regards to listed test methods
– Test method validation statement in both parts
• Part 1 - Clause 4.4 Test Methods
• Part 2 - Clause 4.3 Test Methods
– This statement was included in the previous revision
• ISO 11607 Clause 4.3.2

4.4 Test Methods
11607 part 1
4.4.1 All test methods used to show compliance with this part of ISO 11607 shall be
validated and documented.
NOTE Annex B contains a list of suitable test methods.
4.4.2 The test method validation shall demonstrate the suitability of the method used. The
following elements shall be included:
- establishment of a rationale for the selection of the appropriate test for the packaging system;
- establishment of acceptance criteria;
NOTE Pass/fail is a type of criteria;
- determination of test method repeatability;
- determination of test method reproducibility; and
- establishment of test method sensitivity for integrity tests.

4.3 Test Methods
11607 part 2
4.3.1 All test methods used to show compliance with this part of ISO 11607 shall be
validated and documented.
NOTE Annex B in ISO 11607-1 contains a list of suitable test methods.
4.3.2 The test method validation shall demonstrate the suitability of the method used. The
following elements shall be included:
- establishment of a rationale for the selection of the appropriate test for the packaging system;
- establishment of acceptance criteria;
NOTE Pass/fail is a type of criteria;
- determination of test method repeatability;
- determination of test method reproducibility; and
- establishment of test method sensitivity for integrity tests.

11607 part 1 – Annex B
• What does the list include?
– Standard guides
– Standard practices
– Standard test methods
• Including some that can not be validated
• From various organizations…..

11607 part 1 – Annex B
• Includes standards from the following organizations;
– ASTM (Examples – peel, dye, bubble emission, etc.)
– ISO (Examples – air permeance, biological evaluation, etc.)
– EN (Example – sterilization wrap requirements and test methods)
– USP (Examples – Biological reactivity test, in vitro)
– TAPPI (Example – dirt in paper)
– DIN (Example – testing of paper board – bending stiffness)

Validation 1
General Definitions
confirmation by examination and provision of objective
evidence that the particular requirement for a specific intended use
can be consistently fulfilled
Verification 2
confirmation by examination and provision of objective evidence that
specified requirements have been fulfilled
Qualification
referred to as the process of events used to establish feasibility of a
concept, process, or method
1. 11607 part 1
2. 21 CFR Part 820

ASTM Guidance
ATSM E 177
Standard Practice for Use of Terms
Precision and Bias in ASTM Test Methods
ATSM E 1169
Standard Guide for Conducting Ruggedness Tests
ATSM E 691
Standard Practice for Conducting an Interlaboratory Study
to Determine the Precision of a Test Method

ASTM Guidance cont.
ASTM E 2282
Standard Guide for Defining the
Test Result of a Test Method
ATSM E 1488
Standard Guide for Statistical Procedures to Use in
Developing and Applying Test Methods
ATSM E 456
Standard Terminology Relating to Quality and Statistics

ASTM E 177
Standard Practice for Use of Terms the
Precision and Bias in ASTM Test Methods
– Purpose is to provide a basis for;
• Precision & Bias statements (for quantitative methods)
• Describes methods of expressing precision and bias
• Gives examples of how P&B statements may be written

Definitions
precision
the closeness of agreement among test results obtained under
prescribed conditions
repeatability
addresses variability between independent test results gathered
from within a single laboratory (otherwise known as intralaboratory
testing)
reproducibility
addresses variability among single test results gathered from
different laboratories (otherwise known as interlaboratory testing)

Definitions
bias
a generic concept related to a consistent or systematic difference
between a set of test results from the process and an accepted
reference value of the property being measured
accuracy
a generic concept of exactness related to the closeness of
agreement between the average of one or more test results and an
accepted reference value
– Accuracy should be used when comparing test results, not in terms
of the output result generated from execution of test a process. But
as - is this test result accurate, verses what is the accuracy of these
results compared to previously generated results.

ASTM E 1169
Standard Guide for
Conducting Ruggedness Tests
– Purpose is to provide a basis for;
• Finding the variables that influence measurements
– experimental factors using Plackett-Burman designs
• Determining how closely these factors need to be controlled
– The test does not determine optimum conditions for
the test method

uggedness (robustness)
Definitions
determination of variables associated with the performance of a test
method that is ultimately critical to the accuracy of the precision
statistical control
point when the test results obtained vary in a predictable manner,
showing no unassignable trends, cycles, abrupt changes, excess
scatter, or other unpredictable variations as determined by
application of appropriate statistical methods

ASTM E 691
Standard Practice for Conducting an Interlaboratory
Study to Determine the Precision of a Test Method
– Purpose is to provide a basis for;
• Addressing inherent variability in test methods
– Includes (a.) Operator, (b.) equipment used, (c.) calibration
of equipment, (d.) environment (temp, humidity, & etc.)
• Developing protocol w/ details
– Calibration, specific methods of testing, etc.
• Sample preparation
– Sample preparation tips, including conditioning
– Statistically valid sampling plans
• Statistical evaluation methods

ASTM E 2282
Standard Practice for
Defining the Test Result of a Test Method
– Purpose is to provide a basis for;
• Identifying the elements that comprise the test
results of a test method and to illustrate how
these elements combine into the test result.

Definitions
test method
a definitive procedure for the identification, measurement and evaluation of one or
more qualities, characteristics, or properties of a material, product, system, or service
that produces a test result
observation or observed value
the most elemental single reading or corrected reading obtained in the process of
making a measurement
test determination
(1) the process of calculating from one or more observations a property of a single
test specimen, or (2) the value obtained from the process
test result
the value obtained by carrying out the complete protocol of the test method once,
being either a single test determination or a specified combination of a number of test
determinations

ASTM E 1488
Standard Guide for Statistical Procedures
to Use in Developing and Applying Test Methods
– Purpose is to provide a basis for;
• Using statistical procedures for use in
– developing new test methods or
– revising or evaluating existing test methods
• Types of data
– Numerical vs. Categorical / Attribute Data
• Statistical Analysis Methods
– Analysis of Means (ANOM)
– Analysis of Ranges (ANOR)

ASTM E 456
Standard Terminology Relating to Quality and
Statistics
– Purpose is to provide a basis for;
• Terminology used in quality and statistical based ASTM
methods in which standard definitions appear
– attribute data: observed values or determinations which
indicate the presence of absence of specific characteristics.
– variables data: measurements which vary and may take any
of a specified set of numerical values.

Formal Process to Evaluate
• As taken from 11607;
– Suitability of particular test method
– Robustness (ruggedness)
– Precision and / or bias
– Repeatability (within lab precision)
– Reproducibility (between lab precision)
– Range and / or Sensitivity of Detection

Document the Exercise
• The engineering exercise should include:
–Protocol
• Objective / Scope
• Sampling plan, w/ statistical rationale
• Predetermined acceptance criteria
– Report w/ supporting data
• Demonstrates all requirements have been met
– Statement of validation / successful execution
• Includes “proof of work”

Applying the process….
• ASTM F88-05
– Quantitative test method application
– Statistical comparison of test results w/ controlled variation
• ASTM F1929-98
– Qualitative test method application
– Determining sensitivity levels for pass / fail tests
• “In house” test method validation
– Modified standard test method for use by MDM
– Internal test method developed by MDM

1. Scope
1.1 method covers the measurement of the strength
of seal in flexible materials
1.2 method may be conducted on seals between a
flexible and rigid material
1.3 the seals tested may be from any source,
laboratory, or commercial
1.4 method measures the force required to separate
a test strip of material containing a seal. Also
identifies the mode of specimen failure

Conducting a Test Method Evaluation
• Evaluate / Read the Test Method
– Scope meets intended use
– Understand the sources of variation
• Material combinations
– Flexible vs. rigid application
• Number and types of equipment involved
– Equipment calibration details
» What, how, and when
• Test sample conditioning

Conducting a Test Method Evaluation
• Conduct an evaluation to expose how these
factors affect the test result
– “ruggedness test”
• Objective is to evaluate how the test method variables
affect the test determination
• The results of this test will provide you with an
accurate “sample” to use in the repeatability
phase of your evaluation

Conducting a Test Method Evaluation
• Conduct “reproducibility” studies with other pieces of like
equipment in other locations using “samples” proven in
the ruggedness and repeatability studies
• Use a statistician to help determine the predetermined
“Precision” acceptance criteria for the method in your lab
• Based on the data compiled within your R&R study you
may use a one-way analysis of variance (ASTM E 691) to
evaluate the data
• Compare your data to the “Precision” data recorded in
the actual standard (Section 11)

Applying the process….
• ASTM F88-05
– Quantitative test method application
– Statistical comparison of test results w/ controlled variation
• ASTM F1929-98
– Qualitative test method application
– Determining sensitivity levels for pass / fail tests
• “In house” test method validation
– Modified standard test method for use by MDM
– Internal test method developed by MDM

1. Scope
1.1 defines materials and a procedure that will
detect and locate a leak equal or greater
than a channel of 0.002 in.
1.2 intended for use on packages with an
edge seal formed between a transparent
film and a porous material
1.3 requires a contrasting dye penetrant
solution be used

Conducting a Test Method Evaluation
• Evaluate / Read the Test Method
– Scope meets intended use
• Objective is to determine if the method that
you conduct internally meets the sensitivity
claim in the standardized method
– For this particular method the sensitivity level is
a channel down to 0.002 in. in seal

Conducting a Test Method Evaluation
• Overall evaluation should consider;
– Sufficient sample sizes
– Variation in dye mixture methods
– Variation in sourced ingredients
– Variation in exposure times
• 5 seconds vs. 20 seconds
– Variation in the actual procedure / process
– Variation in voids detected

Conducting a Test Method Evaluation
• While considering those variables you
should prove the sensitivity of the method
– Establish a range, w/ control limits
• Example range –
» Control (no void)
» 0.001” void (min limit)
» 0.002” void (detectable)
» 0.003” void (detectable)
» 0.005” void (detectable)

Applying the process….
• ASTM F88-05
– Quantitative test method application
– Statistical comparison of test results w/ controlled variation
• ASTM F1929-98
– Qualitative test method application
– Determining sensitivity levels for pass / fail tests
• “In house” test method validation
– Modified standard test method for use by MDM
– Internal test method developed by MDM

Non-Standardized Test
Methods
• Acceptable for use but validation is more
difficult because there is no data for
comparison
• You must ensure that the test is measuring the
intended attribute
• US regulators expect more data from the
validation process

Getting started in your lab….
• Review the standard test methods that you
currently execute within your company
– How was the standardized method evaluated?
• Review the ASTM standards mentioned in this
presentation for guidance
• Develop a plan to “validate” (evaluate) your
methods separately and be prepared to employ
varying strategies to fulfill the requirement

Randall Troutman CPP
Thank You
Oliver Medical
800.253.3893 ext. 7914
616.456.7711 ext. 7914
rtroutman@olivermedical.com
www.olivermedical.com

References
� Tyvek® Rx Newsletter (Vol. 15 Issue 2 October 2006)
� “Fact vs. Fiction”, ASTM Standardization News, Pat Picariello
� “Validation and Regulatory Acceptance of Toxicological Test Methods”,
ad hoc Interagency Coordinating Committee on the Validation of
Alternative Methods, 1997
� ISO EN 11607-1 2006
� ISO EN 11607-2 2006
� ASTM F1929-98
� ASTM F88-05
� ASTM E 177
� ASTM E 1169
� ASTM E 691
� ASTM E 2282
� ASTM E 1488
� ASTM E 456

Additional Research
• Test Method Validation Techniques
– Analytical Chemistry Test Methods
– Toxilogical Test Methods
– Forensic Test Methods