Update To The 2009 AMP Molecular Diagnostic Assay Validation White Paper
Update To The 2009 AMP Molecular Diagnostic Assay Validation White Paper
Update To The 2009 AMP Molecular Diagnostic Assay Validation White Paper
Update to the 2009 AMP Molecular Diagnostic Assay Validation White Paper
Members of the 2013 and 2014 Clinical Practice Committees of the Association for Molecular Pathology
The 2013 AMP Clinical Practice Committee consisted of Matthew J. Bankowski, Milena Cankovic, Jennifer Dunlap,
Larissa V. Furtado*, Jerald Gong, Thomas Huard, Linda Jeng*, Loren Joseph (Chair), Annette Kim, Bryan Krock,
Marilyn M. Li, Mary Lowery-Nordberg, Melissa Miller, Caroline Sue Richards, and Paul Rothberg.
*
Project leads
Standard of practice is not defined by this article, and there may be alternatives. See Disclaimer for further details.
INTRODUCTION
The goal of method validation in the molecular diagnostics laboratory is to ensure that a given test is ready for
implementation in the clinical laboratory. To reach that goal, each step of the testing process must be carefully
evaluated and documented. Such validation is relatively standardized for high volume automated assays in fields
such as clinical chemistry. It is challenging to apply those standardized practices to molecular diagnostics
laboratories, which rely heavily on low-volume labor-intensive tests, both FDA approved and laboratory-
developed. Excellent and comprehensive documents have recently been published on this topic 1-10. This overview
is intended to provide a brief practical reference that can be used to guide the validation process in molecular
diagnostics laboratories. In addition, a sample summary checklist is provided in the appendix as a template to
facilitate documentation of laboratory director review and sign off on the validation process when new assays are
developed or adopted.
SUMMARY
CLIA (42 CFR 493.1253) and CAP (MOL 30785-31705) require that laboratories validate the performance of tests
before reporting patient results.
DEFINITIONS
Assay Validation
• WHO definition: “the action (or process) of proving that a procedure, process, system equipment or
method used works as expected and achieves the intended result” 11.
• FDA definition: “confirmation by examination and provision of objective evidence that the particular
requirements for a specific intended use can be consistently fulfilled”12.
• CLSI definition: “confirmation by examination and provision of objective evidence through a defined
process that the particular requirements for a specific intended use can be consistently fulfilled.”13 This
definition is also used by CAP Accreditation Programs 14.
Assay Verification
September 2014
Association for Molecular Pathology
9650 Rockville Pike, Bethesda, MD 20814
www.amp.org 301-634-7939 amp@amp.org
Molecular Diagnostic Assay Validation White Paper 2
• CLIA definition: “confirmation by examination and provision of objective evidence that specified
requirements have been fulfilled”13, or “confirmation that the laboratory using a test can replicate the
manufacturer’s performance claims when the test is used according to the package insert” 15.
• CAP definition: “an abbreviated process to demonstrate that a test performs in substantial compliance to
previously established claims”14
Verification applies to unmodified non-waived (moderate- and high-complexity) tests that have been cleared or
approved by the FDA and are labeled “for in vitro diagnostic use.”
ASSAY TYPES
For UNMODIFIED FDA-approved or FDA-cleared tests, laboratories must verify that test(s) perform(s) as expected
by obtaining data on:
• Accuracy
• Precision
• Reportable range
• Linear range (for quantitative assays)
• Reference intervals (normal values) for laboratory patient population
For MODIFIED FDA-Approved tests or for non-FDA cleared tests (e.g., Laboratory Developed Procedures (LDP),
previously LDTs) laboratories must establish the following performance characteristics:
• Accuracy
• Precision
• Reportable range
• Linear range (for quantitative assays)
• Reference intervals (normal values) for laboratory patient population
• Analytic sensitivity
• Analytic specificity
For some tests there may be additional parameters which are necessary to assess:
• Frequency or call rate for genotyping assays
• Specimen stability
• Carryover (e.g., well-to-well cross-contamination for automated nucleic acid extraction)
ASSAY DESIGN
• Define the requirements of the test (intended use, test method and expected performance characteristics
(e.g. accuracy, precision, reportable range, analytical sensitivity, analytical specificity, reference interval)
• Review the literature to support evidence for clinical utility and clinical validity of the test
• Assess clinical indication for the test
• Define target population
• Define purpose of the test (e.g., screening, diagnosis, prognosis, monitoring)
• Choose pertinent specimen types
• Establish criteria for sample rejection (e.g., sample age and quantity, preferred anticoagulant for
collection tubes)
Molecular Diagnostic Assay Validation White Paper 3
• Establish minimal acceptance performance criteria of the test (e.g., test turnaround time, coefficient of
variation of the assay)
• Consider the role of test result in patient management
• Assess technical feasibility of assay implementation (e.g., right equipment, manpower, enough samples to
justify implementation of the assay)
• Choose appropriate controls
• Perform initial optimization studies to establish assay protocol and parameters before starting the
validation
CLINICAL VALIDATION
Clinical validation assesses the clinical validity and clinical utility of the test in light of clinical characteristics of the
disease/marker being tested. Data for clinical validation can be obtained from studies performed by the
laboratory, or from studies reported in peer-reviewed literature or other reliable sources. CLIA requires
laboratories to have a qualified laboratory director who is responsible for ensuring the clinical utility of the tests
performed in his or her laboratory.
CLINICAL CONDITION
TEST OUTCOME POSITIVE NEGATIVE
Positive Predictive Value
POSITIVE True Positive False Positive
TP / (TP + FP)
Negative Predictive Value
NEGATIVE False Negative True Negative
TN / (TN + FN)
Sensitivity Specificity
TP / (TP + FN) TN / (FP + TN)
ANALYTICAL VALIDATION
NOTE: Please note that the following are only suggestions, not directives. The extent and type of validation that is
needed for a particular test is left to the discretion of the laboratory director.
Accuracy (Trueness)
Definitions:
• Standards for the Reporting of Diagnostic Accuracy (STARD) definition: The amount of agreement
between the information from the test under evaluation (the index test) and the reference standard (the
best available method for establishing the presence or absence of the condition of interest) 16
Molecular Diagnostic Assay Validation White Paper 4
• The closeness of the agreement between the results of a single measurement and the true value of the
analyte 17, 18
Precision
Definitions:
• The closeness of agreement between independent test/measurement results obtained under stipulated
conditions17, 18
• Getting the same results with repetition of the assay.
Elements of precision:
• Repeatability – this is referred to as intra-assay variation or within-run precision. It is the closeness of
agreement between repeated tests of the same sample under the same operating conditions.
• Reproducibility – This is referred to as inter-assay variation or between-run precision. It is the closeness of
agreement between the results of measurements when operating conditions are varied. All potential
sources of variability should be considered. Typically, the same material is analyzed by the same protocol
on different days, by different operators and with different equipment.
• Sources of variability that may have an impact on precision
o Operator (most common source of variability)
o Reagent lot (second most common source of variability)
o Instrument
o Sample concentration
o Sample source
o Run
o Time of the day
o Laboratory environment
Reportable Range
Definition: The span of test result values over which the laboratory can establish or verify the accuracy of the
instrument or test system measurement response 19.
Linear Range
Definition: The range where the test values are proportional to the concentration of the analyte in the sample 6.
Reference Intervals
Definition: The range of test values expected for a designated population of individuals, e.g., 95 percent of
individuals that are presumed to be healthy (or normal) 20.
Analytical Sensitivity
Definitions:
• The ability of a test to detect a mutation when that mutation is present.
o Sensitivity = True positive ÷ (True positive + False negative)
• Also used to refer to the lower limit of detection (LLOD) for the analyte of interest (i.e., the lowest
concentration of analyte that the assay can consistently detect with acceptable precision),
• Or the lowest amount of starting material (nucleic acid) for an assay that will provide consistent results
with acceptable precision.
Analytical Specificity
Definitions:
•
The ability of a test to give a normal (negative) result in specimens that do not have the mutation being
tested.
•
Specificity = True negative ÷ (True negative + False positive)
Molecular Diagnostic Assay Validation White Paper 5
•
Also used to refer to the ability of a test to detect only the intended analyte without cross-reacting with
closely related analytes or potentially interfering substances. For example, a test that is intended to be
specific for cytomegalovirus should not cross-react with herpes simplex virus 1.
• Sources of contamination, admixture, and substances that might interfere with analytical specificity:
o Poor sampling
o Maternal cell contamination in prenatal specimens
o Cross contamination during sample processing
o Inclusion of normal, non-diseased tissue with the diseased tissue of interest
o Tissue from source addition to the desired sample
o Bacteria
o Endogenous substances (e.g. hemoglobin, cholesterol, triglycerides)
o Exogenous substances (e.g. medications, anticoagulants, residual sample processing, stabilization
reagents)
Potential interfering substances and cross-reacting nucleic acids that may have an effect on the accurate
measurement/determination of the nucleic acid sequence of interest should be ruled-out through interference
studies during validation.
RM should be selected based on the needs of the assay, the methodology, and the availability of materials.
Potential benefits and drawbacks should be evaluated with respect to each assay.
• Genomic DNA – mimics patient sample in term of complexity but can only control for one or a few
alleles/genotypes or targets at a time
• Recombinant plasmids or synthesized oligonucleotides – do not resemble complexity of genomic DNA but
can control for multiple alleles in a single reaction
Examples:
Human DNA, bacterial and viral genomic DNA, mitochondrial DNA, synthetic DNA, plasmids containing human
DNA, amplicons, in vitro transcripts, synthetic oligonucleotides, recombinant DNA, phage and phage protein
packaged nucleic acid, genetically modified cell lines
Resources: AMP maintains a collection of links to laboratory testing reference and validation materials and
providers at http://www.amp.org/committees/clinical_practice/ValidationResources.cfm.
Accuracy (Trueness)
• Analyze known samples (either of known concentration or result, or both) and compare the result with
that obtained by a reference (“gold-standard”) or comparative method, when the former is not available.
• Test samples that span the entire reportable range for quantitative assays and different possible
genotypes for genotyping assays.
• For sequencing assays
o It is important to establish that the test is capable of detecting appropriate representative types
of DNA changes (e.g. point mutations, deletions, insertions).
Molecular Diagnostic Assay Validation White Paper 6
o If the disease is uncommon, it may be necessary to obtain reference materials from other
laboratories and/or commercial sources.
o However, the number of possible mutations essentially precludes the use of reference materials
that cover every possible mutation.
• For quantitative assays, compare results between the new method and “reference” method or method
already established in the lab. Evaluation of bias between new and comparative method) can be done in
one of the following ways:
o Calculation of average bias and comparison to CAP limits for passing proficiency testing.
o Calculation of statistically significant difference in the mean by applying the t-test.
o Use linear regression analysis:
• Plot data (reference samples/method: x-axis; measured values/new method: y-axis)
• Calculate linear regression statistics (ideally: slope=1, intercept=0 and r=0.99)
• Establish criteria for accepting results (e.g. >95% confidence interval, +/- 2 SD)
• The appropriate number of specimens depends on many factors including, but not limited to:
o complexity of the assay
o frequency of targets/alleles in the intended use population
o established accuracy of reference methods
o whether test is FDA-approved, FDA-cleared, modified FDA-approved or LDP
Precision
• Within run: Run the same sample several times in one run
• Between runs: Run a set of samples in several different runs over several days
• Determine the mean, standard deviation and coefficient of variation (standard deviation ÷ mean)
• Precision studies should be carried out for the entire assay, as applicable, from extraction and amplification
to detection.
o Use a spreadsheet to calculate mean value, standard deviation and %CV for within-run and
between-run precision, and percent of agreement between tests performed under two different
conditions (confidence interval should be calculated for the observed percent agreement). For
FDA approved assays, compare to the manufacturer’s claim; if higher %CV were obtained, then
evaluate the cause.
• Compare precision to clinically acceptable variation (e.g. for HCV quantitative assays,
changes beyond 0.5 log are considered true changes and not intrinsic test variation).
Reportable Range
Qualitative Assays: For real-time PCR assays, a cutoff CT value can be determined using low positive sensitivity
controls as well as negative validation samples. For fragment analysis assays (e.g. NPM1), positive controls can be
tested to determine the mutant peak size range (e.g. 169-171bp). A positive result can be defined as any
detectable and reproducible mutation that results in an amplification product that falls in the mutant peak size
range. For Sanger sequencing and pyrosequencing assays, the reportable range is any nucleotide mutation
identified in the codon(s)/exon(s) of the gene of interest that is being tested. This can be assessed by testing
positive controls and/or positive inter-laboratory validation samples.
Quantitative Assays:
• Assay a series of standards or serial dilutions of a known standard or sample
• Graph the results
o x-axis: Known values; y-axis: Measured values
o Determine slope (ideal = 1.0) and intercept (ideal = 0), calculate regression
CLIA-88 requires that the reportable range does not exceed the range of the available calibrators
Abnormal range: Include in the abnormal control group individuals who are heterozygous and individuals who are
homozygous, if possible.
Analytical Sensitivity
• Can be obtained by measuring a set of samples at different concentrations a given number of times
Excessive amount of analyte (e.g. nucleic acid) can cause incorrect genotype calls; insufficient amount of
analyte can cause allele drop-out and false-negative results. False-negative rate can be determined by
testing individuals who are known to have the condition being tested.
• Limit of Detection (LoD) - the lowest amount of analyte in a sample that can be detected, but not
quantified as an exact value (WHO). It is a concentration of analyte above the limit of the blank that will
be detected 95% of the time with a 95% confidence interval. Can be determined by testing serial dilutions
of multiple replicates with a known concentration of the target substance in the analytical range of the
expected detection limit.
• For multiplex assays LoD must be established for each target singly, as well as verified with other targets
that could potentially be present in the same sample. If one target of multiplex assay is present at a high
level, the detection of another target present at low levels could be potentially impaired (e.g. fetal DNA in
maternal blood, maternal cell contamination in fetal specimens, mosaics or minimal residual disease).
Molecular Diagnostic Assay Validation White Paper 8
• Compare results with those of a “gold-standard” reference method or of another validated method in use
in the lab.
Analytical Specificity
• Testing of potential interference should be conducted at the highest concentration that would occur in a
clinical setting. The maximum amount of a potentially interfering substance the assay can tolerate
without causing actual interference or adversely affecting the test results should be determined.
Interference studies can be conducted by spiking the specimens with interfering agents (spiked vs.
unspiked). Studies should be performed for each specimen matrix used in the assay.
o RNA copurified with varying levels of DNA and vice versa.
o Residual reagents such as organic solvents, wash buffers, residual high salt concentrations,
heparin, and others.
o Cross reactivity –To determine if analytes other than the ones the assay is designed to measure
cause false positive results, a panel of closely related organisms/alleles should be assessed.
• False-positive rate can be determined by testing individuals known to have condition different than that
being tested, but that is similar enough to be in the differential diagnosis (e.g.: a test that is intended to
be specific for cytomegalovirus should not cross-react with herpes simplex virus 1)..
• Compare results with those of a “gold-standard” reference method or of another validated method in use
in the lab.
Documentation
All steps of the Analytical Validation should be carefully documented.
• Documentation - the laboratory must keep results of validation studies as long as the protocol is in use,
and for at least two years after the protocol is retired.
• Reporting/Laboratory Information System
o The test report should reflect the reason for the patient’s referral.
o If pre-written interpretations (“canned comments”) are used, the laboratory director should
ensure that the correct test interpretation is issued.
DISCLAIMER
The AMP Clinical Practice Guidelines and Reports are developed to be of assistance to laboratory and other health
care professionals by providing guidance and recommendations for particular areas of practice. The Guidelines or
Report should not be considered inclusive of all proper approaches or methods, or exclusive of others. The
Guidelines or Report cannot guarantee any specific outcome, nor do they establish a standard of care. The
Guidelines or Report are not intended to dictate the treatment of a particular patient. Treatment decisions must
be made based on the independent judgment of health care providers and each patient’s individual circumstances.
AMP makes no warranty, expressed or implied, regarding the Guidelines or Report and specifically excludes any
warranties of merchantability and fitness for a particular use or purpose. AMP shall not be liable for direct,
indirect, special, incidental, or consequential damages related to the use of the information contained herein.
REFERENCES
1. Prence EM. A practical guide for the validation of genetic tests. Genetic Testing. 1999; 3:201-205.
2. Ferreira-Gonzalez A, Wilkinson DS, Garrett CT. Establishing a molecular diagnostics laboratory. From: McPherson &
Pincus: Henry’s Clinical Diagnosis and Management by Laboratory Methods, 21st ed. WB Saunders. 2006.
3. Ferreira-Gonzalez A, Garrett CT. Laboratory-developed tests in molecular diagnostics. From: Molecular Diagnostics:
For the Clinical Laboratorian, 2nd ed. WB Coleman and GJ Tsongalis. Humana Press.
4. Rennert H and Leonard DGB. Quality control and quality assessment programs. From: Molecular Pathology in Clinical
Practice, pp. 568-574. DGB Leonard, ed. Springer. 2007.
5. Association for Molecular Pathology. Association for Molecular Pathology Statement: Recommendations for in-house
development and operation of molecular diagnostic tests. Am J Clin Pathol. 1999;111:449-463.
6. Jennings L, Van Deerlin VM, Gulley ML. Recommended principles and practices for validating clinical molecular
pathology tests. Arch Pathol Lab Med. 2009; 133:743-755.
7. Burd EM. Validation of laboratory-developed molecular assays for infectious diseases. Clin Microbiol Rev. 2010
Jul;23(3):550-576.
8. Halling KC, Schrijver I, Persons DL. Test verification and validation for molecular diagnostic assays. Arch Pathol Lab
Med. 2012; 136(1):11-13.
9. Wolk, D. M. and E. M. Marlowe. 2011. Molecular Method Verification, p. 861- 884. In: D. P. Piercing (ed.), Molecular
Microbiology, Diagnostic Principles and Practice. 2 ed. ASM Press, Washington,DC.
10. MM17-A, Verification and Validation of Qualitative Nucleic Acid Tests: A Review of the Approved CLSI Guideline.
Molecular Diagnostic Assay Validation White Paper 10
11. World Health Organization Expert Committee on Biological Standardization. Glossary of terms for biological
substances used for texts of the requirements. WHO unpublished document BS/95.1793. 1995. World Health
Organization, Geneva, Switzerland
12. Code of Federal Regulations. 2009. Title 42. Public health, vol. 4, chapter V. Health Care Financing Administration,
Department of Health and Human Services, part 493. Laboratory requirements, section 493.1253. Standard:
establishment and verification of performance specifications. U.S. Government Printing Office, Washington, DC.
13. Electronic Code of Federal regulation. CLIA”88, Title 21, Volume 8, 21CFR820.3
http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?fr=820.3, Revised as of April 1,
2013
14. Sarewitz SJ, 2013 CAP Accreditation program webinar on validation
http://www.cap.org/apps/docs/education/lapaudio/pdf/011613_presentation.pdf)
15. Electronic Code of Federal Regulations. CLIA ’88, Sec. 493.2, Definitions (http://www.ecfr.gov/cgi-
bin/retrieveECFR?gp=1&SID=8ad4874c7299a6e7e3bb5211e8f30fa0&h=L&n=42y5.0.1.1.9&r=PART&ty=HTML#42:5.0.
1.1.9.1.23.2)
16. Ochodo EA, Bossuyt PM. Reporting the accuracy of diagnostic tests: the STARD initiative 10 years on. Clin Chem. 2013
Jun;59(6):917-9
17. Eurachem Guide, 1998. The Fitness for Purpose of Analytical Methods. A Laboratory Guide to Method Validation and
Related Topics. LGC Ltd. Teddington, Middlesex, United Kingdom.
18. IUPAC Technical Report, 2002. Harmonized Guidelines for Single-Laboratory Validation of Methods of Analysis, Pure
Appl. Chem., Vol. 74, (5): 835-855.
19. International Organization for Standardization. 2003. In vitro diagnostic medical devices—measurement of quantities
in biological samples. Metrological traceability of values assigned to calibrators and control materials. ISO 17511.
International Organization for Standardization, Geneva, Switzerland.
20. International Organization for Standardization. 2006. Statistics—vocabulary and symbols, part 1. Probability and
general statistical terms. ISO 3534-1. International Organization for Standardization, Geneva, Switzerland.
21. Chen L, Hadd A, Sah S, Filipovic-Sadic S, Krosting J, Sekinger E, Pan R, Hagerman PJ, Stenzel TT, Tassone F, Latham GJ.
An information-rich CGG repeat primed PCR that detects the full range of fragile X expanded alleles and minimizes the
need for southern blot analysis. J Mol Diagn. 2010 Sep;12(5):589-600.
Molecular Diagnostic Assay Validation White Paper 11
VALIDATION SUMMARY
ASSAY _____________________________________________________________
NEW / MODIFIED
Assay Design
Clinical Validation
Documentation of the clinical validity and clinical utility of the test in light of clinical characteristics of the
disease/marker being tested
• e.g., prevalence of the disease in a relevant population, prevalence of sequence or copy number
variants
Clinical sensitivity (the proportion of patients with the disorder that will to be positive by this test)
Clinical specificity (the proportion of patients without the disorder that will be negative by this test)
Analytical Validation