Evolution of Translational Omics:

Evolution of Translational Omics:Lessons Learned and the Path ForwardIOM Committee onOmics-based Predictive TestsGil Omenn and E. Albert ReeceAAMC Annual Meeting“Theranostics: A Step toward Personalized Medicine”November 4, 2012

SpeakersGil Omenn, IOM Committee ChairProfessor of Computational Medicine & Bioinformatics, InternalMedicine, Human Genetics, and Public HealthUniversity of Michigan, Ann Arbor, MIE. Albert ReeceVice President for Medical Affairs, University of Maryland;John Z. and Akiko K. Bowers Distinguished Professor;Dean, University of Maryland School of Medicine

Disclosure Information:Name of Speaker: Gilbert S. Omenn, MD, PhDI have the following financial relationships to disclose:Employee: University of MichiganBoard of Directors:Amgen Inc.Armune Biosciences Inc.Consultant for (Scientific Advisory Board): Compendia Biosciences, InnocentiveInnovation, Minerva Biotechnology, Galectin Therapeutics, Institute for SystemsBiologySpeaker’s Bureau for: noneGrant/Research support from: National Institutes of HealthStockholder in:Honoraria from:All companies named abovenoneX I will not discuss off label use and/orinvestigational use in my presentation.

Theranostics and Personalized MedicineTheranostics combines molecularly targeted therapies withcompanion diagnostics to stratify and identify patients forwhom the targeted therapy will be most appropriate.The diagnostics are likely to be based on, or informed by,multi-analyte molecular signatures.

Origin of the Task for the IOM Committee• Omics tests developed at Duke to predict sensitivity to chemoRx• Papers in 2006/2007 suggested major advance in directing therapy• Concerns about accuracy and validity were raised immediately and in2009 Annals of Applied Statistics publication by Baggerly and Coombes:‣ Numerous errors in test development‣ Inconsistencies between primary data and data used in articles‣ Failure to reproduce results• Criticisms rebuffed for 4 years while:• Hundreds of publications cited papers in question• 2 companies launched• 3 Clinical trials initiated in 2007, using tests to direct patient care• Trials scrutinized by NCI statistical staff, but apparently not byfunders or general oncologic scientific community• 2010 letter to director of NCI, signed by more than 30bioinformaticians and statisticians, urged suspension of trials• IOM asked to review situation and provide guidance for the field

Committee Charge1. Develop evaluation criteria to determine when omicsbasedtests are fit for use in a clinical trial.2. Apply these criteria to omics-based tests used in threecancer clinical trials conducted by Duke investigators(and other case studies).3. Recommend ways to ensure adherence to the testdevelopment framework going forward.

IOM Committee Composition20 member committee with expertise in:Clinical medicineEthicsClinical pathologyPatient advocacyBiomarker test development FDA oversightBiostatistics and bioinformatics Scientific publicationMolecular biologyUniversity administrationClinical trial design, conduct, and analysisDiscovery and development of omics-based technologies and tests

IOM CommitteeGILBERT S. OMENN (Chair), University of Michigan Medical SchoolCATHERINE D. DEANGELIS, Johns Hopkins School of MedicineDAVID L. DEMETS, University of Wisconsin-MadisonTHOMAS R. FLEMING, University of WashingtonGAIL GELLER, Johns Hopkins UniversityJOE GRAY, Oregon Health & Science University Knight Cancer InstituteDANIEL F. HAYES, University of Michigan Comprehensive Cancer CenterI. CRAIG HENDERSON, University of California San FranciscoLARRY KESSLER, University of Washington School of Public HealthSTANLEY LAPIDUS, SynapDx CorporationDEBRA LEONARD, Weill Medical College of Cornell UniversityHAROLD L. MOSES, Vanderbilt-Ingram Cancer CenterWILLIAM PAO, Vanderbilt University School of MedicineREBECCA D. PENTZ, Emory School of MedicineNATHAN D. PRICE, Institute for Systems BiologyJOHN QUACKENBUSH, Dana-Farber Cancer InstituteELDA RAILEY, Research Advocacy NetworkDAVID RANSOHOFF, University of North Carolina School of MedicineE. ALBERT REECE, University of Maryland School of MedicineDANIELA M. WITTEN, University of Washington

Omics• Encompasses multiple molecular disciplines• Omics-based Test: composed or derived from multiple molecularmeasurements and interpreted by a fully specified computationalmodel to produce a clinically actionable resultOmics Characteristics• Complex, high dimensional data• Many more variables than samples• High risk that computational models will overfit data

Goals of Committee’s RecommendationsGOAL I: Define best practices for discovery, validation,and translation of an omics-based test into clinical trialsfor specified clinical use.[Recommendations 1-3, Framework]GOAL II: Recommend actions to ensure adoption of andadherence to the development and evaluation processby investigators, institutions, funders, FDA, journals.[Recommendations 4-7]

Omics-Based Test DevelopmentFramework: Three Phases

How are Omics Tests Different?• Difficulty in defining the biological rationale underpinning a test• Biological rationale behind single-analyte tests is often evident➠ Examples: HER2, LDL• Biological rationale for an omics-based test is often not well defined• Challenges in data provenance and sharing• Large, complex datasets used to create computational models• Simple data management errors can easily occur• Sharing of data and code is not routine• Difficult for other scientists to replicate and verify findings

How are Omics Tests Different?• Need for individuals with expertise in multiple disciplines• Biologists, geneticists, statisticians, bioinformaticians, clinical pathologists• Responsibility for omics-based test is shared among many investigators,regulatory agencies, funders, and institutions• No single investigator has the breadth of expertise needed to fully understand allaspects of test development• High hope in omics-enabled medical care, but slow progress thus far• Heterogeneity of patients with a given diagnosis• Expensive and uncertain development pathway• No widely accepted process for translation of omics test into clinics• Omics is still an evolving field• Lack of carefully annotated tissue archives from trials with outcome information

Discovery Phase

Discovery Phase• Candidate test is developed on training set, and locked down.• This candidate test is evaluated on an independent sample set.➠ Statistics and bioinformatics validation occurs throughout thediscovery and test validation stage.

Discovery Phase• Step 1: Data quality control Quality control performed computationally Need to assess reproducibility from run to run Sample run date, machine characteristics may confound analysis• Step 2: Computational model development Overfitting is a major concern Training set / test set or cross-validation must be used Model is locked down before proceeding to Step 3• Step 3: Confirmation with an independent sample set• Step 4: Release data, code, and full computational models

Test Validation Phase

Test Validation PhaseAn omics-based test consists of both the data-generatingassay and the fully specified computational model.Both the laboratory assay and the computationalmodel/classifier must be validated.Recommendations apply to both development pathways:‣FDA approval or clearance as a device‣Development of a “Lab-Developed Test” (LDT), as defined bythe FDA, not requiring FDA clearance or approval

How are Omics Tests Different?• Regulatory oversight for omics tests differs from drug development• Two paths for bringing a test to clinical use:1. Via FDA review2. Via validation in a CLIA-certified lab as a Laboratory DevelopedTest (LDT)• LDT Pathway:‣ FDA hasn’t yet clearly defined a regulatory framework for review‣ Academic medical centers can move omics tests from discoveryto clinical use without external regulatory review‣ Academic centers may be unprepared for such oversightresponsibility

Evaluation for Clinical Utility and Use Phase• Clinical Utility: “Evidence of improved measurable clinicaloutcomes . . . compared with current management without[omics] testing.”• Clinical utility is not assessed by FDA or in the LDT process;utility is assessed in clinical trials and in decisions forreimbursement• Lack of FDA review does not mean lack of clinical utility• Process of gathering evidence to support clinical use shouldbegin before test is introduced into clinical practice

Evaluation for Clinical Utility and Use PhaseThree pathways:• Prospective–retrospective studies using archived specimensfrom previously conducted clinical trials.• Prospective clinical trials that directly address the utility of theomics-based test, where either The test does not direct patient management, or The test does direct patient management.

Evaluation for Clinical Utility and Use PhaseSimon, R. M., et al.; J Natl Cancer Inst; 2009Sargent, D. J., et al.; J Clin Oncol; 2005Freidlin, B., et al.; J Natl Cancer Inst; 2010

Case Studies (see report appendix)• HER2• Oncotype DX• MammaPrint• Tissue of Origin• OVA1• OvaCheck• AlloMap• Corus CAD

Lessons Learned From Case StudiesKeys to a successful case study:• A well-designed development plan• Data and code availability• Separate discovery and validation sets• Lock down prior to clinical validation and use of test• Interaction with the FDA• Assessment of clinical utility• Oversight by investigators and institution

Statistical and Bioinformatics ValidationTest Lockdown Reported? Overlap of Discovery andValidation Datasets at SomePoint in Test Development?Oncotype DX Yes (Paik et al., 2004) NoMammaPrint Yes (Personal communication) a YesTissue of OriginYes (Monzon et al., 2009; Pillai etal., 2011; personal communication) bNoOVA1 Yes (Personal communication) c NoAlloMap Yes (Personal communication) d YesCorus CAD Yes (Rosenberg et al., 2010) NoaLaura van‘t Veer, November 28, 2011bEd Stevens, Pathwork Diagnostics, October 18, 2011cScott Henderson, Vermillion, December 12, 2011dMitch Nelles, XDx, October 12, 2011

Duke Case StudyRecommendations for Responsible PartiesE. Albert ReeceUniversity of Maryland• What happened at Duke?• Who are responsible for scrutinizing omics-based testdevelopment and introduction into clinical studies?• What are their responsibilities?

Disclosure Information:Name of Speaker: E. Albert ReeceI have the following financial relationships to disclose:Employee: University of MarylandBoard of Directors:Consultant for (Scientific Advisory Board):Speaker’s Bureau for:Grant/Research support from:Stockholder in:Honoraria from:Patents:X I will not discuss off label use and/orinvestigational use in my presentation.

Problems Identified by Baggerly and Coombes• First identified in letters to the authors and the journals• Extensively documented in Annals of Applied Statistics in 2009• Revealed a series of errors in a number of articles, including:• Reversal of “sensitive/resistant” labels in training data• Errors in test data, such as:• Only 84/122 test samples were distinct• Some samples labeled as both “sensitive” and “resistant”• “Off by one” errors led to erroneous gene lists• Some genes cited as evidence for biological plausibility werenot output by software; 2 were not even on the arrays used.• Heatmap published in one paper corresponded to data from adifferent paper.• Rejected by lead authors and committees as a “squabbleamong statisticians”; later acknowledged as “numerousmissed signals”

ClinicalTrials.gov ID NCT00636441 NCT00509366 NCT00545948Disease Breast cancer Lung cancer Lung cancerStart date April 2008 February 2007 October 2007Date listed inclinicaltrials.govPatient accrualIntendedActualMarch 2008 July 2007 October 200727056Sponsor DOD Eli Lilly/Duke/NCI Eli Lilly/DukePI Markom, MD (Duke) Vhlahovic, MD, MHS(Duke)Chemosensitivity predictorDoxorubicin anddocetaxel8047Cisplatin11724Ready, MD, PhD (Duke;Comp. Cancer Center)Pemetrexed andVinorelbineTermination date 11/4/2010 11/4/2010 2/3/2011Citations inclinicaltrials.govThree Trials at Duke UniversityPotti et al., 2006aBild et al., 2006; Potti etal., 2006aPotti et al., 2006a,b;Potti et al., 2007b

Duke: Retracted Publications• Key retracted papers by Nevins and Potti: first five paper in the series• 2006 Nature Medicine (Potti et al.)‣ Cited 306 times• 2006 New England Journal of Medicine (Potti et al.)‣ Cited 350 times• 2007 Lancet Oncology (Bonnefoi et al.)‣ Cited 95 times• 2007 Journal of Clinical Oncology (Dressman et al.; Hsu et al.)‣ Cited 111 times, 60 times• Duke leadership identified 40 papers with Potti as co-author• Two thirds will be partially or fully retracted• Others may still be valid; pending evaluation (as of 8/11)• Surveyed 162 co-investigators: none had raised any questions

Importance of:Lessons from the Duke Case‣ Data provenance and data management‣ Locking down the computational model‣ Making data, code, and other information publicly available‣ Independent confirmation of the test‣ Test validation‣ Effective multidisciplinary collaboration‣ Institutional and laboratory culture‣ Institutional oversight, including fresh review of the science when seriouscriticisms are raised or clinical trials or spinoff companies are proposed‣ Consultation with FDA and submission of IDELimitations of:‣ Peer review‣ Ability of funders and journals to address scientific controversies‣ Ability of institutions to objectively review work of their faculty

Recommendation 1: Discovery PhaseIf candidate omics-based tests are intended for clinical development:a. The tests should be confirmed using an independent set ofsamples*.b. Data, code, and metadata should be made available.c. Candidate test should be defined precisely:• Molecular measurements• Computational procedures• Intended clinical use*In some cases (e.g. tests developed using preclinical models or datagenerated in early clinical trials, independent data sets may not exist)

Recommendation 2: Test Validation• Test should be discussed with FDA prior to validationstudies.• Test development and validation should be performed in aCLIA-certified clinical laboratory.• CLIA lab should design, optimize, validate, and implementthe test under current clinical laboratory standards.• Analytical validation and CLIA requirements should be metby each laboratory in which test will be performed.

Recommendation 3: Evaluation for Clinical Utilitya. Investigators should communicate early with the FDAregarding Investigational Device Exemption (IDE) process.b. Omics-based tests should not be changed during theclinical trial without a protocol amendment and discussionwith the FDA. A substantive change to the test may requirerestarting study.

Big PictureRecommendations to ensure adoption of andadherence to the development and evaluation process• Investigators and institutions are responsible for the scientificculture in which omics-based tests are developed.• Recommendations are not intended to create new barriers,but rather to maintain the integrity of the scientific process.

Recommendation 4: Institutions4a: Institutions are responsible for establishing, supporting, andoverseeing the infrastructure and research processes foromics-based test development and evaluation.4b: Institutional leaders should promote a culture of integrity andtransparency by designating officials responsible for:i. IDE and IND requirementsii. Management of financial and non-financial conflicts ofinterest (individual and institutional)iii. A system for preventing, reporting, adjudicating lapsesin integrityiv. Establishing clear procedures for response to inquiries

Recommendation 4: Institutions4c: Institutions should ensure that individuals whocollaborate on omics research and test development are:i. Treated as equal co-investigators and co-owners ofresponsibilityii. Represented on relevant review and oversight bodiesiii. Intellectually independentIf institutions lack the capacity for conducting andoverseeing omics-based research and clinicalapplications, they should not engage in suchresearch.

Recommendation 5: Funders5a: All funders of omics-based translational research should:i. Require investigators to make data, prespecified analysisplans, code, and computational models publicly availableii. Provide continuing support for independent repositories toguarantee ongoing access to omics and clinical dataiii. Support test validation in a CLIA-certified laboratory and theindependent confirmation of a candidate omics-basediv. Alert the institutional leadership about serious questionsv. Establish lines of communication with other funders to beused when serious problems arise5b: Federal funders of omics-based research should have authorityto investigate research being conducted by a funding recipient.

Recommendation 6: FDA6a: FDA should develop and finalize a risk-based guidance ora regulation on:i. Bringing omics-based tests to the FDA for reviewii. Oversight of LDTs6b: FDA should communicate IDE requirements for use ofomics-based tests in clinical trials to OHRP, IRBs, and others.

Recommendation 7: JournalsJournal editors should:7a: Require authors describing clinical evaluations of omicsbasedtests to:i. Register all clinical trialsii. Make data, metadata, analysis plans, code, and fullyspecified computational models publicly availableiii. Provide relevant sections of the research protocoliv. Require authors to state roles and attest to study integrityv. Use appropriate reporting guidelines7b: Develop mechanisms to resolve possible serious errors7c: Alert the institutional leadership and all authors when aserious question of accuracy or integrity has been raised

AcknowledgementsStudy Funders:National Cancer InstituteFood and Drug AdministrationCenters for Disease Control and PreventionU.S. Department of Veterans AffairsAmerican Society for Clinical PathologyCollege of American Pathologists.Study Directors:Christine Micheel, PhDSharyl Nass, PhDResearch Assistant:Niharika Sathe, MHSIOM Board Directors:Roger Herdman, MDAndrew Pope, PhDStaff:Program Officers:Erin Balogh, MPHLaura Levit, JDMirzayan Fellows:Sarah Domnitz, PhDJulia Dooher, PhDSenior Program Assistant:Michael Park

Acknowledgements: External ReviewersWYLIE BURKE, University of WashingtonADAM M. CLARKE, MedTran Health StrategiesSUSAN S. ELLENBERG, University ofPennsylvaniaCHARIS ENG, Case Western Reserve UniversitySchool of MedicineMARCUS FELDMAN, Stanford UniversityDAVID B. FLANNERY, Georgia Health SciencesUniversitySTEPHEN FRIEND, Sage BionetworksLARRY GOLD, SomaLogic, Inc.STEVEN GOODMAN, Stanford University Schoolof MedicineROBERT GRAY, Dana Farber Cancer InstituteSTEVEN GRUBBS, Delaware Christiana CareCommunity Clinical Oncology ProgramDAVID KORN, Harvard UniversityMARC LADANYI, Memorial Sloan-KetteringCancer CenterBERNARD LO, University of California, SanFranciscoDAVID MADIGAN, Columbia UniversityBETTIE SUE SILER MASTERS, University ofTexas Health Science Center at San AntonioCHARLES E. PHELPS, University of RochesterDAN RODEN, Vanderbilt University Medical CenterLARRY SHAPIRO, Washington University Schoolof MedicinePETER SHIELDS, The Ohio State UniversityMedical CenterSTEVE TEUTSCH, Los Angeles County PublicHealthDAVID WONG, Director of the Dental ResearchInstitute, UCLAReview Monitor: LAWRENCE D. BROWN, University of PennsylvaniaReview Coordinator: KRISTINE GEBBIE, Flinders University

To download or read the report online:www.nap.edu

Extra SlidesCase Studies

HER 2 Early single marker test for guiding cancer therapy Identifies patients likely to benefit from HER2-targeted therapy Currently 7 tests approved by FDA Difficulties remain:• different testing modalities• lab-to-lab variability• false-positive and false-negative results• differences in cut-off recommendations➠ Challenges in implementing a single-analyte test areamplified for a multi-analyte omics-based test.

Oncotype Dx 21 gene RT-PCR test to predict risk of distant recurrence for node-,ER+ breast cancer Aims:‣ Prognosis: Identify patients at such a low risk of recurrence thatthe risks of chemotherapy outweigh the potential benefit‣ Prediction: Identify patients whose risk might justify chemotherapybut whose cancers are resistant to it LDT, CLIA-certified laboratory; never submitted to FDA AHRQ review (2008): One of the better-established omics-basedtests due to its validation pathway ASCO, NCCN guidelines suggest routine use for prognosis Extensively reviewed for reimbursement; used for 175,000 patients Prospective TAILORx, RxPONDER Trials ongoing to assess clinicalutility as a predictive factor for chemotherapy

MammaPrint 70 gene microarray expression signature to predict risk of distantrecurrence following surgery for patients with ER+/- breast tumors Aim: Prognosis FDA approved ASCO, NCCN Guidelines do not currently recommend it Discovery microarray and clinical data available Prospective MINDACT trial ongoing to assess clinical utility

Tissue of Origin Test Gene expression-based test to identify the primary tissue oforigin for tumors of unknown origin Limited info available; computational model is proprietary Initially offered as an LDT on FFPE samples FDA clearance (for FFPE in 2008; frozen samples in 2010) No studies yet to evaluate impact on clinical outcomes

OvaCheck Serum proteomics-based approach to ovarian cancer screening Unreliable experimental design/analysis Clinical test was planned, but never commercialized Illustrates benefits of making data and code available

OVA 1 Measures 5 abundant proteins in serum to predict the likelihood ofovarian malignancy in patients with adnexal mass for whom surgeryis being planned Used in decision whether to refer to gynecologic oncologist forsurgery Few details on discovery & development (e.g. computational model) Clinical/biological validation study is published FDA clearance; marketed successfully by Vermillion Inc ACOG/SGO noted potential, but said clinical utility not yetestablished (2011) Current use about 4000 patients per quarter (Q1, 2012)

Allomap Blood test; measures expression of 11 genes Used to aid identification of heart transplant recipients withlow risk of moderate/severe acute cellular rejection Some sample overlap for discovery and a clinical validationstudy; developers called it unavoidable due to sample scarcity Company (XDx) met with FDA early to determine thedevelopment pathway for AlloMap; test cleared in 2008

Corus CAD A less invasive method than angiography to identifyobstructive coronary artery disease (CAD) LDT blood test; measures expression of 23 genes Company (CardioDx) had a pre-IDE meeting with FDA The clinical/biological validation study is published