Comparative Effectiveness Research: Moving Medical Oncology Forward Bradford R. Hirsch, MD, MBA,*,† and S. Yousuf Zafar, MD, MHS*,† Comparative effectiveness research (CER) is critically needed in medical oncology to improve the care being delivered to oncology patients. As medical oncologists are forced to rely on insufficient data as a part of daily treatment decision making, and as the cancer treatment landscape evolves quickly relative to other areas of medicine, CER is particularly pressing in our field. Continued reliance on randomized clinical trials is a part of the solution, but it cannot be the sole answer. As new and richer data sources become available addressing quality of life, resource utilization, and other critical elements, the implementation of CER will advance. Its true power will lie in linkages to “learning health systems” and real-time application to the dayto-day practice of medicine. Semin Radiat Oncol 24:49-53 Published by Elsevier Inc.
Introduction
A
lthough randomized clinical trials (RCTs) are the standard of care before drug approval, there is a dearth of evidence in the postmarketing space as to how to best use agents in routine clinical practice. Examples from our clinical experience in medical oncology abound. For instance, 5 new therapies have been approved for the treatment of renal cell carcinoma (RCC) since 2007,1 leading to 8 first-line options and a myriad of options in the second line, as detailed in the most recent edition of the National Comprehensive Cancer Network clinical guidelines.2 However, many of the newly approved agents were compared in landmark trials to prior standards of care and not against other contemporary agents. The National Institute for Clinical Excellence in the United Kingdom has consistently declined to cover many novel anticancer agents because of limited data proving their clinical utility and costeffectiveness, particularly in third and later lines of therapy.3 However, no such controls exist on a national scale in the *Duke Cancer Institute, Durham, NC. †Duke Clinical Research Institute, Durham, NC. BRH is supported by grant funding from Pfizer, Dendreon, Duke University, and Academy Health/Robert Wood Johnson Foundation, and he also receives research funding from Bristol Meyers Squibb. SYZ is supported by an American Cancer Society Mentored Research Scholar Grant, the Duke Cancer Institute Cancer Control and Prevention Pilot Award, and the Duke Clinical Research Institute Comparative Effectiveness Research KM1 Award. Address reprint requests to Bradford R Hirsch, MD, MBA, Duke Clinical Research Institute, 25165 Morris Building, Box 3436, Durham, NC 27710. E-mail: [email protected]
1053-4296/13/$-see front matter Published by Elsevier Inc. http://dx.doi.org/10.1016/j.semradonc.2013.08.005
United States (US). Complicating the evidentiary uncertainty further is the fact that RCTs often do not reflect populations treated in routine clinical practice, where patients commonly present with multiple comorbidities, poor functional status, and other confounding factors. In light of these complexities, the appeal of comparative effectiveness research (CER) is clear. As defined by the Federal Coordinating Council, and endorsed by the Institute of Medicine among others, CER is the conduct and synthesis of research comparing the benefits and harms of different interventions and strategies to prevent, diagnose, treat and monitor health conditions in ‘real world’ settings. The purpose of this research is to improve health outcomes by developing and disseminating evidence-based information to patients, clinicians, and other decision-makers, responding to their expressed needs, about which interventions are most effective for which patients under specific circumstances.4 Guidelines generated by groups including the National Comprehensive Cancer Network and the American Society of Clinical Oncology (ASCO) attempt to use CER to guide decisions in medical oncology, yet studies have questioned whether adequate evidence exists to assess the relative efficacy of agents and whether professional bodies have the ability to aggregate the available information in a timely and reliable fashion.5 Practicing clinicians deal with these uncertainties in data on a daily basis; indeed, much of the practice of medical oncology is reliant on off-label chemotherapy use based on small, underpowered studies.6-8 This lack of high-quality 49
50 evidence has led to a call for more CERs. However, for many oncologists, the value of CER is reduced to nothing more than a catchphrase because of the complexity in utilizing it reliably in practice. Questions remain as to how challenges in design and implementation of CER can be overcome while its value and relevance is retained. In this review, we highlight a number of key themes emerging in the field of CER and health services research as it relates to medical oncology. First, we cover the role of RCTs in CER, outlining its strengths and weaknesses. Second, we turn to the contribution of observational studies to CER in medical oncology. Finally, we discuss new approaches that are being developed to advance the agenda for CER in the near future. Although CER is critically important to practicing medical oncologists, we must reexamine how to achieve its aims if we hope to move the field forward.
CER and Clinical Trials CER in traditional RCTs is easy to identify in medical oncology as it consists of head-to-head comparisons between agents meant to generate evidence about relative effectiveness. As an example, the recent phase III AXIS trial compared axitinib (Inlyta, Pfizer, New York, NY) directly with sorafenib (Nexavar, Bayer, Leverkusen, Germany) in advanced RCC.9 Both of these drugs are readily available to oncologists and considered the standard of care. The results of the head-to-head comparison represent a step forward in guiding clinical practice. There are a number of advantages to the use of RCTs for CER. Randomization in this setting helps to generate reliable evidence, preventing issues, such as selection bias and confounding from affecting the results. These biases occur for a number of reasons, such as the assignment of a patient to a given agent or clinical course based on underlying factors that may not be known or the existence of factors that influence outcomes that are not easily quantified. Randomization frees researchers from these potential complications by helping to ensure that the patient populations are comparable between study arms. One could then argue that, in an ideal world, all CER would be generated via RCTs. This is both impractical and inappropriate in medical oncology. It is impractical because of structural issues inherent to RCTs: exorbitant costs; large required sample sizes and poor enrollment; trial design complexity; the time horizon from initial enrollment to trial completion; and stifling regulatory requirements. Dilts et al10 showed that to open a phase III cooperative groups trial in medical oncology requires 369 steps, 36 approvals, and a median time of 2.5 years, and Cheng et al11 showed that, among trials within the National Cancer Institute's Cancer Therapy Evaluation Program, 70.8% of phase III trials experienced poor enrollment. In addition, a recent analysis of oncology trials registered between 2007 and 2010 found that 63.9% were nonrandomized, 62.3% were single arm, and 87.8% were nonblinded.12 In addition, most trials were phase I or II. Together, these data suggest that a large proportion of oncology trials fail to meet the needs of stakeholders and forgo the rigor necessary for sound conclusions.
B.R. Hirsch and S.Y. Zafar The nature of RCTs also limits the applicability of findings. The patients in these trials are often free of comorbidities, younger than average, and followed up far more closely than occurs in routine practice, further compromising external validity. To truly generate comparative effectiveness evidence, patients in trials need to accurately reflect those treated in the community setting, where the majority of cancer care occurs. Furthermore, variations in design and end points often make direct comparisons difficult. In light of these limitations in many cancer RCTs, it is hard to imagine a circumstance in which reliance on RCTs alone will provide answers to all the pressing questions in medical oncology. However, progress is being made in RCT design, which is helpful since it remains the standard of care. For instance, ASCO recently published draft recommendations highlighting the clinical outcomes that they felt to be critical to include in medical oncology trials by cancer type. The hope is to forge basic agreement moving forward on items such as the need to incorporate replicable disease-specific quality-of-life data into trials.13 The authors acknowledged that “providing guidelines and benchmarks for clinical trials outcomes across all cancers is a daunting task.” Organizations such as the Center for Medical Technology Policy attempt to drive similar agreements through the generation of evidence guidance documents in areas such as the use of patient-reported outcomes (PROs) and the design of genetic studies.14 Consensus as to which outcomes and trial types to prioritize in RCTs represents a critical step toward the facilitation of CER.
Observational Data for CER Despite the critical steps that have been taken toward improving clinical trial design, we cannot rely solely on traditional RCTs to generate the elements of CER. There are a growing number of established data sources to supplement RCTs in medical oncology, as outlined in a recent white paper by Meyer et al.15 They range from administrative and claims data from insurers like Blue Cross Blue Shield, to databases within health systems such as Kaiser, to linked clinical results and claims systems such as the Surveillance, Epidemiology, and End Results (SEER)Medicare database. Although none of the available data sources are sufficiently robust to adequately answer all CER questions, they provide a critical substrate for CER and are being increasingly used to generate evidence where appropriate. An interesting discussion around the strengths and weaknesses of CER stemmed from a SEER-Medicare analysis reported by Meyerhardt et al.16 The authors used the SEERMedicare database to assess the effectiveness of bevacizumab (Avastin, Genentech, San Francisco, CA) added to traditional fluorouracil-based chemotherapy in patients treated between 2002 and 2007 for colorectal cancer. They found that its addition was associated with improved survival with an adjusted hazard ratio (HR) of 0.85 (95% CI; 0.66-0.97). The survival benefit was less robust when looked at in later years (2004-2007, HR ¼ 0.93, 95% CI; 0.84-1.02) vs pre-2004 use, and less robust in oxaliplatin-based (Eloxatin, Sanofi-Aventis, Paris, France) chemotherapy regimens (HR ¼ 0.96, 95% CI;
CER: Moving medicine forward 0.86-1.07) vs irinotecan-based (Camptosar, Pfizer, New York, NY) regimens (HR ¼ 0.80, 95% CI; 0.66-0.97). They also found a significant increase in the risk of stroke (4.9% vs 2.5%, P o 0.01) and gastrointestinal perforation (2.3% vs 1.0%; P o 0.01) with the addition of bevacizumab. The authors concluded that the use of bevacizumab was associated with only a “small improvement in overall survival” while causing an increased risk of stroke and perforation. An accompanying editorial by Hurwitz and Lyman17 pointed out that this finding “seemed to depart from the majority of previously published data from RCTs, registries, and meta-analyses.” They questioned whether results from a retrospective analysis using a linked-claims database could be used to overturn prior evidence from RCTs with somewhat different outcomes. To call into question the ability of the results to truly change practice, they referenced the role of confounding in the retrospective analysis, evolutions in the way the agents were used in routine practice over time, and variations in end points between different trials and approaches. As is illustrated, one of the difficulties in using observational data for CER in medical oncology is the potential for confounding. Without a comprehensive assessment of baseline patient characteristics, results of observational CER studies might be biased. In place of RCTs, observational data can lead to a risk of erroneous conclusions about the effectiveness of various treatments. There are examples of how observational data have led us astray in medical oncology. The most commonly referenced is the use of bone marrow transplantation in over 40,000 patients with breast cancer in the 1990s; this practice was based on observational data that were later discredited when RCTs were completed on the intervention.18-20 As baseline patient differences may influence outcomes in subtle and immeasurable ways, new approaches to overcome the limitations beyond simple stratification and regression analyses are evolving including the use of instrumental variable analyses and propensity matching as explained elsewhere in this issue. However, even these relatively novel approaches have their limitations. For instance, a recent editorial by Korn and Freidlin21 outlined the limitations of instrumental variable analyses that rely on “nonverifiable assumptions” about the relationship of specific variables to outcomes and have wide confidence intervals that often result in findings that are difficult to interpret. These may be a bigger issue in medical oncology than in other disciplines. Yet, with the limitations of RCTs in mind, we must move forward. Doing so requires identification of areas in which only observational data are or will be available and areas in which the generated insights will be enough to change practice. Those areas where this is not possible should then be the focus of RCTs. As outlined later, the evolution of available data sources will also improve opportunities to generate reliable, nonrandomized data. We must devise a way to reconcile the strengths and weaknesses of observational data in medical oncology, especially as CER will become increasingly important with a growing number of agents that often provide marginal benefit in the setting of real toxicity.
51
Evolving Data Sources In Medical Oncology, one area of progress is in the evolution of data sources. The HITECH Act of the American Recovery and Reinvestment Act and the Patient Protection and Affordable Care Act are driving the uptake of electronic health records (EHRs), moving toward the ubiquitous generation of data from routine clinical practice, and presenting an opportunity for far more robust CER. However, the tools to aggregate and analyze the information are not yet mature. The Center for Learning Health Care at the Duke Clinical Research Institute and numerous other groups such as the West Clinic (a large community oncology practice-based in Memphis, TN) are developing mobile health and informaticsenabled approaches to augment the data from EHRs needed for CER in medical oncology, with potential applications to other specialties.22,23 A key aspect of these efforts is in the generation of PROs, in which surveys are electronically provided to patients and they generate data about their clinical experience in real time. By producing robust data taken directly from the patient, far more insightful conclusions are possible from observational evidence. The patient experience can augment what we learn from hard survival outcomes and claims data.22,23 The inclusion of PROs is critical to enable comparisons across an increasing number of treatments with similar overall outcomes; doing so will help define and personalize outcomes. Quality-of-life data is already being incorporated into clinical research as shown in the COMPARZ24 and PISCES25 trials in which patient preferences are playing a role in helping to define the preferred agent for first-line therapy for RCC. The incorporation of quality-of-life assessments into observational data will help minimize confounding and move CER forward as it provides an entirely new level of detail about why choices are made and the effects thereof.
Potential Effect of CER The ability of CER to truly advance the practice of oncology is likely to evolve with changes in reimbursement structures such as accountable care organizations and pay-for-performance. A high profile example of the increasing focus on the balance of efficacy and cost was seen when leaders at the Memorial Sloan-Kettering Cancer Center did not allow the addition of aflibercept (Eylea, Sanofi-Aventis, Paris, France) to the hospital's formulary. They argued that the evidence supporting the use of aflibercept did not demonstrate superiority to the existing option, bevacizumab, despite being substantially more expensive. 26 This decision generated a great deal of press in the medical oncology community but, more importantly, in the lay press. Soon, the pricing of the agent was changed by Sanofi-Aventis, bringing the price in line with bevacizumab at its standard dosing. As the focus rises on the balance between cost and effectiveness, there will be an increasing need for CER.
52 A fundamental shortcoming of traditional CER is its conduct in a vacuum, whereby data are generated and published, but rarely integrated directly into care. This too is evolving, under the label of the “learning health system.” In this framework, routine platform care generates new data using EHRs, mobile health, PROs, and other tools as outlined previously. The resulting data are analyzed and then used to design new guidelines, clinical decision support, and other mechanisms to improve care. The cycle then repeats, with the generation of new data from routine practice. There are a number of terms for this approach, including rapid learning, design-implementevaluate-adjust, and plan-do-study-act cycles.27,28 The infrastructure to conduct large-scale research in a learning health system has not existed to date as it requires the real-time generation of information, the ability to visualize and analyze data quickly, and the mechanism for integration of results and recommendations back into practice. With the evaluation of the toolset available to clinicians and researchers, this paradigm is shifting. Rapid learning health care, when implemented with appropriate quality, rigor, and controls, will be the key to the success of CER.29 Relying on rapid learning health care as a backbone to CER is gaining increased traction in medical oncology. The most obvious example is in the CancerLinQ project by ASCO.29 ASCO is planning to “harness the power of ‘Big Data’ to improve care,” by analyzing medical data in real time to provide decision support to practicing oncologists. In this process, troves of data will be aggregated at individual institutions and practices, as well as nationally, to analyze and better understand care patterns and outcomes. These data will then be analyzed to provide feedback and advice at the bedside to drive care improvement. Although access to more data is a step toward successful implementation of CER, it is the increase in quality and robustness that will truly help to overcome the challenges outlined previously.
Conclusion CER is critical to the practice of precision medicine. As medical oncologists are forced to rely on insufficient data as a part of daily treatment decision making, and as the cancer treatment landscape evolves quickly relative to other areas of medicine, CER is particularly pressing in our field. Continued reliance on RCTs is a part of the solution, but it cannot be the sole answer. As newer and richer data sources become available—including quality of life, resource utilization, and the like—the implementation of CER will advance. Its true power will lie in linkages to “learning health systems” and real-time application to the day-to-day practice of medicine.
References 1. FDA: Approved drugs by therapeutic area. Available at: 〈http://www. centerwatch.com/drug-information/fda-approvals/drug-areas.aspx? AreaID=12〉. Accessed April 24, 2013 2. National Comprehensive Cancer Network. NCCN Clinical Practice Guidelines in Oncology (NCCN Guidelines): Kidney Cancer. Ft. Washington, PA, 2012
B.R. Hirsch and S.Y. Zafar 3. National Institute of Health and Clinical Excellence. Available at: 〈http:// www.nice.org.uk/〉. Accessed April 24, 2013 4. Services USDoHaH. Report to the President and the Congress on comparative effectiveness research, 2009. Available at: 〈http://www.hhs. gov/recovery/programs/cer/execsummary.html〉. Accessed April 24, 2013 5. Abernethy AP, Raman G, Balk EM, et al: Systematic review: Reliability of compendia methods for off-label oncology indications. Ann Intern Med 150:336-343, 2009 6. Krzyzanowska MK: Off-label use of cancer drugs: A benchmark is established. J Clin Oncol 31:1125-1127, 2013 7. Mullins CD, Montgomery R, Abernethy AP, et al: Recommendations for clinical trials of off-label drugs used to treat advanced-stage cancer. J Clin Oncol 30:661-666, 2012 8. Conti RM, Bernstein AC, Villaflor VM, et al: Prevalence of off-label use and spending in 2010 among patent-protected chemotherapies in a population-based cohort of medical oncologists. J Clin Oncol 31:1134-1139, 2013 9. Rini BI, Escudier B, Tomczak P, et al: Comparative effectiveness of axitinib versus sorafenib in advanced renal cell carcinoma (AXIS): A randomised phase 3 trial. Lancet 378:1931-1939, 2011 10. Dilts DM, Cheng SK, Crites JS, et al: Phase III clinical trial development: A process of chutes and ladders. Clin Cancer Res 16:5381-5389, 2010 11. Cheng SK, Dietrich MS, Dilts DM: A sense of urgency: Evaluating the link between clinical trial development time and the accrual performance of cancer therapy evaluation program (NCI-CTEP) sponsored studies. Clin Cancer Res 16:5557-5563, 2010 12. Hirsch BR, Califf RM, Cheng SK, et al: The state of the oncology clinical trials portfolio: Insights from a systematic analysis of ClinicalTrials.gov. JAMA Intern Med 173:972-979, 2013 13. Ellis L.M., Bernstein D., Voest E., et al. Defining clinically meaningful outcomes: ASCO recommendations to raise the bar for clinical trials. Draft for public commnet. Available at: 〈http://www.asco.org/sites/www.asco. org/files/asco_meaningful_outcomes_4.23.12_draft_for_comment_ap ril_20131.pdf〉. Accessed April 24, 2013 14. Center for Medical Technology Policy (CMTP). Effectiveness Guidance Documents—A method for comparative effectiveness documents. Available at: 〈http://www.cmtpnet.org/effectiveness-guidance-documents/〉. Accessed April 24, 2013 15. Meyer AM, Carpenter WR, Abernethy AP, et al: Data for cancer comparative effectiveness research: Past, present, and future potential. Cancer 118:5186-5197, 2012 16. Meyerhardt JA, Li L, Sanoff HK, et al: Effectiveness of bevacizumab with first-line combination chemotherapy for Medicare patients with stage IV colorectal cancer. J Clin Oncol 30:608-615, 2012 17. Hurwitz HI, Lyman GH: Registries and randomized trials in assessing the effects of bevacizumab in colorectal cancer: Is there a common theme? J Clin Oncol 30:580-581, 2012 18. Mello MM, Brennan TA: The controversy over high-dose chemotherapy with autologous bone marrow transplant for breast cancer. Health Aff 20:101-117, 2001 19. Zujewski J, Nelson A, Abrams J: Much ado about not...enough data: Highdose chemotherapy with autologous stem cell rescue for breast cancer. J Natl Cancer Inst 90:200-209, 1998 20. Hortobagyi GN: What is the role of high-dose chemotherapy in the era of targeted therapies? J Clin Oncol 22:2263-2266, 2004 21. Korn EL, Freidlin B: Methodology for comparative effectiveness research: Potential and limitations. J Clin Oncol 30:4185-4187, 2012 22. Hirsch BR, Abernethy AP: Leveraging informatics, mHealth, and biobanks to treat each patient right. Pers Med 9:849-857, 2012 23. Hirsch BR, Abernethy AP: Structured decision-making: Using personalized medicine to improve the value of cancer care. J Pers Med 3:1-13, 2013 24. Motzer R, Hutson TE, Reeves J, et al: Randomized, open-label, phase III trial of pazopanib versus sunitinib in first-line treatment of patients with metastatic renal cell carcinoma (mRCC): Results of the COMPARZ trial. Ann Oncol 238: 2012 25. Escudier BJ, Porta C, Bono P, et al: Patient preference between pazopanib (Paz) and sunitinib (Sun): Results of a randomized double-blind, placebocontrolled, cross-over study in patients with metastatic renal cell
CER: Moving medicine forward carcinoma (mRCC)—PISCES study, NCT 01064310. J Clin Oncol 30: Suppl; abstr CRA4502, 2012 26. Bach P, Saltz L, Wittes R: In Cancer Care, Cost Matters. New York Times, October 14, 2012:A25 27. Cleghorn GD, Headrick LA: The PDSA cycle at the core of learning in health professions education. Jt Comm J Qual Improv 22:206-212, 1996
53 28. Greene SM, Reid RJ, Larson EB: Implementing the learning health system: From concept to action. Ann Intern Med 157:207-210, 2012 29. Miriovsky BJ, Shulman LN, Abernethy AP: Importance of health information technology, electronic health recrods and continuously aggregating data to comparative effectiveness research and rapid learning systems. J Clin Oncol 30:4243-4248, 2012
Introduction
A
lthough randomized clinical trials (RCTs) are the standard of care before drug approval, there is a dearth of evidence in the postmarketing space as to how to best use agents in routine clinical practice. Examples from our clinical experience in medical oncology abound. For instance, 5 new therapies have been approved for the treatment of renal cell carcinoma (RCC) since 2007,1 leading to 8 first-line options and a myriad of options in the second line, as detailed in the most recent edition of the National Comprehensive Cancer Network clinical guidelines.2 However, many of the newly approved agents were compared in landmark trials to prior standards of care and not against other contemporary agents. The National Institute for Clinical Excellence in the United Kingdom has consistently declined to cover many novel anticancer agents because of limited data proving their clinical utility and costeffectiveness, particularly in third and later lines of therapy.3 However, no such controls exist on a national scale in the *Duke Cancer Institute, Durham, NC. †Duke Clinical Research Institute, Durham, NC. BRH is supported by grant funding from Pfizer, Dendreon, Duke University, and Academy Health/Robert Wood Johnson Foundation, and he also receives research funding from Bristol Meyers Squibb. SYZ is supported by an American Cancer Society Mentored Research Scholar Grant, the Duke Cancer Institute Cancer Control and Prevention Pilot Award, and the Duke Clinical Research Institute Comparative Effectiveness Research KM1 Award. Address reprint requests to Bradford R Hirsch, MD, MBA, Duke Clinical Research Institute, 25165 Morris Building, Box 3436, Durham, NC 27710. E-mail: [email protected]
1053-4296/13/$-see front matter Published by Elsevier Inc. http://dx.doi.org/10.1016/j.semradonc.2013.08.005
United States (US). Complicating the evidentiary uncertainty further is the fact that RCTs often do not reflect populations treated in routine clinical practice, where patients commonly present with multiple comorbidities, poor functional status, and other confounding factors. In light of these complexities, the appeal of comparative effectiveness research (CER) is clear. As defined by the Federal Coordinating Council, and endorsed by the Institute of Medicine among others, CER is the conduct and synthesis of research comparing the benefits and harms of different interventions and strategies to prevent, diagnose, treat and monitor health conditions in ‘real world’ settings. The purpose of this research is to improve health outcomes by developing and disseminating evidence-based information to patients, clinicians, and other decision-makers, responding to their expressed needs, about which interventions are most effective for which patients under specific circumstances.4 Guidelines generated by groups including the National Comprehensive Cancer Network and the American Society of Clinical Oncology (ASCO) attempt to use CER to guide decisions in medical oncology, yet studies have questioned whether adequate evidence exists to assess the relative efficacy of agents and whether professional bodies have the ability to aggregate the available information in a timely and reliable fashion.5 Practicing clinicians deal with these uncertainties in data on a daily basis; indeed, much of the practice of medical oncology is reliant on off-label chemotherapy use based on small, underpowered studies.6-8 This lack of high-quality 49
50 evidence has led to a call for more CERs. However, for many oncologists, the value of CER is reduced to nothing more than a catchphrase because of the complexity in utilizing it reliably in practice. Questions remain as to how challenges in design and implementation of CER can be overcome while its value and relevance is retained. In this review, we highlight a number of key themes emerging in the field of CER and health services research as it relates to medical oncology. First, we cover the role of RCTs in CER, outlining its strengths and weaknesses. Second, we turn to the contribution of observational studies to CER in medical oncology. Finally, we discuss new approaches that are being developed to advance the agenda for CER in the near future. Although CER is critically important to practicing medical oncologists, we must reexamine how to achieve its aims if we hope to move the field forward.
CER and Clinical Trials CER in traditional RCTs is easy to identify in medical oncology as it consists of head-to-head comparisons between agents meant to generate evidence about relative effectiveness. As an example, the recent phase III AXIS trial compared axitinib (Inlyta, Pfizer, New York, NY) directly with sorafenib (Nexavar, Bayer, Leverkusen, Germany) in advanced RCC.9 Both of these drugs are readily available to oncologists and considered the standard of care. The results of the head-to-head comparison represent a step forward in guiding clinical practice. There are a number of advantages to the use of RCTs for CER. Randomization in this setting helps to generate reliable evidence, preventing issues, such as selection bias and confounding from affecting the results. These biases occur for a number of reasons, such as the assignment of a patient to a given agent or clinical course based on underlying factors that may not be known or the existence of factors that influence outcomes that are not easily quantified. Randomization frees researchers from these potential complications by helping to ensure that the patient populations are comparable between study arms. One could then argue that, in an ideal world, all CER would be generated via RCTs. This is both impractical and inappropriate in medical oncology. It is impractical because of structural issues inherent to RCTs: exorbitant costs; large required sample sizes and poor enrollment; trial design complexity; the time horizon from initial enrollment to trial completion; and stifling regulatory requirements. Dilts et al10 showed that to open a phase III cooperative groups trial in medical oncology requires 369 steps, 36 approvals, and a median time of 2.5 years, and Cheng et al11 showed that, among trials within the National Cancer Institute's Cancer Therapy Evaluation Program, 70.8% of phase III trials experienced poor enrollment. In addition, a recent analysis of oncology trials registered between 2007 and 2010 found that 63.9% were nonrandomized, 62.3% were single arm, and 87.8% were nonblinded.12 In addition, most trials were phase I or II. Together, these data suggest that a large proportion of oncology trials fail to meet the needs of stakeholders and forgo the rigor necessary for sound conclusions.
B.R. Hirsch and S.Y. Zafar The nature of RCTs also limits the applicability of findings. The patients in these trials are often free of comorbidities, younger than average, and followed up far more closely than occurs in routine practice, further compromising external validity. To truly generate comparative effectiveness evidence, patients in trials need to accurately reflect those treated in the community setting, where the majority of cancer care occurs. Furthermore, variations in design and end points often make direct comparisons difficult. In light of these limitations in many cancer RCTs, it is hard to imagine a circumstance in which reliance on RCTs alone will provide answers to all the pressing questions in medical oncology. However, progress is being made in RCT design, which is helpful since it remains the standard of care. For instance, ASCO recently published draft recommendations highlighting the clinical outcomes that they felt to be critical to include in medical oncology trials by cancer type. The hope is to forge basic agreement moving forward on items such as the need to incorporate replicable disease-specific quality-of-life data into trials.13 The authors acknowledged that “providing guidelines and benchmarks for clinical trials outcomes across all cancers is a daunting task.” Organizations such as the Center for Medical Technology Policy attempt to drive similar agreements through the generation of evidence guidance documents in areas such as the use of patient-reported outcomes (PROs) and the design of genetic studies.14 Consensus as to which outcomes and trial types to prioritize in RCTs represents a critical step toward the facilitation of CER.
Observational Data for CER Despite the critical steps that have been taken toward improving clinical trial design, we cannot rely solely on traditional RCTs to generate the elements of CER. There are a growing number of established data sources to supplement RCTs in medical oncology, as outlined in a recent white paper by Meyer et al.15 They range from administrative and claims data from insurers like Blue Cross Blue Shield, to databases within health systems such as Kaiser, to linked clinical results and claims systems such as the Surveillance, Epidemiology, and End Results (SEER)Medicare database. Although none of the available data sources are sufficiently robust to adequately answer all CER questions, they provide a critical substrate for CER and are being increasingly used to generate evidence where appropriate. An interesting discussion around the strengths and weaknesses of CER stemmed from a SEER-Medicare analysis reported by Meyerhardt et al.16 The authors used the SEERMedicare database to assess the effectiveness of bevacizumab (Avastin, Genentech, San Francisco, CA) added to traditional fluorouracil-based chemotherapy in patients treated between 2002 and 2007 for colorectal cancer. They found that its addition was associated with improved survival with an adjusted hazard ratio (HR) of 0.85 (95% CI; 0.66-0.97). The survival benefit was less robust when looked at in later years (2004-2007, HR ¼ 0.93, 95% CI; 0.84-1.02) vs pre-2004 use, and less robust in oxaliplatin-based (Eloxatin, Sanofi-Aventis, Paris, France) chemotherapy regimens (HR ¼ 0.96, 95% CI;
CER: Moving medicine forward 0.86-1.07) vs irinotecan-based (Camptosar, Pfizer, New York, NY) regimens (HR ¼ 0.80, 95% CI; 0.66-0.97). They also found a significant increase in the risk of stroke (4.9% vs 2.5%, P o 0.01) and gastrointestinal perforation (2.3% vs 1.0%; P o 0.01) with the addition of bevacizumab. The authors concluded that the use of bevacizumab was associated with only a “small improvement in overall survival” while causing an increased risk of stroke and perforation. An accompanying editorial by Hurwitz and Lyman17 pointed out that this finding “seemed to depart from the majority of previously published data from RCTs, registries, and meta-analyses.” They questioned whether results from a retrospective analysis using a linked-claims database could be used to overturn prior evidence from RCTs with somewhat different outcomes. To call into question the ability of the results to truly change practice, they referenced the role of confounding in the retrospective analysis, evolutions in the way the agents were used in routine practice over time, and variations in end points between different trials and approaches. As is illustrated, one of the difficulties in using observational data for CER in medical oncology is the potential for confounding. Without a comprehensive assessment of baseline patient characteristics, results of observational CER studies might be biased. In place of RCTs, observational data can lead to a risk of erroneous conclusions about the effectiveness of various treatments. There are examples of how observational data have led us astray in medical oncology. The most commonly referenced is the use of bone marrow transplantation in over 40,000 patients with breast cancer in the 1990s; this practice was based on observational data that were later discredited when RCTs were completed on the intervention.18-20 As baseline patient differences may influence outcomes in subtle and immeasurable ways, new approaches to overcome the limitations beyond simple stratification and regression analyses are evolving including the use of instrumental variable analyses and propensity matching as explained elsewhere in this issue. However, even these relatively novel approaches have their limitations. For instance, a recent editorial by Korn and Freidlin21 outlined the limitations of instrumental variable analyses that rely on “nonverifiable assumptions” about the relationship of specific variables to outcomes and have wide confidence intervals that often result in findings that are difficult to interpret. These may be a bigger issue in medical oncology than in other disciplines. Yet, with the limitations of RCTs in mind, we must move forward. Doing so requires identification of areas in which only observational data are or will be available and areas in which the generated insights will be enough to change practice. Those areas where this is not possible should then be the focus of RCTs. As outlined later, the evolution of available data sources will also improve opportunities to generate reliable, nonrandomized data. We must devise a way to reconcile the strengths and weaknesses of observational data in medical oncology, especially as CER will become increasingly important with a growing number of agents that often provide marginal benefit in the setting of real toxicity.
51
Evolving Data Sources In Medical Oncology, one area of progress is in the evolution of data sources. The HITECH Act of the American Recovery and Reinvestment Act and the Patient Protection and Affordable Care Act are driving the uptake of electronic health records (EHRs), moving toward the ubiquitous generation of data from routine clinical practice, and presenting an opportunity for far more robust CER. However, the tools to aggregate and analyze the information are not yet mature. The Center for Learning Health Care at the Duke Clinical Research Institute and numerous other groups such as the West Clinic (a large community oncology practice-based in Memphis, TN) are developing mobile health and informaticsenabled approaches to augment the data from EHRs needed for CER in medical oncology, with potential applications to other specialties.22,23 A key aspect of these efforts is in the generation of PROs, in which surveys are electronically provided to patients and they generate data about their clinical experience in real time. By producing robust data taken directly from the patient, far more insightful conclusions are possible from observational evidence. The patient experience can augment what we learn from hard survival outcomes and claims data.22,23 The inclusion of PROs is critical to enable comparisons across an increasing number of treatments with similar overall outcomes; doing so will help define and personalize outcomes. Quality-of-life data is already being incorporated into clinical research as shown in the COMPARZ24 and PISCES25 trials in which patient preferences are playing a role in helping to define the preferred agent for first-line therapy for RCC. The incorporation of quality-of-life assessments into observational data will help minimize confounding and move CER forward as it provides an entirely new level of detail about why choices are made and the effects thereof.
Potential Effect of CER The ability of CER to truly advance the practice of oncology is likely to evolve with changes in reimbursement structures such as accountable care organizations and pay-for-performance. A high profile example of the increasing focus on the balance of efficacy and cost was seen when leaders at the Memorial Sloan-Kettering Cancer Center did not allow the addition of aflibercept (Eylea, Sanofi-Aventis, Paris, France) to the hospital's formulary. They argued that the evidence supporting the use of aflibercept did not demonstrate superiority to the existing option, bevacizumab, despite being substantially more expensive. 26 This decision generated a great deal of press in the medical oncology community but, more importantly, in the lay press. Soon, the pricing of the agent was changed by Sanofi-Aventis, bringing the price in line with bevacizumab at its standard dosing. As the focus rises on the balance between cost and effectiveness, there will be an increasing need for CER.
52 A fundamental shortcoming of traditional CER is its conduct in a vacuum, whereby data are generated and published, but rarely integrated directly into care. This too is evolving, under the label of the “learning health system.” In this framework, routine platform care generates new data using EHRs, mobile health, PROs, and other tools as outlined previously. The resulting data are analyzed and then used to design new guidelines, clinical decision support, and other mechanisms to improve care. The cycle then repeats, with the generation of new data from routine practice. There are a number of terms for this approach, including rapid learning, design-implementevaluate-adjust, and plan-do-study-act cycles.27,28 The infrastructure to conduct large-scale research in a learning health system has not existed to date as it requires the real-time generation of information, the ability to visualize and analyze data quickly, and the mechanism for integration of results and recommendations back into practice. With the evaluation of the toolset available to clinicians and researchers, this paradigm is shifting. Rapid learning health care, when implemented with appropriate quality, rigor, and controls, will be the key to the success of CER.29 Relying on rapid learning health care as a backbone to CER is gaining increased traction in medical oncology. The most obvious example is in the CancerLinQ project by ASCO.29 ASCO is planning to “harness the power of ‘Big Data’ to improve care,” by analyzing medical data in real time to provide decision support to practicing oncologists. In this process, troves of data will be aggregated at individual institutions and practices, as well as nationally, to analyze and better understand care patterns and outcomes. These data will then be analyzed to provide feedback and advice at the bedside to drive care improvement. Although access to more data is a step toward successful implementation of CER, it is the increase in quality and robustness that will truly help to overcome the challenges outlined previously.
Conclusion CER is critical to the practice of precision medicine. As medical oncologists are forced to rely on insufficient data as a part of daily treatment decision making, and as the cancer treatment landscape evolves quickly relative to other areas of medicine, CER is particularly pressing in our field. Continued reliance on RCTs is a part of the solution, but it cannot be the sole answer. As newer and richer data sources become available—including quality of life, resource utilization, and the like—the implementation of CER will advance. Its true power will lie in linkages to “learning health systems” and real-time application to the day-to-day practice of medicine.
References 1. FDA: Approved drugs by therapeutic area. Available at: 〈http://www. centerwatch.com/drug-information/fda-approvals/drug-areas.aspx? AreaID=12〉. Accessed April 24, 2013 2. National Comprehensive Cancer Network. NCCN Clinical Practice Guidelines in Oncology (NCCN Guidelines): Kidney Cancer. Ft. Washington, PA, 2012
B.R. Hirsch and S.Y. Zafar 3. National Institute of Health and Clinical Excellence. Available at: 〈http:// www.nice.org.uk/〉. Accessed April 24, 2013 4. Services USDoHaH. Report to the President and the Congress on comparative effectiveness research, 2009. Available at: 〈http://www.hhs. gov/recovery/programs/cer/execsummary.html〉. Accessed April 24, 2013 5. Abernethy AP, Raman G, Balk EM, et al: Systematic review: Reliability of compendia methods for off-label oncology indications. Ann Intern Med 150:336-343, 2009 6. Krzyzanowska MK: Off-label use of cancer drugs: A benchmark is established. J Clin Oncol 31:1125-1127, 2013 7. Mullins CD, Montgomery R, Abernethy AP, et al: Recommendations for clinical trials of off-label drugs used to treat advanced-stage cancer. J Clin Oncol 30:661-666, 2012 8. Conti RM, Bernstein AC, Villaflor VM, et al: Prevalence of off-label use and spending in 2010 among patent-protected chemotherapies in a population-based cohort of medical oncologists. J Clin Oncol 31:1134-1139, 2013 9. Rini BI, Escudier B, Tomczak P, et al: Comparative effectiveness of axitinib versus sorafenib in advanced renal cell carcinoma (AXIS): A randomised phase 3 trial. Lancet 378:1931-1939, 2011 10. Dilts DM, Cheng SK, Crites JS, et al: Phase III clinical trial development: A process of chutes and ladders. Clin Cancer Res 16:5381-5389, 2010 11. Cheng SK, Dietrich MS, Dilts DM: A sense of urgency: Evaluating the link between clinical trial development time and the accrual performance of cancer therapy evaluation program (NCI-CTEP) sponsored studies. Clin Cancer Res 16:5557-5563, 2010 12. Hirsch BR, Califf RM, Cheng SK, et al: The state of the oncology clinical trials portfolio: Insights from a systematic analysis of ClinicalTrials.gov. JAMA Intern Med 173:972-979, 2013 13. Ellis L.M., Bernstein D., Voest E., et al. Defining clinically meaningful outcomes: ASCO recommendations to raise the bar for clinical trials. Draft for public commnet. Available at: 〈http://www.asco.org/sites/www.asco. org/files/asco_meaningful_outcomes_4.23.12_draft_for_comment_ap ril_20131.pdf〉. Accessed April 24, 2013 14. Center for Medical Technology Policy (CMTP). Effectiveness Guidance Documents—A method for comparative effectiveness documents. Available at: 〈http://www.cmtpnet.org/effectiveness-guidance-documents/〉. Accessed April 24, 2013 15. Meyer AM, Carpenter WR, Abernethy AP, et al: Data for cancer comparative effectiveness research: Past, present, and future potential. Cancer 118:5186-5197, 2012 16. Meyerhardt JA, Li L, Sanoff HK, et al: Effectiveness of bevacizumab with first-line combination chemotherapy for Medicare patients with stage IV colorectal cancer. J Clin Oncol 30:608-615, 2012 17. Hurwitz HI, Lyman GH: Registries and randomized trials in assessing the effects of bevacizumab in colorectal cancer: Is there a common theme? J Clin Oncol 30:580-581, 2012 18. Mello MM, Brennan TA: The controversy over high-dose chemotherapy with autologous bone marrow transplant for breast cancer. Health Aff 20:101-117, 2001 19. Zujewski J, Nelson A, Abrams J: Much ado about not...enough data: Highdose chemotherapy with autologous stem cell rescue for breast cancer. J Natl Cancer Inst 90:200-209, 1998 20. Hortobagyi GN: What is the role of high-dose chemotherapy in the era of targeted therapies? J Clin Oncol 22:2263-2266, 2004 21. Korn EL, Freidlin B: Methodology for comparative effectiveness research: Potential and limitations. J Clin Oncol 30:4185-4187, 2012 22. Hirsch BR, Abernethy AP: Leveraging informatics, mHealth, and biobanks to treat each patient right. Pers Med 9:849-857, 2012 23. Hirsch BR, Abernethy AP: Structured decision-making: Using personalized medicine to improve the value of cancer care. J Pers Med 3:1-13, 2013 24. Motzer R, Hutson TE, Reeves J, et al: Randomized, open-label, phase III trial of pazopanib versus sunitinib in first-line treatment of patients with metastatic renal cell carcinoma (mRCC): Results of the COMPARZ trial. Ann Oncol 238: 2012 25. Escudier BJ, Porta C, Bono P, et al: Patient preference between pazopanib (Paz) and sunitinib (Sun): Results of a randomized double-blind, placebocontrolled, cross-over study in patients with metastatic renal cell
CER: Moving medicine forward carcinoma (mRCC)—PISCES study, NCT 01064310. J Clin Oncol 30: Suppl; abstr CRA4502, 2012 26. Bach P, Saltz L, Wittes R: In Cancer Care, Cost Matters. New York Times, October 14, 2012:A25 27. Cleghorn GD, Headrick LA: The PDSA cycle at the core of learning in health professions education. Jt Comm J Qual Improv 22:206-212, 1996
53 28. Greene SM, Reid RJ, Larson EB: Implementing the learning health system: From concept to action. Ann Intern Med 157:207-210, 2012 29. Miriovsky BJ, Shulman LN, Abernethy AP: Importance of health information technology, electronic health recrods and continuously aggregating data to comparative effectiveness research and rapid learning systems. J Clin Oncol 30:4243-4248, 2012
