Research Article For reprint orders, please contact: [email protected]
A framework for creating standardized outcome measures for patient registries
Aim: Our objectives were to create a conceptual framework for development of standard outcome measures and to design and pilot test a tool for displaying outcome measures. Materials & methods: Information on outcome measures used in registries was gathered through stakeholder discussions, which informed the development of the outcome measurement framework and the related tool. Results: The outcome measurement framework is a conceptual model for how information relevant to evaluating patient outcomes may be defined and collected in a standard way for a broad range of health areas. The related tool facilitates collecting, displaying and searching for information on outcome measures. Conclusion: The model developed through this process offers a framework that can be used to define outcome measures in a standard way across medical conditions. Keywords: comparative effectiveness research • conceptual model • outcome measure • outcome measurement • patient outcomes • patient registry
Because standardized outcome measures do not exist for most condition areas, clinical studies often use different outcome measures or different definitions for the same outcome measures. Different definitions can have a substantial impact on study findings, as demonstrated by a comparison of bleeding impact using three definitions for a bleeding event [1] . Variations in outcome measures also introduce challenges when comparing or aggregating data across studies and may lead to uncertainty when interpreting study findings in the context of existing evidence. In response to these issues, some groups are developing standardized sets of outcome measures for specific condition areas [2–5] . While the creation of standardized outcome measures is an important step toward reducing variation, it is equally important for these outcome measures to be widely used in clinical research and clinical practice. However, the current lack of a central catalog of outcome measures hampers efforts to identify standardized or widely used measures
10.2217/CER.14.38 © 2014 Future Medicine Ltd
for inclusion in new studies or outcomes measurement efforts. Outcome measure variations and the resulting challenges are evident in patient registry-based research. Patient registries are important tools for addressing a range of clinical and policy questions, including examining the natural history of diseases, assessing the safety of drugs and devices, comparing the effectiveness of diagnosis and treatment options and measuring the quality of medical care delivered. A patient registry is defined as “an organized system that uses observational study methods to collect uniform data (clinical and other) to evaluate specified outcomes for a population defined by a particular disease, condition or exposure and that serves one or more predetermined scientific, clinical, or policy purposes” [6] . In recent years, interest in, and use of, patient registries has increased, with investments in registries from government agencies, industry and provider and patient organizations. In particular, patient registries have been recognized as a potentially
J. Comp. Eff. Res. (2014) 3(5), 473–480
Richard E Gliklich1, Michelle B Leavy*,2, Jannette Karl2, Daniel M Campion3, Daniel Levy4 & Elise Berliner5 Harvard Medical School, Massachusetts Eye & Ear Infirmary, 243 Charles Street, Boston, MA 02114, USA 2 Quintiles, 201 Broadway, Cambridge MA 02139, USA 3 Quintiles, 1801 Rockville Pike, Suite 300, Rockville, MD 20852, USA 4 Amazing Charts, 111 Huntington Ave., 4th Floor, Boston, MA 02199, USA 5 Agency for Healthcare Research & Quality, 540 Gaither Road, Rockville, MD 20850, USA *Author for correspondence: Tel.: +1 617 599 9912 Fax: +1 617 621 1620 [email protected] 1
part of
ISSN 2042-6305
473
Research Article Gliklich, Leavy, Karl, Campion, Levy & Berliner useful source of data for comparative effectiveness research [7–10] . As new registries are developed, opportunities arise to combine or link registry data sets and compare or aggregate results. These approaches offer great potential to extend the value of both new and existing registries, as well as to incorporate data from other sources such as electronic health records, but in practice they are often limited by the lack of standardization in both specific outcome measures and general parameters for measurement. For example, rheumatology registries may select different patientreported outcome measures to assess disease activity, or they may use the same outcome measure but collect it at different time points. Cardiovascular disease registries may collect myocardial infarction as an outcome measure, but use different definitions of the term. Addressing these variations will require both the development of standardized sets of outcome measures and a new approach to cataloging existing outcome measures so that they are easily identified and widely adopted. The primary objectives of our project were to create a conceptual framework for development of standard outcome measures used in patient registries and to design and pilot test a tool for collecting and displaying information about outcome measures. This project was funded under contract number HHSA29020050035I Task Order No. 7 from the Agency for Healthcare Research and Quality, US Department of Health and Human Services as part of the Developing Evidence to Inform Decisions about Effectiveness program. Materials & methods The project involved three phases: discussions with stakeholders; design of a generic outcome measurement framework (OMF); and development and pilot testing of a tool based on the framework for cataloging outcome measures. In the first phase, we held five stakeholder meetings in February and March 2011 to gather information on how outcome measures are collected in existing patient registries and to learn about how stakeholders would like to see information on outcome measures presented. The meetings were organized around the Agency for Healthcare Research and Quality priority condition areas, with each meeting focusing on one or more condition areas. Registry sponsors, clinicians and clinical researchers with expertise in the condition areas of interest were invited to participate in the meetings. A sixth meeting was held April 2011 to discuss the preliminary framework. Following the stakeholder meetings, we conducted background research to identify and examine existing
474
J. Comp. Eff. Res. (2014) 3(5)
systems that collect and present information on data elements, outcome measures and/or quality measures. In early 2012, we created the OMF, a conceptual model for developing standardized outcome measures. The OMF formed the basis for the OMF tool, which was designed to collect and display information about outcome measures in other systems. Between July and November 2012, we conducted three rounds of pilot testing activities for the OMF tool. During pilot testing, participants attempted to enter outcome measures into the OMF tool and reported on their experiences. Pilot testing activities also involved web conferences, web-based surveys and document reviews. Following each round of pilot testing, we revised the OMF tool to incorporate feedback. The OMF tool was finalized in December 2012. In total, 178 stakeholders participated in the meetings and pilot testing activities, representing healthcare provider organizations, professional societies, academia, research and consulting organizations, government agencies, patient/consumer organizations, journal editors, payers and pharmaceutical companies (Figure 1) . For each meeting and pilot testing activity, we attempted to include a group with relatively balanced representation from researchers, industry and government. Results The stakeholder discussions highlighted wide variations in outcome measurement, even within the same medical condition. The outcome measures reviewed for this project were not defined in a consistent manner and included varying levels of detail, making it difficult to identify equivalent measures across registries. Stakeholders also noted the lack of comprehensive outcome measurement sets in most condition areas and described challenges related to identifying existing outcome measures and defining new measures. Background research and literature searches found some resources that partially address these issues. For example, several projects to collect and/or define common data elements and quality measures are ongoing, including the Common Data Elements initiative at the National Institute of Neurological Disorders and Stroke [11] , the United States Health Information Knowledgebase [12] , the Consensus Measures for Phenotypes and eXposures (PhenX) project, the National Quality Measures Clearinghouse and the National Quality Forum. However, none of these initiatives specifically focuses on outcome measures. Relevant models, most notably those developed by Carpenter et al. and Porter, were also found in the literature [13,14] . These models provided
future science group
A framework for creating standardized outcome measures for patient registries
a useful framework for discussion with stakeholders and informed the design of this project’s model. Figure 2 represents this project’s conceptual model for how information relevant to evaluating patient outcomes may be defined and collected in a standard way for a broad range of health conditions and treatments. The OMF is a generic categorization hierarchy with three levels. The top level consists of three broad domains organized from left to right: characteristics, treatment and outcomes. These domains represent the process (indicated by arrows in Figure 2) by which characteristics determine treatment, and characteristics and treatment together determine outcomes. Outcomes may then determine additional courses of treatment. The second level of the model consists of subcategories of data elements that are needed to define an outcome measure. For example, within the characteristics domain, information on ‘participant characteristics,’ ‘disease characteristics’ and ‘provider characteristics’ may be used in an outcome measure. At the third level of the model are categories of data elements that would be used to describe the subcategory. For example, ‘participant characteristics’ may be described by data on demographics, genetics, health behaviors, etc. These data element categories are intentionally broad, so the model can be used across medical condition areas. Because widely used common data elements do not exist for most types of data, the model does not include a fourth level specifying individual data elements. Within the outcomes domain on the right side of the model, outcome measures are divided into five subcategories. The outcome measures included here represent both the final assessed outcomes (i.e., overall mortality and disease-free survival), as well as intermediate measures, such as disease response, patient-reported outcomes and health system utilization. Stakeholders recommended inclusion of these intermediate outcomes for their utility in tracking the natural history of diseases and treatment of chronic conditions over time. The model does not include timeframe as a subcategory. Depending on the purpose of the outcome measurement, multiple timeframes and measurement frequencies may be appropriate. The parameters depicted in the model represent the areas where stakeholders reached a consensus. Stakeholders were not able to reach a consensus in all areas; in particular, the management of time points was an area of substantial discord, with stakeholders noting that different condition areas require different timeframes for assessment of outcome measures. Because of this feedback, the conceptual model does not include standard timeframes.
future science group
Payers 4% Patient/consumer organizations 7%
Goverment agenices 11%
Providers/provider organizations 17%
Funding agency 3%
Research Article
Journal editors 1%
Researchers 38%
Industry 19%
Figure 1. Stakeholders participating in outcome measurement framework design and pilot-testing activities, by type.
Discussion The goal of this project was to use a stakeholderdriven approach to develop a conceptual framework for standardizing outcome measures for patient registries. The model developed through this process (Figure 2) offers a framework that can be used to define outcome measures in a standard way across medical conditions, so that the resulting measures are complete and reproducible in other settings. The model was presented to multiple stakeholder groups and refined through these discussions, which produced valuable information regarding what outcome measures registries currently collect and what common framework may be applicable to outcome measures across clinical areas. Because the model applies to a broad range of medical conditions, not all subcategories or data element categories will be relevant for each outcome measure. As an example, “percentage of patients who experience significant chemotherapy-associated adverse events” is a poorly defined outcome measure. The population to which the measure applies is not specified. The measure does not define specific adverse events, and no timeframe is indicated. Using the model, a complete outcome measure for an event of interest can be constructed. Characteristics are defined first – in this example, patients with a diagnosis of breast cancer. The treatment regimen is then defined as chemotherapy, specifically anthracyclines and taxanes and the adverse event of interest is defined as neuropathy. The timeframe is specified as 1 year post-treatment. The resulting outcome measure, “Percentage of breast cancer patients who develop chemotherapy-associated
www.futuremedicine.com
475
Research Article Gliklich, Leavy, Karl, Campion, Levy & Berliner
Framework data element categories Characteristics
Treatment
Survival
Participant
• Overall mortality • Disease-free survival
• Demographics • Genetics • Family/participant history • Functional/performance status • Health behaviors • Environmental exposures
Disease • Diagnosis • Risk factors • Staging system • Genetics of disease • Tissue or infectious agent • Biomarkers • Comorbidity/symptoms • Assessment/scales
Outcomes
Disease response • Recurrence • Progression
Type • Surgical • Medical • Device • Alternative
Events of interest • Adverse events • Exacerbations Patient/caregiver reported
Intent • Palliative vs curative
Provider • Training/experience • Geography • Practice setting: – Academic vs community
• Physical functioning • Health-related quality of life • Other
Clinician reported • Disease progression • Other Health system utilization • Inpatient hospitalization • Office visits • ER visits • Additional procedures • Productivity/absenteeism • Direct cost
Figure 2. Outcome measures framework model.
neuropathy at one year post-treatment,” is complete and could be reproduced in other studies or clinical practice settings. The development of complete, reproducible outcome measures requires substantial effort, particularly when outcome measures are developed as part of cohesive outcome measurement sets for a specific medical condition. As important as developing sound outcome measures is ensuring that the measures can be found and used in new research and in clinical practice. As stakeholders noted and as the team working on this project experienced, sources of outcome
476
J. Comp. Eff. Res. (2014) 3(5)
measures are fragmented. The second component of this project is the development of a tool to address this issue. The OMF tool, which builds on the foundation of the OMF model, facilitates collecting, displaying and searching for information on outcome measures used in patient registries. The immediate goal of the OMF tool is to collate and characterize the outcome measures that registries currently collect, while the long-term goal is to support efforts to standardize outcome measures [15] . The OMF tool addresses several challenges identified by stakeholders and through background
future science group
A framework for creating standardized outcome measures for patient registries
research, including different types of outcome measures, varying definitions for the same clinical concept and the use of multiple timeframes. This tool can accommodate measures collected on a patient level as well as those calculated on a population level; it can identify and clearly display information about measures that may use different wording but are in fact clinically equivalent to each other; and it can collect and display the timeframe within which a registry collects a particular outcome measure. The OMF tool is designed to be implemented as part of a system such as the Registry of Patient Registries (RoPR) [16] , a database of existing registries that is integrated with ClinicalTrials.gov. Within the RoPR, the OMF tool will provide an interface for those listing a registry to enter information on the outcome measures that they collect and for those searching for registries to view information on outcome measures in use in listed registries. While the OMF tool is not yet available, we are continuing work on its development through a new Agency for Healthcare Research and Quality-funded project (contract number HHSA290201400004C). Compared with other models, the OMF model is more broadly applicable and provides the necessary level of detail to support development of new outcome measures. The model builds on the work of Carpenter et al. in the area of oncology and expands and refines that framework to apply across medical conditions [13] . The model complements the outcome measures hierarchy put forth by Porter [14] . The outcome measures hierarchy groups outcome measures into three tiers based on their relative importance in terms of value. The hierarchy and the accompanying discussion present principles to drive the identification and selection of outcome measures, with an emphasis on outcomes that are most relevant to patients. While the hierarchy is useful for identifying the concepts to measure, it does not provide concrete guidance for developing new outcome measures. As noted above, gaps in existing outcome measurement sets exist for most, if not all, disease areas. New outcome measures will need to be defined in order to create the comprehensive outcome measurement sets envisioned by the hierarchy. The OMF serves as a critical building block for those efforts by providing a standard framework for developing new measures. As the availability of healthcare data grows, opportunities to measure outcomes and to use these data to support clinical research and drive process improvement will increase. Standardized outcome measures will support these efforts by enabling data to be linked, shared and compared across studies and settings. A critical next step toward data harmonization
future science group
Research Article
will be to define and implement common data elements. The use of common data elements as the foundation for standard outcome measures will increase the ability of data from multiple sources, such as, clinical trials, patient registries and electronic health records, to be linked in meaningful ways. Efforts to link different measures so that they are on the same scale of measurement are also critical to increasing the ability of researchers to link, share or compare data from different sources; one such effort is the Patient Reported Outcomes Measurement Information System (PROMIS®) initiative, funded by the NIH [17] . A central resource for identifying outcome measures for use in new areas is also critical for data harmonization. The stakeholder-driven process used in this project examined how stakeholders would like to search for outcome measures, what type of information they would be willing to provide on outcome measures when entering a registry into the RoPR and what type of information they would like to find in the OMF catalog. The design proposed here for the OMF tool was informed by the discussions with stakeholders. The resulting tool has several strengths and some limitations. First, the design is flexible and scalable. These are important attributes, as the OMF will need to collect and display a large number of heterogeneous outcome measures. Second, the proposed design simplifies searching for entries by including comprehensive lists of keywords for each entry. Users who enter a keyword into the search field will see all of the entries with that keyword in their search results, which facilitates searching for measures using synonyms. Last, the level of complexity in the proposed workflow aligns with what stakeholders suggested would be feasible within a voluntary system, such as the RoPR. The issues of user burden and feasibility introduced some design limitations. In particular, the individual data elements that comprise an outcome measure are not collected and displayed in the OMF tool due to stakeholder concerns about the burden of entry. As a result, fields such as ‘Definition’ for outcome measures are entered as text. Comparing detailed definitions, such as those shown in Table 1, is challenging, particularly if a user is comparing three or more outcome measures. In the future, the usability of the OMF tool could be improved by breaking down the definition field into individual data elements. In addition, stakeholders strongly recommended a curated system, in which clinically equivalent entries are identified as such, to reduce the burden of sifting through multiple, equivalent definitions. While this
www.futuremedicine.com
477
478
J. Comp. Eff. Res. (2014) 3(5)
Criteria for acute, evolving or recent MI. Either one of the following criteria satisfies the diagnosis for an acute, evolving or recent MI: • Typical rise and gradual fall (troponin) or more rapid rise and fall (CK-MB) of biochemical markers of myocardial necrosis with at least one of the following: – Ischemic symptoms – Development of pathologic Q waves on the ECG – ECG changes indicative of ischemia (ST segment elevation or depression); or – Coronary artery intervention (e.g., coronary angioplasty) • Pathologic findings of an acute MI
Criteria for acute MI. The term myocardial infarction should be used when there is evidence of myocardial necrosis in a clinical setting consistent with myocardial ischemia. Under these conditions any one of the following criteria meets the diagnosis for myocardial infarction:
• Detection of rise and/or fall of cardiac biomarkers (preferably troponin) with at least one value above the 99th percentile of the URL together with evidence of myocardial ischemia with at least one of the following: – Symptoms of ischemia – ECG changes indicative of new ischemia (new ST-T changes or new LBBB) – Development of pathological Q waves in the ECG – Imaging evidence of new loss of viable myocardium or new regional wall motion abnormality • Sudden, unexpected cardiac death, involving cardiac arrest, often with symptoms suggestive of myocardial ischemia, and accompanied by presumably new ST elevation, or new LBBB, and/ or evidence of fresh thrombus by coronary angiography and/or at autopsy, but death occurring before blood samples could be obtained, or at a time before the appearance of cardiac biomarkers in the blood • For PCI in patients with normal baseline troponin values, elevations of cardiac biomarkers above the 99th percentile URL are indicative of periprocedural myocardial necrosis. By convention, increases of biomarkers greater than 3 × 99th percentile URL have been designated as defining PCI-related myocardial infarction. A subtype related to a documented stent thrombosis is recognized • For CABG in patients with normal baseline troponin values, elevations of cardiac biomarkers above the 99th percentile URL are indicative of periprocedural myocardial necrosis. By convention, increases of biomarkers greater than 5 × 99th percentile URL plus either new pathological Q waves or new LBBB, or angiographically documented new graft or native coronary artery occlusion, or imaging evidence of new loss of viable myocardium have been designated as defining CABG-related myocardial infarction • Pathological findings of an acute myocardial infarction
Definition
• Development of new pathologic Q waves on serial ECGs. The patient may or may not remember previous symptoms. Biochemical markers of myocardial necrosis may have normalized, depending on the length of time that has passed since the infarct developed • Pathologic findings of a healed or healing MI
‡
Data taken from [18]. Data taken from [19]. ACCF: American College of Cardiology Foundation; AHA: American Heart Association; CABG: Coronary artery bypass grafting; CK-MB: Creatine kinase-MB fraction; ESC: European Society of Cardiology; LBBB: Left bundle branch block; MI: Myocardial infarction; OMF: Outcome measurement framework; PCI: Percutaneous coronary intervention; PhenX: Consensus Measures for Phenotypes and eXposures; URL: Upper reference limit; WHF: World Heart Federation.
†
5.0
2007
Version
Criteria for established MI. Any one of the following criteria satisfies the diagnosis for established MI:
No
PhenX
Additional Yes (one) Clinically Equivalent Sources in OMF?
Joint ESC/ACCF/AHA/WHF Task Force for the Redefinition of Myocardial Infarction
Myocardial Infarction
Title
Source
201
151
OMF ID number Myocardial Infarction
Outcome measure example B‡
OMF field Outcome measure example A†
Table 1. Mock-up of outcome measures displayed for comparison within the Registry of Patient Registries system.
Research Article Gliklich, Leavy, Karl, Campion, Levy & Berliner
future science group
A framework for creating standardized outcome measures for patient registries
approach has advantages in terms of content quality, the curation process requires additional time and resources. Despite these limitations, the OMF tool offers practical mechanism for the display of information on the outcome measures currently in use, with the goal of reducing variation. Characterizing the outcome measures currently in use will support long-term efforts to develop standard outcome measures by identifying areas of common ground where standards may be developed relatively quickly and areas that will require additional work.
Research Article
Future perspective While the OMF tool will help identify and characterize outcome measures as they are currently being used, the findings are likely to only highlight the lack of standard outcome measures in most conditions. In order to have clinical medicine become truly outcome-based for both care decisions and payment, the next decade will require focused efforts by clinical specialists, patient organizations and scientific agencies to fill that void through development of numerous specialized outcome measures using a standard model, such as the OMF. Disclaimer
Conclusion The use of standard outcome measures in registries will increase opportunities for collaboration and allow results to be compared and aggregated across registries in the same condition area, thereby increasing the utility of registries for comparative effectiveness and other types of research. Standard outcome measures may also improve the quality of registry data by helping to ensure that key outcomes are collected consistently using clear, concise definitions. In the long term, standardization of outcome measures and data elements has the potential to facilitate the efficient use of research resources and may especially benefit patients, clinicians, researchers and funding agencies involved in designing patient registries and using registry data in quality improvement programs, patient-centered outcomes research and comparative effectiveness research.
The findings and conclusions in this document are those of the authors, who are responsible for its contents; the findings and conclusions do not necessarily represent the views of AHRQ. Therefore, no statement in this report should be construed as an official position of AHRQ or of the US Department of Health and Human Services.
Financial & competing interests disclosure The project described in this article was funded by the Agency for Healthcare Research and Quality under the Developing Evidence to Inform Decisions about Effectiveness (DEcIDE) program (contract number HHSA29020050035I TO7). The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed. No writing assistance was utilized in the production of this manuscript.
Executive summary • The use of different definitions for outcome measures introduces challenges when comparing or aggregating data across studies and may lead to uncertainty when interpreting study findings in the context of existing evidence. • Addressing variations in outcome measurement will require both the development of standardized sets of outcome measures and a new approach to cataloging existing outcome measures so that they are easily identified and widely adopted. • The outcome measurement framework (OMF) is a conceptual model for how information relevant to evaluating patient outcomes may be defined and collected in a standard way for a broad range of health conditions and treatments. • Compared with other models, the OMF model is more broadly applicable and provides the necessary level of detail to support development of new outcome measures. • The OMF tool, which builds on the foundation of the OMF model, facilitates collecting, displaying and searching for information on outcome measures used in patient registries. • The immediate goal of the OMF tool is to collate and characterize the outcome measures that registries currently collect, while the long-term goal is to support efforts to standardize outcome measures.
References
coronary syndromes and undergoing percutaneous coronary interventions and its impact on the apparent safety of antithrombotic drugs. Am. Heart J. 154(1), 3–11 (2007).
Papers of special note have been highlighted as: •• of considerable interest 1
Steinhubl SR, Kastrati A, Berger PB. Variation in the definitions of bleeding in clinical trials of patients with acute
future science group
2
Wang XL, Thompson MM, Dole WP et al. Standardization of outcome measures in clinical trials of pharmacological
www.futuremedicine.com
479
Research Article Gliklich, Leavy, Karl, Campion, Levy & Berliner treatment for abdominal aortic aneurysm. Expert Rev. Cardiovasc. Ther. 10(10), 1251–1260 (2012). 3
Shen J, Johnston M, Hays RD. Asthma outcome measures. Expert Rev. Pharmacoecon. Outcomes Res. 11(4), 447–453 (2011).
11
National Institute of Neurological Disorders and Stroke. Common Data Elements Initiative (2014). www.commondataelements.ninds.nih.gov
4
Sheehy C, Gaffney K, Mukhtyar C. Standardized grip strength as an outcome measure in early rheumatoid arthritis. Scand. J. Rheumatol. 42(4), 289–293 (2013).
12
Agency for Healthcare Research and Quality. United States Health Information Knowledgebase (2014). http://ushik.ahrq.gov/
5
Merkel PA, Herlyn K, Mahr AD et al. Progress towards a core set of outcome measures in small-vessel vasculitis. Report from OMERACT 9. J. Rheumatol. 36(10), 2362–2368 (2009).
13
Carpenter WR, Meyer AM, Abernethy AP, Stürmer T, Kosorok MR. A framework for understanding cancer comparative effectiveness research data needs. J. Clin. Epidemiol. 65(11), 1150–1158 (2012).
6
Registries for Evaluating Patient Outcomes: A User’s Guide 3rd Edition. Gliklich RE, Dreyer NA, Leavy M (Eds). Agency for Healthcare Research and Quality, MD, USA (2014).
••
The framework described in this paper informed the design of the conceptual model developed for this project.
14
••
Detailed discussion of the design, operation, analysis and evaluation of patient registries.
Porter ME. What is value in health care? N. Engl. J. Med. 363(26), 2477–2481 (2010).
••
7
Institute of Medicine. Learning What Works Best: the Nation’s Need for Evidence on Comparative Effectiveness in Health Care. National Academy Press, Washington DC, USA, 1–80 (2007).
This paper and its supplementary materials discuss key concepts related to outcome measurement.
15
Registry of Patient Registries (RoPR). Topic Abstract. Agency for Healthcare Research and Quality (2012). www.effectivehealthcare.ahrq.gov/index.cfm/search-forguides-reviews-and-reports/?pageaction=displayproduct&pr oductid=690
16
Agency for Healthcare Research and Quality. Registry of Patient Registries (RoPR) (2014). https://patientregistry.ahrq.gov
17
National Institutes of Health. PROMIS (2014). www.nihpromis.org
18
Thygesen K, Alpert JS, White HD. Universal definition of myocardial infarction. J. Am. Coll. Cardiol. 50(22), 2173–2195 (2007).
19
RTI International. Consensus Measures for Phenotypes and eXposures (PhenX). Myocardial Infarction (2012). www.phenxtoolkit.org
8
9
10
480
www.pcori.org/research-we-support/research-methodologystandards
Congressional Budget Office. Research on the Comparative Effectiveness of Medical Treatments: Issues and Options for an Expanded Federal Role (2007). www.cbo.gov/sites/default/files/cbofiles/ftpdocs/88xx/ doc8891/12-18-comparativeeffectiveness.pdf. US Department of Health and Human Services. Federal Coordinating Council for Comparative Effectiveness Research. Report to the President and the Congress (2009). www.tuftsctsi.org/∼/media/Files/CTSI/Library%20Files/ FCC%20for%20CER%20Rpt%20to%20Pres%20and%20 Congress_063009.ashx PCORI (Patient-Centered Outcomes Research Institute) Methodology Committee. The PCORI Methodology Report (2013).
J. Comp. Eff. Res. (2014) 3(5)
future science group
A framework for creating standardized outcome measures for patient registries
Aim: Our objectives were to create a conceptual framework for development of standard outcome measures and to design and pilot test a tool for displaying outcome measures. Materials & methods: Information on outcome measures used in registries was gathered through stakeholder discussions, which informed the development of the outcome measurement framework and the related tool. Results: The outcome measurement framework is a conceptual model for how information relevant to evaluating patient outcomes may be defined and collected in a standard way for a broad range of health areas. The related tool facilitates collecting, displaying and searching for information on outcome measures. Conclusion: The model developed through this process offers a framework that can be used to define outcome measures in a standard way across medical conditions. Keywords: comparative effectiveness research • conceptual model • outcome measure • outcome measurement • patient outcomes • patient registry
Because standardized outcome measures do not exist for most condition areas, clinical studies often use different outcome measures or different definitions for the same outcome measures. Different definitions can have a substantial impact on study findings, as demonstrated by a comparison of bleeding impact using three definitions for a bleeding event [1] . Variations in outcome measures also introduce challenges when comparing or aggregating data across studies and may lead to uncertainty when interpreting study findings in the context of existing evidence. In response to these issues, some groups are developing standardized sets of outcome measures for specific condition areas [2–5] . While the creation of standardized outcome measures is an important step toward reducing variation, it is equally important for these outcome measures to be widely used in clinical research and clinical practice. However, the current lack of a central catalog of outcome measures hampers efforts to identify standardized or widely used measures
10.2217/CER.14.38 © 2014 Future Medicine Ltd
for inclusion in new studies or outcomes measurement efforts. Outcome measure variations and the resulting challenges are evident in patient registry-based research. Patient registries are important tools for addressing a range of clinical and policy questions, including examining the natural history of diseases, assessing the safety of drugs and devices, comparing the effectiveness of diagnosis and treatment options and measuring the quality of medical care delivered. A patient registry is defined as “an organized system that uses observational study methods to collect uniform data (clinical and other) to evaluate specified outcomes for a population defined by a particular disease, condition or exposure and that serves one or more predetermined scientific, clinical, or policy purposes” [6] . In recent years, interest in, and use of, patient registries has increased, with investments in registries from government agencies, industry and provider and patient organizations. In particular, patient registries have been recognized as a potentially
J. Comp. Eff. Res. (2014) 3(5), 473–480
Richard E Gliklich1, Michelle B Leavy*,2, Jannette Karl2, Daniel M Campion3, Daniel Levy4 & Elise Berliner5 Harvard Medical School, Massachusetts Eye & Ear Infirmary, 243 Charles Street, Boston, MA 02114, USA 2 Quintiles, 201 Broadway, Cambridge MA 02139, USA 3 Quintiles, 1801 Rockville Pike, Suite 300, Rockville, MD 20852, USA 4 Amazing Charts, 111 Huntington Ave., 4th Floor, Boston, MA 02199, USA 5 Agency for Healthcare Research & Quality, 540 Gaither Road, Rockville, MD 20850, USA *Author for correspondence: Tel.: +1 617 599 9912 Fax: +1 617 621 1620 [email protected] 1
part of
ISSN 2042-6305
473
Research Article Gliklich, Leavy, Karl, Campion, Levy & Berliner useful source of data for comparative effectiveness research [7–10] . As new registries are developed, opportunities arise to combine or link registry data sets and compare or aggregate results. These approaches offer great potential to extend the value of both new and existing registries, as well as to incorporate data from other sources such as electronic health records, but in practice they are often limited by the lack of standardization in both specific outcome measures and general parameters for measurement. For example, rheumatology registries may select different patientreported outcome measures to assess disease activity, or they may use the same outcome measure but collect it at different time points. Cardiovascular disease registries may collect myocardial infarction as an outcome measure, but use different definitions of the term. Addressing these variations will require both the development of standardized sets of outcome measures and a new approach to cataloging existing outcome measures so that they are easily identified and widely adopted. The primary objectives of our project were to create a conceptual framework for development of standard outcome measures used in patient registries and to design and pilot test a tool for collecting and displaying information about outcome measures. This project was funded under contract number HHSA29020050035I Task Order No. 7 from the Agency for Healthcare Research and Quality, US Department of Health and Human Services as part of the Developing Evidence to Inform Decisions about Effectiveness program. Materials & methods The project involved three phases: discussions with stakeholders; design of a generic outcome measurement framework (OMF); and development and pilot testing of a tool based on the framework for cataloging outcome measures. In the first phase, we held five stakeholder meetings in February and March 2011 to gather information on how outcome measures are collected in existing patient registries and to learn about how stakeholders would like to see information on outcome measures presented. The meetings were organized around the Agency for Healthcare Research and Quality priority condition areas, with each meeting focusing on one or more condition areas. Registry sponsors, clinicians and clinical researchers with expertise in the condition areas of interest were invited to participate in the meetings. A sixth meeting was held April 2011 to discuss the preliminary framework. Following the stakeholder meetings, we conducted background research to identify and examine existing
474
J. Comp. Eff. Res. (2014) 3(5)
systems that collect and present information on data elements, outcome measures and/or quality measures. In early 2012, we created the OMF, a conceptual model for developing standardized outcome measures. The OMF formed the basis for the OMF tool, which was designed to collect and display information about outcome measures in other systems. Between July and November 2012, we conducted three rounds of pilot testing activities for the OMF tool. During pilot testing, participants attempted to enter outcome measures into the OMF tool and reported on their experiences. Pilot testing activities also involved web conferences, web-based surveys and document reviews. Following each round of pilot testing, we revised the OMF tool to incorporate feedback. The OMF tool was finalized in December 2012. In total, 178 stakeholders participated in the meetings and pilot testing activities, representing healthcare provider organizations, professional societies, academia, research and consulting organizations, government agencies, patient/consumer organizations, journal editors, payers and pharmaceutical companies (Figure 1) . For each meeting and pilot testing activity, we attempted to include a group with relatively balanced representation from researchers, industry and government. Results The stakeholder discussions highlighted wide variations in outcome measurement, even within the same medical condition. The outcome measures reviewed for this project were not defined in a consistent manner and included varying levels of detail, making it difficult to identify equivalent measures across registries. Stakeholders also noted the lack of comprehensive outcome measurement sets in most condition areas and described challenges related to identifying existing outcome measures and defining new measures. Background research and literature searches found some resources that partially address these issues. For example, several projects to collect and/or define common data elements and quality measures are ongoing, including the Common Data Elements initiative at the National Institute of Neurological Disorders and Stroke [11] , the United States Health Information Knowledgebase [12] , the Consensus Measures for Phenotypes and eXposures (PhenX) project, the National Quality Measures Clearinghouse and the National Quality Forum. However, none of these initiatives specifically focuses on outcome measures. Relevant models, most notably those developed by Carpenter et al. and Porter, were also found in the literature [13,14] . These models provided
future science group
A framework for creating standardized outcome measures for patient registries
a useful framework for discussion with stakeholders and informed the design of this project’s model. Figure 2 represents this project’s conceptual model for how information relevant to evaluating patient outcomes may be defined and collected in a standard way for a broad range of health conditions and treatments. The OMF is a generic categorization hierarchy with three levels. The top level consists of three broad domains organized from left to right: characteristics, treatment and outcomes. These domains represent the process (indicated by arrows in Figure 2) by which characteristics determine treatment, and characteristics and treatment together determine outcomes. Outcomes may then determine additional courses of treatment. The second level of the model consists of subcategories of data elements that are needed to define an outcome measure. For example, within the characteristics domain, information on ‘participant characteristics,’ ‘disease characteristics’ and ‘provider characteristics’ may be used in an outcome measure. At the third level of the model are categories of data elements that would be used to describe the subcategory. For example, ‘participant characteristics’ may be described by data on demographics, genetics, health behaviors, etc. These data element categories are intentionally broad, so the model can be used across medical condition areas. Because widely used common data elements do not exist for most types of data, the model does not include a fourth level specifying individual data elements. Within the outcomes domain on the right side of the model, outcome measures are divided into five subcategories. The outcome measures included here represent both the final assessed outcomes (i.e., overall mortality and disease-free survival), as well as intermediate measures, such as disease response, patient-reported outcomes and health system utilization. Stakeholders recommended inclusion of these intermediate outcomes for their utility in tracking the natural history of diseases and treatment of chronic conditions over time. The model does not include timeframe as a subcategory. Depending on the purpose of the outcome measurement, multiple timeframes and measurement frequencies may be appropriate. The parameters depicted in the model represent the areas where stakeholders reached a consensus. Stakeholders were not able to reach a consensus in all areas; in particular, the management of time points was an area of substantial discord, with stakeholders noting that different condition areas require different timeframes for assessment of outcome measures. Because of this feedback, the conceptual model does not include standard timeframes.
future science group
Payers 4% Patient/consumer organizations 7%
Goverment agenices 11%
Providers/provider organizations 17%
Funding agency 3%
Research Article
Journal editors 1%
Researchers 38%
Industry 19%
Figure 1. Stakeholders participating in outcome measurement framework design and pilot-testing activities, by type.
Discussion The goal of this project was to use a stakeholderdriven approach to develop a conceptual framework for standardizing outcome measures for patient registries. The model developed through this process (Figure 2) offers a framework that can be used to define outcome measures in a standard way across medical conditions, so that the resulting measures are complete and reproducible in other settings. The model was presented to multiple stakeholder groups and refined through these discussions, which produced valuable information regarding what outcome measures registries currently collect and what common framework may be applicable to outcome measures across clinical areas. Because the model applies to a broad range of medical conditions, not all subcategories or data element categories will be relevant for each outcome measure. As an example, “percentage of patients who experience significant chemotherapy-associated adverse events” is a poorly defined outcome measure. The population to which the measure applies is not specified. The measure does not define specific adverse events, and no timeframe is indicated. Using the model, a complete outcome measure for an event of interest can be constructed. Characteristics are defined first – in this example, patients with a diagnosis of breast cancer. The treatment regimen is then defined as chemotherapy, specifically anthracyclines and taxanes and the adverse event of interest is defined as neuropathy. The timeframe is specified as 1 year post-treatment. The resulting outcome measure, “Percentage of breast cancer patients who develop chemotherapy-associated
www.futuremedicine.com
475
Research Article Gliklich, Leavy, Karl, Campion, Levy & Berliner
Framework data element categories Characteristics
Treatment
Survival
Participant
• Overall mortality • Disease-free survival
• Demographics • Genetics • Family/participant history • Functional/performance status • Health behaviors • Environmental exposures
Disease • Diagnosis • Risk factors • Staging system • Genetics of disease • Tissue or infectious agent • Biomarkers • Comorbidity/symptoms • Assessment/scales
Outcomes
Disease response • Recurrence • Progression
Type • Surgical • Medical • Device • Alternative
Events of interest • Adverse events • Exacerbations Patient/caregiver reported
Intent • Palliative vs curative
Provider • Training/experience • Geography • Practice setting: – Academic vs community
• Physical functioning • Health-related quality of life • Other
Clinician reported • Disease progression • Other Health system utilization • Inpatient hospitalization • Office visits • ER visits • Additional procedures • Productivity/absenteeism • Direct cost
Figure 2. Outcome measures framework model.
neuropathy at one year post-treatment,” is complete and could be reproduced in other studies or clinical practice settings. The development of complete, reproducible outcome measures requires substantial effort, particularly when outcome measures are developed as part of cohesive outcome measurement sets for a specific medical condition. As important as developing sound outcome measures is ensuring that the measures can be found and used in new research and in clinical practice. As stakeholders noted and as the team working on this project experienced, sources of outcome
476
J. Comp. Eff. Res. (2014) 3(5)
measures are fragmented. The second component of this project is the development of a tool to address this issue. The OMF tool, which builds on the foundation of the OMF model, facilitates collecting, displaying and searching for information on outcome measures used in patient registries. The immediate goal of the OMF tool is to collate and characterize the outcome measures that registries currently collect, while the long-term goal is to support efforts to standardize outcome measures [15] . The OMF tool addresses several challenges identified by stakeholders and through background
future science group
A framework for creating standardized outcome measures for patient registries
research, including different types of outcome measures, varying definitions for the same clinical concept and the use of multiple timeframes. This tool can accommodate measures collected on a patient level as well as those calculated on a population level; it can identify and clearly display information about measures that may use different wording but are in fact clinically equivalent to each other; and it can collect and display the timeframe within which a registry collects a particular outcome measure. The OMF tool is designed to be implemented as part of a system such as the Registry of Patient Registries (RoPR) [16] , a database of existing registries that is integrated with ClinicalTrials.gov. Within the RoPR, the OMF tool will provide an interface for those listing a registry to enter information on the outcome measures that they collect and for those searching for registries to view information on outcome measures in use in listed registries. While the OMF tool is not yet available, we are continuing work on its development through a new Agency for Healthcare Research and Quality-funded project (contract number HHSA290201400004C). Compared with other models, the OMF model is more broadly applicable and provides the necessary level of detail to support development of new outcome measures. The model builds on the work of Carpenter et al. in the area of oncology and expands and refines that framework to apply across medical conditions [13] . The model complements the outcome measures hierarchy put forth by Porter [14] . The outcome measures hierarchy groups outcome measures into three tiers based on their relative importance in terms of value. The hierarchy and the accompanying discussion present principles to drive the identification and selection of outcome measures, with an emphasis on outcomes that are most relevant to patients. While the hierarchy is useful for identifying the concepts to measure, it does not provide concrete guidance for developing new outcome measures. As noted above, gaps in existing outcome measurement sets exist for most, if not all, disease areas. New outcome measures will need to be defined in order to create the comprehensive outcome measurement sets envisioned by the hierarchy. The OMF serves as a critical building block for those efforts by providing a standard framework for developing new measures. As the availability of healthcare data grows, opportunities to measure outcomes and to use these data to support clinical research and drive process improvement will increase. Standardized outcome measures will support these efforts by enabling data to be linked, shared and compared across studies and settings. A critical next step toward data harmonization
future science group
Research Article
will be to define and implement common data elements. The use of common data elements as the foundation for standard outcome measures will increase the ability of data from multiple sources, such as, clinical trials, patient registries and electronic health records, to be linked in meaningful ways. Efforts to link different measures so that they are on the same scale of measurement are also critical to increasing the ability of researchers to link, share or compare data from different sources; one such effort is the Patient Reported Outcomes Measurement Information System (PROMIS®) initiative, funded by the NIH [17] . A central resource for identifying outcome measures for use in new areas is also critical for data harmonization. The stakeholder-driven process used in this project examined how stakeholders would like to search for outcome measures, what type of information they would be willing to provide on outcome measures when entering a registry into the RoPR and what type of information they would like to find in the OMF catalog. The design proposed here for the OMF tool was informed by the discussions with stakeholders. The resulting tool has several strengths and some limitations. First, the design is flexible and scalable. These are important attributes, as the OMF will need to collect and display a large number of heterogeneous outcome measures. Second, the proposed design simplifies searching for entries by including comprehensive lists of keywords for each entry. Users who enter a keyword into the search field will see all of the entries with that keyword in their search results, which facilitates searching for measures using synonyms. Last, the level of complexity in the proposed workflow aligns with what stakeholders suggested would be feasible within a voluntary system, such as the RoPR. The issues of user burden and feasibility introduced some design limitations. In particular, the individual data elements that comprise an outcome measure are not collected and displayed in the OMF tool due to stakeholder concerns about the burden of entry. As a result, fields such as ‘Definition’ for outcome measures are entered as text. Comparing detailed definitions, such as those shown in Table 1, is challenging, particularly if a user is comparing three or more outcome measures. In the future, the usability of the OMF tool could be improved by breaking down the definition field into individual data elements. In addition, stakeholders strongly recommended a curated system, in which clinically equivalent entries are identified as such, to reduce the burden of sifting through multiple, equivalent definitions. While this
www.futuremedicine.com
477
478
J. Comp. Eff. Res. (2014) 3(5)
Criteria for acute, evolving or recent MI. Either one of the following criteria satisfies the diagnosis for an acute, evolving or recent MI: • Typical rise and gradual fall (troponin) or more rapid rise and fall (CK-MB) of biochemical markers of myocardial necrosis with at least one of the following: – Ischemic symptoms – Development of pathologic Q waves on the ECG – ECG changes indicative of ischemia (ST segment elevation or depression); or – Coronary artery intervention (e.g., coronary angioplasty) • Pathologic findings of an acute MI
Criteria for acute MI. The term myocardial infarction should be used when there is evidence of myocardial necrosis in a clinical setting consistent with myocardial ischemia. Under these conditions any one of the following criteria meets the diagnosis for myocardial infarction:
• Detection of rise and/or fall of cardiac biomarkers (preferably troponin) with at least one value above the 99th percentile of the URL together with evidence of myocardial ischemia with at least one of the following: – Symptoms of ischemia – ECG changes indicative of new ischemia (new ST-T changes or new LBBB) – Development of pathological Q waves in the ECG – Imaging evidence of new loss of viable myocardium or new regional wall motion abnormality • Sudden, unexpected cardiac death, involving cardiac arrest, often with symptoms suggestive of myocardial ischemia, and accompanied by presumably new ST elevation, or new LBBB, and/ or evidence of fresh thrombus by coronary angiography and/or at autopsy, but death occurring before blood samples could be obtained, or at a time before the appearance of cardiac biomarkers in the blood • For PCI in patients with normal baseline troponin values, elevations of cardiac biomarkers above the 99th percentile URL are indicative of periprocedural myocardial necrosis. By convention, increases of biomarkers greater than 3 × 99th percentile URL have been designated as defining PCI-related myocardial infarction. A subtype related to a documented stent thrombosis is recognized • For CABG in patients with normal baseline troponin values, elevations of cardiac biomarkers above the 99th percentile URL are indicative of periprocedural myocardial necrosis. By convention, increases of biomarkers greater than 5 × 99th percentile URL plus either new pathological Q waves or new LBBB, or angiographically documented new graft or native coronary artery occlusion, or imaging evidence of new loss of viable myocardium have been designated as defining CABG-related myocardial infarction • Pathological findings of an acute myocardial infarction
Definition
• Development of new pathologic Q waves on serial ECGs. The patient may or may not remember previous symptoms. Biochemical markers of myocardial necrosis may have normalized, depending on the length of time that has passed since the infarct developed • Pathologic findings of a healed or healing MI
‡
Data taken from [18]. Data taken from [19]. ACCF: American College of Cardiology Foundation; AHA: American Heart Association; CABG: Coronary artery bypass grafting; CK-MB: Creatine kinase-MB fraction; ESC: European Society of Cardiology; LBBB: Left bundle branch block; MI: Myocardial infarction; OMF: Outcome measurement framework; PCI: Percutaneous coronary intervention; PhenX: Consensus Measures for Phenotypes and eXposures; URL: Upper reference limit; WHF: World Heart Federation.
†
5.0
2007
Version
Criteria for established MI. Any one of the following criteria satisfies the diagnosis for established MI:
No
PhenX
Additional Yes (one) Clinically Equivalent Sources in OMF?
Joint ESC/ACCF/AHA/WHF Task Force for the Redefinition of Myocardial Infarction
Myocardial Infarction
Title
Source
201
151
OMF ID number Myocardial Infarction
Outcome measure example B‡
OMF field Outcome measure example A†
Table 1. Mock-up of outcome measures displayed for comparison within the Registry of Patient Registries system.
Research Article Gliklich, Leavy, Karl, Campion, Levy & Berliner
future science group
A framework for creating standardized outcome measures for patient registries
approach has advantages in terms of content quality, the curation process requires additional time and resources. Despite these limitations, the OMF tool offers practical mechanism for the display of information on the outcome measures currently in use, with the goal of reducing variation. Characterizing the outcome measures currently in use will support long-term efforts to develop standard outcome measures by identifying areas of common ground where standards may be developed relatively quickly and areas that will require additional work.
Research Article
Future perspective While the OMF tool will help identify and characterize outcome measures as they are currently being used, the findings are likely to only highlight the lack of standard outcome measures in most conditions. In order to have clinical medicine become truly outcome-based for both care decisions and payment, the next decade will require focused efforts by clinical specialists, patient organizations and scientific agencies to fill that void through development of numerous specialized outcome measures using a standard model, such as the OMF. Disclaimer
Conclusion The use of standard outcome measures in registries will increase opportunities for collaboration and allow results to be compared and aggregated across registries in the same condition area, thereby increasing the utility of registries for comparative effectiveness and other types of research. Standard outcome measures may also improve the quality of registry data by helping to ensure that key outcomes are collected consistently using clear, concise definitions. In the long term, standardization of outcome measures and data elements has the potential to facilitate the efficient use of research resources and may especially benefit patients, clinicians, researchers and funding agencies involved in designing patient registries and using registry data in quality improvement programs, patient-centered outcomes research and comparative effectiveness research.
The findings and conclusions in this document are those of the authors, who are responsible for its contents; the findings and conclusions do not necessarily represent the views of AHRQ. Therefore, no statement in this report should be construed as an official position of AHRQ or of the US Department of Health and Human Services.
Financial & competing interests disclosure The project described in this article was funded by the Agency for Healthcare Research and Quality under the Developing Evidence to Inform Decisions about Effectiveness (DEcIDE) program (contract number HHSA29020050035I TO7). The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed. No writing assistance was utilized in the production of this manuscript.
Executive summary • The use of different definitions for outcome measures introduces challenges when comparing or aggregating data across studies and may lead to uncertainty when interpreting study findings in the context of existing evidence. • Addressing variations in outcome measurement will require both the development of standardized sets of outcome measures and a new approach to cataloging existing outcome measures so that they are easily identified and widely adopted. • The outcome measurement framework (OMF) is a conceptual model for how information relevant to evaluating patient outcomes may be defined and collected in a standard way for a broad range of health conditions and treatments. • Compared with other models, the OMF model is more broadly applicable and provides the necessary level of detail to support development of new outcome measures. • The OMF tool, which builds on the foundation of the OMF model, facilitates collecting, displaying and searching for information on outcome measures used in patient registries. • The immediate goal of the OMF tool is to collate and characterize the outcome measures that registries currently collect, while the long-term goal is to support efforts to standardize outcome measures.
References
coronary syndromes and undergoing percutaneous coronary interventions and its impact on the apparent safety of antithrombotic drugs. Am. Heart J. 154(1), 3–11 (2007).
Papers of special note have been highlighted as: •• of considerable interest 1
Steinhubl SR, Kastrati A, Berger PB. Variation in the definitions of bleeding in clinical trials of patients with acute
future science group
2
Wang XL, Thompson MM, Dole WP et al. Standardization of outcome measures in clinical trials of pharmacological
www.futuremedicine.com
479
Research Article Gliklich, Leavy, Karl, Campion, Levy & Berliner treatment for abdominal aortic aneurysm. Expert Rev. Cardiovasc. Ther. 10(10), 1251–1260 (2012). 3
Shen J, Johnston M, Hays RD. Asthma outcome measures. Expert Rev. Pharmacoecon. Outcomes Res. 11(4), 447–453 (2011).
11
National Institute of Neurological Disorders and Stroke. Common Data Elements Initiative (2014). www.commondataelements.ninds.nih.gov
4
Sheehy C, Gaffney K, Mukhtyar C. Standardized grip strength as an outcome measure in early rheumatoid arthritis. Scand. J. Rheumatol. 42(4), 289–293 (2013).
12
Agency for Healthcare Research and Quality. United States Health Information Knowledgebase (2014). http://ushik.ahrq.gov/
5
Merkel PA, Herlyn K, Mahr AD et al. Progress towards a core set of outcome measures in small-vessel vasculitis. Report from OMERACT 9. J. Rheumatol. 36(10), 2362–2368 (2009).
13
Carpenter WR, Meyer AM, Abernethy AP, Stürmer T, Kosorok MR. A framework for understanding cancer comparative effectiveness research data needs. J. Clin. Epidemiol. 65(11), 1150–1158 (2012).
6
Registries for Evaluating Patient Outcomes: A User’s Guide 3rd Edition. Gliklich RE, Dreyer NA, Leavy M (Eds). Agency for Healthcare Research and Quality, MD, USA (2014).
••
The framework described in this paper informed the design of the conceptual model developed for this project.
14
••
Detailed discussion of the design, operation, analysis and evaluation of patient registries.
Porter ME. What is value in health care? N. Engl. J. Med. 363(26), 2477–2481 (2010).
••
7
Institute of Medicine. Learning What Works Best: the Nation’s Need for Evidence on Comparative Effectiveness in Health Care. National Academy Press, Washington DC, USA, 1–80 (2007).
This paper and its supplementary materials discuss key concepts related to outcome measurement.
15
Registry of Patient Registries (RoPR). Topic Abstract. Agency for Healthcare Research and Quality (2012). www.effectivehealthcare.ahrq.gov/index.cfm/search-forguides-reviews-and-reports/?pageaction=displayproduct&pr oductid=690
16
Agency for Healthcare Research and Quality. Registry of Patient Registries (RoPR) (2014). https://patientregistry.ahrq.gov
17
National Institutes of Health. PROMIS (2014). www.nihpromis.org
18
Thygesen K, Alpert JS, White HD. Universal definition of myocardial infarction. J. Am. Coll. Cardiol. 50(22), 2173–2195 (2007).
19
RTI International. Consensus Measures for Phenotypes and eXposures (PhenX). Myocardial Infarction (2012). www.phenxtoolkit.org
8
9
10
480
www.pcori.org/research-we-support/research-methodologystandards
Congressional Budget Office. Research on the Comparative Effectiveness of Medical Treatments: Issues and Options for an Expanded Federal Role (2007). www.cbo.gov/sites/default/files/cbofiles/ftpdocs/88xx/ doc8891/12-18-comparativeeffectiveness.pdf. US Department of Health and Human Services. Federal Coordinating Council for Comparative Effectiveness Research. Report to the President and the Congress (2009). www.tuftsctsi.org/∼/media/Files/CTSI/Library%20Files/ FCC%20for%20CER%20Rpt%20to%20Pres%20and%20 Congress_063009.ashx PCORI (Patient-Centered Outcomes Research Institute) Methodology Committee. The PCORI Methodology Report (2013).
J. Comp. Eff. Res. (2014) 3(5)
future science group
