RESEARCH ARTICLE For reprint orders, please contact: [email protected]
A tall order on a tight timeframe: stakeholder perspectives on comparative effectiveness research using electronic clinical data Background & significance: The AcademyHealth Electronic Data Methods Forum aims to advance the national dialogue on the use of electronic clinical data (ECD) for comparative effectiveness research (CER), patient-centered outcomes research, and quality improvement by facilitating exchange and collaboration among eleven research projects and external stakeholders. AcademyHealth conducted a mixed-method needs assessment with the Electronic Data Methods Forum’s key stakeholders to assess: stakeholder views on developing new infrastructure for CER using ECD; current gaps in knowledge with respect to CER; and expectations for a learning health system. Methods: AcademyHealth conducted 50 stakeholder interviews between August 2011 and November 2011 with participants from the following seven stakeholder groups: government, business/payer, industry, healthcare delivery, patient/consumer, nonprofit/policy and research. With input from key collaborators, AcademyHealth designed a semi-structured interview guide and a short survey. Reviewers used the qualitative data analysis software NVivo to code the transcripts and to identify and manage complex concepts. Quantitative data from the questionnaire has been integrated with the final analysis as relevant. Results: The analysis of recurring concepts in the interviews focus on five central themes: stakeholders have substantial expectations for CER using ECD, both with respect to addressing the limitations of traditional research studies, and generating meaningful evidence for decision-making and improving patient outcomes; stakeholders are aware of many challenges related to implementing CER with ECD, including the need to develop appropriate governance, assess and manage data quality, and develop methods to address confounding in observational data; stakeholders continue to struggle to define ‘patient-centeredness’ in CER using ECD, adding complexity to attaining this goal; stakeholders express that improving translation and dissemination of CER, and how research can be ‘useful’ at the point of care, can help mitigate negative perceptions of the CER ‘brand’; and stakeholders perceive a need for a substantial ‘culture shift’ to facilitate collaborative science and new ways of conducting biomedical and outcomes research. Many stakeholders proposed approaches or solutions they felt might address the challenges identified.
Erin Holve1, Marianne Hamilton Lopez*1, Lisa Scott2 & Courtney Segal1 AcademyHealth, 1150 17th Street NW, Suite 600, Washington, DC 20036, USA 2 Acuitas Research, 175 W Jackson Boulevard, Chicago, IL 60604, USA *Author for correspondence: Tel.: +1 202 292 6750 Fax: + 1 202 292 6850 1
Keywords: analytic methods n clinical informatics n comparative effectiveness research n electronic clinical data n governance n infrastructure n learning healthcare system n learning health system n stakeholders
Background & significance
Of the US$1.1 billion in funding for comparative effectiveness research (CER) from the American Recovery and Reinvestment Act of 2009, approximately US$417.2 million was directed to improve and enhance the infrastructure and
10.2217/CER.12.47 © 2012 Future Medicine Ltd
1(5), 441–451 (2012)
part of
ISSN 2042-6305
441
Research article
Holve, Hamilton Lopez, Scott & Segal
capacity for conducting CER [101,102]. Within this amount, the Agency for Healthcare Research and Quality (AHRQ ) supported three grant programs: the Prospective Outcome Systems using Patient-specific Electronic data to Compare Tests and therapies (PROSPECT), the Scalable Distributed Research Network (DRN) for CER, and the Enhanced Registry for Quality Improvement (QI) to build infrastructure and methods to conduct CER, patient-centered outcomes research (PCOR), and QI with electronic clinical data (ECD). An additional AHRQ-funded project, the AcademyHealth Electronic Data Methods (EDM) Forum, is a harmonizing entity for the 11 research projects funded by the three grant programs and aims to advance the national dialogue on the use of ECD for CER, PCOR and QI by facilitating exchange and collaboration among the research projects and external stakeholders (e.g., the EDM Forum Steering Committee and relevant funding agencies and nongovernmental organizations) [1–3]. AcademyHealth conducted a mixed-method needs assessment with the EDM Forum’s key stakeholders to assess: stakeholder views on developing new infrastructure for CER using ECD; current gaps in knowledge with respect to CER; and expectations for a ‘learning health system’, in which data generated in the process of care can be used for QI and research to understand ‘what works best and for whom’ [103] and then cycled back into the process of care so that individuals and communities may benefit from the most current data and information to improve health outcomes. Information on the comparative effectiveness of treatments is of great interest to individuals in this process [4], and often requires linking large, heterogeneous data sources from multiple institutions, which necessitates collaborative ECD infrastructure (governance, data, methods and training) [102]. A 2011 JAMA commentary by Navathe, Clancy and Glied discusses current federal investments to build infrastructure for CER and expresses the goal that these investments can address some of the data and methodological issues in outcomes research [5]. Yet, while ECD has the potential to enable research on a myriad of questions to improve patient outcomes, the collection, input, storage, transfer and analysis of these data for research and QI is a substantial and complex effort. Similarly, while there is strong support for the learning health system [6–10,104], it remains
442
J. Compar. Effect. Res. (2012) 1(5)
unclear whether stakeholders involved in the production and use of CER share similar expectations regarding the types of data infrastructure, the research methods and the dissemination efforts required. Methods
AcademyHealth conducted 50 stakeholder interviews between August 2011 and November 2011 by telephone. ■■ Sample population
Stakeholder groups were developed based on a review and synthesis of stakeholder groups developed for previous projects [11] and reviewed by project consultants, the AHRQ project officer and Steering Committee leadership [105]. Individuals selected for interviews were identified by first selecting an initial set of experts based on their work in the published literature, conference proceedings and personal knowledge of relevant project leads. A snowball sampling strategy was employed to achieve relative balance among stakeholder groups. As a result, participants from the following seven stakeholder groups were included: ■■ Government (representatives from local, state, or Federal government [18%]) ■■
Business/payer (private and public entities making coverage decisions [10%])
■■
Industry (designers, producers, distributors or evaluators of products used in health research and care [14%])
■■
Healthcare delivery (healthcare providers and leaders [administrative and clinical] in healthcare delivery systems [10%])
■■
Patient/consumer (patient and consumer advocates [10%])
■■
Nonprofit/policy (representatives impacting policy through analytics, funding and thought leadership [16%])
■■
Research (researchers at universities, non university-based research institutes and organizations [22%])
■■ Instruments
With input from core leadership, AcademyHealth designed a semi-structured interview guide and short survey. The interview guide focused on the participants’ vision for the data infrastructure to support a learning health system, impressions
future science group
Stakeholder perspectives on comparative effectiveness research using electronic clinical data
of what the proliferation of ECD will mean for research and preferences for how CER evidence is translated and disseminated. Because the interviews were exploratory in nature and the interviewees represented diverse perspectives, the instrument followed a semi-structured format (Box 1). For example, standard, structured interview questions such as ‘what … the proliferation of electronic health data (e.g., in electronic health records [EHRs] and registries) is going to mean for research?’ were followed by probing questions to address specific opportunities and challenges for CER based on the interviewees’ specific policy considerations, methodological issues, community perceptions, among others. A set of eight short survey questions were designed to better understand stakeholders’ perspectives on the relative value of specific attributes of CER studies (Figure 1). The survey was developed in part based on the pragmatic-explanatory continuum indicator summary (PRECIS) tool designed to look at attributes of pragmatic trials [12]. The Western Institutional Review Board granted AcademyHealth an institutional review board exemption for this project.
agreement measure reported between the coders was 98.3%, and the weighted k-coefficient was 0.76. Taken together, the measures for this exploratory study suggest an ‘excellent’ rate of agreement [106]. Quantitative data from the questionnaire has been integrated with the final analysis as relevant. Using the codes as a guideline, reviewers applied analysis of word repetition and the constant comparison method [13] to identify commonly mentioned concepts and keywords, and patterns throughout the interviews. As a result, five central themes were identified. Results
The five central themes include: ■■ Stakeholders have substantial expectations for CER using ECD, both with respect to addressing the limitations of traditional research studies, and generating meaningful evidence for decision-making and improving patient outcomes; ■■
Stakeholders are aware of many challenges related to implementing CER with ECD, including the need to develop appropriate governance, assess and managing data quality and develop methods to address confounding in observational data;
■■
Stakeholders continue to struggle to define ‘patient centeredness’ in CER using ECD, adding complexity to attaining this goal;
■■
Stakeholders express that improving translation and dissemination of CER, and how research can be ‘useful’ at the point of care, can help mitigate negative perceptions of the CER ‘brand’;
■■
Stakeholders perceive a need for a substantial ‘culture shift’ to facilitate collaborative science and new ways of conducting biomedical and outcomes research.
■■ Review & coding
Each interview was conducted by a trained interviewer and at least one member of the AcademyHealth staff. With the permission of each participant, each interview was transcribed and recorded for note-taking purposes. AcademyHealth reviewers used the qualitative data analysis software NVivo 9 to code each of the interview transcripts for complex concepts. The analysis was intentionally focused on identifying key themes and diverse viewpoints that emerged within the exploratory, semi-structured design of the study (Box 1). An initial list of codes was developed using a four-step process of carefully reviewing a subset of early interviews (n = 5); reviewing a summary of the interviews developed by the consultant based on her field notes; identifying themes from other EDM Forum research activities including a literature review and site visits; and team/group deliberations. The reviewers conducted a lineby-line analysis of the transcripts. All transcripts were blind-coded by two AcademyHealth reviewers. Throughout the process, inconsistencies between the reviewers’ coding were deliberated, emergent codes were added based on discussions between the two coders and the code list was refined iteratively. The weighted percentage
future science group
Research article
Many stakeholders proposed approaches or solutions they felt might address the challenges identified. ■■ Substantial expectations for CER using ECD to address the limitations of traditional research studies, generate meaningful evidence for decision-making & improve patient outcomes
“We believe quite firmly that better use of electronic health information is a key to real health reform.
www.futuremedicine.com
443
Research article
Holve, Hamilton Lopez, Scott & Segal
Box 1. Electronic Data Methods Forum stakeholder interview guide. 1. Tell us about how your work relates to CER and efforts to build a learning healthcare system? Potential areas to discuss: ■■ What are your priority areas for research/information that would be most helpful to your stakeholders/members? ■■ Priority conditions? ■■ Priority populations? ■■ Do you consider yourself a user or producer of CER? ■■ If a user/reviewer of CER findings: ■■ Which communities or stakeholders do you represent or interact with most often? ■■ What types of information do you want on CER? ■■ Are there key resources you rely on for CER findings? ■■ Are there key resources you rely on for information about CER? ■■ Is there information you cannot obtain? ■■ Do you have suggestions/ideas for how to fill those gaps? ■■ How do you assess if CER findings are credible? ■■ What sources do you rely on to assess whether the CER findings are credible? ■■ What sources do you rely on to assess whether the data and methods used to generate the study are credible? ■■ If a researcher/someone who produces CER: ■■ What types of information do you want on CER? ■■ Are there key resources you rely on for CER findings? ■■ Are there key resources you rely on for information on CER? ■■ Is there information you cannot obtain? ■■ Do you have suggestions/ideas for how to fill those gaps? ■■ How long have you been conducting CER and/or PCOR? ■■ Roughly how many CER or PCOR studies have you undertaken? ■■ What are the primary topical areas of your research? ■■ Which analytic methods have you used to conduct CER? ■■ What approaches have you/do you take to engage patients in your research? ■■ How do you assess if CER findings are credible? ■■ What sources do you rely on to assess whether the CER findings are credible? ■■ What sources do you rely on to assess whether the data and methods used to generate the study are credible? 2. What is your vision for the data infrastructure to support a learning healthcare system? 3. What do you think the proliferation of electronic health data (e.g., in EHRs and registries) is going to mean for research? ■■ Where do you see the biggest opportunities for CER? ■■ What are the greatest challenges for CER? Potential areas to discuss: ■■ Do you perceive there are gaps and/or shortcomings in the current infrastructure or knowledge with regard to: ■■ Analytic methods? ■■ Clinical informatics? ■■ Data governance (data security and privacy)? 4. What is your preference for the way CER evidence is disseminated and translated to your stakeholders/members, among others? 5. Who do you look to/identify as leaders or innovators in this area? Are there specific projects or programs that you think we should look to/review? 6. What types of resources, support, and dialogue are needed to make the most of current investments to build the infrastructure for CER? Potential areas to discuss: ■■ What is your vision for disseminating research findings and information on the new research infrastructure for CER we have described? ■■ Do you think there may be potential challenges for publishing CER findings based on observational data sources? CER: Comparative effectiveness research; EHR: Electronic health record; PCOR: Patient-centered outcomes research.
That’s because it is essential to so many of the decisions that are made at individual patient-care levels and also population and policy levels.” – Nonprofit/policy representative
444
J. Compar. Effect. Res. (2012) 1(5)
Stakeholders have substantial expectations for CER using ECD, especially with respect to the potential for research with ECD to address some limitations of traditional research studies [2]. As
future science group
Stakeholder perspectives on comparative effectiveness research using electronic clinical data
Research article
SE = 0.1 4.9
Patients representing populations of interest?
SE = 0.1
Measure of multiple outcomes of interest to clinicians, decision-makers and community
4.4
Account of stakeholder interests/needs in prioritizing research questions?
4.3
PROs and/or QI measures?
4.3
Interventions tested in multiple settings?
4.3
SE = 0.2
SE = 0.2
SE = 0.2
Peer review?
4.2
Consideration of feasibility of implementation and applicability to clinical settings?
4.1
SE = 0.2
SE = 0.2
Maximum level of internal validity?
SE = 0.2
3.5 1
2
3 Response scale
4
5
Figure 1. Mean values for attributes of comparative effectiveness research as reported by stakeholders. PRO: Patient-reported outcome; QI: Quality improvement; SE: Standard error.
the survey results demonstrate, the mean value of interviewees’ responses was 4.1 or higher across nearly all of the research domains/dimensions, signifying respondents found each of the domains ‘very’ important. In the interviews, many participants commented on the need to accelerate the research cycle so results are timelier and the research process is, ideally, less expensive. The desire to improve representation of vulnerable subpopulations in research was mentioned frequently. With respect to improving representation, there is concordance between stakeholders’ comments and survey findings. The desire for CER with clinical data to be highly representative of populations is the most highly ranked attribute (mean = 4.9; standard deviation = 0.3). As described by an industry representative “I think EHRs can give us an understanding of the real world that we can’t get today.” Another nonprofit/policy representative noted: “I see the potential for much greater capabilities to do research on populations of interest … and the potential to research certain rarer conditions.” A government representative articulated his
future science group
concept of a best-case scenario in which “…the entire medical community is using electronic information tools. Everybody has adopted, everyone is on board. Many, if not most, clinical communities will have registries – sort of the town hall, the community waterholes where you can identify your greatest challenges and get insight from community leaders on the best ways to meet challenges.” ■■ Stakeholders are aware of many challenges related to implementing CER with ECD (e.g., governance; data quality; addressing selection bias)
“The challenge we have is learning how to use it (the data), how to actually harness it, ask questions and actually take the data to the next step. I have to tell you that is a big deal. That is a bigger deal than people anticipated.” – Healthcare delivery representative Many stakeholders raised the challenges of governing data and ensuring data quality. Both the real and perceived methodological challenges of using observational data were also addressed, particularly those related to selection bias.
www.futuremedicine.com
445
Research article
Holve, Hamilton Lopez, Scott & Segal
Governing creation, access & use of ECD for research
The term ‘governance’ is used by the EDM Forum to reflect a set of concepts including research network partnerships, data sharing, privacy, security and data access [14]. However, most stakeholders did not directly refer to ‘governance.’ Instead, stakeholders mentioned ‘privacy’ or ‘security’ concerns, and strategies to manage research networks. Many participants discussed the tension between maintaining the highest degree of security and privacy while conducting research, given the complexities of collecting, storing and transferring personal health data across sites and systems. While the necessity of formal arrangements for research data use agreement was discussed, the time and cost of these efforts was mentioned as a major challenge. The role of institutional review boards (IRBs) and the Health Insurance Portability and Accountability Act Privacy rule in multisite research using ECD was raised by many as an area needing more clarity given that IRBs often establish the rules by which institutional datasharing partnerships function. One researcher emphasized the need to clarify the specific role IRBs should play in the oversight of this process, commenting, “The whole thing is going to be dead on its face if we can’t sort out these IRB issues.” Recommendations to improve governance included centralized or state IRBs, and the development of formal guidance for IRBs on multisite research. Some also expressed a desire to revisit or reframe the HIPAA privacy rule. Assessing & managing ECD data quality
Several methodological concerns were mentioned, most of which stem from the need to enhance existing methods and develop new approaches to evaluate the electronic data that is available. While stakeholders see the availability of more ECD as a significant contribution to research, several expressed concern that access to more data alone is insufficient. One researcher noted “Often when we have so much data – medical records data – people think, ‘Oh, the problem is solved. There is tons of information.’ Not so.” In addition, many practical challenges to using EHR data were enumerated. Chief among these is the issue of semantic interoperability –the ability to match nonstandardized terms used by different institutions to represent common medical conditions, treatments and processes [15]. According to one industry representative: “Individual
446
J. Compar. Effect. Res. (2012) 1(5)
organizations might have found a way to mine their own data, but bringing it across organizations is a nightmare for a zillion reasons … the heart rate might be called pulse; it might be called 29 different things.” As a result, there is a strong desire for developing new methods to assess data quality, including strategies to address incorrect or missing data due to collection error or inaccessible data due to limitations of the EHR. Stakeholders expressed that the issue of missing data is exacerbated because systems and informatics tools at many institutions are designed to facilitate billing rather than to address CER questions. One government representative expressed concern that for ECD a lot of the missing data is “either buried in text or is not in the chart at all.” Analyzing ‘big health data’ & controlling for selection bias
Some stakeholders identified the need to think critically and synthesize volumes of data in innovative ways. One researcher commented: “None of these are statistical problems. They are analytical thinking challenges.” Another researcher noted that, “The heavy lifting is taking streams of data and coming up with a higher-level syntheses … the challenge is not the number of bits or the volume of information, it is that this is informed by expertise about health and medicine and physiology.” Stakeholders also commented on the challenge of controlling for confounding variables when using observational data. One researcher said that “If you get the methods wrong, all the wonderful studies will be limited. In methods, the big umbrella is confounding control. How can we make sure the treatment groups we are comparing are balanced in terms of the baseline risk?” Some strategies to address confounding and selection bias were mentioned, including comparing efforts to standardize adjustment using a minimal level of confounding and comparing these results to analyses in which confounding adjustments are maximized based on available evidence. Others proposed conducting sensitivity analyses in different settings or populations, or using electronic clinical systems to conduct experimental research by randomizing patients at the point of care. ■■ Defining & attaining patient-centeredness in CER with ECD
“With the advent of health information technology, there are many opportunities for more active
future science group
Stakeholder perspectives on comparative effectiveness research using electronic clinical data
engagement [of patients] … research doesn’t have to be something done to patients. It can be done with patients. Patients don’t just supply information but also can be involved in designing the question.” – Patient/consumer representative Although not asked specifically about how CER using ECD would impact patients and their potential role in research, stakeholders from all seven stakeholder groups consistently raised the topic. Many stakeholders ascribed value to conducting CER with ECD based on patient values and outcomes because, as one healthcare delivery representative said, “CER is for patients.” Stakeholders used the terms ‘patient engagement’ and patient-centeredness interchangeably, which may be due to the fact that this field and these concepts are still emerging. While a formal definition of PCOR has recently been released by the Patients Centered Outcomes Research Institute [107] an accepted definition of PCOR was not available at the time the interviews were conducted. Comments focused on the need for including patients’ values and perspectives when developing infrastructure (e.g., EHRs) and research (e.g., defining priorities and desired outcomes). However, an emerging theme from the interviews is that stakeholders continue to struggle to define and attain the goal of patient-centeredness. Some stakeholders questioned which patients should be engaged, at what time and in what capacity. One patient/consumer representative recommended that patients and their representatives be “involved in every single aspect [of CER] from the beginning to the usage of the information,” while an industry representative stated that she was “on the fence about the role of the patient.” Neither support for patient involvement nor uncertainty about their most effective roles was limited to specific stakeholder segments. For example, involving patients in collecting and curating data was suggested by a few participants, including an industry representative who commented, “There should be feedback loops that allow patients to update and correct the data. That’s the data set we should be working with. Otherwise, we are spending a lot of time cleaning up noisy data.” ■■ Improving translation & dissemination of CER & how research can be ‘useful’ at the point of care, can help mitigate negative perception of the CER ‘brand’
“Communication around CER must build public support, dispel fears and, while doing so, manage
future science group
Research article
expectations in a way that still garners support for public and private investment over the decades required. Communication management might be the number one challenge because it is so vital to success and so complicated to do right.” – Nonprofit/policy representative Finding new and more efficient approaches to effectively translate and disseminate CER and its findings was a prominent theme. Several participants explicitly mentioned concern that the public might resist the ‘CER brand,’ especially where CER is perceived as a method for limiting patients’ healthcare choices. One patient/ consumer representative explained: “You may format the best announcement [of CER], but if people get the wrong impression you get an F in reality … don’t call it CER. Off the top of my head it is just effectiveness information…” As described by stakeholders, CER should be disseminated so that research findings inform healthcare decision-making at the point of care. As articulated by one industry representative, “This information needs to get into decisionmaking by many people: by patients, doctors, payers.” Multiple stakeholders, including a patient/ consumer representative, suggested presenting CER findings in the context of decision-making: “Make sure you say, ‘Here are results’. We are going to outline three or four concrete ways in which these data are relevant to you and what you can do now that you have this information.’’ The current heavy reliance on peer-reviewed literature for disseminating research findings was viewed as a problem. While valued for its rigor, many commented that the peer-review process can restrict the impact of CER on patient outcomes because of the lengthy time required for review and publication, and because of the elite and limited readership. One researcher stated, “Only academics read academic journals.” This concern is supported by results from the survey, in which ‘peer review’ was the one of the lower ranked attributes of CER using ECD (mean = 4.2; SE = 0.2); though there were divergent views on the extent to which peer review was a strong indicator of quality research. Several alternatives to traditional peer-reviewed publications were proposed, including social media platforms and communities, targeting messages and presentations for a number of audiences, and asking external stakeholders, such as consumer advocacy organizations, to disseminate findings. These opportunities are enhanced with the opportunity to draw on more timely
www.futuremedicine.com
447
Research article
Holve, Hamilton Lopez, Scott & Segal
data-collection efforts and clinical decision support strategies that can accelerate the uptake of findings at the point of care. Stakeholders suggest that collaboration between stakeholders may also alleviate dissemination challenges because the participants in the research would be key end-users of the results and invested in the findings. As one researcher said, “When knowledge is generated with a community organized around a common purpose, a learning collaborative, the knowledge that is generated is highly desired. In a learning collaborative, that (desire) becomes the dissemination channel.”
Some stakeholders articulated the need for a substantial ‘culture shift’: “The way I can see to solve this [lack of data standardization and lack of specificity in the data] is if you prospectively partner with the right people at the right time in strategic engagements, coupled with having the technological infrastructure” said an industry representative. A researcher commented, “You can’t force it [collaboration]. What you can do is highlight the value of collaboration … Investigators are the first ones, once they realize the added value of collaboration, to jump on the opportunity.” Discussion
■■ The need for a substantial ‘culture shift’ to facilitate collaborative science
“For me, it is really about strategic partnerships. It is a multipronged approach. It is leveraging existing infrastructure. It is recognizing value of strategic partnerships. It is nurturing them. There are people who don’t typically partner. It is understanding what the word partner means.” – Industry representative Many of the stakeholders note that conducting CER using ECD can foster collaboration and new partnerships among researchers, clinicians, other disciplines (e.g., informatics) and research funders. One researcher commented that complex problems require an “intensely transdisciplinary” approach to research, while an industry representative mentioned that partnerships across stakeholders bring value: “There is a space here for shared learning. We can learn from and with each other. I know there are things here that we are totally missing, and we don’t even know it.” Yet, some emphasized the unique practices, goals, standards of quality and entrenched business interests among stakeholder groups, which can create barriers to working together. As described by stakeholders, the research environment is entrenched by disciplines, with each field having its own methods, data and expertise. This ‘silo’ mentality was mentioned by researchers and policymakers as one set of professional disincentives to data sharing since – in the research community – access to unique data is often critical to securing publication and professional acknowledgment, including promotion and tenure. There is also intrinsic value in the data, as described by one industry representative: “People make millions of millions of dollars off their data. I don’t see them giving it up. Under no circumstances do I see that.”
448
J. Compar. Effect. Res. (2012) 1(5)
The findings from the EDM Forum stakeholder interviews underscore the time pressure that CER–American Recovery and Reinvestment Act investigators face to respond to external pressures and expectations among stakeholders. While stakeholders are aware of the complexity of CER and the need to develop robust approaches to address governance, informatics and analytic methods to use ECD for CER, they nonetheless express the need to capitalize on current investments and demonstrate the utility and value of CER now. There is an explicitly stated concern that the slower timeframe and disciplinary silos in which biomedical science traditionally operates and is disseminated will result in a lost opportunity to make a substantial impact on health and healthcare delivery. Stakeholders uniformly expect CER to be rigorous and high quality, but the standards by which quality is judged, such as peer review, are not always viewed as critical. Maximizing the highest level of internal validity (as in randomized control trials), for example, is outweighed by the need for representative and timely research. Perhaps as a result, unreasonably high expectations for the research were a concern expressed by some. As articulated by one government representative, “I think there are grand dreams that I fear have been built up to the point where they are just bound to be disappointed. ECD has been mentioned as a panacea to almost any data problem, whether it’s drug safety or public health surveillance … it is not unreasonable to say that research will be easier with electronic medical records, but the emphasis is on easier, not easy.” Balancing these perspectives is viewed as an important dimension of CER, and portends the enormous responsibility facing the research community, policymakers and engaged stakeholders
future science group
Stakeholder perspectives on comparative effectiveness research using electronic clinical data
charged with generating the best evidence to improve health and healthcare. Research limitations
CER using ECD is a nascent field of inquiry, which requires exploratory research in order to identify key perspectives and synthesize priorities for the field. As with any exploratory, qualitative study, there are limitations to the generalizability of our findings. First, the number of individuals working and writing on CER using ECD is relatively limited and there is no clear sampling frame for CER researchers. As a result, it is possible we missed key perspectives or divergent viewpoints. Second, despite an initial hypothesis that the quantitative portion of the mixed-methods analysis would yield distinct clusters of opinions regarding the relative value of CER attributes – which we further hypothesized might cluster by stakeholder group – this result did not hold true and most stakeholders ranked every attribute highly. This finding may indicate that, despite our efforts to engage divergent points of view, the available universe of individuals with the level of expertise we sought to discuss the (inherently technical) topic of CER using ECD was more homogeneous than anticipated, even for a new field of inquiry. Both limitations suggest that greatly expanding the number of participants and perspectives in the future would be valuable.
Research article
Future perspective
Given the relatively early stage of the PROSPECT, DRN and Enhanced Registry projects at which the stakeholder needs assessment was conducted, and the purpose for which the needs assessment was conducted – as an exploratory effort to understand perspectives on the conduct of CER with ECD to drive the EDM Forum’s activities – the manuscript does not propose policy recommendations beyond the scope of the EDM Forum, although this effort may be part of future work. Input from the stakeholder needs assessment has informed the Forum’s decisions regarding priority topics for commissioned papers, projects and issue briefs such as governance [16,17], patient engagement in research [18,19] and data quality and methods [20]. Moving forward, the findings from the stakeholder needs assessment will continue to guide the topics on which the EDM Forum engages stakeholders in active dialogue. This includes input into the selection of convening topics and research activities the Forum undertakes, as well as awareness of key issues where it is critical to translate and disseminate infrastructure efforts and research generated by the PROSPECT, DRN and Enhanced registry studies. The EDM Forum Stakeholder needs assessment demonstrates there is an emerging research agenda for CER using ECD that is extremely important to a broad set of community members. Understanding and improving patient
Executive summary Stakeholders have substantial expectations for comparative effectiveness research (CER) using electronic clinical data (ECD), both with respect to addressing the limitations of traditional research studies, and generating meaningful evidence for decisionmaking and improving patient outcomes. ■■ Stakeholders are aware of many challenges related to implementing CER with ECD, including the need to develop appropriate governance, assess and manage data quality, and develop methods to address confounding in observational data. ■■ Stakeholders continue to struggle to define ‘patient centeredness’ in CER using ECD, adding complexity to attaining this goal. ■■ Stakeholders express that improving translation and dissemination of CER, and how research can be ‘useful’ at the point of care, can help mitigate negative perceptions of the CER ‘brand.’ ■■ Stakeholders perceive a need for a substantial ‘culture shift’ to facilitate collaborative science and new ways of conducting biomedical and outcomes research. ■■ The findings from the Electronic Data Methods (EDM) Forum stakeholder interviews underscore the time pressure that CER American Recovery and Reinvestment Act investigators face to respond to external pressures and expectations among stakeholders. ■■ There is an explicity stated concern that the slower timeframe and disciplinary silos in which biomedical science traditionally operates and is disseminated will result in a lost opportunity to make a substantial impact on health and healthcare delivery. ■■ Stakeholders express the need to capitalize on current investments and demonstrate the utility and value of CER now. ■■ Input from the stakeholder needs assessment has informed the EDM Forum’s decisions regarding priority topics for commissioned papers, projects, and issue briefs such as governance, patient engagement in research, and data quality and methods. Moving forward, the findings from the stakeholder needs assessment will continue to guide the topics on which the EDM Forum engages stakeholders in active dialogue, as well as dissemination activities. ■■
future science group
www.futuremedicine.com
449
Research article
Holve, Hamilton Lopez, Scott & Segal
engagement in clinical research, addressing legal and technical barriers related to multisite electronic data sharing and improving the quality of data flowing from the EHR are worthwhile investments that stakeholders perceive will contribute to advancing the science of CER to improve patient outcomes. Acknowledgements The authors would like to acknowledge A Rein for her contribution to the construction of the study and data collection and T Trudnak for her guidance and review. Special thanks are given to the 50 stakeholders that participated in the interviews.
Financial & competing interests disclosure AcademyHealth acknowledges the Agency for Healthcare Research and Quality for its support of this work through
n
1
n
2
n
3
450
n
Randhawa GS, Slutsky JR. Building sustainable multi-functional prospective electronic clinical data systems. Med. Care 50(7 Suppl. 1), S3–S6 (2012). Randhawa provides background information about the Agency for Healthcare Research and Quality-supported projects to build flexible prospective clinical electronic data infrastructure that meet the needs of diverse users and the Electronic Data Methods (EDM) Forum to advance the methods in clinical informatics, research analytics and governance by actively engaging investigators from the American Recovery and Reinvestment Act-funded projects as well as external stakeholders. Holve E, Segal C, Hamilton Lopez M, Rein A, Johnson BH. The Electronic Data Methods (EDM) Forum for comparative effectiveness research (CER). Med. Care 50(7 Suppl. 1), S7–S10 (2012). Holve et al. provide detailed information about the creation and function of the AcademyHealth EDM Forum to collect, synthesize and share lessons learned from eleven research projects that are building infrastructure and using electronic clinical data for comparative effectiveness research (CER) and patient-centered outcomes research (PCOR). Holve E, Segal C, Hamilton Lopez M. Opportunities and challenges for comparative effectiveness research (CER) with electronic clinical data: a perspective from the EDM
Ethical conduct of research The authors state that they have obtained appropriate insti tutional review board approval or have followed the princi ples outlined in the Declaration of Helsinki for all human or animal experimental investigations. In addition, for investigations involving human subjects, informed consent has been obtained from the participants involved.
Forum. Med. Care 50(7 Suppl. 1), S11–S18 (2012).
References Papers of special note have been highlighted as: of interest
the American Recovery & Reinvestment Act of 2009, Grant U13 HS19564-01 (Electronic Data Methods Forum for Comparative Effectiveness Research). The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed. No writing assistance was utilized in the production of this manuscript.
4
5
n
6
Provides an overview of the challenges faced by research teams as part of their efforts to develop electronic clinical data (ECD) infrastructure to support CER. The findings reflect a set of opportunities for transdisciplinary learning, and will ideally enhance the transparency and generalizability of CER using ECD. Deverka PA, Lavallee DC, Desai PJ et al. Stakeholder participation in comparative effectiveness research: defining a framework for effective engagement. J. Compar. Effect. Res. 1(2), 181–194 (2012). Navathe AS, Clancy C, Glied S. Advancing research data infrastructure for patient -centered outcomes research. JAMA 306(11), 1254–1255 (2011). The authors reflect on the expectation that current American Recovery and Reinvestment Act investments building the infrastructure for CER, coupled with technological advances, can address some of the data and methodological issues in outcomes research (a prominent theme in the manuscript). The Learning Healthcare System: Workshop Summary (IOM Roundtable on Evidence-Based Medicine). Olsen LA, Aisner D, McGinnis JM (Eds). National Academies Press, Washington, DC, USA (2007).
7
Etheredge LM. A rapid-learning health system. Health Aff. 26(2), 107–118 (2007).
8
Friedman CP, Wong AK, Blumenthal D. Achieving a nationwide learning health system. Sci. Transl. Med. 2(57), 29–31 (2010).
J. Compar. Effect. Res. (2012) 1(5)
9
Abernethy AP, Etheredge LM, Ganz PA et al. Rapid-learning system for cancer care. J. Clin. Oncol. 28(27), 4268–4274 (2010).
10 Gluck M. Early Glimpses of the Learning
Health Care System: The Potential Impact of Health IT. Health IT for Actionable Knowledge report. AcademyHealth, Washington, DC, USA (2012). 11 Holve E, Pittman P. A First Look at the
Volume and Cost of Comparative Effectiveness Research in the United States. AcademyHealth, Washington, DC, USA (2009). 12 Thorpe KE, Zwarenstein M, Oxman AD
et al. A pragmatic explanatory continuum indicator summary (PRECIS): a tool to help trial designers. CMAJ 180(10), E47–E57 (2009). 13 Ryan GW, Bernard HR. Techniques to
identify themes. Field Methods 15(1), 85–109 (2003). 14 Rosenbaum S. Data governance and
stewardship: designing data stewardship entities and advancing data access. Health Serv. Res. 45(5), 1442–1455 (2010). n
Rosenbaum writes about the relationship between US health policy and health information, especially related to data collection and sharing, and provides a working definition of ‘data governance.’
15 Blobel B, Oemig F, Dössel O. What is
needed to finally achieve semantic interoperability? In: World Congress on Medical Physics and Biomedical Engineering. Schlegel WC (Ed.). Springer BerlinHeidelberg, Munich, Germany, 411–414 (2009).
future science group
Stakeholder perspectives on comparative effectiveness research using electronic clinical data
16 Marsolo K. Medical care: approaches to
20 Kahn MG, Raebel MA, Glanz JM,
facilitate institutional review board approval of multicenter research studies. Med. Care 50 (7 Suppl. 1), S77–S81 (2012). n
Marsolo examines the IRB review process for multicenter studies (one of the topics discussed in-depth in the manuscript) and provides strategies used to streamline the IRB review process examples utilized by two existing multicenter CER networks.
17 Sabharwal R, Holve E, Rein A, Segal C.
Approaches to Using Protected Health Information (PHI) for Patient-Centered Outcomes Research (PCOR): Regulatory Requirements, De-identification Strategies, and Policy. EDM Forum, AcademyHealth, Washington, DC, USA (2012). 18 Hamilton Lopez M, Holve E, Rein A,
Winkler J. Involving Patients and Consumers in Research: New Opportunities for Meaningful Engagement in Research and Quality Improvement. EDM Forum, AcademyHealth, Washington, DC, USA (2012). 19 Rein A, Hamilton Lopez M, Winkler J,
Holve E. Involving Patients and Consumers in Research: New Opportunities for Meaningful Engagement in Research and Quality Improvement. EDM Forum, AcademyHealth, Washington, DC, USA (2012) (In Press).
future science group
Riedlinger K, Steiner JF. A pragmatic framework for single-site and multisite data quality assessment in electronic health record-based clinical research. Med. Care. 50 (7 Suppl. 1), S21–S29 (2012). n
The authors examine the issue of multisite data-quality assessment (one of the topics discussed in-depth in the manuscript) and proposes a ‘fit-for-use’ conceptual model for data quality assessment and a process model for planning and conducting single-site and multisite data-quality assessments.
■■ Websites 101 US Department of Health and Human
Services. Comparative Effectiveness Research Funding (HHS Recovery Act). Washington, DC, USA (2010). www.hhs.gov/recovery/programs/cer/index. html (Accessed 17 March 2011) 102 Agency for Healthcare Research and Quality.
American Recovery and Reinvestment Act Investments in Comparative Effectiveness Research for Data Infrastructure. AHRQ, Rockville, MD (2010). www.ahrq.gov/fund/cerfactsheets/osfsinfra. htm (Accessed 19 June 2012)
www.futuremedicine.com
Research article
103 Institute of Medicine. Learning what works
best: the nations need for evidence on comparative effectiveness in health care (2007). www.iom.edu/ebm-effectiveness (Accessed 2 August 2012) n
Articulates the need for a ‘learning healthcare system’ a topic discussed in-depth in the manuscript.
104 Federal Coordinating Council for Comparative
Effectiveness Research. Report to the President and The Congress. US Department of Health and Human Services, NC, USA (2009). www.hhs.gov/recovery/programs/cer/ cerannualrpt.pdf 105 The Electronic Data Methods Forum Steering
Committee (2011). www.edmforum.org/publicgrant/About/ SteeringCommittee (Accessed 5 June 2012). 106 Lombard M, Snyder-Duch J, Campanella
Bracken C. Practical resources for assessing and reporting intercoder reliability in content analysis research projects (2010). http://matthewlombard.com/reliability (Accessed 8 February 2012) 107 Patient Centered Outcomes Research Institute.
Patient-centered outcomes research. www.pcori.org/patient-centered-outcomesresearch (Accessed 6 August 2012)
451
A tall order on a tight timeframe: stakeholder perspectives on comparative effectiveness research using electronic clinical data Background & significance: The AcademyHealth Electronic Data Methods Forum aims to advance the national dialogue on the use of electronic clinical data (ECD) for comparative effectiveness research (CER), patient-centered outcomes research, and quality improvement by facilitating exchange and collaboration among eleven research projects and external stakeholders. AcademyHealth conducted a mixed-method needs assessment with the Electronic Data Methods Forum’s key stakeholders to assess: stakeholder views on developing new infrastructure for CER using ECD; current gaps in knowledge with respect to CER; and expectations for a learning health system. Methods: AcademyHealth conducted 50 stakeholder interviews between August 2011 and November 2011 with participants from the following seven stakeholder groups: government, business/payer, industry, healthcare delivery, patient/consumer, nonprofit/policy and research. With input from key collaborators, AcademyHealth designed a semi-structured interview guide and a short survey. Reviewers used the qualitative data analysis software NVivo to code the transcripts and to identify and manage complex concepts. Quantitative data from the questionnaire has been integrated with the final analysis as relevant. Results: The analysis of recurring concepts in the interviews focus on five central themes: stakeholders have substantial expectations for CER using ECD, both with respect to addressing the limitations of traditional research studies, and generating meaningful evidence for decision-making and improving patient outcomes; stakeholders are aware of many challenges related to implementing CER with ECD, including the need to develop appropriate governance, assess and manage data quality, and develop methods to address confounding in observational data; stakeholders continue to struggle to define ‘patient-centeredness’ in CER using ECD, adding complexity to attaining this goal; stakeholders express that improving translation and dissemination of CER, and how research can be ‘useful’ at the point of care, can help mitigate negative perceptions of the CER ‘brand’; and stakeholders perceive a need for a substantial ‘culture shift’ to facilitate collaborative science and new ways of conducting biomedical and outcomes research. Many stakeholders proposed approaches or solutions they felt might address the challenges identified.
Erin Holve1, Marianne Hamilton Lopez*1, Lisa Scott2 & Courtney Segal1 AcademyHealth, 1150 17th Street NW, Suite 600, Washington, DC 20036, USA 2 Acuitas Research, 175 W Jackson Boulevard, Chicago, IL 60604, USA *Author for correspondence: Tel.: +1 202 292 6750 Fax: + 1 202 292 6850 1
Keywords: analytic methods n clinical informatics n comparative effectiveness research n electronic clinical data n governance n infrastructure n learning healthcare system n learning health system n stakeholders
Background & significance
Of the US$1.1 billion in funding for comparative effectiveness research (CER) from the American Recovery and Reinvestment Act of 2009, approximately US$417.2 million was directed to improve and enhance the infrastructure and
10.2217/CER.12.47 © 2012 Future Medicine Ltd
1(5), 441–451 (2012)
part of
ISSN 2042-6305
441
Research article
Holve, Hamilton Lopez, Scott & Segal
capacity for conducting CER [101,102]. Within this amount, the Agency for Healthcare Research and Quality (AHRQ ) supported three grant programs: the Prospective Outcome Systems using Patient-specific Electronic data to Compare Tests and therapies (PROSPECT), the Scalable Distributed Research Network (DRN) for CER, and the Enhanced Registry for Quality Improvement (QI) to build infrastructure and methods to conduct CER, patient-centered outcomes research (PCOR), and QI with electronic clinical data (ECD). An additional AHRQ-funded project, the AcademyHealth Electronic Data Methods (EDM) Forum, is a harmonizing entity for the 11 research projects funded by the three grant programs and aims to advance the national dialogue on the use of ECD for CER, PCOR and QI by facilitating exchange and collaboration among the research projects and external stakeholders (e.g., the EDM Forum Steering Committee and relevant funding agencies and nongovernmental organizations) [1–3]. AcademyHealth conducted a mixed-method needs assessment with the EDM Forum’s key stakeholders to assess: stakeholder views on developing new infrastructure for CER using ECD; current gaps in knowledge with respect to CER; and expectations for a ‘learning health system’, in which data generated in the process of care can be used for QI and research to understand ‘what works best and for whom’ [103] and then cycled back into the process of care so that individuals and communities may benefit from the most current data and information to improve health outcomes. Information on the comparative effectiveness of treatments is of great interest to individuals in this process [4], and often requires linking large, heterogeneous data sources from multiple institutions, which necessitates collaborative ECD infrastructure (governance, data, methods and training) [102]. A 2011 JAMA commentary by Navathe, Clancy and Glied discusses current federal investments to build infrastructure for CER and expresses the goal that these investments can address some of the data and methodological issues in outcomes research [5]. Yet, while ECD has the potential to enable research on a myriad of questions to improve patient outcomes, the collection, input, storage, transfer and analysis of these data for research and QI is a substantial and complex effort. Similarly, while there is strong support for the learning health system [6–10,104], it remains
442
J. Compar. Effect. Res. (2012) 1(5)
unclear whether stakeholders involved in the production and use of CER share similar expectations regarding the types of data infrastructure, the research methods and the dissemination efforts required. Methods
AcademyHealth conducted 50 stakeholder interviews between August 2011 and November 2011 by telephone. ■■ Sample population
Stakeholder groups were developed based on a review and synthesis of stakeholder groups developed for previous projects [11] and reviewed by project consultants, the AHRQ project officer and Steering Committee leadership [105]. Individuals selected for interviews were identified by first selecting an initial set of experts based on their work in the published literature, conference proceedings and personal knowledge of relevant project leads. A snowball sampling strategy was employed to achieve relative balance among stakeholder groups. As a result, participants from the following seven stakeholder groups were included: ■■ Government (representatives from local, state, or Federal government [18%]) ■■
Business/payer (private and public entities making coverage decisions [10%])
■■
Industry (designers, producers, distributors or evaluators of products used in health research and care [14%])
■■
Healthcare delivery (healthcare providers and leaders [administrative and clinical] in healthcare delivery systems [10%])
■■
Patient/consumer (patient and consumer advocates [10%])
■■
Nonprofit/policy (representatives impacting policy through analytics, funding and thought leadership [16%])
■■
Research (researchers at universities, non university-based research institutes and organizations [22%])
■■ Instruments
With input from core leadership, AcademyHealth designed a semi-structured interview guide and short survey. The interview guide focused on the participants’ vision for the data infrastructure to support a learning health system, impressions
future science group
Stakeholder perspectives on comparative effectiveness research using electronic clinical data
of what the proliferation of ECD will mean for research and preferences for how CER evidence is translated and disseminated. Because the interviews were exploratory in nature and the interviewees represented diverse perspectives, the instrument followed a semi-structured format (Box 1). For example, standard, structured interview questions such as ‘what … the proliferation of electronic health data (e.g., in electronic health records [EHRs] and registries) is going to mean for research?’ were followed by probing questions to address specific opportunities and challenges for CER based on the interviewees’ specific policy considerations, methodological issues, community perceptions, among others. A set of eight short survey questions were designed to better understand stakeholders’ perspectives on the relative value of specific attributes of CER studies (Figure 1). The survey was developed in part based on the pragmatic-explanatory continuum indicator summary (PRECIS) tool designed to look at attributes of pragmatic trials [12]. The Western Institutional Review Board granted AcademyHealth an institutional review board exemption for this project.
agreement measure reported between the coders was 98.3%, and the weighted k-coefficient was 0.76. Taken together, the measures for this exploratory study suggest an ‘excellent’ rate of agreement [106]. Quantitative data from the questionnaire has been integrated with the final analysis as relevant. Using the codes as a guideline, reviewers applied analysis of word repetition and the constant comparison method [13] to identify commonly mentioned concepts and keywords, and patterns throughout the interviews. As a result, five central themes were identified. Results
The five central themes include: ■■ Stakeholders have substantial expectations for CER using ECD, both with respect to addressing the limitations of traditional research studies, and generating meaningful evidence for decision-making and improving patient outcomes; ■■
Stakeholders are aware of many challenges related to implementing CER with ECD, including the need to develop appropriate governance, assess and managing data quality and develop methods to address confounding in observational data;
■■
Stakeholders continue to struggle to define ‘patient centeredness’ in CER using ECD, adding complexity to attaining this goal;
■■
Stakeholders express that improving translation and dissemination of CER, and how research can be ‘useful’ at the point of care, can help mitigate negative perceptions of the CER ‘brand’;
■■
Stakeholders perceive a need for a substantial ‘culture shift’ to facilitate collaborative science and new ways of conducting biomedical and outcomes research.
■■ Review & coding
Each interview was conducted by a trained interviewer and at least one member of the AcademyHealth staff. With the permission of each participant, each interview was transcribed and recorded for note-taking purposes. AcademyHealth reviewers used the qualitative data analysis software NVivo 9 to code each of the interview transcripts for complex concepts. The analysis was intentionally focused on identifying key themes and diverse viewpoints that emerged within the exploratory, semi-structured design of the study (Box 1). An initial list of codes was developed using a four-step process of carefully reviewing a subset of early interviews (n = 5); reviewing a summary of the interviews developed by the consultant based on her field notes; identifying themes from other EDM Forum research activities including a literature review and site visits; and team/group deliberations. The reviewers conducted a lineby-line analysis of the transcripts. All transcripts were blind-coded by two AcademyHealth reviewers. Throughout the process, inconsistencies between the reviewers’ coding were deliberated, emergent codes were added based on discussions between the two coders and the code list was refined iteratively. The weighted percentage
future science group
Research article
Many stakeholders proposed approaches or solutions they felt might address the challenges identified. ■■ Substantial expectations for CER using ECD to address the limitations of traditional research studies, generate meaningful evidence for decision-making & improve patient outcomes
“We believe quite firmly that better use of electronic health information is a key to real health reform.
www.futuremedicine.com
443
Research article
Holve, Hamilton Lopez, Scott & Segal
Box 1. Electronic Data Methods Forum stakeholder interview guide. 1. Tell us about how your work relates to CER and efforts to build a learning healthcare system? Potential areas to discuss: ■■ What are your priority areas for research/information that would be most helpful to your stakeholders/members? ■■ Priority conditions? ■■ Priority populations? ■■ Do you consider yourself a user or producer of CER? ■■ If a user/reviewer of CER findings: ■■ Which communities or stakeholders do you represent or interact with most often? ■■ What types of information do you want on CER? ■■ Are there key resources you rely on for CER findings? ■■ Are there key resources you rely on for information about CER? ■■ Is there information you cannot obtain? ■■ Do you have suggestions/ideas for how to fill those gaps? ■■ How do you assess if CER findings are credible? ■■ What sources do you rely on to assess whether the CER findings are credible? ■■ What sources do you rely on to assess whether the data and methods used to generate the study are credible? ■■ If a researcher/someone who produces CER: ■■ What types of information do you want on CER? ■■ Are there key resources you rely on for CER findings? ■■ Are there key resources you rely on for information on CER? ■■ Is there information you cannot obtain? ■■ Do you have suggestions/ideas for how to fill those gaps? ■■ How long have you been conducting CER and/or PCOR? ■■ Roughly how many CER or PCOR studies have you undertaken? ■■ What are the primary topical areas of your research? ■■ Which analytic methods have you used to conduct CER? ■■ What approaches have you/do you take to engage patients in your research? ■■ How do you assess if CER findings are credible? ■■ What sources do you rely on to assess whether the CER findings are credible? ■■ What sources do you rely on to assess whether the data and methods used to generate the study are credible? 2. What is your vision for the data infrastructure to support a learning healthcare system? 3. What do you think the proliferation of electronic health data (e.g., in EHRs and registries) is going to mean for research? ■■ Where do you see the biggest opportunities for CER? ■■ What are the greatest challenges for CER? Potential areas to discuss: ■■ Do you perceive there are gaps and/or shortcomings in the current infrastructure or knowledge with regard to: ■■ Analytic methods? ■■ Clinical informatics? ■■ Data governance (data security and privacy)? 4. What is your preference for the way CER evidence is disseminated and translated to your stakeholders/members, among others? 5. Who do you look to/identify as leaders or innovators in this area? Are there specific projects or programs that you think we should look to/review? 6. What types of resources, support, and dialogue are needed to make the most of current investments to build the infrastructure for CER? Potential areas to discuss: ■■ What is your vision for disseminating research findings and information on the new research infrastructure for CER we have described? ■■ Do you think there may be potential challenges for publishing CER findings based on observational data sources? CER: Comparative effectiveness research; EHR: Electronic health record; PCOR: Patient-centered outcomes research.
That’s because it is essential to so many of the decisions that are made at individual patient-care levels and also population and policy levels.” – Nonprofit/policy representative
444
J. Compar. Effect. Res. (2012) 1(5)
Stakeholders have substantial expectations for CER using ECD, especially with respect to the potential for research with ECD to address some limitations of traditional research studies [2]. As
future science group
Stakeholder perspectives on comparative effectiveness research using electronic clinical data
Research article
SE = 0.1 4.9
Patients representing populations of interest?
SE = 0.1
Measure of multiple outcomes of interest to clinicians, decision-makers and community
4.4
Account of stakeholder interests/needs in prioritizing research questions?
4.3
PROs and/or QI measures?
4.3
Interventions tested in multiple settings?
4.3
SE = 0.2
SE = 0.2
SE = 0.2
Peer review?
4.2
Consideration of feasibility of implementation and applicability to clinical settings?
4.1
SE = 0.2
SE = 0.2
Maximum level of internal validity?
SE = 0.2
3.5 1
2
3 Response scale
4
5
Figure 1. Mean values for attributes of comparative effectiveness research as reported by stakeholders. PRO: Patient-reported outcome; QI: Quality improvement; SE: Standard error.
the survey results demonstrate, the mean value of interviewees’ responses was 4.1 or higher across nearly all of the research domains/dimensions, signifying respondents found each of the domains ‘very’ important. In the interviews, many participants commented on the need to accelerate the research cycle so results are timelier and the research process is, ideally, less expensive. The desire to improve representation of vulnerable subpopulations in research was mentioned frequently. With respect to improving representation, there is concordance between stakeholders’ comments and survey findings. The desire for CER with clinical data to be highly representative of populations is the most highly ranked attribute (mean = 4.9; standard deviation = 0.3). As described by an industry representative “I think EHRs can give us an understanding of the real world that we can’t get today.” Another nonprofit/policy representative noted: “I see the potential for much greater capabilities to do research on populations of interest … and the potential to research certain rarer conditions.” A government representative articulated his
future science group
concept of a best-case scenario in which “…the entire medical community is using electronic information tools. Everybody has adopted, everyone is on board. Many, if not most, clinical communities will have registries – sort of the town hall, the community waterholes where you can identify your greatest challenges and get insight from community leaders on the best ways to meet challenges.” ■■ Stakeholders are aware of many challenges related to implementing CER with ECD (e.g., governance; data quality; addressing selection bias)
“The challenge we have is learning how to use it (the data), how to actually harness it, ask questions and actually take the data to the next step. I have to tell you that is a big deal. That is a bigger deal than people anticipated.” – Healthcare delivery representative Many stakeholders raised the challenges of governing data and ensuring data quality. Both the real and perceived methodological challenges of using observational data were also addressed, particularly those related to selection bias.
www.futuremedicine.com
445
Research article
Holve, Hamilton Lopez, Scott & Segal
Governing creation, access & use of ECD for research
The term ‘governance’ is used by the EDM Forum to reflect a set of concepts including research network partnerships, data sharing, privacy, security and data access [14]. However, most stakeholders did not directly refer to ‘governance.’ Instead, stakeholders mentioned ‘privacy’ or ‘security’ concerns, and strategies to manage research networks. Many participants discussed the tension between maintaining the highest degree of security and privacy while conducting research, given the complexities of collecting, storing and transferring personal health data across sites and systems. While the necessity of formal arrangements for research data use agreement was discussed, the time and cost of these efforts was mentioned as a major challenge. The role of institutional review boards (IRBs) and the Health Insurance Portability and Accountability Act Privacy rule in multisite research using ECD was raised by many as an area needing more clarity given that IRBs often establish the rules by which institutional datasharing partnerships function. One researcher emphasized the need to clarify the specific role IRBs should play in the oversight of this process, commenting, “The whole thing is going to be dead on its face if we can’t sort out these IRB issues.” Recommendations to improve governance included centralized or state IRBs, and the development of formal guidance for IRBs on multisite research. Some also expressed a desire to revisit or reframe the HIPAA privacy rule. Assessing & managing ECD data quality
Several methodological concerns were mentioned, most of which stem from the need to enhance existing methods and develop new approaches to evaluate the electronic data that is available. While stakeholders see the availability of more ECD as a significant contribution to research, several expressed concern that access to more data alone is insufficient. One researcher noted “Often when we have so much data – medical records data – people think, ‘Oh, the problem is solved. There is tons of information.’ Not so.” In addition, many practical challenges to using EHR data were enumerated. Chief among these is the issue of semantic interoperability –the ability to match nonstandardized terms used by different institutions to represent common medical conditions, treatments and processes [15]. According to one industry representative: “Individual
446
J. Compar. Effect. Res. (2012) 1(5)
organizations might have found a way to mine their own data, but bringing it across organizations is a nightmare for a zillion reasons … the heart rate might be called pulse; it might be called 29 different things.” As a result, there is a strong desire for developing new methods to assess data quality, including strategies to address incorrect or missing data due to collection error or inaccessible data due to limitations of the EHR. Stakeholders expressed that the issue of missing data is exacerbated because systems and informatics tools at many institutions are designed to facilitate billing rather than to address CER questions. One government representative expressed concern that for ECD a lot of the missing data is “either buried in text or is not in the chart at all.” Analyzing ‘big health data’ & controlling for selection bias
Some stakeholders identified the need to think critically and synthesize volumes of data in innovative ways. One researcher commented: “None of these are statistical problems. They are analytical thinking challenges.” Another researcher noted that, “The heavy lifting is taking streams of data and coming up with a higher-level syntheses … the challenge is not the number of bits or the volume of information, it is that this is informed by expertise about health and medicine and physiology.” Stakeholders also commented on the challenge of controlling for confounding variables when using observational data. One researcher said that “If you get the methods wrong, all the wonderful studies will be limited. In methods, the big umbrella is confounding control. How can we make sure the treatment groups we are comparing are balanced in terms of the baseline risk?” Some strategies to address confounding and selection bias were mentioned, including comparing efforts to standardize adjustment using a minimal level of confounding and comparing these results to analyses in which confounding adjustments are maximized based on available evidence. Others proposed conducting sensitivity analyses in different settings or populations, or using electronic clinical systems to conduct experimental research by randomizing patients at the point of care. ■■ Defining & attaining patient-centeredness in CER with ECD
“With the advent of health information technology, there are many opportunities for more active
future science group
Stakeholder perspectives on comparative effectiveness research using electronic clinical data
engagement [of patients] … research doesn’t have to be something done to patients. It can be done with patients. Patients don’t just supply information but also can be involved in designing the question.” – Patient/consumer representative Although not asked specifically about how CER using ECD would impact patients and their potential role in research, stakeholders from all seven stakeholder groups consistently raised the topic. Many stakeholders ascribed value to conducting CER with ECD based on patient values and outcomes because, as one healthcare delivery representative said, “CER is for patients.” Stakeholders used the terms ‘patient engagement’ and patient-centeredness interchangeably, which may be due to the fact that this field and these concepts are still emerging. While a formal definition of PCOR has recently been released by the Patients Centered Outcomes Research Institute [107] an accepted definition of PCOR was not available at the time the interviews were conducted. Comments focused on the need for including patients’ values and perspectives when developing infrastructure (e.g., EHRs) and research (e.g., defining priorities and desired outcomes). However, an emerging theme from the interviews is that stakeholders continue to struggle to define and attain the goal of patient-centeredness. Some stakeholders questioned which patients should be engaged, at what time and in what capacity. One patient/consumer representative recommended that patients and their representatives be “involved in every single aspect [of CER] from the beginning to the usage of the information,” while an industry representative stated that she was “on the fence about the role of the patient.” Neither support for patient involvement nor uncertainty about their most effective roles was limited to specific stakeholder segments. For example, involving patients in collecting and curating data was suggested by a few participants, including an industry representative who commented, “There should be feedback loops that allow patients to update and correct the data. That’s the data set we should be working with. Otherwise, we are spending a lot of time cleaning up noisy data.” ■■ Improving translation & dissemination of CER & how research can be ‘useful’ at the point of care, can help mitigate negative perception of the CER ‘brand’
“Communication around CER must build public support, dispel fears and, while doing so, manage
future science group
Research article
expectations in a way that still garners support for public and private investment over the decades required. Communication management might be the number one challenge because it is so vital to success and so complicated to do right.” – Nonprofit/policy representative Finding new and more efficient approaches to effectively translate and disseminate CER and its findings was a prominent theme. Several participants explicitly mentioned concern that the public might resist the ‘CER brand,’ especially where CER is perceived as a method for limiting patients’ healthcare choices. One patient/ consumer representative explained: “You may format the best announcement [of CER], but if people get the wrong impression you get an F in reality … don’t call it CER. Off the top of my head it is just effectiveness information…” As described by stakeholders, CER should be disseminated so that research findings inform healthcare decision-making at the point of care. As articulated by one industry representative, “This information needs to get into decisionmaking by many people: by patients, doctors, payers.” Multiple stakeholders, including a patient/ consumer representative, suggested presenting CER findings in the context of decision-making: “Make sure you say, ‘Here are results’. We are going to outline three or four concrete ways in which these data are relevant to you and what you can do now that you have this information.’’ The current heavy reliance on peer-reviewed literature for disseminating research findings was viewed as a problem. While valued for its rigor, many commented that the peer-review process can restrict the impact of CER on patient outcomes because of the lengthy time required for review and publication, and because of the elite and limited readership. One researcher stated, “Only academics read academic journals.” This concern is supported by results from the survey, in which ‘peer review’ was the one of the lower ranked attributes of CER using ECD (mean = 4.2; SE = 0.2); though there were divergent views on the extent to which peer review was a strong indicator of quality research. Several alternatives to traditional peer-reviewed publications were proposed, including social media platforms and communities, targeting messages and presentations for a number of audiences, and asking external stakeholders, such as consumer advocacy organizations, to disseminate findings. These opportunities are enhanced with the opportunity to draw on more timely
www.futuremedicine.com
447
Research article
Holve, Hamilton Lopez, Scott & Segal
data-collection efforts and clinical decision support strategies that can accelerate the uptake of findings at the point of care. Stakeholders suggest that collaboration between stakeholders may also alleviate dissemination challenges because the participants in the research would be key end-users of the results and invested in the findings. As one researcher said, “When knowledge is generated with a community organized around a common purpose, a learning collaborative, the knowledge that is generated is highly desired. In a learning collaborative, that (desire) becomes the dissemination channel.”
Some stakeholders articulated the need for a substantial ‘culture shift’: “The way I can see to solve this [lack of data standardization and lack of specificity in the data] is if you prospectively partner with the right people at the right time in strategic engagements, coupled with having the technological infrastructure” said an industry representative. A researcher commented, “You can’t force it [collaboration]. What you can do is highlight the value of collaboration … Investigators are the first ones, once they realize the added value of collaboration, to jump on the opportunity.” Discussion
■■ The need for a substantial ‘culture shift’ to facilitate collaborative science
“For me, it is really about strategic partnerships. It is a multipronged approach. It is leveraging existing infrastructure. It is recognizing value of strategic partnerships. It is nurturing them. There are people who don’t typically partner. It is understanding what the word partner means.” – Industry representative Many of the stakeholders note that conducting CER using ECD can foster collaboration and new partnerships among researchers, clinicians, other disciplines (e.g., informatics) and research funders. One researcher commented that complex problems require an “intensely transdisciplinary” approach to research, while an industry representative mentioned that partnerships across stakeholders bring value: “There is a space here for shared learning. We can learn from and with each other. I know there are things here that we are totally missing, and we don’t even know it.” Yet, some emphasized the unique practices, goals, standards of quality and entrenched business interests among stakeholder groups, which can create barriers to working together. As described by stakeholders, the research environment is entrenched by disciplines, with each field having its own methods, data and expertise. This ‘silo’ mentality was mentioned by researchers and policymakers as one set of professional disincentives to data sharing since – in the research community – access to unique data is often critical to securing publication and professional acknowledgment, including promotion and tenure. There is also intrinsic value in the data, as described by one industry representative: “People make millions of millions of dollars off their data. I don’t see them giving it up. Under no circumstances do I see that.”
448
J. Compar. Effect. Res. (2012) 1(5)
The findings from the EDM Forum stakeholder interviews underscore the time pressure that CER–American Recovery and Reinvestment Act investigators face to respond to external pressures and expectations among stakeholders. While stakeholders are aware of the complexity of CER and the need to develop robust approaches to address governance, informatics and analytic methods to use ECD for CER, they nonetheless express the need to capitalize on current investments and demonstrate the utility and value of CER now. There is an explicitly stated concern that the slower timeframe and disciplinary silos in which biomedical science traditionally operates and is disseminated will result in a lost opportunity to make a substantial impact on health and healthcare delivery. Stakeholders uniformly expect CER to be rigorous and high quality, but the standards by which quality is judged, such as peer review, are not always viewed as critical. Maximizing the highest level of internal validity (as in randomized control trials), for example, is outweighed by the need for representative and timely research. Perhaps as a result, unreasonably high expectations for the research were a concern expressed by some. As articulated by one government representative, “I think there are grand dreams that I fear have been built up to the point where they are just bound to be disappointed. ECD has been mentioned as a panacea to almost any data problem, whether it’s drug safety or public health surveillance … it is not unreasonable to say that research will be easier with electronic medical records, but the emphasis is on easier, not easy.” Balancing these perspectives is viewed as an important dimension of CER, and portends the enormous responsibility facing the research community, policymakers and engaged stakeholders
future science group
Stakeholder perspectives on comparative effectiveness research using electronic clinical data
charged with generating the best evidence to improve health and healthcare. Research limitations
CER using ECD is a nascent field of inquiry, which requires exploratory research in order to identify key perspectives and synthesize priorities for the field. As with any exploratory, qualitative study, there are limitations to the generalizability of our findings. First, the number of individuals working and writing on CER using ECD is relatively limited and there is no clear sampling frame for CER researchers. As a result, it is possible we missed key perspectives or divergent viewpoints. Second, despite an initial hypothesis that the quantitative portion of the mixed-methods analysis would yield distinct clusters of opinions regarding the relative value of CER attributes – which we further hypothesized might cluster by stakeholder group – this result did not hold true and most stakeholders ranked every attribute highly. This finding may indicate that, despite our efforts to engage divergent points of view, the available universe of individuals with the level of expertise we sought to discuss the (inherently technical) topic of CER using ECD was more homogeneous than anticipated, even for a new field of inquiry. Both limitations suggest that greatly expanding the number of participants and perspectives in the future would be valuable.
Research article
Future perspective
Given the relatively early stage of the PROSPECT, DRN and Enhanced Registry projects at which the stakeholder needs assessment was conducted, and the purpose for which the needs assessment was conducted – as an exploratory effort to understand perspectives on the conduct of CER with ECD to drive the EDM Forum’s activities – the manuscript does not propose policy recommendations beyond the scope of the EDM Forum, although this effort may be part of future work. Input from the stakeholder needs assessment has informed the Forum’s decisions regarding priority topics for commissioned papers, projects and issue briefs such as governance [16,17], patient engagement in research [18,19] and data quality and methods [20]. Moving forward, the findings from the stakeholder needs assessment will continue to guide the topics on which the EDM Forum engages stakeholders in active dialogue. This includes input into the selection of convening topics and research activities the Forum undertakes, as well as awareness of key issues where it is critical to translate and disseminate infrastructure efforts and research generated by the PROSPECT, DRN and Enhanced registry studies. The EDM Forum Stakeholder needs assessment demonstrates there is an emerging research agenda for CER using ECD that is extremely important to a broad set of community members. Understanding and improving patient
Executive summary Stakeholders have substantial expectations for comparative effectiveness research (CER) using electronic clinical data (ECD), both with respect to addressing the limitations of traditional research studies, and generating meaningful evidence for decisionmaking and improving patient outcomes. ■■ Stakeholders are aware of many challenges related to implementing CER with ECD, including the need to develop appropriate governance, assess and manage data quality, and develop methods to address confounding in observational data. ■■ Stakeholders continue to struggle to define ‘patient centeredness’ in CER using ECD, adding complexity to attaining this goal. ■■ Stakeholders express that improving translation and dissemination of CER, and how research can be ‘useful’ at the point of care, can help mitigate negative perceptions of the CER ‘brand.’ ■■ Stakeholders perceive a need for a substantial ‘culture shift’ to facilitate collaborative science and new ways of conducting biomedical and outcomes research. ■■ The findings from the Electronic Data Methods (EDM) Forum stakeholder interviews underscore the time pressure that CER American Recovery and Reinvestment Act investigators face to respond to external pressures and expectations among stakeholders. ■■ There is an explicity stated concern that the slower timeframe and disciplinary silos in which biomedical science traditionally operates and is disseminated will result in a lost opportunity to make a substantial impact on health and healthcare delivery. ■■ Stakeholders express the need to capitalize on current investments and demonstrate the utility and value of CER now. ■■ Input from the stakeholder needs assessment has informed the EDM Forum’s decisions regarding priority topics for commissioned papers, projects, and issue briefs such as governance, patient engagement in research, and data quality and methods. Moving forward, the findings from the stakeholder needs assessment will continue to guide the topics on which the EDM Forum engages stakeholders in active dialogue, as well as dissemination activities. ■■
future science group
www.futuremedicine.com
449
Research article
Holve, Hamilton Lopez, Scott & Segal
engagement in clinical research, addressing legal and technical barriers related to multisite electronic data sharing and improving the quality of data flowing from the EHR are worthwhile investments that stakeholders perceive will contribute to advancing the science of CER to improve patient outcomes. Acknowledgements The authors would like to acknowledge A Rein for her contribution to the construction of the study and data collection and T Trudnak for her guidance and review. Special thanks are given to the 50 stakeholders that participated in the interviews.
Financial & competing interests disclosure AcademyHealth acknowledges the Agency for Healthcare Research and Quality for its support of this work through
n
1
n
2
n
3
450
n
Randhawa GS, Slutsky JR. Building sustainable multi-functional prospective electronic clinical data systems. Med. Care 50(7 Suppl. 1), S3–S6 (2012). Randhawa provides background information about the Agency for Healthcare Research and Quality-supported projects to build flexible prospective clinical electronic data infrastructure that meet the needs of diverse users and the Electronic Data Methods (EDM) Forum to advance the methods in clinical informatics, research analytics and governance by actively engaging investigators from the American Recovery and Reinvestment Act-funded projects as well as external stakeholders. Holve E, Segal C, Hamilton Lopez M, Rein A, Johnson BH. The Electronic Data Methods (EDM) Forum for comparative effectiveness research (CER). Med. Care 50(7 Suppl. 1), S7–S10 (2012). Holve et al. provide detailed information about the creation and function of the AcademyHealth EDM Forum to collect, synthesize and share lessons learned from eleven research projects that are building infrastructure and using electronic clinical data for comparative effectiveness research (CER) and patient-centered outcomes research (PCOR). Holve E, Segal C, Hamilton Lopez M. Opportunities and challenges for comparative effectiveness research (CER) with electronic clinical data: a perspective from the EDM
Ethical conduct of research The authors state that they have obtained appropriate insti tutional review board approval or have followed the princi ples outlined in the Declaration of Helsinki for all human or animal experimental investigations. In addition, for investigations involving human subjects, informed consent has been obtained from the participants involved.
Forum. Med. Care 50(7 Suppl. 1), S11–S18 (2012).
References Papers of special note have been highlighted as: of interest
the American Recovery & Reinvestment Act of 2009, Grant U13 HS19564-01 (Electronic Data Methods Forum for Comparative Effectiveness Research). The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed. No writing assistance was utilized in the production of this manuscript.
4
5
n
6
Provides an overview of the challenges faced by research teams as part of their efforts to develop electronic clinical data (ECD) infrastructure to support CER. The findings reflect a set of opportunities for transdisciplinary learning, and will ideally enhance the transparency and generalizability of CER using ECD. Deverka PA, Lavallee DC, Desai PJ et al. Stakeholder participation in comparative effectiveness research: defining a framework for effective engagement. J. Compar. Effect. Res. 1(2), 181–194 (2012). Navathe AS, Clancy C, Glied S. Advancing research data infrastructure for patient -centered outcomes research. JAMA 306(11), 1254–1255 (2011). The authors reflect on the expectation that current American Recovery and Reinvestment Act investments building the infrastructure for CER, coupled with technological advances, can address some of the data and methodological issues in outcomes research (a prominent theme in the manuscript). The Learning Healthcare System: Workshop Summary (IOM Roundtable on Evidence-Based Medicine). Olsen LA, Aisner D, McGinnis JM (Eds). National Academies Press, Washington, DC, USA (2007).
7
Etheredge LM. A rapid-learning health system. Health Aff. 26(2), 107–118 (2007).
8
Friedman CP, Wong AK, Blumenthal D. Achieving a nationwide learning health system. Sci. Transl. Med. 2(57), 29–31 (2010).
J. Compar. Effect. Res. (2012) 1(5)
9
Abernethy AP, Etheredge LM, Ganz PA et al. Rapid-learning system for cancer care. J. Clin. Oncol. 28(27), 4268–4274 (2010).
10 Gluck M. Early Glimpses of the Learning
Health Care System: The Potential Impact of Health IT. Health IT for Actionable Knowledge report. AcademyHealth, Washington, DC, USA (2012). 11 Holve E, Pittman P. A First Look at the
Volume and Cost of Comparative Effectiveness Research in the United States. AcademyHealth, Washington, DC, USA (2009). 12 Thorpe KE, Zwarenstein M, Oxman AD
et al. A pragmatic explanatory continuum indicator summary (PRECIS): a tool to help trial designers. CMAJ 180(10), E47–E57 (2009). 13 Ryan GW, Bernard HR. Techniques to
identify themes. Field Methods 15(1), 85–109 (2003). 14 Rosenbaum S. Data governance and
stewardship: designing data stewardship entities and advancing data access. Health Serv. Res. 45(5), 1442–1455 (2010). n
Rosenbaum writes about the relationship between US health policy and health information, especially related to data collection and sharing, and provides a working definition of ‘data governance.’
15 Blobel B, Oemig F, Dössel O. What is
needed to finally achieve semantic interoperability? In: World Congress on Medical Physics and Biomedical Engineering. Schlegel WC (Ed.). Springer BerlinHeidelberg, Munich, Germany, 411–414 (2009).
future science group
Stakeholder perspectives on comparative effectiveness research using electronic clinical data
16 Marsolo K. Medical care: approaches to
20 Kahn MG, Raebel MA, Glanz JM,
facilitate institutional review board approval of multicenter research studies. Med. Care 50 (7 Suppl. 1), S77–S81 (2012). n
Marsolo examines the IRB review process for multicenter studies (one of the topics discussed in-depth in the manuscript) and provides strategies used to streamline the IRB review process examples utilized by two existing multicenter CER networks.
17 Sabharwal R, Holve E, Rein A, Segal C.
Approaches to Using Protected Health Information (PHI) for Patient-Centered Outcomes Research (PCOR): Regulatory Requirements, De-identification Strategies, and Policy. EDM Forum, AcademyHealth, Washington, DC, USA (2012). 18 Hamilton Lopez M, Holve E, Rein A,
Winkler J. Involving Patients and Consumers in Research: New Opportunities for Meaningful Engagement in Research and Quality Improvement. EDM Forum, AcademyHealth, Washington, DC, USA (2012). 19 Rein A, Hamilton Lopez M, Winkler J,
Holve E. Involving Patients and Consumers in Research: New Opportunities for Meaningful Engagement in Research and Quality Improvement. EDM Forum, AcademyHealth, Washington, DC, USA (2012) (In Press).
future science group
Riedlinger K, Steiner JF. A pragmatic framework for single-site and multisite data quality assessment in electronic health record-based clinical research. Med. Care. 50 (7 Suppl. 1), S21–S29 (2012). n
The authors examine the issue of multisite data-quality assessment (one of the topics discussed in-depth in the manuscript) and proposes a ‘fit-for-use’ conceptual model for data quality assessment and a process model for planning and conducting single-site and multisite data-quality assessments.
■■ Websites 101 US Department of Health and Human
Services. Comparative Effectiveness Research Funding (HHS Recovery Act). Washington, DC, USA (2010). www.hhs.gov/recovery/programs/cer/index. html (Accessed 17 March 2011) 102 Agency for Healthcare Research and Quality.
American Recovery and Reinvestment Act Investments in Comparative Effectiveness Research for Data Infrastructure. AHRQ, Rockville, MD (2010). www.ahrq.gov/fund/cerfactsheets/osfsinfra. htm (Accessed 19 June 2012)
www.futuremedicine.com
Research article
103 Institute of Medicine. Learning what works
best: the nations need for evidence on comparative effectiveness in health care (2007). www.iom.edu/ebm-effectiveness (Accessed 2 August 2012) n
Articulates the need for a ‘learning healthcare system’ a topic discussed in-depth in the manuscript.
104 Federal Coordinating Council for Comparative
Effectiveness Research. Report to the President and The Congress. US Department of Health and Human Services, NC, USA (2009). www.hhs.gov/recovery/programs/cer/ cerannualrpt.pdf 105 The Electronic Data Methods Forum Steering
Committee (2011). www.edmforum.org/publicgrant/About/ SteeringCommittee (Accessed 5 June 2012). 106 Lombard M, Snyder-Duch J, Campanella
Bracken C. Practical resources for assessing and reporting intercoder reliability in content analysis research projects (2010). http://matthewlombard.com/reliability (Accessed 8 February 2012) 107 Patient Centered Outcomes Research Institute.
Patient-centered outcomes research. www.pcori.org/patient-centered-outcomesresearch (Accessed 6 August 2012)
451
