SPECIAL FOCUS y The ARRA investment in CER Commentary For reprint orders, please contact: [email protected]
The importance of collaboration in comparative effectiveness research “Collaboration across institutions and disciplines is widely regarded to be essential to the creation and conduct of comparative effectiveness research projects.” Keywords: collaboration • data infrastructure • institutional review board • multidisciplinary • teamwork
Collaboration has been viewed as crucial to the production of the type of comparative effectiveness research (CER) needed to inform real-world decisions by patients, providers and health systems [1] . Collaborations across institutions and disciplines may allow for sharing and further development of new research methods and a focus on comparisons of the intervention and outcome measures that matter most to typical decision makers. Collaborative research in CER may also facilitate the inclusion of patients, clinicians and delivery settings that are representative of the populations and locations where the CER findings will ultimately be applied in point-of-care decisions. The evaluation of the American Recovery and Reinvestment Act (ARRA) CER portfolio created an opportunity to learn more about types of collaborative relationships in CER, the potential benefits of collaboration and factors that facilitate or hinder progress on CER projects. Types of collaboration in CER projects funded by ARRA Among the 424 projects in the ARRA CER portfolio, 60.4% included investigators from multiple institutions. Some types of projects were more likely to include multiple institutions: projects that primarily focused on generating or synthesizing research evidence were most likely to have a multi-institutional team (71.7%), followed by projects that primarily focused on creating data infrastructure (58.8%). Career development and training projects were least likely to include
10.2217/CER.14.58 © 2014 Future Medicine Ltd
multiple institutions, with 40.7 and 31.3%, respectively. In order to better understand the multidisciplinary nature of CER, we looked at the makeup of teams for 80 multi-institutional projects and 48 single-institution projects. All projects had a primary focus on generating new research evidence and responded to a survey sent by the evaluation team. The most common types of expertise that both multiple- and single-institution projects required were the same: biostatistics, clinical practice and health services research. A larger share of teams conducting multi- institutional projects required expertise in health services research, economics or econometrics, and survey methodology or psycho metrics than did projects at a single institution. For example, 77.5% of multi-institutional teams required expertise in health services research and 57.5% required expertise in economics or econometrics, compared with 58.3 and 43.8%, respectively, for projects at a single institution. Single-institution projects were more likely to require expertise in quality improvement research than were multiinstitutional projects (29.2 vs 21.3% of multi-institutional projects).
Laura D Kimmey Author for correspondence: Mathematica Policy Research, 1100 1st Street NE, 12th Floor, Washington, DC 20002, USA Tel.: +1 202 484 9220 Fax: +1 202 863 1763 lkimmey@ mathematica-mpr.com
Eugene Rich Mathematica Policy Research, 1100 1st Street NE, 12th Floor, Washington, DC 20002, USA
Benefits of collaboration in CER projects funded by ARRA During discussions conducted as part of an evaluation of the ARRA CER portfolio, several investigators and project officers identified successful collaboration as a facilitator of project achievement [2] . In some cases, collaboration was reported to be essential
J. Comp. Eff. Res. (2014) 3(6), 577–579
part of
ISSN 2042-6305
577
Commentary Rich & Kimmey to creating the project; as one investigator stated, “[ARRA-funded data infrastructure projects] help build infrastructure that would be very hard for any one institution to be able to embark on alone … They are changing the way we do business. It’s a collaborative project that only works if several institutions sign up to it.”
potential for further collaboration among the study team.
Facilitators of collaboration in CER projects funded by ARRA Our evaluation offered the opportunity to explore several factors that may facilitate collaboration in CER. The information here comes largely from interviews ...respondents who worked on the observational with investigators from the RCT and observational studies mentioned previously as well as comments study team suggested that their project might made during a technical expert panel (TEP) consistencourage other health plans to participate in ing of investigators of ARRA CER projects. Prior comparative effectiveness research... relationships, effective communication and decision In other cases, collaboration was recognized as a making based on a participatory process were idenfacilitator of project progress to a successful conclu- tified by many people as facilitators of multi-institusion. During interviews focused on collaboration in tional collaboration. Multiple interview respondents CER conducted with investigators from two ARRA and TEP members also noted that understanding difCER projects, respondents reported both short-term ferent perspectives facilitated collaboration on their and potential longer-term benefits from collaboration ARRA-funded CER projects. For example, in the on their CER projects. Two academic institutions con- observational study involving an academic institution ducting a randomized controlled trial (RCT) reported and two healthcare payers, respondents reported that that collaboration allowed enrollment of a more diverse understanding differences between organizations was group of patients than would have been possible with an important part of understanding each collaboraonly one institution. Respondents from that project tor’s ability to contribute to the project. Finally, in also said that collaboration produced improvements interviews as well as TEP discussion, investigators in study design and processes, such as screening crite- highlighted the importance of having an information ria. As one respondent explained, “If you’re doing the system to support entering, storing and sharing data study at only one place, it’s too easy to think that your across institutions and sites. For example, in the RCT way is the only way and hard to think out of the box.” involving two academic institutions, respondents Another respondent from the same RCT reported that reported that real-time data sharing enabled project including the perspectives of different clinical special- staff to view the same patient data, discuss questions ties also helps investigators determine the most appro- and resolve issues without delays resulting from having priate research design: “Different specialties have dif- to transfer data securely. ferent levels of expertise; if you fail to include an area, you aren’t taking into account potential successes and Barriers to collaboration in CER projects failures. It’s important to have everyone at the table so funded by ARRA you can look at outcomes properly and find the most Investigators of ARRA CER projects who participated effective approach.” in interviews or in the TEP discussion of collaboraThe benefits of collaboration on CER extended tion in CER identified several barriers to collaboration. beyond the traditional RCT setting. The second Several people reported that the institutional review project that we examined was an observational study board (IRB) process has the potential to slow progress. that included an academic institution and two health One person said that his ARRA-funded CER project plans. A health plan respondent reported that the had to be approved by four IRBs and that there was collaboration among the three organizations and the substantial variation in the IRBs’ interpretation of the resulting team mindset served as a stimulus to con- study protocol. Another reported a delay in obtaining tinue work at times when the respondent’s health plan approval because the IRB at her institution had little was not making as much progress with the interven- experience reviewing CER studies. Several investigation as it would have liked. Some respondents who tors reported facing challenges with data access and worked on the observational study team suggested sharing, such as electronic health record platforms that that their project might encourage other health plans could not communicate with one another. to participate in CER and other academicians to conJust as understanding diverse perspectives was idensider working with health plans. Respondents from tified as a facilitator of collaboration, multiple people both the trial and observational study cited possible reported that different priorities among collaborators longer-term benefits of their collaborations such as the sometimes hindered progress on their project. One
“
”
578
J. Comp. Eff. Res. (2014) 3(6)
future science group
The importance of collaboration in comparative effectiveness research
respondent said that the researchers and non-researchers on his project team operated at dissimilar paces. In another case, an investigator reported that one of the collaborators on her project slowed progress by trying to retain ownership of parts of the study. Finally, several people said that lack of experience with federally funded research among non-research institutions posed a challenge to their CER projects. For example, one person stated that the community providers working with their research team did not understand how to obtain informed consent.
or CER funders could support project teams spending time building and strengthening relationships prior to and during the course of collaborative projects. Further exploration of factors that facilitate or hinder progress on CER projects and how to enhance or minimize those factors is important if the field is to continue to produce evidence that contributes to decision making by diverse patients, providers, health systems and payers. Elsewhere in this issue, we discuss one potential method for studying the development of collaborations in CER, social network analysis [3] .
Conclusion The evaluation of this portfolio of ARRA CER investments allowed us to describe the collaborative relationships developed by many different project directors during the initiation of a diverse array of CER projects. It also provided the opportunity to explore further the potential benefits of collaboration and the factors that may facilitate or hinder progress on CER projects. Collaboration across institutions and disciplines is widely regarded to be essential to the creation and conduct of CER projects. There may be ways that project teams, the research community, the federal government and other entities can facilitate more collaboration in CER or address common barriers. For example, the federal government could provide additional guidance for IRBs in interpreting regulations on patient privacy,
Disclaimer The views in the article are those of the authors and do not necessarily reflect the views of the Department of Health and Human Services.
Financial & competing interests disclosure The work was funded by the Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation (ASPE), under contract with Mathematica Policy Research (Contract HHSP23320095642WC/HHSP23337014T). The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed .No writing assistance was utilized in the production of this manuscript.
References 1
2
Federal Coordinating Council for Comparative Effectiveness Research. Report to the President and the Congress. U.S. Department of Health and Human Services, DC, USA (2009). Esposito D, Geonnotti K, Peterson S et al. Research Evaluation and Impact Assessment of ARRA Comparative Effectiveness
future science group
Commentary
Research Portfolio: Interim Report for the Project Specific Analysis. Mathematica Policy Research, DC, USA (2013). 3
Zickafoose JS, Kimmey LD, Esposito DE, Rich ER. Evaluating collaborations in CER: the opportunities and challenges for social network analysis. J. Comp. Eff. Res. 3(6), 667–675 (2014).
www.futuremedicine.com
579
The importance of collaboration in comparative effectiveness research “Collaboration across institutions and disciplines is widely regarded to be essential to the creation and conduct of comparative effectiveness research projects.” Keywords: collaboration • data infrastructure • institutional review board • multidisciplinary • teamwork
Collaboration has been viewed as crucial to the production of the type of comparative effectiveness research (CER) needed to inform real-world decisions by patients, providers and health systems [1] . Collaborations across institutions and disciplines may allow for sharing and further development of new research methods and a focus on comparisons of the intervention and outcome measures that matter most to typical decision makers. Collaborative research in CER may also facilitate the inclusion of patients, clinicians and delivery settings that are representative of the populations and locations where the CER findings will ultimately be applied in point-of-care decisions. The evaluation of the American Recovery and Reinvestment Act (ARRA) CER portfolio created an opportunity to learn more about types of collaborative relationships in CER, the potential benefits of collaboration and factors that facilitate or hinder progress on CER projects. Types of collaboration in CER projects funded by ARRA Among the 424 projects in the ARRA CER portfolio, 60.4% included investigators from multiple institutions. Some types of projects were more likely to include multiple institutions: projects that primarily focused on generating or synthesizing research evidence were most likely to have a multi-institutional team (71.7%), followed by projects that primarily focused on creating data infrastructure (58.8%). Career development and training projects were least likely to include
10.2217/CER.14.58 © 2014 Future Medicine Ltd
multiple institutions, with 40.7 and 31.3%, respectively. In order to better understand the multidisciplinary nature of CER, we looked at the makeup of teams for 80 multi-institutional projects and 48 single-institution projects. All projects had a primary focus on generating new research evidence and responded to a survey sent by the evaluation team. The most common types of expertise that both multiple- and single-institution projects required were the same: biostatistics, clinical practice and health services research. A larger share of teams conducting multi- institutional projects required expertise in health services research, economics or econometrics, and survey methodology or psycho metrics than did projects at a single institution. For example, 77.5% of multi-institutional teams required expertise in health services research and 57.5% required expertise in economics or econometrics, compared with 58.3 and 43.8%, respectively, for projects at a single institution. Single-institution projects were more likely to require expertise in quality improvement research than were multiinstitutional projects (29.2 vs 21.3% of multi-institutional projects).
Laura D Kimmey Author for correspondence: Mathematica Policy Research, 1100 1st Street NE, 12th Floor, Washington, DC 20002, USA Tel.: +1 202 484 9220 Fax: +1 202 863 1763 lkimmey@ mathematica-mpr.com
Eugene Rich Mathematica Policy Research, 1100 1st Street NE, 12th Floor, Washington, DC 20002, USA
Benefits of collaboration in CER projects funded by ARRA During discussions conducted as part of an evaluation of the ARRA CER portfolio, several investigators and project officers identified successful collaboration as a facilitator of project achievement [2] . In some cases, collaboration was reported to be essential
J. Comp. Eff. Res. (2014) 3(6), 577–579
part of
ISSN 2042-6305
577
Commentary Rich & Kimmey to creating the project; as one investigator stated, “[ARRA-funded data infrastructure projects] help build infrastructure that would be very hard for any one institution to be able to embark on alone … They are changing the way we do business. It’s a collaborative project that only works if several institutions sign up to it.”
potential for further collaboration among the study team.
Facilitators of collaboration in CER projects funded by ARRA Our evaluation offered the opportunity to explore several factors that may facilitate collaboration in CER. The information here comes largely from interviews ...respondents who worked on the observational with investigators from the RCT and observational studies mentioned previously as well as comments study team suggested that their project might made during a technical expert panel (TEP) consistencourage other health plans to participate in ing of investigators of ARRA CER projects. Prior comparative effectiveness research... relationships, effective communication and decision In other cases, collaboration was recognized as a making based on a participatory process were idenfacilitator of project progress to a successful conclu- tified by many people as facilitators of multi-institusion. During interviews focused on collaboration in tional collaboration. Multiple interview respondents CER conducted with investigators from two ARRA and TEP members also noted that understanding difCER projects, respondents reported both short-term ferent perspectives facilitated collaboration on their and potential longer-term benefits from collaboration ARRA-funded CER projects. For example, in the on their CER projects. Two academic institutions con- observational study involving an academic institution ducting a randomized controlled trial (RCT) reported and two healthcare payers, respondents reported that that collaboration allowed enrollment of a more diverse understanding differences between organizations was group of patients than would have been possible with an important part of understanding each collaboraonly one institution. Respondents from that project tor’s ability to contribute to the project. Finally, in also said that collaboration produced improvements interviews as well as TEP discussion, investigators in study design and processes, such as screening crite- highlighted the importance of having an information ria. As one respondent explained, “If you’re doing the system to support entering, storing and sharing data study at only one place, it’s too easy to think that your across institutions and sites. For example, in the RCT way is the only way and hard to think out of the box.” involving two academic institutions, respondents Another respondent from the same RCT reported that reported that real-time data sharing enabled project including the perspectives of different clinical special- staff to view the same patient data, discuss questions ties also helps investigators determine the most appro- and resolve issues without delays resulting from having priate research design: “Different specialties have dif- to transfer data securely. ferent levels of expertise; if you fail to include an area, you aren’t taking into account potential successes and Barriers to collaboration in CER projects failures. It’s important to have everyone at the table so funded by ARRA you can look at outcomes properly and find the most Investigators of ARRA CER projects who participated effective approach.” in interviews or in the TEP discussion of collaboraThe benefits of collaboration on CER extended tion in CER identified several barriers to collaboration. beyond the traditional RCT setting. The second Several people reported that the institutional review project that we examined was an observational study board (IRB) process has the potential to slow progress. that included an academic institution and two health One person said that his ARRA-funded CER project plans. A health plan respondent reported that the had to be approved by four IRBs and that there was collaboration among the three organizations and the substantial variation in the IRBs’ interpretation of the resulting team mindset served as a stimulus to con- study protocol. Another reported a delay in obtaining tinue work at times when the respondent’s health plan approval because the IRB at her institution had little was not making as much progress with the interven- experience reviewing CER studies. Several investigation as it would have liked. Some respondents who tors reported facing challenges with data access and worked on the observational study team suggested sharing, such as electronic health record platforms that that their project might encourage other health plans could not communicate with one another. to participate in CER and other academicians to conJust as understanding diverse perspectives was idensider working with health plans. Respondents from tified as a facilitator of collaboration, multiple people both the trial and observational study cited possible reported that different priorities among collaborators longer-term benefits of their collaborations such as the sometimes hindered progress on their project. One
“
”
578
J. Comp. Eff. Res. (2014) 3(6)
future science group
The importance of collaboration in comparative effectiveness research
respondent said that the researchers and non-researchers on his project team operated at dissimilar paces. In another case, an investigator reported that one of the collaborators on her project slowed progress by trying to retain ownership of parts of the study. Finally, several people said that lack of experience with federally funded research among non-research institutions posed a challenge to their CER projects. For example, one person stated that the community providers working with their research team did not understand how to obtain informed consent.
or CER funders could support project teams spending time building and strengthening relationships prior to and during the course of collaborative projects. Further exploration of factors that facilitate or hinder progress on CER projects and how to enhance or minimize those factors is important if the field is to continue to produce evidence that contributes to decision making by diverse patients, providers, health systems and payers. Elsewhere in this issue, we discuss one potential method for studying the development of collaborations in CER, social network analysis [3] .
Conclusion The evaluation of this portfolio of ARRA CER investments allowed us to describe the collaborative relationships developed by many different project directors during the initiation of a diverse array of CER projects. It also provided the opportunity to explore further the potential benefits of collaboration and the factors that may facilitate or hinder progress on CER projects. Collaboration across institutions and disciplines is widely regarded to be essential to the creation and conduct of CER projects. There may be ways that project teams, the research community, the federal government and other entities can facilitate more collaboration in CER or address common barriers. For example, the federal government could provide additional guidance for IRBs in interpreting regulations on patient privacy,
Disclaimer The views in the article are those of the authors and do not necessarily reflect the views of the Department of Health and Human Services.
Financial & competing interests disclosure The work was funded by the Department of Health and Human Services, Office of the Assistant Secretary for Planning and Evaluation (ASPE), under contract with Mathematica Policy Research (Contract HHSP23320095642WC/HHSP23337014T). The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed .No writing assistance was utilized in the production of this manuscript.
References 1
2
Federal Coordinating Council for Comparative Effectiveness Research. Report to the President and the Congress. U.S. Department of Health and Human Services, DC, USA (2009). Esposito D, Geonnotti K, Peterson S et al. Research Evaluation and Impact Assessment of ARRA Comparative Effectiveness
future science group
Commentary
Research Portfolio: Interim Report for the Project Specific Analysis. Mathematica Policy Research, DC, USA (2013). 3
Zickafoose JS, Kimmey LD, Esposito DE, Rich ER. Evaluating collaborations in CER: the opportunities and challenges for social network analysis. J. Comp. Eff. Res. 3(6), 667–675 (2014).
www.futuremedicine.com
579
