Journal of Clinical Epidemiology 67 (2014) 425e432
ORIGINAL ARTICLES
The refinement of topics for systematic reviews: lessons and recommendations from the Effective Health Care Program David I. Buckleya,*, Mohammed T. Ansarib, Mary Butlerc, Clara Sohd, Christine S. Change a Oregon Health & Science University, 3181 SW Sam Jackson Park Road, Mail Code: FM, Portland, OR 97239, USA Clinical Epidemiology Program, Methods Centre, Box 201B, General Campus, The Ottawa Hospital, 501 Smyth Road, Ottawa, ON K1H 8L6, Canada c Division of Health Policy and Management, School of Public Health, University of Minnesota, 420 Delaware Street SE, Minneapolis, MN 55455, USA d Kaiser Permanente Center for Health Research, Evidence-based Practice Center, 3800 N Interstate Ave, Portland, OR 97227, USA e Center for Outcomes and Evidence, Agency for Healthcare Research and Quality, 540 Gaither Road, Rockville, MD 20850, USA b
Accepted 7 October 2013
Abstract Objectives: The Agency for Healthcare Research and Quality (AHRQ) Effective Health Care Program conducts systematic reviews of health-care topics nominated by stakeholders. Topics undergo refinement to ensure relevant questions of appropriate scope and useful reviews. Input from key informants, experts, and a literature scan informs changes in the nominated topic. AHRQ convened a work group to assess approaches and develop recommendations for topic refinement. Study Design and Setting: Work group members experienced in topic refinement generated a list of questions and guiding principles relevant to the refinement process. They discussed each issue and reached agreement on recommendations. Results: Topics should address important health-care questions or dilemmas, consider stakeholder priorities and values, reflect the state of the science, and be consistent with systematic review research methods. Guiding principles of topic refinement are fidelity to the nomination, relevance, research feasibility, responsiveness to stakeholder inputs, reduced investigator bias, transparency, and suitable scope. Suggestions for stakeholder engagement, synthesis of input, and reporting are discussed. Refinement decisions require judgment and balancing guiding principles. Variability in topics precludes a prescriptive approach. Conclusion: Accurate, rigorous, and useful systematic reviews require well-refined topics. These guiding principles and methodological recommendations may help investigators refine topics for reviews. Ó 2014 Elsevier Inc. All rights reserved. Keywords: Systematic review; Evidence-based practice; Stakeholder engagement; Methods; Decision making; Patient-centered care; Research design
1. Introduction
‘‘A prudent question is one-half of wisdom.’’ dFrancis Bacon Systematic reviews aim to improve health outcomes by developing evidence-based information about which interventions are most effective for which patients under
Funding: This project was funded under Contract No. HHSA 290-2007-10057-I from the Agency for Healthcare Research and Quality, US Department of Health and Human Services. The authors of this article are responsible for its content, including any clinical treatment recommendations. No statement in this article should be construed as an official position of AHRQ or of the US Department of Health and Human Services. Conflict of interest: None. * Corresponding author. Tel.: þ1-503-494-8367; fax: þ1-503-4941513. 0895-4356/$ - see front matter Ó 2014 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jclinepi.2013.10.023
specific circumstances [1]. Systematic reviews are used by a variety of organizations to inform clinical guidelines [2], health-care policies [3], and insurance coverage decisions [4]. The Evidence-based Practice Center (EPC) Program, part of the Agency for Healthcare Research and Quality (AHRQ) Effective Health Care (EHC) Program, conducts systematic reviews on health-care topics nominated by stakeholders. Stakeholders may represent patients, consumers, advocacy organizations, clinicians, researchers, agencies that issue guidelines, policy makers, industry, or health-care organizations. Involving stakeholders in the nomination process provides an opportunity for end users of research to participate in asking and answering questions about health care. To provide useful answers, systematic reviews must ask the right questions. Challenges arise when stakeholdernominated topics are not ideally formulated for the broadest public health and/or clinical relevance or are inconsistent with accepted systematic review methods or EPC Program
426
D.I. Buckley et al. / Journal of Clinical Epidemiology 67 (2014) 425e432
2. Methods What is new? This article provides methodological guidance for refining topics for systematic review in Agency for Healthcare Research and Quality Evidencebased Practice Center (EPC) Program. The goal of topic refinement is to produce a topic that addresses important health-care questions and dilemmas, considers the priorities and values of relevant stakeholders, reflects the state of the science, and allows for application of systematic review research methods. The article describes various EPC approaches to the refinement process, with their advantages and disadvantages, and provides guidance for a consistent approach to refinement and the reporting of refinement decisions in light of guiding principles. This can promote transparency, patient centeredness, and more useful and relevant systematic reviews. The guiding principles are fidelity to the original nomination, relevance, research feasibility, responsiveness to stakeholder inputs, reduction of potential investigator bias, transparency, and suitable scope. The nominated topic is refined with information from a literature scan and stakeholder inputs in an iterative manner. Synthesis of input requires judgment of the topic refinement team and consideration of the guiding principles. The investigators may balance certain principles when making decisions about whether and how to include input, especially when they are conflicting. Variability of topics in the EPC program makes it impractical to apply the guiding principles in a prescriptive manner.
principles for conducting reviews [5]. To ensure relevant and useful systematic reviews, topics nominated by stakeholders generally need to be refined. In 2007, investigators with the EPC Program began developing methods for topic refinement. Although the EPC Program stipulates the elements of topic refinement, different EPCs have approached specific aspects in both similar and different ways. This variation provided an excellent opportunity to consider the advantages and disadvantages of different approaches and produce recommendations to guide the topic refinement process. To that end, we convened a work group of investigators experienced in conducting topic refinements from four EPCs and one AHRQ project officer.
Our work group followed previously described principles for developing EPC Program methods guidance [6]. We used a best-practice approach based on our experience, our assessment of topic refinement reports from other EPCs, and input on the draft methods guidance report [7] from investigators across the EPC program. We compiled a prioritized list of issues and concepts that were challenging for many EPCs, incompletely articulated in topic refinement summary reports, variable across EPCs, and/or consistent across EPCs. These items were consolidated into three main categories: (1) the purpose of topic refinement, (2) guiding principles, and (3) the mechanics of conducting a topic refinement. Through a structured process in 18 meetings, we discussed each issue and reached agreement on recommendations. Recognizing the legitimate variability in topics and approaches, we described different viable approaches and discussed their relative merits. Experts in systematic review methods provided a peer review of a draft report, which was also posted for public comment. The methods are described in more detail in the full report [7].
3. Results 3.1. What is refinement? Refinement implies making changes to attain a better fit with a certain standard. In this sense, the goal of topic refinement is to improve a nominated topic so that it is a good and accurate fit with specified criteria (Table 1). Topic refinement is one stage in the process of producing a systematic review through the EPC Program (Fig. 1). It is preceded by topic nomination and development, in which investigators review stakeholder-nominated topics and determine if they meet program selection criteria and should proceed with topic refinement and systematic review [8]. The primary goal of topic refinement is to formulate research questions that can be addressed by a systematic review, not to answer the questions. A refined topic includes three principal elements: (1) clearly articulated population(s), intervention(s), comparator(s), outcome(s), timing, and setting(s) of interestdcollectively referred to as the PICOTS [8,9]; (2) key questions that are precise, detailed, and focused; and (3) an analytical framework that represents the relationships between the PICOTS and the key questions [6,10e12]. Typically, unrefined topic nominations present the PICOTS elements in a general form. Nominated topics may be inadequately precise, overly inclusive, or overly narrow. Hence, refinement of a topic may narrow the focus of some elements of the PICOTS, expand some elements, or bothd a process more closely resembling sculpting in clay rather than marble.
D.I. Buckley et al. / Journal of Clinical Epidemiology 67 (2014) 425e432
427
Table 1. Criteria that a refined topic should fit
Table 2. Guiding principles for topic refinement
The health-care question or dilemma the systematic review aims to address The priorities and values of relevant stakeholders and users of the systematic review The state of the science and technical aspects of the topic Systematic review research methods
Fidelity to the original nomination retains the essential intent of the nominator and does not necessarily strive to satisfy the specific purpose of a given nominator. This assures that topics and systematic reviews are based on real-world issues that are important to stakeholders and that the systematic review will have relevance to a ready audience. Topics have relevance to those who would make decisions with the findings of the systematic review, as well as those who would be affected by those decisions. Research feasibility pertains to the practicality of conducting a review using systematic review methods within available resources. Responsiveness to stakeholder input assures that topics are tied to real-world concerns and decisional dilemmas but does not require integration of all inputs. Each investigator brings their experience, expertise, perspective, and values, which could introduce bias. Aspects of the topic refinement process can reduce possible investigator bias. Transparency in reporting includes a clear description of topic refinement decisions and the underlying rationale. This is important for public accountability and the integrity of the topic refinement process. A topic scope is the degree of inclusiveness reflected in the PICOTS, key questions, and analytical framework. Defining a suitable scope for a topic requires the investigator to consider numerous factors that affect the complexity and level of detail of the key questions.
3.2. Guiding principles We identified seven guiding principles to systematically consider when refining a topic for a systematic review (Table 2). These interrelated principles are consistent with those previously described in the EPC guidance for conducting systematic reviews [5]. To satisfy one principle, an investigator may compromise on another. We do not prescribe how these principles should be applied or balanced, only that they be considered. Investigators will use their judgment in refining topics, often making trade-offs. We envision investigators using the guiding principles for more systematic and explicit decision making. 3.2.1. Fidelity to the original nomination The EHC Program is committed to addressing patientcentered health-care questions that are tied to the concerns and decisional dilemmas of a broad range of stakeholders. Although the program does not necessarily strive to satisfy the specific purposes of given nominators, fidelity to the nomination assures that systematic reviews are relevant and based on real-world issues important to stakeholders. Investigators should be mindful of the initial intent of the nominator as they narrow or broaden a topic. 3.2.2. Relevance Topics should be relevant to decisional issues that matter to the users of the systematic review and should include
Fig. 1. Major stages in producing a systematic review.
outcomes that matter to patients [13]. Some nominated topics of high relevance to the nominator may be too narrowly framed to be of great use to a broader audience. Thus, topic refinement investigators may broaden or change the scope of the topic to increase its relevance. The investigator also refines the topic to reflect the underlying clinical or system logic, which includes the relevant clinical concepts and beliefs about the mechanism by which interventions may improve health outcomes [10]. A topic might be generally relevant for a particular issue or audience, but its relevance is limited if the analytical framework and key questions do not reflect the topic’s clinical logic. For example, the nomination on pressure ulcer treatment [14] included as an outcome the progression of an ulcer to a more advanced stage. Key informants emphasized that traditional staging systems imply a natural progression in wound severity that ignores variability in etiology. Furthermore, a progression of stage may not always be most important. Therefore, the refined topic did not include progression of stage as an outcome. 3.2.3. Research feasibility Topics should be formulated so as to be feasibly researched. Research feasibility pertains to the practicality of conducting a systematic review within a specified time frame and budget. Factors that affect research feasibility are the clarity and precision of the key questions, the relative heterogeneity of the PICOTS elements, the scope and complexity of the topic, and the size and nature of the evidence base.
428
D.I. Buckley et al. / Journal of Clinical Epidemiology 67 (2014) 425e432
The clarity and precision of the key questions and PICOTS influence systematic review inclusion and exclusion criteria. Unclear or vague questions may be cumbersome or complex to answer. Precise key questions allow for clearer decisions about the evidence and its synthesis, producing more accurate and efficient reviews. Similarly, overly heterogeneous PICOTS may make a topic more cumbersome to research, evidence synthesis more complicated, and presentation of the findings less clear. If the topic scope includes numerous health-care issues, or aspects of an issue, the synthesis of the evidence and communication of findings may be challenging. In addition, a large evidence base may be unwieldy and impractical for extraction and synthesis of the relevant literature within available resources. 3.2.4. Responsiveness to stakeholder inputs To assure that topics are tied to real-world concerns and decisional dilemmas, the topic refinement team is responsive to the input of diverse stakeholders. Responsiveness to stakeholder inputs does not imply that the topic refinement team must incorporate all inputs. Stakeholders provide the investigators with a diversity of perspectives to consider, but refinement decisions belong to the refinement team. 3.2.5. Reducing investigator bias A topic refinement investigator serves as an arbiter who weighs and integrates information and viewpoints from various sources. Each investigator also brings their experience, expertise, perspective, and values. Aspects of the refinement process can reduce the possible effect of investigator bias. First, as a deliberative process among team members, assumptions and viewpoints can be made explicit and discussed. Second, the EHC Program enforces a conflict of interest policy for investigators [15]. Third, a refinement team considers stakeholder inputs that may challenge assumptions, identify gaps or inconsistencies in thinking, and provide insight into different values related to the key questions. Finally, refinement is a structured process that helps assure that investigators openly and judiciously consider various viewpoints, including those different than their own. 3.2.6. Transparency The evidence that influenced crucial topic refinement decisions and the rationale underlying refinements should be clearly and explicitly described and documented. This is important for accountability and for efficiency and continuity in the subsequent steps of the systematic review. The unavoidable subjectivity in the topic refinement process implies that two investigators or topic refinement teams presented with the same topic nomination could make different decisions and refinements and, thereby, produce two systematic reviews with different PICOTS and key questions. Documentation allows others to understand
the basis on which particular refinements were made, and how decisions affected the outcome. 3.2.7. Suitable scope The scope of a topic refers to its relative degree of inclusiveness as reflected in the PICOTS, key questions, and analytical framework. A topic of narrow scope restricted to a single intervention, subpopulation, outcome, and/or setting may lack the greatest relevance. In contrast, a topic of broad scope including many interventions in the general population with multiple outcomes and settings may present challenges for research feasibility. A suitable scope is sufficiently inclusive to have high relevance and usefulness for decision makers, and yet is not so broad as to reduce the coherence of the review and the precision of its findings. The topic’s scope may also be a function of the level of detail in the key questions and complexity of the PICOTS elements. In general, higher specificity and detail in the PICOTS and key questions will constrain the focus and scope of the topic. However, a topic with highly detailed key questions is not always of narrow scope as a topic of broad scope by virtue of addressing numerous issues with many key questions might have a high level of detail in those questions. 3.3. The mechanics of conducting a topic refinement Although the topic refinement process follows a logical sequence (Fig. 2), the changes in the topic may not always flow in a linear and predictable way. Decisions about an element of the topic may be reconsidered after additional input. The degree of refinement required will vary. Some topics begin with clear and relevant key questions and well-defined PICOTS that accurately reflect the clinical logic; in these cases, little may change. Other topics may be less clear or complete and require more substantial refinement. 3.3.1. Initial topic refinement phase During the initial topic refinement phase, the team will conduct a literature scan (Fig. 2). Its purpose is twofold: (1) to help the investigators better understand the topic, its clinical logic, and the decisional dilemmas and (2) to familiarize the team with the relevant literature. The topic refinement team may conduct interviews with topical experts, who provide insight into technical issues, controversies, and the current state of knowledge. They can also clarify issues or uncertainties that arise when reviewing the topic nomination materials and the literature scan. Guided by the literature scan, topical expert input, and team expertise, investigators develop the provisional PICOTS, analytical framework, and key questions. These form the basis of subsequent key informant interviews. Refining the PICOTS often involves a balance and tradeoffs between the different PICOTS elements; inclusion of one element might have restrictive implications for others.
D.I. Buckley et al. / Journal of Clinical Epidemiology 67 (2014) 425e432
429
Fig. 2. The process of topic refinement. EPC, Evidence-based Practice Center; AHRQ, Agency for Healthcare Research and Quality.
For example, an outcome may not be applicable to certain subpopulations, or constraining the population of interest may limit the relevance to certain interventions. The full report contains more details [7]. General guidance for development of key questions and analytical frameworks has been described elsewhere [6,10e12]. 3.3.2. Engaging stakeholders as key informants The topic refinement team obtains inputs from stakeholder groups through the engagement of key informants, who reflect the perspectives of those who would make decisions with the report findings, as well as those who would be affected by those decisions. The team elicits input on
issues that cannot be resolved with a limited literature search and/or that require stakeholder input. With this input, the team can better understand real-world context, decisional dilemmas from diverse perspectives, and controversies and reasons for divergent views. In addition to the following, the full report contains more detailed suggestions on composing key informant groups and conducting interviews [7]. 3.3.2.1. Identifying key informants. The topic refinement team first identifies relevant stakeholder categories representing the diversity of viewpoints. Unless clearly not relevant, patients or their representatives should be included.
430
D.I. Buckley et al. / Journal of Clinical Epidemiology 67 (2014) 425e432
For controversial topics, key informants representing important opposing viewpoints should be enlisted. 3.3.2.2. Composition of key informant interview groups. The refinement team considers various factors when grouping key informants for interviews. These include the number of individuals, the types and variety of stakeholder groups, and the specific issues to be addressed. For example, if the interview were to focus primarily on an issue requiring particular expertise, the size and heterogeneity of the group could be limited. Similarly, if investigators sought to explore the tension between differing views of an issue, a larger and more heterogeneous group might be desirable. Patients or consumers may be more comfortable expressing their views when in a single stakeholder group. The size of the group in a single interview may affect the quality of engagement, the detail and depth of the discussion, and the ease of facilitating the interview. A large group may not allow all key informants to fully express their views within the allotted time; it may also preclude exploration of a particular question to the desired level of detail. Compared with smaller groups, a large group is more likely to include participants with a diversity of opinions, personalities, and communication styles, which may challenge the interviewer’s ability to guide and focus the discussion. 3.3.2.3. Conducting key informant interviews. The interviews are an efficient way of eliciting input from stakeholders in as complete and thorough a manner as possible within a practical time frame. Although consensus among participants is not the goal, it may arise spontaneously, suggesting that the PICOTS and key questions are on target. However, a lack of consensus may be equally useful in highlighting an area of disagreement that the team may further explore. The discussion may cover issues that the team has not been able to adequately address with the literature scan or topical experts; questions that require stakeholders’ perspectives or experience; questions about which outcomes are important in decision making; and comments on the provisional PICOTS, analytical framework, and key questions. Effective facilitation is essential for effective key informant interviews and has been described elsewhere [13]. 3.3.3. Integration and synthesis An essential aspect of the topic refinement process is the integration and synthesis of the information that the team gathers from various sources. Some of these refinement decisions are straightforward and may be addressed easily as information is gathered. For issues that are complex, controversial, or best addressed by another information source, the team may intentionally delay a decision to gather additional input. Occasionally, a previously settled issue is reconsidered in light of additional information or a subsequent decision about another issue.
Table 3. Key points The goal of topic refinement is to produce a topic that addresses important health-care questions and dilemmas, considers the priorities and values of relevant stakeholders, reflects the state of the science, and allows for application of systematic review research methods. The guiding principles are fidelity to the original nomination, relevance, research feasibility, responsiveness to stakeholder input, reduction of potential investigator bias, transparency, and suitable scope. B These principles are consistent with EPC program topic selection criteria for systematic reviews and the principles for conducting systematic reviews. Variability of topics in the EPC program makes it impractical to apply the guiding principles in a prescriptive manner. Topic refinement is an iterative and phased process. The stages of topic refinement (Fig. 2) are B Initial topic refinement B Key informant interviews B Public comment period B Synthesis and reporting Initial topic refinement gathers information from topical experts and a literature scan to develop the provisional PICOTS, analytical framework (AF), and key questions (KQs) of the topic to present to key informants for input. The key informant panel is comprised of 6e12 individuals. They reflect the perspectives of stakeholder groups who would make decisions with the findings of the report, as well as those affected by those decisions. Their input can improve the relevance and applicability of the systematic review. B Commonly, 2e4 individuals are engaged at a time for interviews to allow for sufficient opportunity to express opinions and for interaction. Consensus is not a goal. B The team ensures that the group’s mix of expertise and viewpoints are complementary. B Interviews are usually 60e90 minutes in duration and conducted over 3e4 weeks. B The interviews are generally facilitated by a core member of the topic refinement team, with part or all of the topic refinement team in attendance. Public comment on the topic allows for input from a broader range of individuals. Synthesis of input requires judgment of the topic refinement team and consideration of the guiding principles. The investigators may balance certain principles when making decisions about whether and how to include comments from individual stakeholders or other sources of input, especially when they are conflicting. The topic refinement team is comprised of independent investigators; ultimately, they are responsible for decisions about integration of input. In reporting, all decisions should be concisely and transparently documented in the topic refinement summary report. This report may be used by the topic refinement team, systematic review team, and AHRQ program officer to understand decisions made during topic refinement. Abbreviations: EPC, Evidence-based Practice Center; AHRQ, Agency for Healthcare Research and Quality.
The team may encounter challenges in synthesis, particularly when sources of input conflict. For example, the topic nominator may specify particular interventions. Key informants may identify additional interventions and comparators that reflect clinical practice and additional decisional dilemmas. The team will then balance fidelity to
D.I. Buckley et al. / Journal of Clinical Epidemiology 67 (2014) 425e432
the nomination with responsiveness to stakeholder input and suitable scope to ensure that the systematic review is relevant and useful to the nominator and other stakeholders. The team may not be responsive to all input and must decide which input to integrate. In making decisions, topic refinement investigators consider the nature of the evidence, the opinions of experts, the team’s own expertise, and systematic review methods, while using the guiding principles to reach a balanced compromise that considers the relative importance and/or practicality of various criteria. 3.3.4. Public posting The refined PICOTS, key questions, and analytical framework are posted online for broader stakeholder input before being finalized. Public comments may provide additional insights about the relative importance of PICOTS elements and key questions to specific stakeholders, point to relevant literature and guidelines, identify ongoing work, or highlight areas of clinical uncertainty. 3.3.5. Reporting The multiple opportunities for modifying a topic underscore the importance of consistent reporting. The EHC program uses a standard document to guide reporting [7]. This report documents the evolution of a topic through the refinement process, lays out the rationale for decisions, and may be used as a reference during the systematic review. Additionally, the report may help to inform discussion of similar issues and assist with framing controversial issues and future research needs in the evidence report. Although detailed documentation of every aspect of the process is not required, disclosure is important for establishing confidence in the refined document.
4. Conclusion Our work group reviewed the topic refinement methods used by EPCs and identified lessons learned; we developed guiding principles and described practical approaches (Table 3). These recommendations are not meant to be prescriptive but rather to promote more systematic and explicit decision making. Our approach was limited in several ways. We were not able to assess the effect of topic refinement on systematic reviews or its effect on the uptake and usefulness of systematic reviews by stakeholders. The work group had a limited number of individuals and, thus, a limited number of perspectives. However, our work group represented a diversity of approaches across EPCs, and we received input from investigators across the program. Questions remain about the topic refinement process. How to most effectively identify and engage stakeholders? How to better understand the effects of the inherent subjectivity of the process and to modulate those effects when
431
possible? We expect that methods will continue to evolve and more will be learned about the best approaches to these and other challenges.
Acknowledgments The authors gratefully acknowledge and thank the following individuals for their contributions to this project: Basmah Rahman, MPH, and Ngoc Wasson, MPH, for their administrative and project management support; and Stephanie Chang, MD, MPH, and Mark Helfand, MD, MPH, MS, for their valuable input and continuing support.
Appendix Supplementary data Supplementary data related to this article can be found at http://dx.doi.org/10.1016/j.jclinepi.2013.10.023. References [1] Federal Coordinating Council for Comparative Effectiveness Research. Report to the President and the Congress. Rockville, MD: U.S. Dept. of Health and Human Services; 2009. Available at http://www.hhs.gov/recovery/programs/cer/cerannualrpt.pdf. Accessed February 5, 2013. [2] U.S. Preventive Services Task Force. Rockville, MD: Agency for Healthcare Research and Quality; 2012. Available at http://www. ahrq.gov/clinic/uspstfix.htm. Accessed May 22, 2012. [3] Medicare uses of AHRQ research fact sheet. AHRQ publication no. 02P019. Rockville, MD: Agency for Healthcare Research and Quality; 2002. Available at http://www.ahrq.gov/news/focus/mediuses.htm. Accessed February 5, 2013. [4] Drug Effectiveness Review Project. 2012. Available at http://www. ohsu.edu/xd/research/centers-institutes/evidence-based-policy-center/ derp/index.cfm/. Accessed May 23, 2012. [5] Helfand M, Balshem H. Principles in developing and applying guidance. In: Methods guide for effectiveness and comparative effectiveness reviews. Rockville, MD: Agency for Healthcare Research and Quality; 2009. Available at www.effectivehealthcare.ahrq.gov/reports/final.cfm. Accessed February 5, 2013. [6] Helfand M, Balshem H. AHRQ series paper 2: principles for developing guidance: AHRQ and the Effective Health-Care Program. J Clin Epidemiol 2010;63:484e90. [7] Buckley DI, Ansari M, Butler M, Williams C, Chang C. The refinement of topics for systematic reviews: lessons and recommendations from the Effective Health Care Program. Methods research report. (Prepared by the Oregon Evidence-based Practice Center under Contract No. 290-2007-10057-I.) Publication No. 13-EHC023-EF. Rockville, MD: Agency for Healthcare Research and Quality; 2013. Available at http://effectivehealthcare.ahrq.gov/ehc/products/456/1385/ Methods-Refinement-of-Topics-130514.pdf. Accessed August 30, 2013. [8] Whitlock E, Lopez SA, Chang S, Helfand M, Eder M, Floyd N. AHRQ series paper 3: identifying, selecting, and refining topics for comparative effectiveness systematic reviews: AHRQ and the Effective Health-Care Program. J Clin Epidemiol 2010;63:491e501. [9] Counsell CC. Formulating questions and locating primary studies for inclusion in systematic reviews. Ann Intern Med 1997;127:380e7. [10] Battista RN, Fletcher SW. Making recommendations on preventive practices: methodological issues. Am J Prev Med 1988;4(4 Suppl): 53e76.
432
D.I. Buckley et al. / Journal of Clinical Epidemiology 67 (2014) 425e432
[11] Harris RP, Helfand M, Woolf SH, Lohr KN, Mulrow CD, Teutsch SM, et al, Third U. S. Preventive Services Task Force Methods Work Group. Current methods of the US Preventive Services Task Force: a review of the process. Am J Prev Med 2001; 20(3 Suppl):21e35. [12] Methods guide for medical test reviews. Rockville, MD: Agency for Healthcare Research and Quality; 2012. Available at. www. effectivehealthcare.ahrq.gov/reports/final.cfm. Accessed February 5, 2013. [13] Effective Health Care Program. The facilitation primer: strategies, tools & considerations to get you started. Rockville, MD: Agency for Healthcare Research and Quality; 2012. Available at http:// effectivehealthcare.ahrq.gov/tasks/sites/ehc/assets/File/Facilitation_Primer_ 20120124.pdf. Accessed February 5, 2013. [14] Saha S, Smith MEB, Totten A, Fu R, Wasson N, Rahman B, Motu’apuaka M, Hickam DH. Pressure Ulcer Treatment Strategies: Comparative Effectiveness. Comparative Effectiveness Review No.
90. (Prepared by the Oregon Evidence-based Practice Center under Contract No. 290-2007-10057-I.) AHRQ Publication No. 13EHC003-EF. Rockville, MD: Agency for Healthcare Research and Quality; May 2013. Available at http://effectivehealthcare.ahrq.gov/ ehc/products/308/1491/pressure-ulcer-treatment-report-130508.pdf. Accessed December 19, 2013. [15] Viswanathan M, Carey TS, Belinson SE, Berliner E, Chang S, Graham E, Guise JM, Ip SS, Maglione MA, McCrory D, McPheeters M, Newberry SJ, Sista P, White CM. Identifying and Managing Nonfinancial Conflicts of Interest for Systematic Reviews. Methods Research Report. (Prepared by the RTI International–University of North Carolina at Chapel Hill Evidence-based Practice Center under Contract No. 290-2007-10056-I.) AHRQ Publication No.13EHC085-EF. Rockville, MD: Agency for Healthcare Research and Quality; May 2013. Available at http://effectivehealthcare.ahrq. gov/ehc/products/467/1514/conflict-of-interest-methods.pdf. Accessed December 19, 2013.
ORIGINAL ARTICLES
The refinement of topics for systematic reviews: lessons and recommendations from the Effective Health Care Program David I. Buckleya,*, Mohammed T. Ansarib, Mary Butlerc, Clara Sohd, Christine S. Change a Oregon Health & Science University, 3181 SW Sam Jackson Park Road, Mail Code: FM, Portland, OR 97239, USA Clinical Epidemiology Program, Methods Centre, Box 201B, General Campus, The Ottawa Hospital, 501 Smyth Road, Ottawa, ON K1H 8L6, Canada c Division of Health Policy and Management, School of Public Health, University of Minnesota, 420 Delaware Street SE, Minneapolis, MN 55455, USA d Kaiser Permanente Center for Health Research, Evidence-based Practice Center, 3800 N Interstate Ave, Portland, OR 97227, USA e Center for Outcomes and Evidence, Agency for Healthcare Research and Quality, 540 Gaither Road, Rockville, MD 20850, USA b
Accepted 7 October 2013
Abstract Objectives: The Agency for Healthcare Research and Quality (AHRQ) Effective Health Care Program conducts systematic reviews of health-care topics nominated by stakeholders. Topics undergo refinement to ensure relevant questions of appropriate scope and useful reviews. Input from key informants, experts, and a literature scan informs changes in the nominated topic. AHRQ convened a work group to assess approaches and develop recommendations for topic refinement. Study Design and Setting: Work group members experienced in topic refinement generated a list of questions and guiding principles relevant to the refinement process. They discussed each issue and reached agreement on recommendations. Results: Topics should address important health-care questions or dilemmas, consider stakeholder priorities and values, reflect the state of the science, and be consistent with systematic review research methods. Guiding principles of topic refinement are fidelity to the nomination, relevance, research feasibility, responsiveness to stakeholder inputs, reduced investigator bias, transparency, and suitable scope. Suggestions for stakeholder engagement, synthesis of input, and reporting are discussed. Refinement decisions require judgment and balancing guiding principles. Variability in topics precludes a prescriptive approach. Conclusion: Accurate, rigorous, and useful systematic reviews require well-refined topics. These guiding principles and methodological recommendations may help investigators refine topics for reviews. Ó 2014 Elsevier Inc. All rights reserved. Keywords: Systematic review; Evidence-based practice; Stakeholder engagement; Methods; Decision making; Patient-centered care; Research design
1. Introduction
‘‘A prudent question is one-half of wisdom.’’ dFrancis Bacon Systematic reviews aim to improve health outcomes by developing evidence-based information about which interventions are most effective for which patients under
Funding: This project was funded under Contract No. HHSA 290-2007-10057-I from the Agency for Healthcare Research and Quality, US Department of Health and Human Services. The authors of this article are responsible for its content, including any clinical treatment recommendations. No statement in this article should be construed as an official position of AHRQ or of the US Department of Health and Human Services. Conflict of interest: None. * Corresponding author. Tel.: þ1-503-494-8367; fax: þ1-503-4941513. 0895-4356/$ - see front matter Ó 2014 Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.jclinepi.2013.10.023
specific circumstances [1]. Systematic reviews are used by a variety of organizations to inform clinical guidelines [2], health-care policies [3], and insurance coverage decisions [4]. The Evidence-based Practice Center (EPC) Program, part of the Agency for Healthcare Research and Quality (AHRQ) Effective Health Care (EHC) Program, conducts systematic reviews on health-care topics nominated by stakeholders. Stakeholders may represent patients, consumers, advocacy organizations, clinicians, researchers, agencies that issue guidelines, policy makers, industry, or health-care organizations. Involving stakeholders in the nomination process provides an opportunity for end users of research to participate in asking and answering questions about health care. To provide useful answers, systematic reviews must ask the right questions. Challenges arise when stakeholdernominated topics are not ideally formulated for the broadest public health and/or clinical relevance or are inconsistent with accepted systematic review methods or EPC Program
426
D.I. Buckley et al. / Journal of Clinical Epidemiology 67 (2014) 425e432
2. Methods What is new? This article provides methodological guidance for refining topics for systematic review in Agency for Healthcare Research and Quality Evidencebased Practice Center (EPC) Program. The goal of topic refinement is to produce a topic that addresses important health-care questions and dilemmas, considers the priorities and values of relevant stakeholders, reflects the state of the science, and allows for application of systematic review research methods. The article describes various EPC approaches to the refinement process, with their advantages and disadvantages, and provides guidance for a consistent approach to refinement and the reporting of refinement decisions in light of guiding principles. This can promote transparency, patient centeredness, and more useful and relevant systematic reviews. The guiding principles are fidelity to the original nomination, relevance, research feasibility, responsiveness to stakeholder inputs, reduction of potential investigator bias, transparency, and suitable scope. The nominated topic is refined with information from a literature scan and stakeholder inputs in an iterative manner. Synthesis of input requires judgment of the topic refinement team and consideration of the guiding principles. The investigators may balance certain principles when making decisions about whether and how to include input, especially when they are conflicting. Variability of topics in the EPC program makes it impractical to apply the guiding principles in a prescriptive manner.
principles for conducting reviews [5]. To ensure relevant and useful systematic reviews, topics nominated by stakeholders generally need to be refined. In 2007, investigators with the EPC Program began developing methods for topic refinement. Although the EPC Program stipulates the elements of topic refinement, different EPCs have approached specific aspects in both similar and different ways. This variation provided an excellent opportunity to consider the advantages and disadvantages of different approaches and produce recommendations to guide the topic refinement process. To that end, we convened a work group of investigators experienced in conducting topic refinements from four EPCs and one AHRQ project officer.
Our work group followed previously described principles for developing EPC Program methods guidance [6]. We used a best-practice approach based on our experience, our assessment of topic refinement reports from other EPCs, and input on the draft methods guidance report [7] from investigators across the EPC program. We compiled a prioritized list of issues and concepts that were challenging for many EPCs, incompletely articulated in topic refinement summary reports, variable across EPCs, and/or consistent across EPCs. These items were consolidated into three main categories: (1) the purpose of topic refinement, (2) guiding principles, and (3) the mechanics of conducting a topic refinement. Through a structured process in 18 meetings, we discussed each issue and reached agreement on recommendations. Recognizing the legitimate variability in topics and approaches, we described different viable approaches and discussed their relative merits. Experts in systematic review methods provided a peer review of a draft report, which was also posted for public comment. The methods are described in more detail in the full report [7].
3. Results 3.1. What is refinement? Refinement implies making changes to attain a better fit with a certain standard. In this sense, the goal of topic refinement is to improve a nominated topic so that it is a good and accurate fit with specified criteria (Table 1). Topic refinement is one stage in the process of producing a systematic review through the EPC Program (Fig. 1). It is preceded by topic nomination and development, in which investigators review stakeholder-nominated topics and determine if they meet program selection criteria and should proceed with topic refinement and systematic review [8]. The primary goal of topic refinement is to formulate research questions that can be addressed by a systematic review, not to answer the questions. A refined topic includes three principal elements: (1) clearly articulated population(s), intervention(s), comparator(s), outcome(s), timing, and setting(s) of interestdcollectively referred to as the PICOTS [8,9]; (2) key questions that are precise, detailed, and focused; and (3) an analytical framework that represents the relationships between the PICOTS and the key questions [6,10e12]. Typically, unrefined topic nominations present the PICOTS elements in a general form. Nominated topics may be inadequately precise, overly inclusive, or overly narrow. Hence, refinement of a topic may narrow the focus of some elements of the PICOTS, expand some elements, or bothd a process more closely resembling sculpting in clay rather than marble.
D.I. Buckley et al. / Journal of Clinical Epidemiology 67 (2014) 425e432
427
Table 1. Criteria that a refined topic should fit
Table 2. Guiding principles for topic refinement
The health-care question or dilemma the systematic review aims to address The priorities and values of relevant stakeholders and users of the systematic review The state of the science and technical aspects of the topic Systematic review research methods
Fidelity to the original nomination retains the essential intent of the nominator and does not necessarily strive to satisfy the specific purpose of a given nominator. This assures that topics and systematic reviews are based on real-world issues that are important to stakeholders and that the systematic review will have relevance to a ready audience. Topics have relevance to those who would make decisions with the findings of the systematic review, as well as those who would be affected by those decisions. Research feasibility pertains to the practicality of conducting a review using systematic review methods within available resources. Responsiveness to stakeholder input assures that topics are tied to real-world concerns and decisional dilemmas but does not require integration of all inputs. Each investigator brings their experience, expertise, perspective, and values, which could introduce bias. Aspects of the topic refinement process can reduce possible investigator bias. Transparency in reporting includes a clear description of topic refinement decisions and the underlying rationale. This is important for public accountability and the integrity of the topic refinement process. A topic scope is the degree of inclusiveness reflected in the PICOTS, key questions, and analytical framework. Defining a suitable scope for a topic requires the investigator to consider numerous factors that affect the complexity and level of detail of the key questions.
3.2. Guiding principles We identified seven guiding principles to systematically consider when refining a topic for a systematic review (Table 2). These interrelated principles are consistent with those previously described in the EPC guidance for conducting systematic reviews [5]. To satisfy one principle, an investigator may compromise on another. We do not prescribe how these principles should be applied or balanced, only that they be considered. Investigators will use their judgment in refining topics, often making trade-offs. We envision investigators using the guiding principles for more systematic and explicit decision making. 3.2.1. Fidelity to the original nomination The EHC Program is committed to addressing patientcentered health-care questions that are tied to the concerns and decisional dilemmas of a broad range of stakeholders. Although the program does not necessarily strive to satisfy the specific purposes of given nominators, fidelity to the nomination assures that systematic reviews are relevant and based on real-world issues important to stakeholders. Investigators should be mindful of the initial intent of the nominator as they narrow or broaden a topic. 3.2.2. Relevance Topics should be relevant to decisional issues that matter to the users of the systematic review and should include
Fig. 1. Major stages in producing a systematic review.
outcomes that matter to patients [13]. Some nominated topics of high relevance to the nominator may be too narrowly framed to be of great use to a broader audience. Thus, topic refinement investigators may broaden or change the scope of the topic to increase its relevance. The investigator also refines the topic to reflect the underlying clinical or system logic, which includes the relevant clinical concepts and beliefs about the mechanism by which interventions may improve health outcomes [10]. A topic might be generally relevant for a particular issue or audience, but its relevance is limited if the analytical framework and key questions do not reflect the topic’s clinical logic. For example, the nomination on pressure ulcer treatment [14] included as an outcome the progression of an ulcer to a more advanced stage. Key informants emphasized that traditional staging systems imply a natural progression in wound severity that ignores variability in etiology. Furthermore, a progression of stage may not always be most important. Therefore, the refined topic did not include progression of stage as an outcome. 3.2.3. Research feasibility Topics should be formulated so as to be feasibly researched. Research feasibility pertains to the practicality of conducting a systematic review within a specified time frame and budget. Factors that affect research feasibility are the clarity and precision of the key questions, the relative heterogeneity of the PICOTS elements, the scope and complexity of the topic, and the size and nature of the evidence base.
428
D.I. Buckley et al. / Journal of Clinical Epidemiology 67 (2014) 425e432
The clarity and precision of the key questions and PICOTS influence systematic review inclusion and exclusion criteria. Unclear or vague questions may be cumbersome or complex to answer. Precise key questions allow for clearer decisions about the evidence and its synthesis, producing more accurate and efficient reviews. Similarly, overly heterogeneous PICOTS may make a topic more cumbersome to research, evidence synthesis more complicated, and presentation of the findings less clear. If the topic scope includes numerous health-care issues, or aspects of an issue, the synthesis of the evidence and communication of findings may be challenging. In addition, a large evidence base may be unwieldy and impractical for extraction and synthesis of the relevant literature within available resources. 3.2.4. Responsiveness to stakeholder inputs To assure that topics are tied to real-world concerns and decisional dilemmas, the topic refinement team is responsive to the input of diverse stakeholders. Responsiveness to stakeholder inputs does not imply that the topic refinement team must incorporate all inputs. Stakeholders provide the investigators with a diversity of perspectives to consider, but refinement decisions belong to the refinement team. 3.2.5. Reducing investigator bias A topic refinement investigator serves as an arbiter who weighs and integrates information and viewpoints from various sources. Each investigator also brings their experience, expertise, perspective, and values. Aspects of the refinement process can reduce the possible effect of investigator bias. First, as a deliberative process among team members, assumptions and viewpoints can be made explicit and discussed. Second, the EHC Program enforces a conflict of interest policy for investigators [15]. Third, a refinement team considers stakeholder inputs that may challenge assumptions, identify gaps or inconsistencies in thinking, and provide insight into different values related to the key questions. Finally, refinement is a structured process that helps assure that investigators openly and judiciously consider various viewpoints, including those different than their own. 3.2.6. Transparency The evidence that influenced crucial topic refinement decisions and the rationale underlying refinements should be clearly and explicitly described and documented. This is important for accountability and for efficiency and continuity in the subsequent steps of the systematic review. The unavoidable subjectivity in the topic refinement process implies that two investigators or topic refinement teams presented with the same topic nomination could make different decisions and refinements and, thereby, produce two systematic reviews with different PICOTS and key questions. Documentation allows others to understand
the basis on which particular refinements were made, and how decisions affected the outcome. 3.2.7. Suitable scope The scope of a topic refers to its relative degree of inclusiveness as reflected in the PICOTS, key questions, and analytical framework. A topic of narrow scope restricted to a single intervention, subpopulation, outcome, and/or setting may lack the greatest relevance. In contrast, a topic of broad scope including many interventions in the general population with multiple outcomes and settings may present challenges for research feasibility. A suitable scope is sufficiently inclusive to have high relevance and usefulness for decision makers, and yet is not so broad as to reduce the coherence of the review and the precision of its findings. The topic’s scope may also be a function of the level of detail in the key questions and complexity of the PICOTS elements. In general, higher specificity and detail in the PICOTS and key questions will constrain the focus and scope of the topic. However, a topic with highly detailed key questions is not always of narrow scope as a topic of broad scope by virtue of addressing numerous issues with many key questions might have a high level of detail in those questions. 3.3. The mechanics of conducting a topic refinement Although the topic refinement process follows a logical sequence (Fig. 2), the changes in the topic may not always flow in a linear and predictable way. Decisions about an element of the topic may be reconsidered after additional input. The degree of refinement required will vary. Some topics begin with clear and relevant key questions and well-defined PICOTS that accurately reflect the clinical logic; in these cases, little may change. Other topics may be less clear or complete and require more substantial refinement. 3.3.1. Initial topic refinement phase During the initial topic refinement phase, the team will conduct a literature scan (Fig. 2). Its purpose is twofold: (1) to help the investigators better understand the topic, its clinical logic, and the decisional dilemmas and (2) to familiarize the team with the relevant literature. The topic refinement team may conduct interviews with topical experts, who provide insight into technical issues, controversies, and the current state of knowledge. They can also clarify issues or uncertainties that arise when reviewing the topic nomination materials and the literature scan. Guided by the literature scan, topical expert input, and team expertise, investigators develop the provisional PICOTS, analytical framework, and key questions. These form the basis of subsequent key informant interviews. Refining the PICOTS often involves a balance and tradeoffs between the different PICOTS elements; inclusion of one element might have restrictive implications for others.
D.I. Buckley et al. / Journal of Clinical Epidemiology 67 (2014) 425e432
429
Fig. 2. The process of topic refinement. EPC, Evidence-based Practice Center; AHRQ, Agency for Healthcare Research and Quality.
For example, an outcome may not be applicable to certain subpopulations, or constraining the population of interest may limit the relevance to certain interventions. The full report contains more details [7]. General guidance for development of key questions and analytical frameworks has been described elsewhere [6,10e12]. 3.3.2. Engaging stakeholders as key informants The topic refinement team obtains inputs from stakeholder groups through the engagement of key informants, who reflect the perspectives of those who would make decisions with the report findings, as well as those who would be affected by those decisions. The team elicits input on
issues that cannot be resolved with a limited literature search and/or that require stakeholder input. With this input, the team can better understand real-world context, decisional dilemmas from diverse perspectives, and controversies and reasons for divergent views. In addition to the following, the full report contains more detailed suggestions on composing key informant groups and conducting interviews [7]. 3.3.2.1. Identifying key informants. The topic refinement team first identifies relevant stakeholder categories representing the diversity of viewpoints. Unless clearly not relevant, patients or their representatives should be included.
430
D.I. Buckley et al. / Journal of Clinical Epidemiology 67 (2014) 425e432
For controversial topics, key informants representing important opposing viewpoints should be enlisted. 3.3.2.2. Composition of key informant interview groups. The refinement team considers various factors when grouping key informants for interviews. These include the number of individuals, the types and variety of stakeholder groups, and the specific issues to be addressed. For example, if the interview were to focus primarily on an issue requiring particular expertise, the size and heterogeneity of the group could be limited. Similarly, if investigators sought to explore the tension between differing views of an issue, a larger and more heterogeneous group might be desirable. Patients or consumers may be more comfortable expressing their views when in a single stakeholder group. The size of the group in a single interview may affect the quality of engagement, the detail and depth of the discussion, and the ease of facilitating the interview. A large group may not allow all key informants to fully express their views within the allotted time; it may also preclude exploration of a particular question to the desired level of detail. Compared with smaller groups, a large group is more likely to include participants with a diversity of opinions, personalities, and communication styles, which may challenge the interviewer’s ability to guide and focus the discussion. 3.3.2.3. Conducting key informant interviews. The interviews are an efficient way of eliciting input from stakeholders in as complete and thorough a manner as possible within a practical time frame. Although consensus among participants is not the goal, it may arise spontaneously, suggesting that the PICOTS and key questions are on target. However, a lack of consensus may be equally useful in highlighting an area of disagreement that the team may further explore. The discussion may cover issues that the team has not been able to adequately address with the literature scan or topical experts; questions that require stakeholders’ perspectives or experience; questions about which outcomes are important in decision making; and comments on the provisional PICOTS, analytical framework, and key questions. Effective facilitation is essential for effective key informant interviews and has been described elsewhere [13]. 3.3.3. Integration and synthesis An essential aspect of the topic refinement process is the integration and synthesis of the information that the team gathers from various sources. Some of these refinement decisions are straightforward and may be addressed easily as information is gathered. For issues that are complex, controversial, or best addressed by another information source, the team may intentionally delay a decision to gather additional input. Occasionally, a previously settled issue is reconsidered in light of additional information or a subsequent decision about another issue.
Table 3. Key points The goal of topic refinement is to produce a topic that addresses important health-care questions and dilemmas, considers the priorities and values of relevant stakeholders, reflects the state of the science, and allows for application of systematic review research methods. The guiding principles are fidelity to the original nomination, relevance, research feasibility, responsiveness to stakeholder input, reduction of potential investigator bias, transparency, and suitable scope. B These principles are consistent with EPC program topic selection criteria for systematic reviews and the principles for conducting systematic reviews. Variability of topics in the EPC program makes it impractical to apply the guiding principles in a prescriptive manner. Topic refinement is an iterative and phased process. The stages of topic refinement (Fig. 2) are B Initial topic refinement B Key informant interviews B Public comment period B Synthesis and reporting Initial topic refinement gathers information from topical experts and a literature scan to develop the provisional PICOTS, analytical framework (AF), and key questions (KQs) of the topic to present to key informants for input. The key informant panel is comprised of 6e12 individuals. They reflect the perspectives of stakeholder groups who would make decisions with the findings of the report, as well as those affected by those decisions. Their input can improve the relevance and applicability of the systematic review. B Commonly, 2e4 individuals are engaged at a time for interviews to allow for sufficient opportunity to express opinions and for interaction. Consensus is not a goal. B The team ensures that the group’s mix of expertise and viewpoints are complementary. B Interviews are usually 60e90 minutes in duration and conducted over 3e4 weeks. B The interviews are generally facilitated by a core member of the topic refinement team, with part or all of the topic refinement team in attendance. Public comment on the topic allows for input from a broader range of individuals. Synthesis of input requires judgment of the topic refinement team and consideration of the guiding principles. The investigators may balance certain principles when making decisions about whether and how to include comments from individual stakeholders or other sources of input, especially when they are conflicting. The topic refinement team is comprised of independent investigators; ultimately, they are responsible for decisions about integration of input. In reporting, all decisions should be concisely and transparently documented in the topic refinement summary report. This report may be used by the topic refinement team, systematic review team, and AHRQ program officer to understand decisions made during topic refinement. Abbreviations: EPC, Evidence-based Practice Center; AHRQ, Agency for Healthcare Research and Quality.
The team may encounter challenges in synthesis, particularly when sources of input conflict. For example, the topic nominator may specify particular interventions. Key informants may identify additional interventions and comparators that reflect clinical practice and additional decisional dilemmas. The team will then balance fidelity to
D.I. Buckley et al. / Journal of Clinical Epidemiology 67 (2014) 425e432
the nomination with responsiveness to stakeholder input and suitable scope to ensure that the systematic review is relevant and useful to the nominator and other stakeholders. The team may not be responsive to all input and must decide which input to integrate. In making decisions, topic refinement investigators consider the nature of the evidence, the opinions of experts, the team’s own expertise, and systematic review methods, while using the guiding principles to reach a balanced compromise that considers the relative importance and/or practicality of various criteria. 3.3.4. Public posting The refined PICOTS, key questions, and analytical framework are posted online for broader stakeholder input before being finalized. Public comments may provide additional insights about the relative importance of PICOTS elements and key questions to specific stakeholders, point to relevant literature and guidelines, identify ongoing work, or highlight areas of clinical uncertainty. 3.3.5. Reporting The multiple opportunities for modifying a topic underscore the importance of consistent reporting. The EHC program uses a standard document to guide reporting [7]. This report documents the evolution of a topic through the refinement process, lays out the rationale for decisions, and may be used as a reference during the systematic review. Additionally, the report may help to inform discussion of similar issues and assist with framing controversial issues and future research needs in the evidence report. Although detailed documentation of every aspect of the process is not required, disclosure is important for establishing confidence in the refined document.
4. Conclusion Our work group reviewed the topic refinement methods used by EPCs and identified lessons learned; we developed guiding principles and described practical approaches (Table 3). These recommendations are not meant to be prescriptive but rather to promote more systematic and explicit decision making. Our approach was limited in several ways. We were not able to assess the effect of topic refinement on systematic reviews or its effect on the uptake and usefulness of systematic reviews by stakeholders. The work group had a limited number of individuals and, thus, a limited number of perspectives. However, our work group represented a diversity of approaches across EPCs, and we received input from investigators across the program. Questions remain about the topic refinement process. How to most effectively identify and engage stakeholders? How to better understand the effects of the inherent subjectivity of the process and to modulate those effects when
431
possible? We expect that methods will continue to evolve and more will be learned about the best approaches to these and other challenges.
Acknowledgments The authors gratefully acknowledge and thank the following individuals for their contributions to this project: Basmah Rahman, MPH, and Ngoc Wasson, MPH, for their administrative and project management support; and Stephanie Chang, MD, MPH, and Mark Helfand, MD, MPH, MS, for their valuable input and continuing support.
Appendix Supplementary data Supplementary data related to this article can be found at http://dx.doi.org/10.1016/j.jclinepi.2013.10.023. References [1] Federal Coordinating Council for Comparative Effectiveness Research. Report to the President and the Congress. Rockville, MD: U.S. Dept. of Health and Human Services; 2009. Available at http://www.hhs.gov/recovery/programs/cer/cerannualrpt.pdf. Accessed February 5, 2013. [2] U.S. Preventive Services Task Force. Rockville, MD: Agency for Healthcare Research and Quality; 2012. Available at http://www. ahrq.gov/clinic/uspstfix.htm. Accessed May 22, 2012. [3] Medicare uses of AHRQ research fact sheet. AHRQ publication no. 02P019. Rockville, MD: Agency for Healthcare Research and Quality; 2002. Available at http://www.ahrq.gov/news/focus/mediuses.htm. Accessed February 5, 2013. [4] Drug Effectiveness Review Project. 2012. Available at http://www. ohsu.edu/xd/research/centers-institutes/evidence-based-policy-center/ derp/index.cfm/. Accessed May 23, 2012. [5] Helfand M, Balshem H. Principles in developing and applying guidance. In: Methods guide for effectiveness and comparative effectiveness reviews. Rockville, MD: Agency for Healthcare Research and Quality; 2009. Available at www.effectivehealthcare.ahrq.gov/reports/final.cfm. Accessed February 5, 2013. [6] Helfand M, Balshem H. AHRQ series paper 2: principles for developing guidance: AHRQ and the Effective Health-Care Program. J Clin Epidemiol 2010;63:484e90. [7] Buckley DI, Ansari M, Butler M, Williams C, Chang C. The refinement of topics for systematic reviews: lessons and recommendations from the Effective Health Care Program. Methods research report. (Prepared by the Oregon Evidence-based Practice Center under Contract No. 290-2007-10057-I.) Publication No. 13-EHC023-EF. Rockville, MD: Agency for Healthcare Research and Quality; 2013. Available at http://effectivehealthcare.ahrq.gov/ehc/products/456/1385/ Methods-Refinement-of-Topics-130514.pdf. Accessed August 30, 2013. [8] Whitlock E, Lopez SA, Chang S, Helfand M, Eder M, Floyd N. AHRQ series paper 3: identifying, selecting, and refining topics for comparative effectiveness systematic reviews: AHRQ and the Effective Health-Care Program. J Clin Epidemiol 2010;63:491e501. [9] Counsell CC. Formulating questions and locating primary studies for inclusion in systematic reviews. Ann Intern Med 1997;127:380e7. [10] Battista RN, Fletcher SW. Making recommendations on preventive practices: methodological issues. Am J Prev Med 1988;4(4 Suppl): 53e76.
432
D.I. Buckley et al. / Journal of Clinical Epidemiology 67 (2014) 425e432
[11] Harris RP, Helfand M, Woolf SH, Lohr KN, Mulrow CD, Teutsch SM, et al, Third U. S. Preventive Services Task Force Methods Work Group. Current methods of the US Preventive Services Task Force: a review of the process. Am J Prev Med 2001; 20(3 Suppl):21e35. [12] Methods guide for medical test reviews. Rockville, MD: Agency for Healthcare Research and Quality; 2012. Available at. www. effectivehealthcare.ahrq.gov/reports/final.cfm. Accessed February 5, 2013. [13] Effective Health Care Program. The facilitation primer: strategies, tools & considerations to get you started. Rockville, MD: Agency for Healthcare Research and Quality; 2012. Available at http:// effectivehealthcare.ahrq.gov/tasks/sites/ehc/assets/File/Facilitation_Primer_ 20120124.pdf. Accessed February 5, 2013. [14] Saha S, Smith MEB, Totten A, Fu R, Wasson N, Rahman B, Motu’apuaka M, Hickam DH. Pressure Ulcer Treatment Strategies: Comparative Effectiveness. Comparative Effectiveness Review No.
90. (Prepared by the Oregon Evidence-based Practice Center under Contract No. 290-2007-10057-I.) AHRQ Publication No. 13EHC003-EF. Rockville, MD: Agency for Healthcare Research and Quality; May 2013. Available at http://effectivehealthcare.ahrq.gov/ ehc/products/308/1491/pressure-ulcer-treatment-report-130508.pdf. Accessed December 19, 2013. [15] Viswanathan M, Carey TS, Belinson SE, Berliner E, Chang S, Graham E, Guise JM, Ip SS, Maglione MA, McCrory D, McPheeters M, Newberry SJ, Sista P, White CM. Identifying and Managing Nonfinancial Conflicts of Interest for Systematic Reviews. Methods Research Report. (Prepared by the RTI International–University of North Carolina at Chapel Hill Evidence-based Practice Center under Contract No. 290-2007-10056-I.) AHRQ Publication No.13EHC085-EF. Rockville, MD: Agency for Healthcare Research and Quality; May 2013. Available at http://effectivehealthcare.ahrq. gov/ehc/products/467/1514/conflict-of-interest-methods.pdf. Accessed December 19, 2013.
