Identification of topics for comparative effectiveness systematic reviews in the field of cancer imaging.

RESEARCH ARTICLE For reprint orders, please contact: [email protected]

Identification of topics for comparative effectiveness systematic reviews in the field of cancer imaging Aim: With rapid innovations in diagnostic and therapeutic interventions in cancer care, comparative effectiveness reviews (CERs) are essential to inform clinical practice and guide future research. However, the optimal means to identify priority CER topics are uninvestigated. We aimed to devise a transparent and reproducible process to identify ten to 12 CER topics in the area of cancer imaging relevant to a wide range of stakeholders. Materials & methods: Environmental scans and explicit prioritization criteria supported interactions (email communications, web-based discussions and live teleconferences) with experts and stakeholders culminating in a three-phase deductive exercise for prioritization of CER topics. Results: We prioritized 12 CER topics in breast, lung and gastrointestinal cancers that addressed screening, diagnosis, staging, monitoring and evaluating response to treatment. Conclusion: Our project developed and implemented a transparent and reproducible process for research prioritization and topic nomination that can be further refined to improve the relevance of future CERs. KEYWORDS: cancer imaging n comparative effectiveness reviews n environmental scan n research prioritization n stakeholders n topic identification

Cancer is the second leading cause of death in the USA. It is also among the most costly conditions to treat [1,2]. The increasing availability of novel and innovative healthcare technologies for the diagnosis and treatment of cancer along with the rising costs of these technologies have created an urgent need to systematically ascertain the relative effectiveness of these options. In response, the oncology community has undertaken a number of initiatives to address this need and make explicit comparisons of diagnostic and therapeutic technologies available to providers and guideline developers [3]. While demand and support for comparative effectiveness research have risen in recent years, the ideal methods for prioritizing research questions have not been established. Both public and private proponents of comparative effectiveness and patientcentered outcomes research have strongly advocated for engaging the full range of stakeholders – consumers, providers and payers alike – in setting research priorities, based on the notion that engagement can improve the relevance of research, increase its transparency and accelerate its adoption into practice [4–7]. The Agency for Healthcare Research and Quality (AHRQ) has been a leader in working with stakeholders nationally and locally to identify, develop and prioritize topics for comparative effectiveness reviews (CERs), and in identifying future research needs. The CER process comprises a systematic review of existing research, and is designed to offer guidance to healthcare decision-makers; support improvements in the quality, effectiveness and efficiency of healthcare; and offer further guidance on research priorities [101]. In this report, we describe the results of a CER topic identification exercise sponsored by AHRQ and carried out by the Tufts Medical Center Evidence-based Practice

10.2217/CER.13.61 © 2013 Future Medicine Ltd

2(5), 483–495 (2013)

Madhu Rao1,2, Thomas W Concannon3, Ramon Iovin1, Winifred W Yu1, Jeffrey A Chan1, Georgios Lypas4,5, Teruhiko Terasawa1,6, James M Gaylor1, Lina Kong1, Andrew C Rausch1, Joseph Lau7 & Georgios D Kitsios*1,8 Institute for Clinical Research & Health Policy Studies (ICRHPS), Tufts Medical Center & Tufts University School of Medicine, MA, USA 2 Division of Nephrology, Tufts Medical Center, MA, USA 3 The RAND Corporation, 20 Park Plaza Suite 920, Boston, MA 02111, USA 4 Department of Medical Oncology, Dana Farber Cancer Institute, MA, USA 5 Genetic Oncology Unit, Hygeia Hospital, Athens, Greece 6 Department of Internal Medicine, Fujita Health University School of Medicine, Tsu, Mie, Japan 7 Brown Center for Evidence Based Medicine, Brown University, Providence, RI, USA 8 Department of Internal Medicine, Lahey Hospital & Medical Center, 41 Mall Road, Burlington, MA 01805, USA *Author for correspondence: Tel.: +1 781 744 7000 [email protected] 1

part of

ISSN 2042-6305

483

RESEARCH ARTICLE

Rao, Concannon, Iovin et al.

Center (EPC). Our primary aim was to identify ten to 12 CER topics in the area of cancer imaging that would be relevant to a wide range of stakeholders and could be used by AHRQ or other research-sponsoring organizations for development of complete CERs. Our secondary purpose was to present a framework that can be adapted for future topic-identification activities for comparative effectiveness research across a range of patient-care domains. Methods & results

We organized the project to meet three milestones (Figure 1). Milestone one was a prioritized list of two or three broad cancer subcategories on which to focus the remainder of the project. Milestone two was the identification of

Engagement activities

30–40 specific research topics within the two or three cancer subcategories. Finally, milestone three consisted of a list of ten to 12 prioritized research topics for CERs. Our project was structured deductively in three milestones with specific goals for end products in each phase. This was done for two purposes: first, to be able to consider, in a comprehensive fashion, the totality of potential research in the field of cancer imaging; and second, to generate lists of projects that would be sufficiently diverse but also of manageable breadth for further detailed consideration and analyses. We based our research prioritization approach on a set of guiding principles found to be common across various related priority-setting models and approaches [8]. These principles include:

Step

Milestone

One Cancer map development Teleconference discussions Two Review prioritization criteria Technical expert panel Three Environmental scan Teleconference discussions Four Analysis of environmental scan

Email and telephone orientation

One List of two to three subcategories

Five Stakeholder identification and recruitment

Online discussions Shareholder panel

Six Topic nomination

Two List of 30–40 cancer topics

Seven Topic prioritization

Three List of ten to 12 cancer topics

Online discussions Webinar discussions

Figure 1. Project steps and milestones. The entire project comprised seven steps (middle column) to achieve three key milestones (right column). The technical expert panel and/or stakeholder panel were engaged in each step of the project (left column).

484

J. Compar. Effect. Res. (2013) 2(5)

future science group

Identification of topics for comparative effectiveness systematic reviews in cancer imaging

alignment with the overall strategic purpose of the Effective Health Care (EHC) Program of AHRQ; application of clear and consistent criteria for prioritization; involvement of stake holders; maintenance of adequate transparency to allow for public accountability; and standardized evaluation of the process itself. To this end, we adopted the prioritization criteria used by the EHC Program for CERs to guide our work [8]. These criteria – appropriateness, importance, feasibility/desirability and potential value – provided a framework to guide our selection of cancer subcategories as well as potential CER topics.

guiding treatment, determining if a treatment is working, and monitoring for recurrence. This initial mapping activity informed subsequent steps of the project in two ways: each of the ten subcategories of the anatomical cancer map was considered as a potential candidate for prioritization (step four) to two or three cancer subcategories of milestone one; and the cancer imaging map was subsequently used in steps six and seven to generate an exhaustive list of potential topics (based on the type of cancer and the imaging modality and application) within the two or three prioritized cancer subcategories of milestone one.

Milestone one: list of two to three priority cancer subcategories

■■ Step two: review & application of prioritization criteria

■■ Step one: cancer map development

As previously mentioned, to prioritize cancer subcategories, we employed the prioritization criteria used by the EHC Program for CERs (appropriateness, importance, feasibility/desirability and potential value) [8]. As cancer is one of the priority health conditions designated by the EHC Program, all subcategories of cancer were considered to meet the criterion of appropriateness and, as such, this criterion did not play any further role in assigning relative value. The remaining prioritization criteria were used to assess the importance of each cancer subcategory (milestone one) as well as the remaining two milestone products.

Step one consisted of two parallel mapping efforts. The end products of these efforts were: an anatomical cancer map, classifying all malignant neoplasms into a systems/organs hierarchy and thus defining anatomical cancer subcategories; and a cancer imaging map, classifying all imaging technologies and applications of interest (diagnostic or therapeutic) used in oncology. Each mapping effort was guided by the physicians on our team, and a technical expert panel (TEP) consisting of academic radiologists and oncologists recruited specifically for this project. The anatomical cancer map was adapted from the classification system used by the American Cancer Society (ACS) and the Surveillance, Epidemiology and End Results Program of the National Cancer Institute (NCI) [9]. We generated a list of all malignant neoplasms and mapped them according to subcategory of affected anatomical site (e.g., thoracic and gastrointestinal), a framework also used by the ACS and NCI to map information for disease-specific statistics (e.g., incidence and fatality rates). The final map consisted of ten broad cancer subcategories organized by anatomical system (head and neck, breast, musculoskeletal, thoracic, gastrointestinal, hematological, genital, urinary, skin and unknown primary organ cancers) [9]. The cancer imaging map was developed by compiling a list of all imaging technologies utilized in clinical oncology, based on input by physicians in our group. We then classified this list according to the system proposed by the Cancer Imaging Program of the NCI, with applications sorted according to the purpose of the imaging application: screening, diagnosis/staging,


RESEARCH ARTICLE

■■ Step three: environmental scan

In step three, we conducted environmental scans of a number of data sources to inform our prioritization criteria for each cancer subcategory. Broadly speaking, environmental scanning is the identification and subsequent analysis of emerging issues and trends for topics of interest, in order to guide future planning, through screening multiple and diverse information sources [10]. Our environmental scan was tailored to address the prioritization criterion of importance and of feasibility/desirability. Importance was measured by the relative public health burden and intensity of public interest. The relative public health burden of each cancer subcategory was assessed by examining cancer-related statistics from the ACS [102], the Surveillance, Epidemiology and End Results Database [103], and the NCI [104], focusing on parameters such as incidence and mortality rates and their trends, 5-year survival rates, lifetime risks and prevalence. We gauged the extent of public interest for each cancer subcategory by analyzing data

www.futuremedicine.com

485

RESEARCH ARTICLE


collected from multiple databases that provided complementary information. We analyzed search engine-usage patterns via queries to Google® Insights for Searches [105], and the volume of posts and ‘hit’ frequency per cancer subcategory on cancer-related forums and webbased discussions within the American Cancer Society’s Cancer Survivors Network [106]. To capture another dimension of potential public interest, we queried the PsycINFO® database to measure the volume of peer-reviewed literature on quality-of-life assessments for each subcategory of cancer patients. We focused our searches on PsycINFO, since cancer study citations in this database are enriched for studies on patient-centered psychological outcomes and these studies are not always included in the Medline® database [11]. We assessed feasibility/desirability by the quantity of published literature available for future evidence synthesis in CERs. Our data sources included indexes of published literature and ongoing research, national report statistics, online patient forums and search engine statistics. ■■ Step four: analysis of the environmental scan

We assigned a five-level score for each of the ten public health burden parameters (based on incidence, prevalence, mortality and survival statistics) to each of the ten candidate cancer subcategories, from which we derived a summary score (numeric-weighted average) for each cancer. To visually appraise this score information, we created star graphs (Figure 2) that allowed us to visualize the consistency of high-ranking scores across many parameters for certain cancer subcategories. We reviewed this procedure with the TEP and jointly selected cancer subcategories that ranked above median summary scores for consideration in the next step, yielding five priority cancer subcategories: gastrointestinal, thoracic, breast, genital and hematological. To test the robustness of this approach, we then performed a sensitivity analysis by examining data for cancers of individual organs in these subcategories. The top three organ-specific cancers according to this analysis were lung and bronchus, colon and rectum, and breast, which belonged in the thoracic, gastrointestinal and breast cancer subcategories, respectively, that were high-ranking cancer subcategories in our analysis. The level of public interest for each cancer subcategory – as captured from the PsycINFO

486


database, Cancer Survivors Network online forum [106] and Google Insights [105] – is summarized in Figure 3A . We aimed to summarize this information in a single graph in order to identify the subcategories that scored highly across all three of these independent parameters. Breast, gastrointestinal, thoracic, head and neck, and genital cancers were among the top categories. We assessed the relative feasibility/desirability of each subcategory for future CERs via the volume of systematic reviews and human primary research literature in Medline. We queried Medline using broad (medical subject headings) search terms for all cancers included in a subcategory and limited results to diagnostic imaging studies only. We also searched an internal database of meta-analyses of diagnostic tests maintained and curated by our EPC that includes all meta-analyses of imaging technologies used in cancer until December 2010 [12], and estimated the volume of ongoing research in cancer imaging by conducting targeted searches in ClinicalTrials.gov [107]. Similarly, an analysis of literature search results in Medline indicated breast, gastrointestinal, genital, head and neck, and thoracic cancers as the most predominant in terms of available literature, as shown in Figure 3B. Using this information, we worked with the TEP to select three consensus cancer subcategories – breast, thoracic and gastrointestinal – for inclusion in the next stage of activities. Milestones two & three: list of 30–40 cancer topics & prioritization of two to three cancer topics ■■ Step five: stakeholder recruitment & engagement

In pursuit of our second milestone, we initially identified stakeholders within each of the seven stakeholder categories, working from the ‘7Ps of CER’ framework for stakeholder engagement [13]. The 7Ps refer to patients and the public, providers, purchasers, payers, policy-makers, product makers and principal investigators. Our panel was designed to represent the full spectrum of groups with an interest in the outcomes of evidence synthesis in cancer imaging. Potential stakeholders were identified by scanning inhouse stakeholder databases, searching Medline for expert panelists in the principal investigator and provider categories, and inquiring through organizational and personal contacts and through the snowball method after initial



Breast

Head and neck

RESEARCH ARTICLE

Musculoskeletal

Incidence rate in 2010 Cost Incidence rate 5-year 5 Trend in 3 survival incidence rate 1 -1 Trend in Lifetime risk mortality rate Prevalence Mortality rate Mortality rate in 2010

Incidence rate in 2010 Cost Incidence rate 5-year 5 Trend in 3 survival incidence rate 1 Trend in -1 Lifetime risk mortality rate Prevalence Mortality rate Mortality rate in 2010

Incidence rate in 2010 Cost Incidence rate 5-year 5 Trend in 3 survival incidence rate 1 Trend in -1 Lifetime risk mortality rate Prevalence Mortality rate Mortality rate in 2010

Thoracic Incidence rate in 2010 Cost 5 Incidence rate 3 5-year Trend in 1 survival incidence rate -1 Trend in Lifetime risk mortality rate Prevalence Mortality rate Mortality rate in 2010

Gastrointestinal Incidence rate in 2010 Cost 5 Incidence rate 3 5-year Trend in 1 survival incidence rate -1 Trend in Lifetime risk mortality rate Prevalence Mortality rate Mortality rate in 2010

Hematological Incidence rate in 2010 Cost 4 Incidence rate 5-year Trend in 2 survival incidence rate 0 Trend in Lifetime risk mortality rate Prevalence Mortality rate Mortality rate in 2010

Genital

Cost 5-year survival Trend in mortality rate Mortality rate

Incidence rate in 2010 5 Incidence rate 4 3 Trend in 2 1 incidence rate 0

Lifetime risk

Prevalence Mortality rate in 2010

Cost 5-year survival Trend in mortality rate

Urinary

Skin

Incidence rate in 2010 5 Incidence rate 3 Trend in 1 incidence rate -1 Lifetime risk

Incidence rate in 2010 5 Incidence rate 3 Trend in 1 incidence rate -1 Lifetime risk

Mortality rate


Cost 5-year survival Trend in mortality rate Mortality rate


Figure 2. Epidemiology and public health burden: star plots of cancer subcategories. Each axis of the star represents the ranking score for a parameter; graphs with larger surface areas indicate subcategories that have consistently higher-ranking scores across all parameters (i.e., likely to represent more important subcategories in terms of public health burden). The cancer subcategories with the largest areas were gastrointestinal, thoracic, breast, genital and hematological. Please note that the subcategory ‘unknown primary’ did not fit this framework for several parameters and was not included in this analysis.

requests for participation. Participating stakeholders were required to make full disclosures of financial, professional and business interests and submit a curriculum vitae. No incentives were provided. Over the course of the recruitment period, 135 stakeholders were invited to participate, with a 24% refusal rate and 56% nonresponse rate, leaving a stakeholder panel of 24 individuals (20%) who participated in at least one step of the process. In total, 16–18 stakeholders, representing between four and six stakeholder categories, contributed at each individual step. Six of seven


stakeholder groups were included on the final panel. We considered purchasers and payers to share the same interests with respect to cancer research; both seek to restrain low value and spread high value interventions to insured populations. Therefore, we did not include purchasers as a distinct stakeholder group. ■■ Step six: topic generation

Following the prioritization of three subcate gories of cancer (milestone one), we compiled a comprehensive list of cancer imaging topics for discussion. We obtained nominations from


487

RESEARCH ARTICLE


1000 Breast

PsycINFO® search results

800 600 400

Genital Hematological Skin

200

Gastrointestinal

Thoracic Head and neck

0

Urinary Musculoskeletal

-200 0

100

200

300

400

500

600

MEDLINE® search results for imaging RCTs

Forum discussion search results 300 Breast 250 Gastrointestinal 200 150

Genital

Thoracic Head and neck

100

Unknown primary

50

Skin

0

Hematological

Musculoskeletal

Urinary

-50 0

5000

10,000

15,000

20,000

25,000

30,000

MEDLINE search results (overall) ®

Figure 3. Cancer subcategory ranking by parameters of public interest and quantity of published literature. (A) The y-axis represents the volume of patient-reported outcome literature in the PsycINFO® database from inception. The x-axis displays the relative volume of forum discussion threads generated over the same timeframe. Each subcategory is depicted by a circle, whose size represents the corresponding Google® Insights intensity metric; we devised this intensity metric as a ratio of numbers of recorded Google searches for different cancers: the common denominator was the cancer with the smallest number of searches (‘hematological’), then each cancer’s number of searches was used as nominator for each subcategory’s metic. Circles of larger size and in the upperright quadrant of the figure represent subcategories of higher public interest. Breast, gastrointestinal, genital, thoracic, and head and neck cancers were the subcategories with the indication of more intense public interest. (B) The y-axis represents the number of Medline® search results for imaging RCTs, the total number of results returned for each cancer subcategory is on the x-axis, and the total number of results for meta-analyses and systematic reviews is represented by the size of each circle (cancer subcategory). In general, subcategories with the largest numbers of imaging RCTs are considered as fulfilling the criterion of feasibility/desirability, as these studies will serve as the primary building blocks of evidence for future comparative effectiveness reviews. Breast, gastrointestinal, genital, thoracic, and head and neck cancers were the subcategories with the most available evidence. RCT: Randomized controlled trial.

488




stakeholders via email and generated additional nominations through consultation with oncology experts. We aimed to capture all potential applications of imaging (screening, staging and so on) of all up-to-date imaging techniques (PET-computed tomography, ultrasound, MRI, computed tomography and so on) and a wide array of technologies and clinical applications for all clinically relevant disease entities within a cancer subcategory (e.g., within the gastrointestinal subcategory: stomach, esophageal, colorectal cancer and so on). Topics were evaluated for redundancy and were refined and merged into brief research questions restated according to a population, intervention, comparator, outcome (PICO)-based rubric. Each topic was thus formulated with the following title structure: ‘comparative effectiveness of (imaging technique[s] and comparator[s]; e.g., computed tomography vs chest x-rays) for a specific (application of imaging; e.g. screening among others) in a (target cancer population; e.g., lung cancer).’ The final list of candidate topics was then sent to stakeholders for prioritization. The topic nomination process yielded a collaboratively determined list of 32 unique candidate topics in breast (12 topics), thoracic (nine topics) and gastrointestinal (11 topics) cancer imaging, including all potential uses of imaging in cancer (five topics on screening, nine on primary diagnosis, four on recurrence monitoring, nine on staging, six on response to treatment, two on prognosis and one on guiding treatment delivery). Three of these topics were duplicative and were, therefore, merged to yield a final compiled list of 29 topics for consideration in the next step (Table 1). ■■ Step seven: prioritization

For milestone three, the topics nominated in step six were subjected to an iterative process of prioritization (Figure 1) through facilitated discussion (via email, web-based discussions and live teleconferences) with stakeholders. At the beginning of this stage, we circulated the initial list of 29 topics via email to each stakeholder in order to solicit feedback and gauge preferences. Each stakeholder was requested to review the list of topics and select a priority set of ten topics for further discussion. Summarizing the preferences of the stakeholder group yielded a list of 21 topics (seven in each topic area). The intent of this initial prioritization was to foster a more focused discussion. These 21 topics were


RESEARCH ARTICLE

subsequently subjected to online deliberations in a custom web forum, which helped condense the list to 19 topics for discussion in the next stage. Some previously rejected topics that emerged as relevant during these stakeholder interactions were reinstated. Following this activity, stakeholders were engaged through web-based discussions, followed-up by a series of small-group webinars and teleconferences in which the EPC team presented stakeholders with an expedited environmental scan of the prioritized topics. As for milestones one and two, our environmental scan considered the same set of disease burden and public interest parameters to provide support for the importance criterion. For this environmental scan, we focused on databases that could help us gauge the availability of primary research for systematic synthesis and pre-existing systematic reviews. We thus performed keyword searches in ClinicalTrials. gov, PubMed, the AHRQ website and the UK Centre for Reviews and Dissemination Health Technology Assessment website [108] to identify CERs, health-technology assessments, systematic reviews, randomized controlled trials and ongoing clinical trials from 2005, to evaluate the feasibility and desirability of conducting a new CER on each proposed topic. Stakeholders were encouraged to evaluate and comment on the potential value of a new CER for each topic. They were then directed to specify their top five preferred topics for a new CER. These rankings were then synthesized to yield the final list of the 12 highest-priority topics (Table 1), which included five topics on breast cancer, three on lung cancer, two on colorectal, one for liver and one topic on pancreatic cancer. Of these topics, four involved the use of imaging for screening, five for primary diagnosis, five for staging, two for recurrence monitoring and one for response to treatment. These 12 topic nominations were then submitted to AHRQ for future topic triage. Discussion

At a time when the USA is seeking to establish robust patient-centered outcomes research programs that incorporate up-to-date evidence into clinical practice, two priorities have consistently emerged as vital to the process: the need to identify the questions with the greatest relevance to healthcare decision-makers; and the related need to engage with stakeholders in shaping


489

RESEARCH ARTICLE


Table 1. Candidate comparative effectiveness review topics. Topic

Cancer subcategory: role

Initial rank

Final rank

CE of different imaging techniques (e.g., mammography, scintimammography, US Breast: screening and MRI mammography) for screening of breast cancer in increased-risk populations

22–28

1

CE of imaging techniques (and strategies) for the screening for lung cancer among high-risk as well as the general population

Lung: screening

2–8

1

CE of MDCT (CT colonography) vs colonoscopy for screening colorectal neoplasms, such as polyps, and primary, recurrent and de novo CRC

CRC: screening and 2–8 primary diagnosis; recurrence monitoring

3

CE of ultrasound, conventional CT, MDCT, MRI (using conventional vs novel contrast agents) for the primary diagnosis and staging of hepatocellular carcinoma

Liver: primary diagnosis and staging

2–8

4–5

CE of conventional CT, MDCT (CT colonography), FDG-PET or FDG-PET/CT, MRI, US and EUS for the pretherapy staging (including the assessment of liver metastasis) of CRC

CRC: staging

2–8

4–5

CE of conventional CT, MDCT (including ‘virtual CT bronchoscopy’), PET/PET-CT and MRI for the diagnosis or pretreatment staging of small-cell and non-small-cell lung cancer

Lung: primary diagnosis and staging

2–8

6

CE of imaging techniques (e.g., CT scans vs chest x-rays) for monitoring of stable lung nodules

Lung: primary diagnosis

22–28

7–8

CE of imaging techniques for the surveillance/follow-up after the treatment of non-metastatic breast cancer

Breast: recurrence monitoring

9–15

7–8

CE of imaging techniques (e.g., sonography, CT and MRI) for the locoregional staging of breast cancer

Breast: staging

22–28

9–11

CE of US, conventional CT, MDCT, MRI and FDG-PET or FDG-PET-CT for the primary diagnosis and staging of pancreatic cancer

Pancreas: primary diagnosis and staging

9–15

9–11

CE of different imaging modalities (e.g., mammography, scintimammography, US and MRI mammography) for screening of breast cancer in the general population

Breast: screening

9–15

9–11

CE of imaging techniques for the assessment of response to treatment in metastatic breast cancer

Breast: response to treatment

9–15

12

CE of functional imaging techniques (e.g., scintimammography, functional CT, dynamic MRI, spectroscopy and PET) for the assessment of response to neoadjuvant therapy in breast cancer

Breast: response to treatment

1

13

CE of FDG-PET or FDG-PET/CT, EUS, conventional CT and MDCT (including ‘virtual CT Esophagus: staging laparoscopy’) for the staging of esophageal cancer

2–8

14

CE of conventional CT, MDCT (CT colonography), FDG-PET or FDG-PET/CT, MRI, US, and EUS for the post-therapy response assessment of CRC

CRC: response to treatment

9–15

15

CE of imaging techniques for the primary diagnosis and staging of neuroendocrine pancreatic tumors

Pancreas: primary diagnosis and staging

9–15

15

CE of FDG-PET or FDG-PET/CT, EUS, conventional CT and MDCT (including ‘virtual CT Esophagus: response laparoscopy’) for the post(neoadjuvant) therapy response assessment of esophageal to treatment cancer

22–28

15

CE of sequential bilateral MRI mammography vs other imaging strategies for the evaluation of solitary breast lesions

Breast: recurrence monitoring

16–21

18

CE of conventional CT, FDG-PET or FDG-PET/CT, and MRI for the assessment of mediastinal lesions

Unknown: primary diagnosis

9–15

19

CE of MRI, EUS and CT in predicting complete response of neoadjuvant chemotherapy, radiotherapy and chemoradiotherapy in gastric and gastroesophageal junction cancers

Gastric: response to treatment

9–15

Not ranked

Topics with a rank of 12 or lower in the righthand column were selected as priorities in milestone three. CE: Comparative effectiveness; CRC: Colorectal cancer; CT: Computed tomography; EUS: Endoscopic ultrasound; FDG: 18F-fluorodeoxyglucose; MDCT: Multidetector computed tomography; SPECT: Single-photon emission computed tomography; US: Ultrasonography.

490




RESEARCH ARTICLE

Table 1. Candidate comparative effectiveness review topics (cont.). Topic

Cancer subcategory: role

Initial rank

Final rank

CE of different imaging techniques for the primary diagnosis and staging of gastric cancer

Gastric: primary diagnosis and staging

9–15

Not ranked

CE of US, conventional CT, MDCT, MRI and FDG-PET or FDG-PET-CT for the post-therapy response assessment of pancreatic cancer

Pancreas: response to treatment

22–28

Not ranked

CE of diagnostic accuracy among CT, MRI, thallium-201 scintigraphy, and FDG-PET or FDG-PET/CT in the diagnosis of recurrent laryngeal carcinoma

Laryngeal: recurrence monitoring

22–28

Not ranked

CE of imaging techniques vs clinical assessment for screening of breast cancer in the general population

Breast: screening

0

Not ranked

CE of different imaging techniques (e.g., mammography, scintimammography, US, MRI mammography, PET/CT and SPECT) for the primary diagnosis of breast cancer

Breast: primary diagnosis

2–8

Not ranked

CE of different imaging techniques (e.g., MRI mammography, scintimammography, PET/CT and SPECT) for prognostication in metastatic breast cancer

Breast: prognosis

29–31

Not ranked

CE of different imaging techniques in the assessment of baseline breast cancer risk

Breast: prognosis

32

Not ranked

CE of imaging techniques for identifying bone metastases of breast cancer

Breast: staging

29–31

Not ranked

Topics with a rank of 12 or lower in the righthand column were selected as priorities in milestone three. CE: Comparative effectiveness; CRC: Colorectal cancer; CT: Computed tomography; EUS: Endoscopic ultrasound; FDG: 18F-fluorodeoxyglucose; MDCT: Multidetector computed tomography; SPECT: Single-photon emission computed tomography; US: Ultrasonography.

these questions. The objective of our project was to devise and implement a transparent and reproducible, stakeholder- and data-driven process to identify topics for CERs within the topic area of cancer imaging. Implicitly, the process was guided by the AHRQ’s priority areas, and therefore, the final CER topics were those identified as the most important within that framework. The results we obtained can be compared with similar undertakings by other research teams [14,15]. One of this project’s major challenges was achieving a balanced sampling of stakeholder input. The 7Ps framework lent itself well to this end, as it guided recruitment from the full spectrum of individuals and groups with an interest in cancer imaging [13]. To identify and retain a sufficient pool of stakeholder candidates – we targeted a panel size of 20–25 representatives as sufficiently diverse but still manageable – we used a combination of several strategies: Medline/literature searches; ‘cold’ calls from contact lists; following leads gathered from those calls; and outreach to personal contacts for identifying potential participants, as previously described [16,17]. The most effective source of participants was obtained through personal recommendation, providing us with 44% of panel membership. This route of stakeholder recruitment does not indicate,


by any means, that any particular views were over-represented in the final panel, but rather underscores that familiarity with the methodology of the CERs and scope of AHRQ research agenda in general can enhance participation in stakeholder-engagement activities. Four participants provided feedback on the process and final product of this project, chiefly highlighting the challenging time commitments necessary for full participation and expressing concerns about their expertise. Sustaining participation through all phases of the project and facilitating stakeholder interactions was resource- and time-intensive, but we found that maintaining a continual dialog with all stakeholders helped our team and the other project participants retain focus. As with balancing stakeholder input, informing stakeholder deliberations was a similarly daunting task. However, we found that the prioritization criteria used (importance, feasibility and desirability, and potential value) served as a useful (although not mandatory) lens through which stakeholders could evaluate research topics. Similarly, environmental scanning proved to be valuable in providing stakeholders with data about past and ongoing research [10]. Environmental scanning is particularly relevant to any area characterized by rapid technological progress, of which cancer imaging is a prime


491

RESEARCH ARTICLE


example. It enables adaptation to an everchanging research landscape. It also helps to identify trends, needs and uncertainties in research, thereby increasing the relevance of the findings for stakeholders. Particularly for the criterion of importance and the assessment of ‘public health burden’, our environmental scans had evident construct validity, utilizing high-quality epidemiologic data (from the Surveillance, Epidemiology and End Results database) and allowing formal quantitative analyses for prioritization. Although repeating such analyses in the future may lead to different results owing to the evolving landscape of cancer epidemiology, our approach represents a reproducible and valid approach for appreciating the relative public health burden of different cancers. We noted an interesting difference between the topics represented in milestones two and three. Initial preferences appeared to be weighted toward topics covering either the preor post-therapy staging of diagnosed cancer. However, final selections were weighted toward screening in general and high-risk populations. This shift may be interpreted as the single most important effect of engaging stakeholders in this iterative process. Environmental scans, coupled with online discussions and webinars, could have had a significant impact on stakeholder thinking and empowered participants to make considered choices, since in this phase, stakeholders were exposed to the results of systematic literature searches showing availability of existing CERs and also of primary ongoing research that would be amenable to systematic synthesis. Thus, their decisions may have taken a more pragmatic focus in the final phases of prioritization. It is also possible that the complexity of imaging modalities in the treatment and staging of cancer may have influenced decisions toward more familiar and easily understood applications of imaging for screening in the panel, particularly among nonclinical expert participants. Finally, we did not assess participating stakeholders’ personal experiences with cancer, such as cancer survivorship. Such histories may be influential in shaping individual decisions beyond their expertise in a particular stakeholder category. Despite the success of certain aspects of our approach, our methods and results were not without limitations. First, environmental scanning is a resource-intensive activity, and the data

492


generated is only useful if interpreted correctly and utilized effectively for strategic planning and decision-making. Since no gold standard methods exist to define and capture some of our parameters for the prioritization criteria (i.e., intensity of public interest), we had to rely on proxy measures through multiple, complementary searches in different databases. However, recent data indicate that analyses of the volume of internet search activity can serve as reasonable surrogates to disease activity for epidemiologic research [18]. Moreover, although we did find that the initial prioritization of cancer sub categories was both robust to sensitivity analysis and concordant with TEP member decisions, the reproducibility of environmental scanning methods is unestablished. It is also important to note that our claims of reproducibility apply strictly to our process and not the outcome results, and we acknowledge that both the ever-evolving nature of technological advances in cancer imaging and the independent perspectives of various shareholders could produce a different list of prioritized topics in the future. Nonetheless, our process was transparent, utilizing standardized systematic search methods and quantitative synthesis methods, when applicable, and thus represents a reproducible framework of prioritizing topics for CERs in the field of cancer, and beyond. The information sources we utilized are widely available and continuously updated, and our methods for stakeholder interaction followed previously described standards [14,16,17]. Second, our recruitment rate of 20%, refusal rate of 24% and nonresponse rate of over 50% from a sample of 135 invited stakeholders, underscores the difficulties of stakeholder recruitment. Our method of identifying panelists by referral or recommendation and their own subjective views may have played an important role in the outcomes of this topic prioritization activity; it is possible that different individuals, representing the same groups of stakeholders, may have developed different topic rankings. The overall low response rate may also signify that certain views may not have been adequately represented in the final panel, since individuals with familiarity with the methodology and scope of CERs, and also motivation to advocate for certain conditions, may have been keener to participate in this voluntary activity. However, this limitation of representativeness applies to any stakeholder-related activity, and we made



every effort to recruit participants from a wide range of sources, and review any potential conflicts of interests and academic/professional backgrounds. Hence, our emphasis on recruiting and retaining engaged stakeholders from each of the 7P categories was mainly targeted towards maintaining a balance across groups with competing interests. Engagement of the full panel proved challenging: stakeholders only nominated 50% of the initial topics and only six participated in this process (step five). Barriers to participation included the time commitment and the perception of the required level of expertise. Unfamiliarity with a new process, or the perceived technical complexities of the topic area, may have also contributed to low participation rates during the nomination phase. However, we had significantly more success in maintaining wider participation in later stages. Prior experience with stakeholder-engaged research has shown that short-duration teleconferences with a large number of participants of diverse backgrounds are difficult to implement efficiently [8,14]. We utilized open and interactive formats of webinars (as a modified Delphi method) that proved to be an effective engagement strategy in our project, as reflected in the 100% stakeholder response rate for final selections and the constructive feedback we received regarding our methods and the final selection of topics [16,17]. At first glance, a recent prioritization exercise on diagnostic cancer technologies, using payers’ internal claims data and stakeholder engagement, offers an attractive comparison to our work [15]. The highest priority topic from both this and our group was the same: the use of MRI to guide treatment decisions in breast cancer. While this may appear to validate our outcome, the projects differed in an important way: our process was designed to identify priorities for evidence syntheses where sufficient primary studies already exist; the other was designed to identify priorities where sufficient primary research does not already exist. Especially for the case of breast cancer, for which there may be a disproportionate awareness and volume of research, this may explain why this specific cancer category accounted for five of the final 12 CER topics. Further research is needed to understand whether stakeholder-engaged prioritization lives up to its intention: that it will help to improve the relevance, transparency and utility


RESEARCH ARTICLE

of healthcare research. One way to approach this need would be to prospectively compare independent prioritization exercises in two or more teams of researchers and stakeholders. Insights might also be gleaned from collaboration with experts from other industries about what will and will not work in healthcare. Utility assessments could also be conducted on healthcare stakeholders themselves to assess the relevance, transparency and use of recent prioritization activities. Conclusion

In this project, we identified ten to 12 priority CER topics for the diverse communities of stakeholders in cancer and imaging, and we present a framework that can be adapted to prioritization activities for comparative effectiveness research in other patient-care domains. Our findings may stimulate further discussion in the oncology community about priorities for future research. In addition, our methods for gathering stakeholder input may be helpful in similar activities across a range of clinical disciplines. A portion of this work may be carried forward in AHRQ-sponsored topic development activities that lead to the conduct of formal CERs and technology assessments, while others may be supported through other research-sponsoring entities. Future perspective

The next decade is going to continue to see a rapid evolution of technologies and a growing demand for their applications to patient care. Research-sponsoring entities, such as the Patient-Centered Outcome Research Institute and the AHRQ, will play an increasingly significant role in establishing research priorities and using CERs for informed healthcare decision-making. Increasing participation from the multiple stakeholders in each research field – including patients and the public, providers, purchasers, payers, policy-makers, productmakers and principal investigators – is going to be instrumental in shaping research priorities and carrying research forward. The optimal means of stakeholder engagement will continue to evolve, while further development and refinement of transparent quantitative and qualitative methods for incorporating stakeholder input in research prioritization will improve the public health relevance of future research.


493

RESEARCH ARTICLE


Financial & competing interests disclosure This work was supported by the Agency for Healthcare Research and Quality, Contract HHSA-290-2007-10055-I, Task Order 7. The authors have no other relevant affiliations or financial involvement with any organization or entity

with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed. No writing assistance was utilized in the production of this manuscript.

Executive summary Background ■■ There is an escalating need for timely comparative effectiveness reviews (CERs) to inform clinical practice in cancer diagnosis and treatment. ■■ Optimal methods for identifying priority topics for CERs remain undefined. Purpose ■■ The purpose of this work was to: ūū Identify ten to 12 CER topics in the area of cancer imaging relevant to a wide range of stakeholders. ūū Devise a transparent and reproducible process for such activities. Methods & results ■■ Evidence-based topic prioritization activities were conducted in three consecutive milestones. ■■ Explicit prioritization criteria were utilized: appropriateness, importance, feasibility/desirability and potential value. ■■ A deductive process of topic generation was applied, from broad cancer anatomical categories to specific clinical uses of imaging to candidate-refined CER topics. ■■ An environmental scan of various information sources was conducted, which informed the process throughout. ■■ A large, balanced panel of stakeholders was recruited, according to the 7P taxonomy: patients and the public, providers, payers, purchasers, product manufacturers, policy-makers and principal investigators. ■■ Stakeholder interactions were conducted via email communications, web-based discussions and live teleconferences. ■■ A final prioritization of 12 CER topics in the subcategories of breast, lung and gastrointestinal cancer was prioritized; topics encompassed screening, primary diagnosis, staging, monitoring for recurrence and evaluating response to treatment. Conclusion ■■ A transparent and reproducible process for research prioritization was developed. ■■ In total, 12 high-priority CER topics in cancer imaging were identified. ■■ Refinements to our methods by others will probably improve the relevance of future CERs. research agenda. JAMA 307(15), 1583–1584 (2012).

References Papers of special note have been highlighted as: of interest n

1

Iglehart JK. Prioritizing comparative-effectiveness research – IOM recommendations. N. Engl. J. Med. 361, 325–328 (2009).

2

Tangka FK, Trogdon JG, Richardson LC et al. Cancer treatment cost in the United States: has the burden shifted over time? Cancer 116, 3477–34784 (2010).

3

4

5

494

6

7

Roehr B. More stakeholder engagement is needed to improve quality of research, say US experts. Br. Med. J. 341, c4193 (2010).

8

Whitlock EP, Lopez SA, Chang S et al. AHRQ series paper 3: identifying, selecting, and refining topics for comparative effectiveness systematic reviews: AHRQ and the effective healthcare program. J. Clin. Epidemiol. 63, 491–501 (2010).

Li EC, DeMartino J. Preliminary report: the development of the NCCN Comparative Therapeutic Index as a clinical evaluative process for existing data in oncology. J. Natl Compr. Canc. Netw. 8, S1–S9 (2010). US Department of Health and Human Services. Federal Coordinating Council for Comparative Effectiveness Research. Report to the President and Congress. Washington, DC, USA 30 June 2009. Selby JV, Beal AC, Frank L. The Patient-Centered Outcomes Research Institute national priorities for resarch and

Tunis SR, Benner J, McClellan M. Comparative effectiveness research: policy context, methods development and research infrastructure. Stat. Med. 29(19), 1963–1967 (2010).

n

9

Key paper describing the processes of identification, selection and refinement of topics for comparative effectiveness systematic reviews within the Agency for Healthcare Research and Quality program. National Cancer Institute. SEER summary staging manual – 2000. In: Codes and


Coding Instructions. Young JL Jr, Roffers SD, Gloeckler Ries LA, Fritz AG, Hurlbut AA (Eds). US NIH, MD, USA, 2–15 (2001). 10 Choo CW, Auster E. Environmental

scanning: acquisition and use of information by managers. In: Annual Review of Information Science and Technology. Williams ME (Ed.). Learned Information Inc., NJ, USA, 279–394 (1993). 11 Eady AM, Wilczynski NL, Haynes RB.

PsycINFO search strategies identified methodologically sound therapy studies and review articles for use by clinicians and researchers. J. Clin. Epidemiol. 61, 34–40 (2008). 12 Dahabreh IJ, Chung M, Kitsios GD et al.

Comprehensive Overview of Methods and Reporting of Meta-Analyses of Test Accuracy. Agency for Healthcare Research and Quality, MD, USA (2012). 13 Concannon TW, Meissner P, Grunbaum JA

et al. A new taxonomy for stakeholder engagement in patient-centered outcomes research. J. Gen. Intern. Med. 27, 985–991 (2012).



n

Describes an explicit taxonomy for categories of stakeholders involved in patient-centered outcomes research and defines their roles and responsibilities.

14 Chalkidou K, Whicher D, Kary W, Tunis

SR. Comparative effectiveness research priorities: identifying critical gaps in evidence for clinical and health policy decision making. Int. J. Technol. Assess. Health Care 25, 241–248 (2009). n

Exemplary pilot project using an interactive nominal group process to prioritize a research agenda across a multistakeholder workgroup, highlighting challenges and successes in the process.

15 Klein G, Gold LS, Sullivan SD et al.

Prioritizing comparative effectiveness research for cancer diagnostics using a regional stakeholder approach. J. Comp. Eff. Res. 1, 241–255 (2012). n

Project engaging regional stakeholders for prioritizing comparative effectiveness research in cancer diagnostics in the western Washington State (USA) region.

16 Guise JM, O’Haire C, McPheeters M et al.

A practice-based tool for engaging


stakeholders in future research: a synthesis of current practices. J. Clin. Epidemiol. 66, 666–674 (2013). 17 Deverka PA, Lavallee DC, Desai PJ et al.

Stakeholder participation in comparative effectiveness research: defining a framework for effective engagement. J. Comp. Eff. Res. 1, 181–194 (2012). 18 Willard SD, Nguyen MM. Internet search

trends analysis tools can provide real-time data on kidney stone disease in the United States. Urology 81, 37–42 (2013).

■■ Websites 101 Agency for Healthcare Research and Quality.

What is comparative effectiveness research (2012). www.effectivehealthcare.ahrq.gov/index.cfm/ what-is-comparative-effectiveness-research1 (Accessed 16 March 2011) 102 American Cancer Society. Cancer facts and

figures 2010 (2012). www.cancer.org/Research/ CancerFactsFigures/CancerFactsFigures/ cancer-facts-and-figures-2010 (Accessed 16 March 2011)


RESEARCH ARTICLE

103 National Cancer Institute. Surveillance

epidemiology and end results. Cancer stat fact sheets (2012). http://seer.cancer.gov/statfacts (Accessed 16 March 2011) 104 National Cancer Institute. The cost of cancer

(2011). www.cancer.gov/aboutnci/servingpeople/ cancer-statistics/costofcancer (Accessed 16 March 2011) 105 Google. Google insights for search (2012).

www.google.com/insights/search (Accessed 1 February 2011) 106 American Cancer Society. Cancer survivors

network (2012). http://csn.cancer.org/forum (Accessed 1 February 2011) 107 Clinical trials database.

http://clinicaltrials.gov (Accessed 16 March 2011) 108 The University of York and the National

Health Service. Centre for reviews and dissemination (2012). www.crd.york.ac.uk/crdweb/SearchPage.asp. (Accessed 1 July 2011)

495

Collider bias in trauma comparative effectiveness research: the stratification blues for systematic reviews.

Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews.

The refinement of topics for systematic reviews: lessons and recommendations from the Effective Health Care Program.

Effectiveness of acupuncture and related therapies for palliative care of cancer: overview of systematic reviews.

Effectiveness of Chinese herbal medicine for cancer palliative care: overview of systematic reviews with meta-analyses.

Comparative effectiveness of follow-up imaging approaches in pancreatic cancer.

Assessment for Risk of Bias in Systematic Reviews and Meta-Analyses in the Field of Hepatology.

Comparative effectiveness topics from a large, integrated delivery system.

Reviews on cutting-edge topics in toxicology.

Using text mining for study identification in systematic reviews: a systematic review of current approaches.

Comparative effectiveness research for molecular imaging.

Improving the uptake of systematic reviews: a systematic review of intervention effectiveness and relevance.

Surgical interventions for gastric cancer: a review of systematic reviews.

The comparative recall of Google Scholar versus PubMed in identical searches for biomedical systematic reviews: a review of searches used in systematic reviews.

Integration of existing systematic reviews into new reviews: identification of guidance needs.

Effectiveness of mHealth interventions for patients with diabetes: An overview of systematic reviews.

Evaluating the Effectiveness of Physical Exercise Interventions in Persons Living With HIV: Overview of Systematic Reviews.

Mental mechanisms for topics identification.

Examining the quality of evidence to support the effectiveness of interventions: an analysis of systematic reviews.

Literature reviews vs systematic reviews.

Cochrane systematic reviews in the field of addiction: past and future.

Salutogenically focused outcomes in systematic reviews of intrapartum interventions: a systematic review of systematic reviews.

Commentary on "critical appraisal of published systematic reviews assessing the cost-effectiveness of telemedicine studies".

Critical appraisal of published systematic reviews assessing the cost-effectiveness of telemedicine studies.