Research
Original Investigation
Identifying Otolaryngology Systematic Review Research Gaps Comparing Global Burden of Disease 2010 Results With Cochrane Database of Systematic Review Content Hannah Pederson, BS; Tyler Okland, BS; Lindsay N. Boyers, BA; Chante Karimkhani, BA; Richard M. Rosenfeld, MD, MPH; Mona Nasser, DDS, MSc; Sze Lin Yoong, PhD; Luke Wolfenden, PhD; Hmwe H. Kyu, MBBS, MPH, PhD; Peter T. Serina, BA; Megan Coggeshall, BA; Robert P. Dellavalle, MD, PhD, MSPH
IMPORTANCE Burden of disease should inform research prioritization.
Supplemental content at jamaotolaryngology.com
OBJECTIVE To determine whether systematic reviews and protocols published in the Cochrane Database of Systematic Reviews (CDSR) appropriately reflect disease burden for otolaryngologic conditions as measured by the Global Burden of Disease (GBD) 2010 project. DESIGN Two investigators independently assessed 10 otolaryngologic conditions in CDSR for systematic review and protocol representation from March to June 2014. The otolaryngologic diseases were matched to their respective GBD 2010 disability-adjusted life-years (DALYs) to assess their correlation. MAIN OUTCOMES AND MEASURES Relationship of CDSR representation (based on systematic reviews and protocols) with percentage of total 2010 DALYs, 2010 DALY rank, and DALY percentage change from 1990 to 2010 for 10 otolaryngologic conditions. RESULTS All 10 otolaryngologic conditions were represented by at least 1 systematic review in CDSR. The number of reviews and protocols in CDSR was well matched with GBD 2010 disability metrics for only 1 disease, mouth cancer. Upper respiratory infections, otitis media, thyroid cancer, and cleft lip and cleft palate were overrepresented in CDSR, and esophageal cancer, “other hearing loss,” nasopharynx cancer, larynx cancer, and “cancer of other part of pharynx and oropharynx” were underrepresented. CONCLUSIONS AND RELEVANCE The representation of otolaryngologic conditions in CDSR correlates poorly with DALY metrics. The results of this study may guide future research prioritization and allocation of funds. JAMA Otolaryngol Head Neck Surg. 2015;141(1):67-72. doi:10.1001/jamaoto.2014.2700 Published online October 30, 2014.
R
esearch priority setting is essential for efficient and equitable use of limited health resources.1 This is a challenging process because of the large number of stakeholders involved, the numerous methods available for prioritization, and the lack of consensus on which method is appropriate.1 Some factors that contribute to decision making regarding research funding and efforts include interest group advocacy, disease transmissibility, political support, infrastructure building, and portfolio diversification.2,3 The appropriate allocation of available resources is in the best interest of all those involved with the health system. This system includes policy makers, health professionals, and patients, with research and service provision funds coming from the same pool of scarce health resources.1,4 jamaotolaryngology.com
Author Affiliations: Author affiliations are listed at the end of this article. Corresponding Author: Robert P. Dellavalle, MD, PhD, MSPH, Dermatology Service, Department of Veteran Affairs Medical Center, 1055 Clermont St, PO Box 165, Denver, CO 80220 ([email protected]).
There are various efforts under way to help drive agendas and priorities within the field of otolaryngology. For instance, ENT UK5 launched the GENERATE project to raise awareness of the importance of otolaryngologic research and to partner with patients and professionals to prioritize future research. This project determines what questions people want answered through surveys, focus groups, and interviews. The information collected is used to prioritize, develop, and find funding for the top projects based on popular interest. In addition, the Cochrane Ear, Nose and Throat (ENT) Disorders Group6 has partnered with the James Lind Alliance7 to identify and prioritize the top 10 uncertainties about the effects of treatments that they agree are most important. This effort brings together patients and clinicians to drive research agen-
JAMA Otolaryngology–Head & Neck Surgery January 2015 Volume 141, Number 1
Copyright 2014 American Medical Association. All rights reserved.
Downloaded From: http://archotol.jamanetwork.com/ by a Georgetown University Medical Center User on 05/25/2015
67
Research Original Investigation
Identifying Otolaryngology Systematic Review Gaps
das based on the most pressing unanswered questions in the field. While both of these prioritization efforts take patient views and evidence gaps into account, neither necessarily reflects the impact of certain diseases on populations. Disease burden is one factor that can be used to steer research prioritization, among other factors such as financial cost. The Global Burden of Disease (GBD) 2010 project is a systematic, epidemiologic collaboration of more than 500 physicians and researchers across the world that quantifies global health disparities to elucidate disease patterns and trends.8-10 This project was funded by the Bill and Melinda Gates Foundation and was led by the Institute of Health Metrics and Evaluation (IHME). 8,9 The GBD 2010 project used disabilityadjusted life years (DALYs) to estimate disease burden for 291 diseases and injuries in 187 countries from 1990 to 2010.8-10 This measure is useful in analyzing mortality and morbidity across various diseases, time periods, and geographic regions.8,9 DALYs measure the years of life in perfect health lost and are the sum of the years of life lost (YLL) to premature mortality and the years lived with disability (YLD). 4,8,9 DALYs are comparable to quality-adjusted life years (QALYs), which are a measure of years of life in perfect health gained and do not incorporate an age-weighting function.11 While the concepts of QALYs and DALYs are similar, they are not interchangeable, since they are based on slightly different assumptions and are calculated by slightly different methods.11 The use of DALYs in the GBD study was aimed at supporting costeffectiveness on global scale; thus, a standard life expectancy was assumed to avoid disadvantaging populations with shorter life expectancies.11 The GBD 2010 project specifically analyzed 10 major otolaryngologic conditions on the basis of prevalence, common case definitions, and data availability. This investigation is part of a larger series intended to map all diseases studied by GBD 2010 to their representation in major research databases, including the Cochrane Database of Systematic Reviews (CDSR).12,13 The main aim of this study was to assess the representation of 10 otolaryngologic conditions studied by GBD 2010 within CDSR and determine whether representation corresponds to disease burden estimates for each condition.
Methods Selection of Conditions and Generation of Search Terms The following 10 otolaryngologic categories studied by GBD 2010 were assessed in this study: esophageal cancer, otitis media, mouth cancer, larynx cancer, nasopharynx cancer, upper respiratory infections, thyroid cancer, cleft lip and cleft palate, “cancer of other part of pharynx and oropharynx,” and “other hearing loss.” The 10 conditions analyzed in this study were chosen from the GBD cause list, which includes each of the conditions investigated by GBD 2010.14 Every condition on this list that is either diagnosed or treated primarily by otolaryngologists was selected for this study. The categories “other sense disorders” and “other chronic respiratory disease” were not included in this study because they contained conditions 68
outside the field of otolaryngology, and thus the disability measurements could not be exclusively attributed to otolaryngologic conditions. The International Classification of Diseases (ICD) codes that define the GBD categories have been previously published.14 Briefly, the ICD, 10th Revision (ICD-10) codes that define the 10 otolaryngologic conditions were used to generate search terms for the present analysis (eTable 1, eTable 2, and eTable 3 in the Supplement). The category cancer of other part of pharynx and oropharynx includes malignant neoplasms of tonsil, oropharynx, piriform sinus, and hypopharynx. The other hearing loss category includes congenital deafness, conductive hearing loss, sensorineural hearing loss, ototoxic hearing loss, presbycusis, and sudden idiopathic hearing loss. In addition, the upper respiratory infections category is defined to include only acute infections.
Collection of Data From CDSR This study compares CDSR research prioritization to GBD 2010 disease burden measurements. The CDSR comprises 53 editorial groups that produce systematic reviews on topics across medical specialties as well as public health and health systems issues.15 Systematic reviews adhere to a strict scientific design based on explicit prespecified and reproducible methods; therefore, they provide reliable estimates about the effects of interventions.16 Cochrane reviews investigate the effects of interventions for prevention, treatment, and rehabilitation in a health care setting. They are designed to facilitate the choices that physicians, patients, policy makers, and others face in health care while demonstrating where knowledge is lacking.16,17 Cochrane systematic reviews undergo extensive editorial processing, are more methodologically rigorous, and are updated more frequently than other reviews.18,19 Additionally, Cochrane protocols are electronically published by CDSR to outline proposals for future systematic reviews. Taken together, systematic reviews and protocols published in the Cochrane library provide a representative spectrum of research prioritization. The CDSR was interrogated by searching each otolaryngologic condition under “title, abstract, keywords.”20 Both systematic reviews and protocols were quantified in the CDSR with the exclusion of withdrawn titles. Each systematic review and protocol was matched to 1 of the 10 otolaryngologic conditions according to its subject content; they were classified under a category when that disease was a predominant focus of the abstract objectives and/or main results. If a review or protocol did not contain the search term or if the primary focus was determined to be outside the field of otolaryngology, it was not included in this study. If the review or protocol included the search term in the title, it was automatically included. If a particular review or protocol included more than 1 topic as its predominant focus, it was counted once for each respective topic. For example, a review focusing on both esophageal cancer and otitis media was counted twice. In addition, the particular Cochrane group that published each review and the date of online publication were determined. Two of us (H.P. and T.O.) performed independent data collection with consensus review from March to June 2014.
JAMA Otolaryngology–Head & Neck Surgery January 2015 Volume 141, Number 1
Copyright 2014 American Medical Association. All rights reserved.
Downloaded From: http://archotol.jamanetwork.com/ by a Georgetown University Medical Center User on 05/25/2015
jamaotolaryngology.com
Identifying Otolaryngology Systematic Review Gaps
Original Investigation Research
Figure 1. Comparison of Otolaryngology Representation in the Cochrane Database of Systematic Reviews With Percentage of Total 2010 DALYs
Cochrane Reviews and Protocols, No.
50
Other hearing loss Esophageal cancer Otitis media Mouth cancer Cancer of other part of pharynx and orophatynx Larynx cancer Nasopharynx cancer Upper respiratory infections Thyroid cancer Cleft lip and cleft palate
40
30
20
10
0 0
10
20
30
40
50
70
Total 2010 DALYs, %
Collection of Data From GBD 2010 The methods used by the GBD 2010 project to generate DALY estimates have been previously described.10 DALY metrics for each of the 10 otolaryngologic conditions, expressed as percentage of total DALYs of all 291 conditions measured in GBD 2010, were obtained from the GBD Compare interactive time plot.21 DALY rankings in 2010 for the 10 otolaryngologic conditions compared with 176 conditions were obtained from the GBD interactive arrow diagram.22 The GBD project studied 291 conditions, but DALY rankings include only 176 conditions because conditions for which the project made no explicit estimates were excluded. Trends in disability were determined via median percentage change of global DALYs from 1990 to 2010 for each otolaryngologic condition from the GBD interactive arrow diagram.22
Comparison of CDSR Representation With GBD Data The CDSR representation was compared with GBD 2010 disability estimates by calculating the “ideal” number of reviews and protocols for each disease given its DALY. This was accomplished by reallocating the total number of reviews and protocols for the 10 otolaryngologic conditions to the various diseases based on their relative DALY metrics. The DALY metrics for the 10 conditions were normalized by summing the percentage of total 2010 DALYs for each condition and then dividing the DALY percentage for each condition by this total. In this manner, the percentage of total otolaryngologic DALYs adds to 100% for the 10 conditions studied. The number of reviews and protocols was then reallocated by multiplying the percentage of total otolaryngologic DALYs by the total number of reviews and protocols. Using this method, a plot of the ideal number of reviews and protocols vs percentage of total DALYs (nonnormalized) yields a straight line (Figure 1). The actual number of reviews and protocols for each disease was then compared qualitatively to this ideal line to assess overrepresentation and underrepresentation in CDSR.
Results A total of 116 systematic reviews and protocols from multiple Cochrane groups represented the 10 otolaryngologic condijamaotolaryngology.com
The line represents a calculation of the “ideal” number of systematic reviews and protocols based on normalizing the DALY metrics for the 10 otolaryngologic conditions and reallocating the total number of reviews and protocols to reflect the normalized DALY metrics. DALY indicates disability-adjusted life-years.
tions (Table; eTable 4 and eTable 5 in the Supplement). A majority (39.7%) of the reviews and protocols assessed were classified into the upper respiratory infections category. Otitis media had the next greatest representation in the database at 19.8%. The diseases with intermediate CDSR representation were esophageal cancer (9.5%), other hearing loss (6.9%), mouth cancer (6.9%), cleft lip and cleft palate (6.0%), and thyroid cancer (5.2%). The diseases with the least representation in CDSR were cancer of other part of pharynx and oropharynx (2.6%), nasopharynx cancer (2.6%), and larynx cancer (0.9%). The percentage of total 2010 DALYs was found for each of the 10 otolaryngologic diseases studied (of the 291 conditions evaluated by GBD 2010) (Table). Other hearing loss had the greatest percentage of total DALYs at 0.64%, and cleft lip and cleft palate had the smallest percentage of total DALYs at 0.02%. The median percentage of DALYs for each otolaryngologic condition experienced a variable decrease from 1990 to 2010 (range, −6% to −29%). Disability for esophageal cancer and larynx cancer decreased the most (−29% and −28%, respectively). Thyroid cancer and mouth cancer experienced more modest decreases in DALYs (−6% and −9%, respectively). The 2010 DALY rank of 176 conditions studied by GBD 2010 was found for each of the 10 otolaryngologic conditions as an additional measure of disease burden relative to diseases in all fields of medicine. Other hearing loss had the highest rank (40th of 176), and thyroid cancer had the lowest rank (148th of 176). The GBD DALY rank data and median percentage change of DALYs was unavailable for cleft lip and cleft palate. For each systematic review and protocol examined in this study, the particular Cochrane group that published the article and the date of online publication were determined. The Acute Respiratory Infections Group contributed 41.4% of the reviews and protocols, and the ENT Disorders Group contributed 29.3%. Of the 116 total systematic reviews and protocols included in this study, 78% were published after the year 2010, and 35% were published after the year 2013. Review and protocol representation were compared with GBD 2010 DALY metrics to determine the correlation between disability and research allocation. Only 1 disease, mouth cancer, correlated well. Upper respiratory infections and otitis me-
JAMA Otolaryngology–Head & Neck Surgery January 2015 Volume 141, Number 1
Copyright 2014 American Medical Association. All rights reserved.
Downloaded From: http://archotol.jamanetwork.com/ by a Georgetown University Medical Center User on 05/25/2015
69
Research Original Investigation
Identifying Otolaryngology Systematic Review Gaps
Table. CDSR Representations and DALY Metrics for 10 Otolaryngologic Conditions Studied by the GBD 2010 Projecta CDSR Representations, No. Condition
Total
Group Contributions (SRs + PRs)
20-Year Change (Median), %c
2010 Rankd
8 (5 SRs, 3 PRs)
ENT Disorders (n = 8)
0.640
−15
40
Esophageal cancer
11 (8 SRs, 3 PRs)
Upper Gastrointestinal and Pancreatic Diseases (n = 11)
0.360
−29
55
Otitis media
23 (21 SRs, 2 PRs)
ENT Disorders (n = 14); Acute Respiratory Infections (n = 9)
0.190
−10
79
Mouth cancer
8 (6 SRs, 2 PRs)
Oral Health (n = 8)
0.130
−9
94
Cancer of other part of pharynx and oropharynx
3 (1 SRs, 2 PRs)
ENT Disorders (n = 3)
0.110
−17
105
Larynx cancer
1 (1 SR)
ENT Disorders (n = 1)
0.095
−28
113
Nasopharynx cancer
3 (1 SRs, 2 PRs)
ENT Disorders (n = 3)
0.081
−12
120
Acute Respiratory Infections (n = 39); ENT Disorders (n = 5); Anaesthesia (n = 1); Airways (n = 1)
0.075
−11
122
46 (41 SRs, 5 PRs)
Thyroid cancer
6 (4 SRs, 2 PRs)
Metabolic and Endocrine Disorders (n = 6)
0.034
−6
148
Cleft lip and cleft palate
7 (4 SRs, 3 PRs)
Oral Health (n = 5); Developmental Psychosocial and Learning Problems (n = 1); Pain, Palliative and Supportive Care (n = 1)
0.023
NA
NA
Abbreviations: CDSR, Cochrane Database of Systemic Reviews; DALY, disability-adjusted life-years; ENT, ear, nose, and throat; GBD, Global Burden of Disease8-10; PR, protocol; SR, systematic review. Otolaryngologic conditions are listed from top to bottom in decreasing percentages of 2010 DALYs.
dia were overrepresented in CDSR when compared with their relatively low percentage of total DALYs. In addition, thyroid cancer and cleft lip and cleft palate were slightly overrepresented. Conversely, esophageal cancer and other hearing loss were substantially underrepresented in CDSR relative to their higher DALYs. Finally, qualitative analysis showed that nasopharynx cancer, larynx cancer, and cancer of other part of pharynx and oropharynx were slightly underrepresented.
Discussion A unique feature of the Cochrane Collaboration is the creation of individual editorial review groups responsible for specialized topic coverage.23 Several of the diseases investigated in this study were covered solely by the Cochrane ENT Disorders Group, including larynx cancer, nasopharynx cancer, cancer of other part of pharynx and oropharynx, and other hearing loss. Other otolaryngologic conditions are covered by multiple Cochrane groups; for example, upper respiratory infections received contributions from Acute Respiratory Infections, ENT Disorders, Anaesthesia, and Airways Cochrane review groups. The multiplicity of groups contributing to this disease may impact its overrepresentation in CDSR compared with its associated disability. While a specific Cochrane group for ENT Disorders exists, it did not contribute the majority of systematic reviews and protocols for the 10 otolaryngologic conditions analyzed in this study. The ENT Disorders Group contributed 29.3% of the reviews and protocols, whereas the Acute Respiratory Infections Group contributed 41.4%. Contributions from 10 different Cochrane review groups were found for the conditions 70
Proportion, %b
Other hearing loss
Upper respiratory infections
a
DALY Metrics
b
Of 291 conditions evaluated by GBD 2010.
c
From 1990 to 2010.
d
Of 176 conditions evaluated by the GBD project.
studied, indicating that otolaryngologic disorders have widereaching implications in numerous fields of medicine. In addition, of the 116 total systematic reviews and protocols included in this study, 78% were published after the year 2010, and 35% were published after the year 2013. This highlights the efforts by CDSR to create a frequently updated and current source of medical information. Of the 10 otolaryngologic conditions studied, mouth cancer was the only condition found to have proportionate CDSR representation relative to its DALY metrics. The remaining 9 diseases were either overrepresented or underrepresented in the database.
Conditions Overrepresented in CDSR Compared With GBD 2010 DALYs One of the more dramatic findings of this study was the overrepresentation of upper respiratory infections in CDSR. This category was represented by more reviews and protocols than any other category (41 reviews, 5 protocols) but had the third lowest percentage of total 2010 DALYs of the 10 otolaryngologic conditions assessed. This might reflect the fact that the upper respiratory infections category encompasses many different conditions. Similarly, otitis media was overrepresented, with more systematic reviews and protocols than other hearing loss and esophageal cancer combined, which had the 2 highest DALYs of the conditions assessed. This overrepresentation most likely reflects the high prevalence of these conditions and greater availability of randomized clinical trials (RCTs) for acute conditions compared with chronic diseases or surgical interventions. Alternatively, the overrepresentation could reflect disparities in funding and/or research between high-income and low-income countries.
JAMA Otolaryngology–Head & Neck Surgery January 2015 Volume 141, Number 1
Copyright 2014 American Medical Association. All rights reserved.
Downloaded From: http://archotol.jamanetwork.com/ by a Georgetown University Medical Center User on 05/25/2015
jamaotolaryngology.com
Identifying Otolaryngology Systematic Review Gaps
Original Investigation Research
Two conditions were slightly overrepresented in CDSR: thyroid cancer and cleft lip and cleft palate. These conditions had the lowest percentages of total DALYs of the conditions examined, but they did not receive the lowest number of reviews and protocols, as would be expected given their associated disability. The ENT Disorders Group did not contribute any reviews or protocols to these conditions. Of note, diseases of the thyroid and parathyroid glands (benign and malignant) became the responsibility of the Metabolic and Endocrine Disorders Group following their registration in February 2000.24
Conditions Underrepresented in CDSR Compared With GBD 2010 DALYs The most significantly underrepresented category in CDSR was other hearing loss. This condition was ranked first of the 10 otolaryngologic conditions according to DALY metrics, yet it received only 5 reviews and 3 protocols, all from the ENT Disorders Group. This category includes many different conditions, from congenital deafness to ototoxic hearing loss and is the most disabling of the diseases investigated in this study. Esophageal cancer, which is ranked second of the 10 otolaryngologic conditions in terms of disability, was also underrepresented in CDSR. Of note, the Upper Gastrointestinal and Pancreatic Diseases Group contributed all of the 11 systematic reviews and protocols for esophageal cancer, with no contributions from the ENT Disorders Group. Additionally, this disease experienced the greatest decrease in DALY percentage from 1990 to 2010 of the 10 diseases studied; thus, its underrepresentation may be appropriate for its decreasing disability over the last 20 years. Finally, nasopharynx cancer, larynx cancer, and cancer of other part of pharynx and oropharynx are slightly underrepresented in CDSR for their associated disabilities. Together, these 3 categories had a total of only 7 reviews and protocols, all of which came from the ENT Disorders Group. Larynx cancer had the lowest CDSR representation, with only 1 systematic review. This disease also had the second greatest median percentage change in DALYs from 1990 to 2010 (−28%). Similar to esophageal cancer, the low representation of larynx cancer may be appropriate given its decrease in DALYs over the 20-year time span.
Limitations and Future Directions We acknowledge several limitations of our study as well as areas for further investigation. To begin, the study is limited by the bounds of the field of otolaryngology. For example, esophageal cancer is a disease often diagnosed by otolaryngologists, but it is typically treated outside of the specialty by a multidisciplinary team.25 Therefore, it may be inaccurate to compare CDSR representation of esophageal cancer with other otolaryngologic disorders. Systematic reviews published by CDSR generally synthesize the best evidence in the published literature while adhering to previously specified eligibility criteria. Therefore, metaanalyses often cannot be performed for topics without an adequate base of published RCTs on similar interventions to justify data synthesis.26 Additionally, the scope of reviews is variable. For instance, authors may prepare a large review of jamaotolaryngology.com
multiple interventions (“lumping”) or several reviews of individual interventions (“splitting”). Thus, the consideration of a systematic review or protocol as a unit of representation may not be appropriate for every topic. An additional limitation of this study is that equal weight was given to all systematic reviews and protocols regardless of the number of studies included in each of them. For example, one review might contain 10 RCTs while another contained 100, but the reviews were counted as equals. In future reports, a weighted comparison could be used to further quantify the research being conducted on certain otolaryngologic conditions. While other means to quantify research prioritization may exist, this study used only the number of published reviews and protocols as a surrogate marker of research prioritization. Limitations also exist in database interrogation. A standardized search term technique and independent data collection by 2 authors were used to minimize this source of error.13 Databases other than CDSR can be used for future data collection. The Database of Abstracts of Reviews of Effects (DARE)27 contains details of systematic reviews that evaluate the effects of health care interventions, the delivery and organization of health services, and the wider determinants of health. This database complements CDSR by identifying and including systematic reviews that have not been conducted by the Cochrane Collaboration. It can therefore be used to expand future searches in determining database representation of certain diseases. The present study is also subject to the limitations of the GBD 2010 project itself. The number of conditions in this analysis was confined to those evaluated and categorized by the GBD 2010. Several conditions, such as vertigo, could not be included, since they were not included as separate categories in GBD 2010. As the outcomes of this study depend on which diseases were evaluated, there are limitations inherent in not including several highly prevalent and morbid chronic diseases. As future GBD data become available on an annual basis, the relationship between database representation and GBD disability metrics can be explored further by including additional otolaryngologic conditions to inform future research prioritization.28 A final limitation of this study is the use of a global perspective to drive decisions on a country-specific level. Health priorities vary substantially from country to country and DALYs can vary depending on current infrastructure of health care delivery. Many competing factors need to be considered in guiding research prioritization, including disease prevalence, direct and indirect costs of disease, quality of current health care delivery systems, and value of current delivery of care systems. Future investigations could include countryspecific data to inform prioritization decisions.
Conclusions Research prioritization should be guided in part by the morbidity and mortality of diseases, yet there is currently a lack of transparency in many databases regarding the availability and quality of data and criteria that inform these critical decisions. Global
JAMA Otolaryngology–Head & Neck Surgery January 2015 Volume 141, Number 1
Copyright 2014 American Medical Association. All rights reserved.
Downloaded From: http://archotol.jamanetwork.com/ by a Georgetown University Medical Center User on 05/25/2015
71
Research Original Investigation
Identifying Otolaryngology Systematic Review Gaps
burden of disease may involve any of a number of factors that impact research prioritization by CDSR. Cochrane reviews are often driven by availability of RCTs and diagnostic test studies, neither of which necessarily reflects the importance of a condition under study, especially in terms of global health or World Health Organization priorities. Therefore, it is not surprising that Cochrane reviews represent a relatively small percentage of overall GBD DALYs for otolaryngology.
ARTICLE INFORMATION Submitted for Publication: August 2, 2014; final revision received August 25, 2014; accepted September 9, 2014. Published Online: October 30, 2014. doi:10.1001/jamaoto.2014.2700. Author Affiliations: University of Colorado School of Medicine, Aurora (Pederson, Okland); Georgetown University School of Medicine, Washington, DC (Boyers); Columbia University College of Physicians and Surgeons, New York, New York (Karimkhani); Department of Otolaryngology, State University of New York Downstate Medical Center, Brooklyn, New York (Rosenfeld); Peninsula Dental School, Plymouth University, Plymouth, England (Nasser); Hunter New England Population Health, Wallsend, New South Wales, Australia (Yoong); School of Medicine and Public Health, the University of Newcastle, Newcastle, New South Wales, Australia (Yoong, Wolfenden); Institute for Health Metrics and Evaluation, University of Washington, Seattle (Kyu, Serina, Coggeshall); Department of Dermatology, University of Colorado Anschutz Medical Campus, Aurora (Dellavalle); Department of Epidemiology, Colorado School of Public Health, Aurora (Dellavalle); Dermatology Service, US Department of Veterans Affairs Medical Center, Denver, Colorado (Dellavalle). Author Contributions: Ms Pederson and Dr Dellavalle had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Boyers, Karimkhani, Nasser, Yoong, Dellavalle. Acquisition, analysis, or interpretation of data: Pederson, Okland, Boyers, Karimkhani, Rosenfeld, Wolfenden, Kyu, Serina, Coggeshall, Dellavalle. Drafting of the manuscript: Pederson, Rosenfeld, Wolfenden, Coggeshall. Critical revision of the manuscript for important intellectual content: Okland, Boyers, Karimkhani, Nasser, Yoong, Wolfenden, Kyu, Serina, Dellavalle. Administrative, technical, or material support: Pederson, Boyers, Karimkhani, Yoong, Coggeshall, Dellavalle. Study supervision: Boyers, Karimkhani, Rosenfeld, Nasser, Dellavalle. Conflict of Interest Disclosures: None reported. Funding/Support: Dr Dellavalle receives a salary from the US Department of Veterans Affairs (VA) and is also supported by grants from the Centers for Disease Control and Prevention (CDC) and National Institutes of Health (NIH). Luke Wolfenden receives funding from the National Health and Medical Research Council, and Hunter New England Local Health District. All other authors have no relevant funding/support.
72
The present study shows that systematic reviews and protocols can be mapped against burden of disease to guide prioritization of future research. We present data for 10 otolaryngologic conditions to identify research gaps and inform future prioritization decisions. As new data and scientific models provide an unprecedented understanding of the global burden of disease, technology allows us minimize research disparities and advance human health in the 21st century.
Role of the Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Cochrane database of systematic reviews. JAMA Dermatol. 2014;150(9):945-951.
Disclaimer: Any opinions expressed herein do not necessarily reflect the opinions of the VA, CDC, or NIH.
15. The Cochrane Collaboration. About the Cochrane Library. http://www.thecochranelibrary .com/view/0/AboutTheCochraneLibrary .html#CDSR. Accessed June 23, 2014.
Additional Information: Ms Boyers and Dr Dellavalle are both employees for the Department of Veterans Affairs, and Dr Dellavalle is also chair of the Colorado Skin Cancer Task Force. REFERENCES 1. Fleurence RL, Torgerson DJ. Setting priorities for research. Health Policy. 2004;69(1):1-10. 2. Gross CP, Anderson GF, Powe NR. The relation between funding by the National Institutes of Health and the burden of disease. N Engl J Med. 1999;340(24):1881-1887. 3. Gillum LA, Gouveia C, Dorsey ER, et al. NIH disease funding levels and burden of disease. PLoS One. 2011;6(2):e16837. 4. Michaud CM, Murray CJ, Bloom BR. Burden of disease. JAMA. 2001;285(5):535-539. 5. ENT UK. Working together to develop research. https://www.entuk.org/professionals/generate. Accessed July 19, 2014.
14. Lozano R, Naghavi M, Foreman K, et al. Global and regional mortality from 235 causes of death for 20 age groups in 1990 and 2010. Lancet. 2012; 380(9859):2095-2128.
16. Norman I, Griffiths P. The rise and rise of the systematic review. Int J Nurs Stud. 2014;51(1):1-3. 17. The Cochrane Collaboration. Cochrane ENT Group: our reviews. http://ent.cochrane.org/our -reviews. Accessed July 19, 2014. 18. Jadad AR, Cook DJ, Jones A, et al. Methodology and reports of systematic reviews and meta-analyses. JAMA. 1998;280(3):278-280. 19. Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007;4(3):e78. 20. The Cochrane Collaboration. Search the Cochrane library. http://www.thecochranelibrary .com/view/0/index.html. Accessed June 1, 2014. 21. Institute for Health Metrics and Evaluation. GBD compare. http://viz.healthmetricsandevaluation .org/gbd-compare/. Accessed June 1, 2014.
6. The Cochrane Collaboration. Prioritising review topics. http://ent.cochrane.org/prioritising-review -topics. Accessed July 10, 2014.
22. Institute for Health Metrics and Evaluation. GBD arrow diagram. http://www .healthmetricsandevaluation.org/gbd/visualization /gbd-arrow-diagram. Accessed June 1, 2014.
7. The James Lind Alliance. ENT Aspects of Balance Priority Setting Partnership Top 10. http://www .lindalliance.org/top-tens.asp. Accessed July 19, 2014.
23. Dickersin K, Manheimer E; The Cochrane Collaboration. The Cochrane Collaboration. Clin Obstet Gynecol. 1998;41(2):315-331.
8. Vos T, Flaxman AD, Naghavi M, et al. Years lived with disability (YLDs) for 1160 sequelae of 289 diseases and injuries 1990-2010. Lancet. 2012;380 (9859):2163-2196.
24. The Cochrane Collaboration. Cochrane ENT Group: scope of our work. http://ent.cochrane.org /scope-our-work. Accessed June 15, 2014.
9. Murray CJ, Vos T, Lozano R, et al. Disability-adjusted life years (DALYs) for 291 diseases and injuries in 21 regions, 1990-2010. Lancet. 2012;380(9859):2197-2223. 10. Murray CJ, Ezzati M, Flaxman AD, et al. GBD 2010: design, definitions, and metrics. Lancet. 2012;380(9859):2063-2066. 11. Sassi F. Calculating QALYs, comparing QALY and DALY calculations. Health Policy Plan. 2006;21(5): 402-408. 12. Cochrane Agenda and Priority Setting Methods Group. Global Burden of Disease (GBD). http: //capsmg.cochrane.org/global-burden-disease-gbd -cochrane-project. Accessed July 19, 2014. 13. Karimkhani C, Boyers LN, Prescott L, et al. Global burden of skin disease as reflected in
25. Hoppo T, Jobe BA. Personalizing therapy for esophageal cancer patients. Thorac Surg Clin. 2013; 23(4):471-478. 26. The Cochrane Collaboration. Cochrane Handbook for Systematic Reviews of Interventions. http://handbook.cochrane.org. Accessed June 1, 2014. 27. The Cochrane Collaboration. Database of Abstracts of Reviews of Effects (DARE). http://www .cochrane.org/editorial-and-publishing-policy -resource/database-abstracts-reviews-effects-dare. Accessed July 23, 2014. 28. Institute for Health Metrics and Evaluation. Global Burden of Diseases, Injuries, and Risk Factors Study 2013 (GBD 2013). http://www .healthmetricsandevaluation.org/gbd/2013. Accessed June 15, 2014.
JAMA Otolaryngology–Head & Neck Surgery January 2015 Volume 141, Number 1
Copyright 2014 American Medical Association. All rights reserved.
Downloaded From: http://archotol.jamanetwork.com/ by a Georgetown University Medical Center User on 05/25/2015
jamaotolaryngology.com
Original Investigation
Identifying Otolaryngology Systematic Review Research Gaps Comparing Global Burden of Disease 2010 Results With Cochrane Database of Systematic Review Content Hannah Pederson, BS; Tyler Okland, BS; Lindsay N. Boyers, BA; Chante Karimkhani, BA; Richard M. Rosenfeld, MD, MPH; Mona Nasser, DDS, MSc; Sze Lin Yoong, PhD; Luke Wolfenden, PhD; Hmwe H. Kyu, MBBS, MPH, PhD; Peter T. Serina, BA; Megan Coggeshall, BA; Robert P. Dellavalle, MD, PhD, MSPH
IMPORTANCE Burden of disease should inform research prioritization.
Supplemental content at jamaotolaryngology.com
OBJECTIVE To determine whether systematic reviews and protocols published in the Cochrane Database of Systematic Reviews (CDSR) appropriately reflect disease burden for otolaryngologic conditions as measured by the Global Burden of Disease (GBD) 2010 project. DESIGN Two investigators independently assessed 10 otolaryngologic conditions in CDSR for systematic review and protocol representation from March to June 2014. The otolaryngologic diseases were matched to their respective GBD 2010 disability-adjusted life-years (DALYs) to assess their correlation. MAIN OUTCOMES AND MEASURES Relationship of CDSR representation (based on systematic reviews and protocols) with percentage of total 2010 DALYs, 2010 DALY rank, and DALY percentage change from 1990 to 2010 for 10 otolaryngologic conditions. RESULTS All 10 otolaryngologic conditions were represented by at least 1 systematic review in CDSR. The number of reviews and protocols in CDSR was well matched with GBD 2010 disability metrics for only 1 disease, mouth cancer. Upper respiratory infections, otitis media, thyroid cancer, and cleft lip and cleft palate were overrepresented in CDSR, and esophageal cancer, “other hearing loss,” nasopharynx cancer, larynx cancer, and “cancer of other part of pharynx and oropharynx” were underrepresented. CONCLUSIONS AND RELEVANCE The representation of otolaryngologic conditions in CDSR correlates poorly with DALY metrics. The results of this study may guide future research prioritization and allocation of funds. JAMA Otolaryngol Head Neck Surg. 2015;141(1):67-72. doi:10.1001/jamaoto.2014.2700 Published online October 30, 2014.
R
esearch priority setting is essential for efficient and equitable use of limited health resources.1 This is a challenging process because of the large number of stakeholders involved, the numerous methods available for prioritization, and the lack of consensus on which method is appropriate.1 Some factors that contribute to decision making regarding research funding and efforts include interest group advocacy, disease transmissibility, political support, infrastructure building, and portfolio diversification.2,3 The appropriate allocation of available resources is in the best interest of all those involved with the health system. This system includes policy makers, health professionals, and patients, with research and service provision funds coming from the same pool of scarce health resources.1,4 jamaotolaryngology.com
Author Affiliations: Author affiliations are listed at the end of this article. Corresponding Author: Robert P. Dellavalle, MD, PhD, MSPH, Dermatology Service, Department of Veteran Affairs Medical Center, 1055 Clermont St, PO Box 165, Denver, CO 80220 ([email protected]).
There are various efforts under way to help drive agendas and priorities within the field of otolaryngology. For instance, ENT UK5 launched the GENERATE project to raise awareness of the importance of otolaryngologic research and to partner with patients and professionals to prioritize future research. This project determines what questions people want answered through surveys, focus groups, and interviews. The information collected is used to prioritize, develop, and find funding for the top projects based on popular interest. In addition, the Cochrane Ear, Nose and Throat (ENT) Disorders Group6 has partnered with the James Lind Alliance7 to identify and prioritize the top 10 uncertainties about the effects of treatments that they agree are most important. This effort brings together patients and clinicians to drive research agen-
JAMA Otolaryngology–Head & Neck Surgery January 2015 Volume 141, Number 1
Copyright 2014 American Medical Association. All rights reserved.
Downloaded From: http://archotol.jamanetwork.com/ by a Georgetown University Medical Center User on 05/25/2015
67
Research Original Investigation
Identifying Otolaryngology Systematic Review Gaps
das based on the most pressing unanswered questions in the field. While both of these prioritization efforts take patient views and evidence gaps into account, neither necessarily reflects the impact of certain diseases on populations. Disease burden is one factor that can be used to steer research prioritization, among other factors such as financial cost. The Global Burden of Disease (GBD) 2010 project is a systematic, epidemiologic collaboration of more than 500 physicians and researchers across the world that quantifies global health disparities to elucidate disease patterns and trends.8-10 This project was funded by the Bill and Melinda Gates Foundation and was led by the Institute of Health Metrics and Evaluation (IHME). 8,9 The GBD 2010 project used disabilityadjusted life years (DALYs) to estimate disease burden for 291 diseases and injuries in 187 countries from 1990 to 2010.8-10 This measure is useful in analyzing mortality and morbidity across various diseases, time periods, and geographic regions.8,9 DALYs measure the years of life in perfect health lost and are the sum of the years of life lost (YLL) to premature mortality and the years lived with disability (YLD). 4,8,9 DALYs are comparable to quality-adjusted life years (QALYs), which are a measure of years of life in perfect health gained and do not incorporate an age-weighting function.11 While the concepts of QALYs and DALYs are similar, they are not interchangeable, since they are based on slightly different assumptions and are calculated by slightly different methods.11 The use of DALYs in the GBD study was aimed at supporting costeffectiveness on global scale; thus, a standard life expectancy was assumed to avoid disadvantaging populations with shorter life expectancies.11 The GBD 2010 project specifically analyzed 10 major otolaryngologic conditions on the basis of prevalence, common case definitions, and data availability. This investigation is part of a larger series intended to map all diseases studied by GBD 2010 to their representation in major research databases, including the Cochrane Database of Systematic Reviews (CDSR).12,13 The main aim of this study was to assess the representation of 10 otolaryngologic conditions studied by GBD 2010 within CDSR and determine whether representation corresponds to disease burden estimates for each condition.
Methods Selection of Conditions and Generation of Search Terms The following 10 otolaryngologic categories studied by GBD 2010 were assessed in this study: esophageal cancer, otitis media, mouth cancer, larynx cancer, nasopharynx cancer, upper respiratory infections, thyroid cancer, cleft lip and cleft palate, “cancer of other part of pharynx and oropharynx,” and “other hearing loss.” The 10 conditions analyzed in this study were chosen from the GBD cause list, which includes each of the conditions investigated by GBD 2010.14 Every condition on this list that is either diagnosed or treated primarily by otolaryngologists was selected for this study. The categories “other sense disorders” and “other chronic respiratory disease” were not included in this study because they contained conditions 68
outside the field of otolaryngology, and thus the disability measurements could not be exclusively attributed to otolaryngologic conditions. The International Classification of Diseases (ICD) codes that define the GBD categories have been previously published.14 Briefly, the ICD, 10th Revision (ICD-10) codes that define the 10 otolaryngologic conditions were used to generate search terms for the present analysis (eTable 1, eTable 2, and eTable 3 in the Supplement). The category cancer of other part of pharynx and oropharynx includes malignant neoplasms of tonsil, oropharynx, piriform sinus, and hypopharynx. The other hearing loss category includes congenital deafness, conductive hearing loss, sensorineural hearing loss, ototoxic hearing loss, presbycusis, and sudden idiopathic hearing loss. In addition, the upper respiratory infections category is defined to include only acute infections.
Collection of Data From CDSR This study compares CDSR research prioritization to GBD 2010 disease burden measurements. The CDSR comprises 53 editorial groups that produce systematic reviews on topics across medical specialties as well as public health and health systems issues.15 Systematic reviews adhere to a strict scientific design based on explicit prespecified and reproducible methods; therefore, they provide reliable estimates about the effects of interventions.16 Cochrane reviews investigate the effects of interventions for prevention, treatment, and rehabilitation in a health care setting. They are designed to facilitate the choices that physicians, patients, policy makers, and others face in health care while demonstrating where knowledge is lacking.16,17 Cochrane systematic reviews undergo extensive editorial processing, are more methodologically rigorous, and are updated more frequently than other reviews.18,19 Additionally, Cochrane protocols are electronically published by CDSR to outline proposals for future systematic reviews. Taken together, systematic reviews and protocols published in the Cochrane library provide a representative spectrum of research prioritization. The CDSR was interrogated by searching each otolaryngologic condition under “title, abstract, keywords.”20 Both systematic reviews and protocols were quantified in the CDSR with the exclusion of withdrawn titles. Each systematic review and protocol was matched to 1 of the 10 otolaryngologic conditions according to its subject content; they were classified under a category when that disease was a predominant focus of the abstract objectives and/or main results. If a review or protocol did not contain the search term or if the primary focus was determined to be outside the field of otolaryngology, it was not included in this study. If the review or protocol included the search term in the title, it was automatically included. If a particular review or protocol included more than 1 topic as its predominant focus, it was counted once for each respective topic. For example, a review focusing on both esophageal cancer and otitis media was counted twice. In addition, the particular Cochrane group that published each review and the date of online publication were determined. Two of us (H.P. and T.O.) performed independent data collection with consensus review from March to June 2014.
JAMA Otolaryngology–Head & Neck Surgery January 2015 Volume 141, Number 1
Copyright 2014 American Medical Association. All rights reserved.
Downloaded From: http://archotol.jamanetwork.com/ by a Georgetown University Medical Center User on 05/25/2015
jamaotolaryngology.com
Identifying Otolaryngology Systematic Review Gaps
Original Investigation Research
Figure 1. Comparison of Otolaryngology Representation in the Cochrane Database of Systematic Reviews With Percentage of Total 2010 DALYs
Cochrane Reviews and Protocols, No.
50
Other hearing loss Esophageal cancer Otitis media Mouth cancer Cancer of other part of pharynx and orophatynx Larynx cancer Nasopharynx cancer Upper respiratory infections Thyroid cancer Cleft lip and cleft palate
40
30
20
10
0 0
10
20
30
40
50
70
Total 2010 DALYs, %
Collection of Data From GBD 2010 The methods used by the GBD 2010 project to generate DALY estimates have been previously described.10 DALY metrics for each of the 10 otolaryngologic conditions, expressed as percentage of total DALYs of all 291 conditions measured in GBD 2010, were obtained from the GBD Compare interactive time plot.21 DALY rankings in 2010 for the 10 otolaryngologic conditions compared with 176 conditions were obtained from the GBD interactive arrow diagram.22 The GBD project studied 291 conditions, but DALY rankings include only 176 conditions because conditions for which the project made no explicit estimates were excluded. Trends in disability were determined via median percentage change of global DALYs from 1990 to 2010 for each otolaryngologic condition from the GBD interactive arrow diagram.22
Comparison of CDSR Representation With GBD Data The CDSR representation was compared with GBD 2010 disability estimates by calculating the “ideal” number of reviews and protocols for each disease given its DALY. This was accomplished by reallocating the total number of reviews and protocols for the 10 otolaryngologic conditions to the various diseases based on their relative DALY metrics. The DALY metrics for the 10 conditions were normalized by summing the percentage of total 2010 DALYs for each condition and then dividing the DALY percentage for each condition by this total. In this manner, the percentage of total otolaryngologic DALYs adds to 100% for the 10 conditions studied. The number of reviews and protocols was then reallocated by multiplying the percentage of total otolaryngologic DALYs by the total number of reviews and protocols. Using this method, a plot of the ideal number of reviews and protocols vs percentage of total DALYs (nonnormalized) yields a straight line (Figure 1). The actual number of reviews and protocols for each disease was then compared qualitatively to this ideal line to assess overrepresentation and underrepresentation in CDSR.
Results A total of 116 systematic reviews and protocols from multiple Cochrane groups represented the 10 otolaryngologic condijamaotolaryngology.com
The line represents a calculation of the “ideal” number of systematic reviews and protocols based on normalizing the DALY metrics for the 10 otolaryngologic conditions and reallocating the total number of reviews and protocols to reflect the normalized DALY metrics. DALY indicates disability-adjusted life-years.
tions (Table; eTable 4 and eTable 5 in the Supplement). A majority (39.7%) of the reviews and protocols assessed were classified into the upper respiratory infections category. Otitis media had the next greatest representation in the database at 19.8%. The diseases with intermediate CDSR representation were esophageal cancer (9.5%), other hearing loss (6.9%), mouth cancer (6.9%), cleft lip and cleft palate (6.0%), and thyroid cancer (5.2%). The diseases with the least representation in CDSR were cancer of other part of pharynx and oropharynx (2.6%), nasopharynx cancer (2.6%), and larynx cancer (0.9%). The percentage of total 2010 DALYs was found for each of the 10 otolaryngologic diseases studied (of the 291 conditions evaluated by GBD 2010) (Table). Other hearing loss had the greatest percentage of total DALYs at 0.64%, and cleft lip and cleft palate had the smallest percentage of total DALYs at 0.02%. The median percentage of DALYs for each otolaryngologic condition experienced a variable decrease from 1990 to 2010 (range, −6% to −29%). Disability for esophageal cancer and larynx cancer decreased the most (−29% and −28%, respectively). Thyroid cancer and mouth cancer experienced more modest decreases in DALYs (−6% and −9%, respectively). The 2010 DALY rank of 176 conditions studied by GBD 2010 was found for each of the 10 otolaryngologic conditions as an additional measure of disease burden relative to diseases in all fields of medicine. Other hearing loss had the highest rank (40th of 176), and thyroid cancer had the lowest rank (148th of 176). The GBD DALY rank data and median percentage change of DALYs was unavailable for cleft lip and cleft palate. For each systematic review and protocol examined in this study, the particular Cochrane group that published the article and the date of online publication were determined. The Acute Respiratory Infections Group contributed 41.4% of the reviews and protocols, and the ENT Disorders Group contributed 29.3%. Of the 116 total systematic reviews and protocols included in this study, 78% were published after the year 2010, and 35% were published after the year 2013. Review and protocol representation were compared with GBD 2010 DALY metrics to determine the correlation between disability and research allocation. Only 1 disease, mouth cancer, correlated well. Upper respiratory infections and otitis me-
JAMA Otolaryngology–Head & Neck Surgery January 2015 Volume 141, Number 1
Copyright 2014 American Medical Association. All rights reserved.
Downloaded From: http://archotol.jamanetwork.com/ by a Georgetown University Medical Center User on 05/25/2015
69
Research Original Investigation
Identifying Otolaryngology Systematic Review Gaps
Table. CDSR Representations and DALY Metrics for 10 Otolaryngologic Conditions Studied by the GBD 2010 Projecta CDSR Representations, No. Condition
Total
Group Contributions (SRs + PRs)
20-Year Change (Median), %c
2010 Rankd
8 (5 SRs, 3 PRs)
ENT Disorders (n = 8)
0.640
−15
40
Esophageal cancer
11 (8 SRs, 3 PRs)
Upper Gastrointestinal and Pancreatic Diseases (n = 11)
0.360
−29
55
Otitis media
23 (21 SRs, 2 PRs)
ENT Disorders (n = 14); Acute Respiratory Infections (n = 9)
0.190
−10
79
Mouth cancer
8 (6 SRs, 2 PRs)
Oral Health (n = 8)
0.130
−9
94
Cancer of other part of pharynx and oropharynx
3 (1 SRs, 2 PRs)
ENT Disorders (n = 3)
0.110
−17
105
Larynx cancer
1 (1 SR)
ENT Disorders (n = 1)
0.095
−28
113
Nasopharynx cancer
3 (1 SRs, 2 PRs)
ENT Disorders (n = 3)
0.081
−12
120
Acute Respiratory Infections (n = 39); ENT Disorders (n = 5); Anaesthesia (n = 1); Airways (n = 1)
0.075
−11
122
46 (41 SRs, 5 PRs)
Thyroid cancer
6 (4 SRs, 2 PRs)
Metabolic and Endocrine Disorders (n = 6)
0.034
−6
148
Cleft lip and cleft palate
7 (4 SRs, 3 PRs)
Oral Health (n = 5); Developmental Psychosocial and Learning Problems (n = 1); Pain, Palliative and Supportive Care (n = 1)
0.023
NA
NA
Abbreviations: CDSR, Cochrane Database of Systemic Reviews; DALY, disability-adjusted life-years; ENT, ear, nose, and throat; GBD, Global Burden of Disease8-10; PR, protocol; SR, systematic review. Otolaryngologic conditions are listed from top to bottom in decreasing percentages of 2010 DALYs.
dia were overrepresented in CDSR when compared with their relatively low percentage of total DALYs. In addition, thyroid cancer and cleft lip and cleft palate were slightly overrepresented. Conversely, esophageal cancer and other hearing loss were substantially underrepresented in CDSR relative to their higher DALYs. Finally, qualitative analysis showed that nasopharynx cancer, larynx cancer, and cancer of other part of pharynx and oropharynx were slightly underrepresented.
Discussion A unique feature of the Cochrane Collaboration is the creation of individual editorial review groups responsible for specialized topic coverage.23 Several of the diseases investigated in this study were covered solely by the Cochrane ENT Disorders Group, including larynx cancer, nasopharynx cancer, cancer of other part of pharynx and oropharynx, and other hearing loss. Other otolaryngologic conditions are covered by multiple Cochrane groups; for example, upper respiratory infections received contributions from Acute Respiratory Infections, ENT Disorders, Anaesthesia, and Airways Cochrane review groups. The multiplicity of groups contributing to this disease may impact its overrepresentation in CDSR compared with its associated disability. While a specific Cochrane group for ENT Disorders exists, it did not contribute the majority of systematic reviews and protocols for the 10 otolaryngologic conditions analyzed in this study. The ENT Disorders Group contributed 29.3% of the reviews and protocols, whereas the Acute Respiratory Infections Group contributed 41.4%. Contributions from 10 different Cochrane review groups were found for the conditions 70
Proportion, %b
Other hearing loss
Upper respiratory infections
a
DALY Metrics
b
Of 291 conditions evaluated by GBD 2010.
c
From 1990 to 2010.
d
Of 176 conditions evaluated by the GBD project.
studied, indicating that otolaryngologic disorders have widereaching implications in numerous fields of medicine. In addition, of the 116 total systematic reviews and protocols included in this study, 78% were published after the year 2010, and 35% were published after the year 2013. This highlights the efforts by CDSR to create a frequently updated and current source of medical information. Of the 10 otolaryngologic conditions studied, mouth cancer was the only condition found to have proportionate CDSR representation relative to its DALY metrics. The remaining 9 diseases were either overrepresented or underrepresented in the database.
Conditions Overrepresented in CDSR Compared With GBD 2010 DALYs One of the more dramatic findings of this study was the overrepresentation of upper respiratory infections in CDSR. This category was represented by more reviews and protocols than any other category (41 reviews, 5 protocols) but had the third lowest percentage of total 2010 DALYs of the 10 otolaryngologic conditions assessed. This might reflect the fact that the upper respiratory infections category encompasses many different conditions. Similarly, otitis media was overrepresented, with more systematic reviews and protocols than other hearing loss and esophageal cancer combined, which had the 2 highest DALYs of the conditions assessed. This overrepresentation most likely reflects the high prevalence of these conditions and greater availability of randomized clinical trials (RCTs) for acute conditions compared with chronic diseases or surgical interventions. Alternatively, the overrepresentation could reflect disparities in funding and/or research between high-income and low-income countries.
JAMA Otolaryngology–Head & Neck Surgery January 2015 Volume 141, Number 1
Copyright 2014 American Medical Association. All rights reserved.
Downloaded From: http://archotol.jamanetwork.com/ by a Georgetown University Medical Center User on 05/25/2015
jamaotolaryngology.com
Identifying Otolaryngology Systematic Review Gaps
Original Investigation Research
Two conditions were slightly overrepresented in CDSR: thyroid cancer and cleft lip and cleft palate. These conditions had the lowest percentages of total DALYs of the conditions examined, but they did not receive the lowest number of reviews and protocols, as would be expected given their associated disability. The ENT Disorders Group did not contribute any reviews or protocols to these conditions. Of note, diseases of the thyroid and parathyroid glands (benign and malignant) became the responsibility of the Metabolic and Endocrine Disorders Group following their registration in February 2000.24
Conditions Underrepresented in CDSR Compared With GBD 2010 DALYs The most significantly underrepresented category in CDSR was other hearing loss. This condition was ranked first of the 10 otolaryngologic conditions according to DALY metrics, yet it received only 5 reviews and 3 protocols, all from the ENT Disorders Group. This category includes many different conditions, from congenital deafness to ototoxic hearing loss and is the most disabling of the diseases investigated in this study. Esophageal cancer, which is ranked second of the 10 otolaryngologic conditions in terms of disability, was also underrepresented in CDSR. Of note, the Upper Gastrointestinal and Pancreatic Diseases Group contributed all of the 11 systematic reviews and protocols for esophageal cancer, with no contributions from the ENT Disorders Group. Additionally, this disease experienced the greatest decrease in DALY percentage from 1990 to 2010 of the 10 diseases studied; thus, its underrepresentation may be appropriate for its decreasing disability over the last 20 years. Finally, nasopharynx cancer, larynx cancer, and cancer of other part of pharynx and oropharynx are slightly underrepresented in CDSR for their associated disabilities. Together, these 3 categories had a total of only 7 reviews and protocols, all of which came from the ENT Disorders Group. Larynx cancer had the lowest CDSR representation, with only 1 systematic review. This disease also had the second greatest median percentage change in DALYs from 1990 to 2010 (−28%). Similar to esophageal cancer, the low representation of larynx cancer may be appropriate given its decrease in DALYs over the 20-year time span.
Limitations and Future Directions We acknowledge several limitations of our study as well as areas for further investigation. To begin, the study is limited by the bounds of the field of otolaryngology. For example, esophageal cancer is a disease often diagnosed by otolaryngologists, but it is typically treated outside of the specialty by a multidisciplinary team.25 Therefore, it may be inaccurate to compare CDSR representation of esophageal cancer with other otolaryngologic disorders. Systematic reviews published by CDSR generally synthesize the best evidence in the published literature while adhering to previously specified eligibility criteria. Therefore, metaanalyses often cannot be performed for topics without an adequate base of published RCTs on similar interventions to justify data synthesis.26 Additionally, the scope of reviews is variable. For instance, authors may prepare a large review of jamaotolaryngology.com
multiple interventions (“lumping”) or several reviews of individual interventions (“splitting”). Thus, the consideration of a systematic review or protocol as a unit of representation may not be appropriate for every topic. An additional limitation of this study is that equal weight was given to all systematic reviews and protocols regardless of the number of studies included in each of them. For example, one review might contain 10 RCTs while another contained 100, but the reviews were counted as equals. In future reports, a weighted comparison could be used to further quantify the research being conducted on certain otolaryngologic conditions. While other means to quantify research prioritization may exist, this study used only the number of published reviews and protocols as a surrogate marker of research prioritization. Limitations also exist in database interrogation. A standardized search term technique and independent data collection by 2 authors were used to minimize this source of error.13 Databases other than CDSR can be used for future data collection. The Database of Abstracts of Reviews of Effects (DARE)27 contains details of systematic reviews that evaluate the effects of health care interventions, the delivery and organization of health services, and the wider determinants of health. This database complements CDSR by identifying and including systematic reviews that have not been conducted by the Cochrane Collaboration. It can therefore be used to expand future searches in determining database representation of certain diseases. The present study is also subject to the limitations of the GBD 2010 project itself. The number of conditions in this analysis was confined to those evaluated and categorized by the GBD 2010. Several conditions, such as vertigo, could not be included, since they were not included as separate categories in GBD 2010. As the outcomes of this study depend on which diseases were evaluated, there are limitations inherent in not including several highly prevalent and morbid chronic diseases. As future GBD data become available on an annual basis, the relationship between database representation and GBD disability metrics can be explored further by including additional otolaryngologic conditions to inform future research prioritization.28 A final limitation of this study is the use of a global perspective to drive decisions on a country-specific level. Health priorities vary substantially from country to country and DALYs can vary depending on current infrastructure of health care delivery. Many competing factors need to be considered in guiding research prioritization, including disease prevalence, direct and indirect costs of disease, quality of current health care delivery systems, and value of current delivery of care systems. Future investigations could include countryspecific data to inform prioritization decisions.
Conclusions Research prioritization should be guided in part by the morbidity and mortality of diseases, yet there is currently a lack of transparency in many databases regarding the availability and quality of data and criteria that inform these critical decisions. Global
JAMA Otolaryngology–Head & Neck Surgery January 2015 Volume 141, Number 1
Copyright 2014 American Medical Association. All rights reserved.
Downloaded From: http://archotol.jamanetwork.com/ by a Georgetown University Medical Center User on 05/25/2015
71
Research Original Investigation
Identifying Otolaryngology Systematic Review Gaps
burden of disease may involve any of a number of factors that impact research prioritization by CDSR. Cochrane reviews are often driven by availability of RCTs and diagnostic test studies, neither of which necessarily reflects the importance of a condition under study, especially in terms of global health or World Health Organization priorities. Therefore, it is not surprising that Cochrane reviews represent a relatively small percentage of overall GBD DALYs for otolaryngology.
ARTICLE INFORMATION Submitted for Publication: August 2, 2014; final revision received August 25, 2014; accepted September 9, 2014. Published Online: October 30, 2014. doi:10.1001/jamaoto.2014.2700. Author Affiliations: University of Colorado School of Medicine, Aurora (Pederson, Okland); Georgetown University School of Medicine, Washington, DC (Boyers); Columbia University College of Physicians and Surgeons, New York, New York (Karimkhani); Department of Otolaryngology, State University of New York Downstate Medical Center, Brooklyn, New York (Rosenfeld); Peninsula Dental School, Plymouth University, Plymouth, England (Nasser); Hunter New England Population Health, Wallsend, New South Wales, Australia (Yoong); School of Medicine and Public Health, the University of Newcastle, Newcastle, New South Wales, Australia (Yoong, Wolfenden); Institute for Health Metrics and Evaluation, University of Washington, Seattle (Kyu, Serina, Coggeshall); Department of Dermatology, University of Colorado Anschutz Medical Campus, Aurora (Dellavalle); Department of Epidemiology, Colorado School of Public Health, Aurora (Dellavalle); Dermatology Service, US Department of Veterans Affairs Medical Center, Denver, Colorado (Dellavalle). Author Contributions: Ms Pederson and Dr Dellavalle had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: Boyers, Karimkhani, Nasser, Yoong, Dellavalle. Acquisition, analysis, or interpretation of data: Pederson, Okland, Boyers, Karimkhani, Rosenfeld, Wolfenden, Kyu, Serina, Coggeshall, Dellavalle. Drafting of the manuscript: Pederson, Rosenfeld, Wolfenden, Coggeshall. Critical revision of the manuscript for important intellectual content: Okland, Boyers, Karimkhani, Nasser, Yoong, Wolfenden, Kyu, Serina, Dellavalle. Administrative, technical, or material support: Pederson, Boyers, Karimkhani, Yoong, Coggeshall, Dellavalle. Study supervision: Boyers, Karimkhani, Rosenfeld, Nasser, Dellavalle. Conflict of Interest Disclosures: None reported. Funding/Support: Dr Dellavalle receives a salary from the US Department of Veterans Affairs (VA) and is also supported by grants from the Centers for Disease Control and Prevention (CDC) and National Institutes of Health (NIH). Luke Wolfenden receives funding from the National Health and Medical Research Council, and Hunter New England Local Health District. All other authors have no relevant funding/support.
72
The present study shows that systematic reviews and protocols can be mapped against burden of disease to guide prioritization of future research. We present data for 10 otolaryngologic conditions to identify research gaps and inform future prioritization decisions. As new data and scientific models provide an unprecedented understanding of the global burden of disease, technology allows us minimize research disparities and advance human health in the 21st century.
Role of the Sponsor: The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Cochrane database of systematic reviews. JAMA Dermatol. 2014;150(9):945-951.
Disclaimer: Any opinions expressed herein do not necessarily reflect the opinions of the VA, CDC, or NIH.
15. The Cochrane Collaboration. About the Cochrane Library. http://www.thecochranelibrary .com/view/0/AboutTheCochraneLibrary .html#CDSR. Accessed June 23, 2014.
Additional Information: Ms Boyers and Dr Dellavalle are both employees for the Department of Veterans Affairs, and Dr Dellavalle is also chair of the Colorado Skin Cancer Task Force. REFERENCES 1. Fleurence RL, Torgerson DJ. Setting priorities for research. Health Policy. 2004;69(1):1-10. 2. Gross CP, Anderson GF, Powe NR. The relation between funding by the National Institutes of Health and the burden of disease. N Engl J Med. 1999;340(24):1881-1887. 3. Gillum LA, Gouveia C, Dorsey ER, et al. NIH disease funding levels and burden of disease. PLoS One. 2011;6(2):e16837. 4. Michaud CM, Murray CJ, Bloom BR. Burden of disease. JAMA. 2001;285(5):535-539. 5. ENT UK. Working together to develop research. https://www.entuk.org/professionals/generate. Accessed July 19, 2014.
14. Lozano R, Naghavi M, Foreman K, et al. Global and regional mortality from 235 causes of death for 20 age groups in 1990 and 2010. Lancet. 2012; 380(9859):2095-2128.
16. Norman I, Griffiths P. The rise and rise of the systematic review. Int J Nurs Stud. 2014;51(1):1-3. 17. The Cochrane Collaboration. Cochrane ENT Group: our reviews. http://ent.cochrane.org/our -reviews. Accessed July 19, 2014. 18. Jadad AR, Cook DJ, Jones A, et al. Methodology and reports of systematic reviews and meta-analyses. JAMA. 1998;280(3):278-280. 19. Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med. 2007;4(3):e78. 20. The Cochrane Collaboration. Search the Cochrane library. http://www.thecochranelibrary .com/view/0/index.html. Accessed June 1, 2014. 21. Institute for Health Metrics and Evaluation. GBD compare. http://viz.healthmetricsandevaluation .org/gbd-compare/. Accessed June 1, 2014.
6. The Cochrane Collaboration. Prioritising review topics. http://ent.cochrane.org/prioritising-review -topics. Accessed July 10, 2014.
22. Institute for Health Metrics and Evaluation. GBD arrow diagram. http://www .healthmetricsandevaluation.org/gbd/visualization /gbd-arrow-diagram. Accessed June 1, 2014.
7. The James Lind Alliance. ENT Aspects of Balance Priority Setting Partnership Top 10. http://www .lindalliance.org/top-tens.asp. Accessed July 19, 2014.
23. Dickersin K, Manheimer E; The Cochrane Collaboration. The Cochrane Collaboration. Clin Obstet Gynecol. 1998;41(2):315-331.
8. Vos T, Flaxman AD, Naghavi M, et al. Years lived with disability (YLDs) for 1160 sequelae of 289 diseases and injuries 1990-2010. Lancet. 2012;380 (9859):2163-2196.
24. The Cochrane Collaboration. Cochrane ENT Group: scope of our work. http://ent.cochrane.org /scope-our-work. Accessed June 15, 2014.
9. Murray CJ, Vos T, Lozano R, et al. Disability-adjusted life years (DALYs) for 291 diseases and injuries in 21 regions, 1990-2010. Lancet. 2012;380(9859):2197-2223. 10. Murray CJ, Ezzati M, Flaxman AD, et al. GBD 2010: design, definitions, and metrics. Lancet. 2012;380(9859):2063-2066. 11. Sassi F. Calculating QALYs, comparing QALY and DALY calculations. Health Policy Plan. 2006;21(5): 402-408. 12. Cochrane Agenda and Priority Setting Methods Group. Global Burden of Disease (GBD). http: //capsmg.cochrane.org/global-burden-disease-gbd -cochrane-project. Accessed July 19, 2014. 13. Karimkhani C, Boyers LN, Prescott L, et al. Global burden of skin disease as reflected in
25. Hoppo T, Jobe BA. Personalizing therapy for esophageal cancer patients. Thorac Surg Clin. 2013; 23(4):471-478. 26. The Cochrane Collaboration. Cochrane Handbook for Systematic Reviews of Interventions. http://handbook.cochrane.org. Accessed June 1, 2014. 27. The Cochrane Collaboration. Database of Abstracts of Reviews of Effects (DARE). http://www .cochrane.org/editorial-and-publishing-policy -resource/database-abstracts-reviews-effects-dare. Accessed July 23, 2014. 28. Institute for Health Metrics and Evaluation. Global Burden of Diseases, Injuries, and Risk Factors Study 2013 (GBD 2013). http://www .healthmetricsandevaluation.org/gbd/2013. Accessed June 15, 2014.
JAMA Otolaryngology–Head & Neck Surgery January 2015 Volume 141, Number 1
Copyright 2014 American Medical Association. All rights reserved.
Downloaded From: http://archotol.jamanetwork.com/ by a Georgetown University Medical Center User on 05/25/2015
jamaotolaryngology.com
