DOI: 10.1111/hir.12041

Review Article Comparison of search strategies in systematic reviews of adverse effects to other systematic reviews Su Golder*, Yoon K. Loke† & Liliane Zorzela‡ *Centre for Reviews and Dissemination (CRD), University of York, York, UK, †School of Medicine Health Policy and Practice, University of East Anglia, Norwich, UK, and ‡Department of Pediatrics, University of Alberta, Alberta, Canada

Abstract Background: Research indicates that the methods used to identify data for systematic reviews of adverse effects may need to differ from other systematic reviews. Objectives: To compare search methods in systematic reviews of adverse effects with other reviews. Methods: The search methodologies in 849 systematic reviews of adverse effects were compared with other reviews. Results: Poor reporting of search strategies is apparent in both systematic reviews of adverse effects and other types of systematic reviews. Systematic reviews of adverse effects are less likely to restrict their searches to MEDLINE or include only randomised controlled trials (RCTs). The use of other databases is largely dependent on the topic area and the year the review was conducted, with more databases searched in more recent reviews. Adverse effects search terms are used by 72% of reviews and despite recommendations only two reviews report using floating subheadings. Conclusions: The poor reporting of search strategies in systematic reviews is universal, as is the dominance of searching MEDLINE. However, reviews of adverse effects are more likely to include a range of study designs (not just RCTs) and search beyond MEDLINE. Keywords: bibliographic databases; database searching; information retrieval; review, literature; MEDLINE; meta analysis; review, systematic; search strategies; searching Key Messages

• • • • •

The reporting of search strategies in all types of systematic reviews needs improvement. The relatively low proportion of systematic reviews of adverse effects limiting their searches to MEDLINE or RCTs only is reassuring. Authors of systematic reviews of adverse effects should consider using subheadings more frequently. Database providers need to index adverse effects more consistently in order for searchers to have more confidence in using these terms in search strategies. Guidance on searching for adverse effects data needs updating and further dissemination to authors of systematic reviews.

Introduction Previous research has suggested that a different approach may be required when searching for adverse

effects as opposed to searching for effectiveness data for systematic reviews, in terms of the sources used to identify data and the search strategies used.1,2 However, how the search techniques used

Correspondence: Su Golder, Centre for Reviews and Dissemination (CRD), University of York, York YO10 5DD, UK. E-mail: [email protected] This article is published with the permission of the Controller of HMSO and the Queen’s Printer for Scotland.

92

© 2014 Crown Copyright. Health Information & Libraries Journal, 31, pp. 92–105

Searches in systematic reviews of adverse effects, Su Golder et al.

in systematic reviews of adverse effects compare to other types of systematic review is unknown. Guidance indicates that different resources may need to be accessed for systematic reviews of adverse effects compared with systematic reviews of effectiveness.3,4 For instance, specific adverse effects databases, such as TOXLINE, or drug databases, such as Derwent Drug File, may be appropriate, and MEDLINE may not be the most relevant source for adverse effects.2 Other sources of information may also be particularly useful in identifying adverse effects data, such as the Food and Drug Administration (FDA) website, textbooks (such as Meyler’s Side Effects of Drugs), bulletins (such as Medicines Safety Update) and industry data.2,5 In addition, the categories of terms used when devising a search strategy for a systematic review of adverse effects may differ from clinical effectiveness reviews. For example, whereas effectiveness reviews typically rely on the population, disease and intervention terms, systematic reviews of adverse effects may rely more heavily on the outcome (or adverse effect terms – either generic terms, such as, ‘side effect’ and ‘adverse reaction’ or specific named adverse effects terms, such as, ‘headache’ and ‘mortality’)1 and may not even require limits to any particular population, disease or even intervention.1,6 Further, the use of floating subheadings in MEDLINE and EMBASE, such as ‘drug toxicity’ and ‘adverse effects’, is advocated in systematic reviews of adverse effects, but subheadings are rarely thought useful in other types of reviews.7,8 This different approach may be hypothesised to lead to differences in the number of records retrieved and the precision of searches for adverse effects as compared to other searches. There are, however, aspects of searches for adverse effects that should be similar to other types of reviews, such as, the reporting of the search strategies, so that they are reproducible, the minimal use of search restrictions (such as date or language) without good reason and the involvement of an information professional or librarian. Previous research suggests an urgent need for improved reporting of information retrieval methods and the methodology used for searching in systematic reviews of adverse effects.9–13 Our © 2014 Crown Copyright. Health Information & Libraries Journal, 31, pp. 92–105

study improves upon previous surveys of systematic reviews of adverse effects by including a larger sample of reviews over a longer time period (1994–2011). In addition, our survey compares the methods used in systematic reviews to identify information on adverse effects to the methods used in other types of systematic reviews to identify included studies. Methods Search strategy Systematic reviews of adverse effects were identified by screening all records published in the Cochrane Database of Systematic Reviews (CDSR) (via The Cochrane Library, Issue 6:2011) and the Database of Abstracts of Reviews of Effects (DARE) [via the Centre for Reviews and Dissemination (CRD) website, June 2011]. These databases were chosen because they are the most accessible major collections of systematic reviews of health care. DARE is compiled by rigorous weekly searches of bibliographic databases (including MEDLINE and EMBASE) as well as handsearching of key journals, grey literature and regular searches of the Internet. CDSR contains all Cochrane reviews including new and updated reviews. Inclusion/exclusion criteria A review was included if the primary aim was to evaluate adverse effects associated with any health care intervention (as stated in the objectives of the review by the author). Adverse effects were either known in advance, and therefore named in the objectives, or the authors primary aim was to include all adverse effects, known in advance or otherwise. Reviews in which adverse effects were a secondary outcome or whose objective was to evaluate the ‘effectiveness and safety’ of an intervention were excluded. In addition, reviews which examined the prevention of adverse effects or the treatment of adverse effects were excluded. Two researchers independently screened titles and abstracts and selected full papers for inclusion. Any discrepancies between the researchers were resolved by discussion.

93

94

Searches in systematic reviews of adverse effects, Su Golder et al.

Data extraction For each review, data were collected on: publication year, type of adverse effects evaluated (such as pre-specified named adverse effects or any adverse effects), types of study design included [such as randomised controlled trials (RCTs) or cohort studies], databases searched (such as MEDLINE or EMBASE), interfaces used (such as OVID or PubMed) and non-database sources searched (such as reference checking or textbooks). Details of search strategies were also extracted: • on the category of search terms used from the standard categories of patient group, disease or condition, intervention, comparator and outcome (PICO)3,4 • in which database fields, such as title, abstract or indexing, the terms were searched • whether any synonyms were used for text word searches and whether or not truncation was used • whether a search filter was applied, for example to identify a particular study design, such as RCTs • whether any language or date restrictions were applied • whether the search strategy was reproducible. Searches were judged to be reproducible if the review authors provided details of the combinations of search terms used, including Boolean logic, field restrictions, truncation and search filters, as well as any date or language restrictions applied. Finally, the qualifications of the searcher, the number of records retrieved and the precision of the searches were recorded.

Comparison with other systematic reviews The search methods reported in this set of systematic reviews of adverse effects were then compared with the search methods reported in surveys of other systematic reviews.12–35 These surveys were identified from the published literature, via the Cochrane Methodology Register (CMR), the internal administration database of DARE and examining the bibliographies of all relevant articles. We selected any surveys of systematic reviews in any topic area which reported comparable data. In order to facili-

tate meaningful comparison with our work, we selected surveys that evaluated similar parameters to our dataset, such as databases and sources searched in the sample of reviews, details and reproducibility of search strategies, presence or absence of language and date restrictions. We were unable to compare our findings against surveys that simply reported on ‘adequate searches’ with no further details. Results From 4656 Cochrane reviews and 11062 DARE abstracts screened, 918 full reports were retrieved and 849 reviews met our inclusion criteria (799 from DARE, 50 from CDSR). The reviews dated from 1994 to 2011. Overall 1% (50/4656) of Cochrane reviews and 7% (799/11062) of DARE reviews focused on adverse effects. This low proportion of total reviews focusing on adverse effects is consistent with that reported in the published literature.12,27,36–38 A summary of the comparisons made of the results from our survey to the results of other surveys is presented in Table 1. Scope of adverse effects evaluation Most of the reviews in our survey (663/849, 78%) concentrate on pre-specified named adverse effect outcomes rather than analysing all potential adverse effects for a given intervention (186/849, 22%). Similarly, a recent survey of drug safety reviews also found that 80% of reviews evaluate pre-specified adverse outcomes of interest.27 Study designs included A third of reviews in our survey (280/849, 33%) limit themselves to data from RCTs. This is lower than in surveys of non-Cochrane systematic reviews, which indicate that around half of reviews are based on RCTs.12,28,29 Number of databases searched As with previous surveys25,26, nearly all the reviews in our survey (837/849, 99%) list the methods used to identify primary studies for the review. The median number of databases searched is 3 (range 0–25). This was similar to that reported in © 2014 Crown Copyright. Health Information & Libraries Journal, 31, pp. 92–105

Searches in systematic reviews of adverse effects, Su Golder et al. Table 1 Comparison of our survey of adverse effects reviews with surveys of other reviews Our survey of adverse effects reviews

Other surveys of systematic reviews

Pre-specified adverse effects Limited to RCTs Median number of databases (range)

78% 33% 3 (0 to 25)

MEDLINE only At least one non-database source Reproducible searches No indication of database platform Qualifications of searcher stated Librarian/information professional Precision (range)

20% 88% 9% 68% 13% 9% 3% (0.0026% to 63%)

80% 50% 2 (1 to 13) 3 (1-25) 5 (1 to 23) 56% 91% 0% to 8.2% 64%, 83% 11% 15% 3% (0.7% to 36%)

systematic reviews of cancer (median 3, range 1–25),30 but lower than the median number of databases searched in systematic reviews of qualitative data (median 5, range 1–23),14 and higher than in systematic reviews of adverse drug reactions (median 2, range 1–13).13 MEDLINE only searched A survey of systematic reviews of orthodontics reports 2000–2004 found that 56% of such reviews search only MEDLINE17 and a survey of paediatric oncology reviews 1998–2007 found 31% of such reviews search only MEDLINE.30 Our survey of systematic reviews of adverse effects identified a lower percentage of reviews searching only MEDLINE (20% of those reviews that reported on the named databases searched (164/837) or 19% of all reviews (164/849)). Non-database sources searched Many reviews in our survey (743/849, 88%) also report searching at least one source other than databases. A survey of Cochrane and non-Cochrane reviews also identifies a high proportion of reviews using non-database sources (91%).25

of adverse effects with sources searched in other surveys of reviews is presented in Table 2. Comparisons could only be made with those surveys of other reviews, which published data on the number or percentage of individual sources searched. Although our survey is not limited to any particular type of intervention, the majority of the reviews evaluate adverse drug effects (621/849, 73%). The percentage of reviews in our survey that search each resource is fairly similar to the other surveys of reviews of adverse drug effects (Table 2), with the exception of The Cochrane Library.13,27 For example, MEDLINE is searched in 96% of the reviews in our survey, compared with 100% in the survey by Cornelius et al.13 and over 89% in the survey by Alves et al. 2012.27 EMBASE is searched in 54% of reviews in our survey, compared with 47% in the survey by Cornelius et al. 2009.13 Other sources searched in the reviews from our survey There are a number of sources searched in the systematic reviews in our survey that were not mentioned in the other surveys of systematic reviews (Table 3). Reporting of search strategies

Comparisons of percentages of reviews searching each source A comparison of the database and non-database sources searched in our survey of systematic reviews © 2014 Crown Copyright. Health Information & Libraries Journal, 31, pp. 92–105

Reproducible searches. As with other surveys of reviews21,22,25,26,29 a high proportion of reviews in our survey (725/849, 85%) provide some information on their search strategy. However, few reviews

95

96

Searches in systematic reviews of adverse effects, Su Golder et al. Table 2 Sources searched in systematic reviews of adverse effects in our survey as compared to reviews in other surveys Data Source

Percentage of reviews, topic area and publication year

MEDLINE

100% Adverse drug effects 200913 100% Medical education 201121 99% Paediatric oncology30 97% Dentistry 200318 96% Adverse effects (our survey) 93% Complementary medicine and safety35 93% Effectiveness 199815,16 91% Physiotherapy 200924 89 + % Drug safety 201227 79% Emergency medicine 200120 68% Qualitative data 200614 68% Paediatric complementary and alternative medicine 200223 66% Asthma treatment 200019

Reference lists of published studies/ bibliographies

87% Physiotherapy 200924 76% Adverse effects (our survey) 72% Complementary medicine and safety35 72% Paediatric oncology30 72% Paediatric complementary and alternative medicine 200223 70 + % Drug safety 201227 62% Emergency medicine 200120 54% Asthma treatment 200019

EMBASE

66% Complementary medicine and safety35 65% Drug safety 201227 59% Physiotherapy 200924 54% Adverse effects (our survey) 49% Paediatric oncology30 47% Adverse drug effects 200913 28% Paediatric complementary and alternative medicine 200223 26% Dentistry 200318 24% Effectiveness 199815,16 3% Emergency medicine 200120

CENTRAL

32% Paediatric oncology30 24% Adverse effects (our survey) 15% Dentistry 200318

Cochrane Library*

82% 68% 55% 44% 36% 21% 12%

Contacting experts/Personal Communication

43% Physiotherapy 200924 34% Paediatric complementary and alternative medicine 200223 28% Asthma treatment 200019 28% Paediatric oncology30 24 + % Drug safety 201227 18% Adverse effects (our survey) 14% Emergency medicine 200120

Drug safety 201227 Complementary medicine and safety35 Physiotherapy 200924 Adverse drug effects 200913 Paediatric complementary and alternative medicine 200223 Adverse effects (our survey) Effectiveness 199815,16

(continued)

© 2014 Crown Copyright. Health Information & Libraries Journal, 31, pp. 92–105

Searches in systematic reviews of adverse effects, Su Golder et al. Table 2. (continued) Data Source

Percentage of reviews, topic area and publication year

Scanned conference reports

35% 22% 17% 17%

Industry data (from contacting industry or using industry websites)

43% Drug safety 201227 13% Adverse Effects (our survey) 13% Paediatric complementary and alternative medicine 200223

CINAHL

70% Qualitative data 200614 52% Physiotherapy 200924 42% Medical Education 201121 24% Complementary medicine and safety35 21% Paediatric complementary and alternative medicine 200223 15% Paediatric oncology30 13% Adverse Effects (our survey) 7% Effectiveness 199815,16 5% Drug safety 201227

Handsearching (as defined by the authors of the systematic reviews)

79% Medical education 201121 26% Dentistry 200318 24% Asthma treatment 200019 23% Paediatric complementary and alternative medicine 200223 10% Emergency medicine 200120 8% Adverse effects (our survey) 6% Drug safety 201227

BIOSIS Previews/Biological Abstracts

13% Paediatric complementary and alternative medicine 2002234 8% Adverse effects (our survey) 4% Physiotherapy 200924 10% Effectiveness 199815,16 7% Adverse effects (our survey) 7% Physiotherapy 200924 6% Paediatric complementary and alternative medicine 200223 57% Qualitative data 200614 34% Paediatric complementary and alternative medicine 200223 18% Physiotherapy 200924 6% Adverse effects (our survey) 4% Drug safety 201227 11% Drug safety 201227 5% Adverse effects (our survey) 14% Physiotherapy 200924 11% Paediatric complementary and alternative medicine 200223 5% Adverse effects (our survey) 12% Paediatric oncology30 4% Adverse effects (our survey) 4% Drug safety 201227 6% Paediatric complementary and alternative medicine 200223 4% Adverse effects (our survey) 8% Physiotherapy 200924 4% Paediatric complementary and alternative medicine 200223 3% Adverse effects (our survey) 2% Drug safety 201227

Current Contents

PsycINFO/PsycLit

Web of Science† Science Citation Index (SCI)

Textbooks/bulletins

Personal files HealthStar (no longer available)

Drug safety 201227 Paediatric oncology30 Adverse Effects (our survey) Paediatric complementary and alternative medicine 200223

(continued)

© 2014 Crown Copyright. Health Information & Libraries Journal, 31, pp. 92–105

97

98

Searches in systematic reviews of adverse effects, Su Golder et al. Table 2. (continued) Data Source

Percentage of reviews, topic area and publication year

International Pharmaceutical Abstracts (IPA)

3% Adverse effects (our survey) 2% Drug safety 201227 2% Adverse effects (our survey) 49% Complementary medicine and safety35 25% Paediatric oncology30 2% Adverse effects (our survey) 15% Paediatric complementary and alternative medicine 200223 7% Physiotherapy 200924 2% Adverse effects (our survey) 2% Adverse effects (our survey) 2% Drug safety 201227 2% Adverse effects (our survey) 2% Drug safety 201227 1% Adverse effects (our survey) 32% Complementary medicine and safety35 4% Drug safety 201227 1% Adverse effects (our survey) 2% Drug safety 201227 1% Adverse effects (our survey)

AMED CancerLit Dissertations and Theses: Abstract and Index (ProQuest) Google Scholar Manufacturers Package Insert CISCOM OVID† OPEN SIGLE (System for Information on Grey Literature)

*In our survey, the terminology used by the authors of the systematic reviews was accepted. For example, where authors stated that they searched CDSR, this was categorised as CDSR and where they stated that they had searched ‘The Cochrane Library’, this was categorised separately as ‘The Cochrane Library’. Other surveys may have accepted statements such as ‘we searched CDSR’ to mean that ‘The Cochrane Library’ was searched. †interface described in review as if database.

in our survey give sufficient detail to allow the search to be reproduced according to the criteria stated earlier (74/849, 9%). Other studies have highlighted the failure of systematic reviews to report whether a comprehensive search strategy was undertaken,14–20,22,23,39–52 with reproducibility of search strategies identified as between 0% and 8.2%.15,16,21,31,35 Interfaces used. The majority of the reviews in our survey gave no indication of the database interface used (574/849, 68%). This is consistent with a survey of Cochrane reviews26 in which 83% do not mention the database platform and a survey of Cochrane and non-Cochrane reviews25 in which 64% do not mention the database platform. Search restrictions As with other surveys, not all the reviews in our survey report on the date ranges of the searches23,25,32 or whether the searches are restricted by language.21,22,25,26,29,30,35 Twenty-two percentage

(188/849) of the reviews in our survey do not give a date range for the searches, and in 50% (421/ 849), it is unclear whether there are any language restrictions. Conducting the searches Only 13% (109/849) of the reviews in our survey gave any indication as to the qualifications of the person who conducted the searches. This is similar to a survey of Cochrane and non-Cochrane reviews in which 11% gave some indication of the qualifications of the searcher.25 Overall, 9% (73/849) of the reviews from our survey were conducted by a librarian or information professional. Low numbers of medical education reviews also reported the involvement of an information professional (15%).21 In the reviews from our survey, the literature searches performed by the information professionals tend to be carried out in more databases (median 4, range 1–25) than those performed by the non-information professionals (median 2, range 0–10) or than those where the qualifications of the © 2014 Crown Copyright. Health Information & Libraries Journal, 31, pp. 92–105

Searches in systematic reviews of adverse effects, Su Golder et al. Table 3 Other sources searched in more than four systematic reviews of adverse effects from our survey but not reported in other surveys of systematic reviews

Data Source Cochrane Database of Systematic Reviews (CDSR) Database of Abstracts of Reviews of Effects (DARE) FDA website Cochrane Registers (such as the Cochrane Schizophrenia Group’s register) Internet Citation searches ClinicalTrials.gov LILACS Surveillance data Scopus ACP Journal Club TOXLINE POPLINE Current controlled trials.gov Health Technology Assessment (HTA) Database ISI Proceedings National Research Register (NRR) NHS Economic Evaluation Database (NHS EED) National Institutes of Health website Reprotox Web of Knowledge* Australian New Zealand Clinical Trials Registry (ANZCTR) Iowa Drug Information Service (IDIS) PASCAL

Reviews that searched each source, n = 849 69 (8%) 48 (6%) 47 (6%) 38 (4%)

34 27 25 25 22 21 17 17 15 12 11

(4%) (3%) (3%) (3%) (3%) (2%) (2%) (2%) (2%) (1%) (1%)

10 (1%) 10 (1%) 9 (1%) 8 9 6 5

(1%) (1%) (1%) (1%)

Search terms Of those reviews in our survey that provide information on their search strategies, few (152/725, 21%) report which fields (such as title, abstract or indexing) the terms were restricted to. However, most reviews indicate the actual terms used or the category of terms used (such as the population, disease, intervention, comparator, and outcome) (Table 4). Of the 725 reported search strategies in our survey, the majority (702/725, 97%) search on terms for the intervention (such as aspirin or oral contraceptives). It would also be expected that the vast majority of reviews of effectiveness would also include intervention terms. Unlike search strategies for clinical effectiveness, however, many (522/725, 72%) of the reviews in our survey search for terms for the outcomes (i.e. adverse effects). Those search strategies that incorporate terms for the outcomes/adverse effects (in 522 reviews) tend to search on specific adverse effects (such as ‘thrombosis’ or ‘headache’) (393/522, 75%) or a combination of specific adverse effects and generic adverse effects (such as adverse effects, side effects or complications) (90/522, 17%) rather than using solely generic outcome terms (39/522, 7%). Five reviews rely on outcome terms (adverse effects terms) alone, such as liver damage, root sensitivity, thrombocytopenia or ectopic pregnancy with no other category of terms used.

5 (1%) 5 (1%)

*interface described in review as if database

searcher were not reported (median 3, range 0–17). Booth 2006 also found that more databases were searched in reviews that involved an information professional.14 Reviews from our survey that list reproducible strategies were more likely (25/73, 34%) to have been conducted by an information specialist than reviews that did not report reproducible searches – only 6% (48/775) of these were carried out by an information specialist. Another study also found that the reporting of the role of the searcher was associated with greater reproducibility of searches.33 © 2014 Crown Copyright. Health Information & Libraries Journal, 31, pp. 92–105

Table 4 Categories of search terms used in database search strategies in reviews from our survey

Category Population (e.g. elderly) Disease Intervention Comparison Outcome: named adverse effect (e.g. headache) Outcome: generic adverse effect (e.g. complications) Methodological filters (e.g. randomised controlled trials) Search strategy not stated

Reviews using terms in category, n = 849 97 179 702 10 483

(11%) (21%) (83%) (1%) (57%)

129 (15%) 199 (23%) 124 (15%)

99

100

Searches in systematic reviews of adverse effects, Su Golder et al.

Only two reviews in our survey report using ‘floating’ subheadings, that is, did not attach the subheadings to any indexing terms. One review reports using the floating subheading ‘adverse events’ in MEDLINE (although the authors probably meant ‘adverse effects’) and the other review reports using the subheadings ‘adverse effects’ and ‘complications’. Number of records retrieved The number of records retrieved by the searches in the reviews in our survey was recorded in 449 reviews and ranged from 13 to 76977 records with a median of 631. The number of included studies was reported in all the reviews and ranged from one to 561 with a median of 18. Although some reviews in our survey included a flow diagram, the starting point was not always the number of records identified by the searches, and it was unclear what the starting point was. In some instances, it could be the number of potentially relevant records (after a first sift) or the number of full-text papers examined. Due to the arduous and impractical nature of repeating searches, we did not gather this information. This problem with flow charts has been identified in previous research.52 Precision of searches Precision could be calculated in those reviews in our survey, which stated both the number of included studies and the total number of studies identified by the searches. The overall precision of the searches varied widely from 0.0026% to 63% with a median of 3%. A study by Sampson et al.34 recorded the precision of searches in systematic reviews of health care interventions, identifying comparable results to our survey with a median of 3% (range 0.7–36%). Discussion Differences between adverse effects reviews and other systematic reviews Our survey identified a number of differences, but also some similarities in the search methods of

systematic reviews of adverse effects as compared to other reviews. The reporting of searches in systematic reviews of adverse effects is as incomplete as that for other systematic reviews. This was particularly apparent in terms of date and language restrictions, the reporting of the qualifications of the searcher and in the low numbers of reproducible search strategies. MEDLINE was commonly searched in most reviews in our survey with EMBASE being less commonly used. However, The Cochrane Library was used less often, and this may stem from the finding that adverse effects reviews in our survey are more likely to try to include non-randomised data compared with other systematic reviews. Reviewers may hold the belief that The Cochrane Library is primarily a repository for RCT data, and they would therefore prefer to search other sources for potentially relevant nonrandomised studies (particularly regarding rare, long-term harmful effects that are difficult to capture in RCTs). The historical emphasis on RCTs in Cochrane reviews is well described,12, 24, 31, 32, 37 and indeed, nearly all the Cochrane reviews in our sample chose to focus solely on RCT data. In addition to the need to consider other study designs, some other differences in search strategies may arise because of diversity in the nature of adverse effects outcomes. Trial interventions are typically targeted at eliciting a particular benefit (usually known as the primary outcome) in patients with a defined disease or condition. Conversely, adverse effects can encompass a huge range of events from common, mild ailments (such as nausea or rash), to serious rare occurrences that can afflict the general population (e.g. penicillin allergy). Hence, reviewers may choose not to specify a particular population in the search, and they may prefer to narrow down the search using particular outcome terms related to the adverse effects of interest. Equally, use of non-specific adverse effects terms (e.g. side effect or adverse effect in title/abstract) or subheadings (e.g. ‘adverse effects’ -/ae) may retrieve a large number of less relevant articles addressing a multitude of adverse effects that are not the main focus of the review. These particular nuances in adverse effects searches may reflect the pragmatic decisions made when formulating the study question, and constructing a search strategy that balances sensitivity with precision. © 2014 Crown Copyright. Health Information & Libraries Journal, 31, pp. 92–105

Searches in systematic reviews of adverse effects, Su Golder et al.

Many of the differences in the sources searched compared with other reviews can be explained in terms of topic area or date. For example, CINAHL, which specialises in nursing and allied health, is searched more often in reviews of qualitative research,14 physiotherapy,24 medical education21 and paediatric complementary and alternative medicine.23 EMBASE, which is a well-known source for drug and general medical information, is searched more often in drug and physiotherapy reviews or more recent reviews. PsycINFO/PsycLit is searched more often in reviews of qualitative data 14 and reviews of paediatric complementary and alternative medicine.23 Fewer reviews in our survey were restricted to MEDLINE than a previous survey of orthodontic reviews 2000–2004.17 This may reflect the more diverse databases required for searching for adverse effects data or the time period studied. There is a general trend for more recent reviews to include more databases, particularly with EMBASE and The Cochrane Library. Another survey also identified an increase over time in the searching of MEDLINE, EMBASE, CINAHL, The Cochrane Library and reference checking.53 Differences between adverse effects reviews strategies and good practice guidance Current guidance highlights that a wider range of information may be required for adverse effects reviews than just RCTs, including data from, for example, individual case reports and case series. Observational studies are required, in particular, for assessing new, unexpected, rare or long-term adverse effects. Around a third of reviews of adverse effects, however, are still restricted to RCTs. There are numerous specialist databases for drug information and adverse effects data that are rarely searched in the 849 systematic reviews. In addition, 20% of reviews only search MEDLINE, yet research has indicated that MEDLINE is not the most fruitful source of adverse effects data and that searching a range of sources is necessary.2,54 In one case study, Science Citation Index (SCI) was identified as the most useful database for a drug related adverse effects review, followed by BIOSIS previews and EMBASE, yet these data© 2014 Crown Copyright. Health Information & Libraries Journal, 31, pp. 92–105

bases are included in 5%, 8% and 54% of the reviews here, respectively.54 Industry and handsearching have been reported as good sources of unique adverse effects data, yet only 13% and 8% of the 849 reviews use these sources.2,54 British Library Direct, Medscape DrugInfo, AHFS First, Thomson Reuters Integrity and Conference Papers Index (CPI) all provided unique relevant references in a systematic review of adverse effects of rosiglitazone, yet are not included in any of the reviews here.54 It was reassuring to note, however, that reference checking, which has also been found to identify unique relevant references, is used in over three quarters of the reviews in our survey. Many (522/725, 72%) of the reviews of adverse effects search for terms for outcomes (i.e. adverse effects), and this may reflect a pragmatic decision in order to restrict searches rather than the adherence to guidance. Guidance is currently based on previous research, which has indicated that specific adverse effects strategies that include both text words and indexing terms for adverse effects in both MEDLINE and EMBASE fail to retrieve nearly a quarter of papers with relevant information on adverse effects.55 However, failure to restrict to outcome terms can lead to unmanageable numbers of records to sift and particularly low precision.1,54,56,57 The low precision (from 0.0026%) recorded in some of the systematic reviews of adverse effects may reflect those reviews, which do not restrict to outcome terms. Since guidance3,4 was published, however, the sensitivity of search strategies with adverse effects terms has improved. Recent research suggests that improvements in both indexing and the descriptions in title and abstract have meant that this approach may now only fail to retrieve approximately 8% of relevant papers,57 and that high sensitivity (over 90%) can be achieved with adverse effects search terms/filters in MEDLINE and EMBASE.57 Guidance on searching for effectiveness does not advocate the use of floating subheadings, whereas guidance on searching for adverse effects suggests that floating subheadings may be useful.3,4 However, only two reviews of the 849 in our survey reported using ‘floating’ subheadings.

101

102

Searches in systematic reviews of adverse effects, Su Golder et al.

Suggested improvements The reporting of search strategies in all systematic reviews needs to be improved. Simple measures include more accurate descriptions of the sources searched. For instance, as there are six databases on The Cochrane Library, it is more informative for readers if authors of reviews list the individual databases searched. In addition, named databases rather than the platforms used should be stated, as statements such as, ‘Web of Knowledge was searched’ does not indicate which databases from this interface were selected. Full search strategies could also be made available as additional files when publishing electronically. The variation in search methods used highlights the need for further development of guidelines and a consensus on standards for retrieving adverse effects data. Further research may help inform guidance and enable more definitive recommendations in the complexities of creating search strategies and the selection of sources. Guidance also needs to be updated in light of more recent research and disseminated to a wide audience. The support for authors of reviews of adverse effects (through groups such as the Cochrane Adverse Effects Methods Group) also needs to continue to grow with the continuation of regular workshops targeted at this difficult area.

Limitations The data extraction relied upon what the authors reported in the published systematic reviews. Reporting was often unclear, for example study design descriptions such as ‘prospective’ or ‘retrospective studies’ were used and searching descriptions such as ‘handsearching’ without stating the sources hand searched. Some of the published surveys of other systematic reviews may not be comparable to our survey of systematic reviews of adverse effects as the reviews were retrieved from different sources (such as MEDLINE and EMBASE as opposed to CDSR and DARE) or were focused on specific subject areas (such as dentistry). Our survey only examines systematic reviews of adverse effects identified from CDSR and DARE. Cochrane reviews adhere to strict guidelines in their produc-

tion3 and have been found to be more rigorous than reviews published in peer reviewed journals19,24,29,58–60 and reviews included in DARE have passed a number of quality criteria checks.61,62 In addition, our survey may have missed some systematic reviews with a main focus on adverse effects. The difference with respect to searching some of the databases in the reviews from our survey compared with reviews from other surveys may reflect more detailed reporting in some surveys than others. For instance, searches of The Cochrane Library databases can be listed individually (such as CDSR, CENTRAL or DARE) or collectively as ‘The Cochrane Library’ and some surveys did not report those sources searched in a limited number of reviews. Conclusions The reporting of search strategies in systematic reviews of adverse effects is as incomplete as that in other systematic reviews. Although many similarities were apparent in the search approaches adopted in systematic reviews of adverse effects as compared to other types of reviews there were also many differences. Adverse effects reviews tend to search beyond MEDLINE, include a wider range of sources and are less likely to be restricted to RCTs. The differences in sources searched, such as CINAHL and EMBASE, could largely be explained by the topic area coverage. In addition, more recent reviews searched more databases. It is difficult to compare the search facets (such as population, intervention, comparators and outcomes) used in systematic reviews of adverse effects with other reviews because of differences in how questions regarding adverse effects are formulated. However, it is likely that systematic reviews of adverse effects rely more heavily on outcome terms (which are often terms that describe specific events). Despite guidance on the use of floating subheadings in systematic reviews of adverse effects, very few search strategies included these. Acknowledgements We thank Kath Wright of CRD, York for her kind assistance in data extraction. We also would like © 2014 Crown Copyright. Health Information & Libraries Journal, 31, pp. 92–105

Searches in systematic reviews of adverse effects, Su Golder et al.

to thank Professor Lesley Stewart and Dr Gill Norman of CRD, York for commenting on an earlier draft. Conflict of interest

10

11

No conflict of interests have been declared. Source of funding This research was undertaken by Su Golder as part of a Medical Research Council (MRC) fellowship. The views expressed in this presentation are those of the authors and not necessarily those of the MRC. The funders had no role in study design, data collection and analysis, decision to publish or preparation of the manuscript.

12

13

14

15

References 1 Golder, S. & Loke, Y. K. Search strategies to identify information on adverse effects: a systematic review. Journal of the Medical Library Association 2009, 97, 84–92. 2 Golder, S. & Loke, Y. K. Sources of information on adverse effects: a systematic review. Health Information and Libraries Journal 2010, 27, 176–190. 3 Higgins, J. P. T. & Green, S. (editors). Cochrane Handbook for Systematic Reviews of Interventions 5.1.0 [updated March 2011]. The Cochrane Collaboration 2011. Accessible at: http://www.cochrane-handbook.org 4 Centre for Reviews and Dissemination Systematic Reviews: CRD’s Guidance for Undertaking Reviews in Health care 3rd ed. York: University of York, Centre for Reviews and Dissemination, 2009. 5 Loke, Y., Golder, S. & Vandenbrouche, J. Comprehensive evaluations of the adverse effects of drugs: importance of appropriate study selection and data sources. Therapeutic Advances in Drug Safety 2011, 2, 59–68. 6 Wieland, S. & Dickersin, K. Selective exposure reporting and Medline indexing limited the search sensitivity for observational studies of the adverse effects of oral contraceptives. Journal of Clinical Epidemiology 2005, 58, 560– 567. 7 Badgett, R., Chiquette, E., Anagnostelis, B. & Mulrow, C. Locating reports of serious adverse drug reactions (PowerPoint presentation). 1999. [cited 2005 Jan 7]. Accessible at: http://medinformatics.uthscsa.edu/#FILTERS. 8 Golder, S., McIntosh, H. M., Duffy, S. & Glanville, J. Developing efficient search strategies to identify reports of adverse effects in MEDLINE and EMBASE. Health Information and Libraries Journal 2006, 23, 3–12. 9 Golder, S., Loke, Y. & McIntosh, H. M. Room for improvement? A survey of the methods used in systematic reviews

© 2014 Crown Copyright. Health Information & Libraries Journal, 31, pp. 92–105

16

17

18

19

20

21

22

23

of adverse effects. BMC Medical Research Methodology 2006, 6, 3. Cornelius, V., Perrio, M., Smith, L. & Shakir, S. A survey of the conduct and reporting of systematic reviews assessing adverse effects of drug interventions. Drug Safety 2008, 31, 891. Golder, S., Loke, Y. & McIntosh, H. M. Poor reporting and inadequate searches were apparent in systematic reviews of adverse effects. Journal of Clinical Epidemiology 2008, 61, 440–448. Hopewell, S., Wolfenden, L. & Clarke, M. Reporting of adverse events in systematic reviews can be improved: survey results. Journal of Clinical Epidemiology 2008, 61, 597–602. Cornelius, V. R., Perrio, M. J., Shakir, S. A. W. & Smith, L. A. Systematic reviews of adverse effects of drug interventions: a survey of their conduct and reporting quality. Pharmacoepidemiology and Drug Safety 2009, 18, 1223–1231. Booth, A. “Brimful of STARLITE”: toward standards for reporting literature searches. Journal of the Medical Library Association 2006, 94, 421–429. Brazier, H. Systematic Reviews depend on Systematic Literature Searches. The Sixth European Conference of Medical and Health Libraries. Libraries without limits: changing needs - changing roles; EAHIL/AEBIS; 22-27 June 1998; Utrecht, The Netherlands. Accessible at: http://www.eahil. net/utrecht.htm Brazier, H. Poorly executed and inadequately documented? An analysis of the literature searches on which systematic reviews are based. In: Presented at the 2nd Symposium on Systematic Reviews: Beyond the Basics; 5–7 January 1999; Oxford, UK; 1999. Flores-Mir, C., Major, M. P. & Major, M. P. Search and selection methodology of systematic reviews in orthodontics [2000–2004]. American Journal of Orthodontics and Dentofacial Orthopedics, 2006, 130, 214–217. Glenny, A., Esposito, M., Coulthard, P. & Worthington, H. The assessment of systematic reviews in dentistry. European Journal of Oral Sciences 2003, 111, 85–92. Jadad, A. R., Moher, M., Browman, G. P., Booker, L., Sigouin, C., Fuentes, M. & Stevens, R. Systematic reviews and meta-analyses on treatment of asthma: critical evaluation. British Medical Journal 2000, 320, 537–540. Kelly, K., Travers, A., Dorgan, M., Slater, L. & Rowe, B. Evaluating the quality of systematic reviews in the emergency medicine literature. Annals of Emergency Medicine 2001, 38, 518–526. Maggio, L. A., Tannery, N. H. & Kanter, S. L. Reproducibility of literature search reporting in medical education reviews. Academic Medicine 2011, 86, 1049–1054. Major, M. P., Major, P. W. & Flores-Mir, C. Benchmarking of reported search and selection methods of systematic reviews by dental speciality. Evidence-Based Dentistry 2007, 8, 66–70. Moher, D., Soeken, K., Sampson, M., Ben-Porat, L. & Berman, B. Assessing the quality of reports of systematic

103

104

Searches in systematic reviews of adverse effects, Su Golder et al.

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

reviews in pediatric complementary and alternative medicine. BMC Pediatrics, 2002, 2, 1021–1030. Moseley, A. M., Elkins, M. R., Herbert, R. D., Maher, C. G. & Sherrington, C. Cochrane reviews used more rigorous methods than non-cochrane reviews: survey of systematic reviews in physiotherapy. Journal of Clinical Epidemiology 2009, 62, 1–10. Sampson, M., McGowan, J., Tetzlaff, J., Cogo, E. & Moher, D. No consensus exists on search reporting methods for systematic reviews. Journal of Clinical Epidemiology 2008, 61, 748–754. Yoshii, A., Plaut, D., McGraw, K. A., Anderson, M. J. & Wellik, K. E. Analysis of the reporting of search strategies in Cochrane systematic reviews. Journal of the Medical Library Association 2009, 97, 21–29. Alves, C., Batel-Marques, F. & Filipa Macedo, A. Data sources on drug safety evaluation: a review of recent published meta-analyses. Pharmacoepidemiology and Drug Safety, 2012, 21, 21–33. Egger, M., Schneider, M. & Davey Smith, G. Spurious precision? Meta-analysis of observational studies British Medical Journal, 1998, 316, 140–144. Moher, D., Tetzlaff, J., Tricco, A. C., Sampson, M. & Altman, D. G. Epidemiology and reporting characteristics of systematic reviews. PLoS Medicine 2007, 4, e78. Lundh, A., Knijnenburg, S. L., Jorgensen, A. W., Dalen, E. C. & Kremer, L. C. Quality of systematic reviews in pediatric oncology – a systematic review. Cancer Treatment Reviews 2009, 35, 645–652. Roundtree, A., Kallen, M., Lopezolivo, M., Kimmel, B., Skidmore, B., Ortiz, Z., & Cox, V. Suarez-Almazor, M. E. Poor reporting of search strategy and conflict of interest in over 250 narrative and systematic reviews of two biologic agents in arthritis: a systematic review. Journal of Clinical Epidemiology 2009, 62, 128–137. Bader, J. & Ismail, A. Survey of systematic reviews in dentistry. Journal of the American Dental Association, 2004, 135, 464–473. Sampson, M. & McGowan, J. Reporting the role of the expert searcher in Cochrane reviews. Cochrane Colloquium; 22-26 October; Melbourne, Australia 2005. Sampson, M., Tetzlaff, J. & Urquhart, C. Precision of health care systematic review searches in a cross-sectional sample. Research Synthesis Methods 2011, 2, 119–125. Pilkington, K. & Boshnakova, A. Complementary medicine and safety: a systematic investigation of design and reporting of systematic reviews. Complementary Therapies in Medicine 2012, 20, 73–82. Ernst, E. & Pittler, M. H. Systematic reviews neglect safety issues. Archives of Internal Medicine 2001, 161, 125– 126. Ernst, E. & Pittler, M. H. Assessment of therapeutic safety in systematic reviews: literature review. British Medical Journal 2001, 323, 546. Vandermeer, B., Bialy, L., Johnston, B., Hooton, N., Hartling, L., Klassen, T., Johnston, B. C. & Wiebe, N. Meta-analyses of safety data: a comparison of exact versus

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

asymptotic methods. 14th Cochrane Colloquium; 23-26 October 2006; Dublin, Ireland. Dixon, E., Hameed, M., Sutherland, F., Cook, D. J. & Doig, C. Evaluating meta-analyses in the general surgical literature. A critical appraisal. Annals of Surgery 2005, 241, 450–459. Linde, K. & Willich, S. N. How objective are systematic reviews? Differences between reviews on complementary medicine. Journal of the Royal Society of Medicine 2003, 96, 17–22. Jadad, A. R. & McQuay, H. J. Meta-analyses to evaluate analgesic interventions: a systematic qualitative review of their methodology. Journal of Clinical Epidemiology 1996, 49, 235–243. Delaney, A., Bagshaw, S. M., Ferland, A., Manns, B., Laupland, K. B. & Doig, C. J. A systematic evaluation of the quality of meta-analyses in the critical care literature. Critical Care 2005, 9, R575–R582. Rigby, K., Silagy, C. & Crockett, A. Health economic reviews. Are they compiled systematically? International Journal of Technology Assessment in Health Care 1996, 12, 450–459. Sacks, H., Berrier, J., Reitman, D., Ancona-Berk, V. A. & Chalmers, T. C. Meta-analyses of randomized controlled trials. The New England Journal of Medicine 1987, 316, 450– 455. Shang, H., Zhang, J., Zhang, B. & Youping, L. Methodology and quality of reporting in systematic reviews and meta-analyses of traditional chinese medicine. 15th Cochrane Colloquium; 23-27 October 2007; Sao Paulo, Brazil. Breslow, R. A., Ross, S. A. & Weed, D. L. Quality of reviews in epidemiology. American Journal of Public Health 1998, 88, 475–477. Shea, B., Moher, D., Graham, I., Pham, B. A. & Tugwell, P. A comparison of the quality of Cochrane reviews and systematic reviews published in paper-based journals. Evaluation and the Health Professions 2002, 25, 116–129. Choi, C., Halpern, S. D., Malik, N., Jadad, A. R., Tramer, M. R. & Walder, B. Examining the evidence in anesthesia literature: a critical appraisal of systematic reviews. Anesthesia and Analgesia 2001, 92, 700–709. Smith, A. F. An analysis of review articles published in four anaesthesia journals. Canadian Journal of Anaesthesia 1997, 44, 405–409. Sampson, M. & McGowan, J. Errors in search strategies were identified by type and frequency. Journal of Clinical Epidemiology 2006, 59, 1057–1063. Shea, B., Boutrer, L. M., Grimshaw, J. M., Francis, D., Ortiz, Z., Wells, G. A., Tugwell, P. S. & Boers, M. Scope for improvement in the quality of reporting of systematic reviews. From the cochrance musculoskeletal group. Journal of Rheumatology 2006, 33, 9–15. Fehrmann, P. & Thomas, J. Comprehensive computer searches and reporting in systematic reviews. Research Synthesis Methods 2011, 2, 15–32. Parekh-Bhurke, S., Kwok, C. S., Pang, C., Hooper, L., Loke, Y. K., Ryder, J. J., Sutton, A. J., Hing, C. B., Harvey, I. & Song, F. Uptake of methods to deal with publica-

© 2014 Crown Copyright. Health Information & Libraries Journal, 31, pp. 92–105

Searches in systematic reviews of adverse effects, Su Golder et al.

54

55

56

57

58

tion bias in systematic reviews has increased over time, but there is still much scope for improvement. Journal of Clinical Epidemiology 2011, 64, 349–357. Golder, S. & Loke, Y. K. The contribution of different information sources for adverse effects data. International Journal of Technology Assessment in Health Care 2012, 28, 133–137. Derry, S., Loke, Y. K. & Aronson, J. K. Incomplete evidence: the inadequacy of databases in tracing published adverse drug reactions in clinical trials. BMC Medical Research Methodology, 2001, 1, 7. Golder, S., Loke, Y. K. & Zorzela, L. Some improvements are apparent in identifying adverse effects in systematic reviews from 1994 to 2011. Journal of Clinical Epidemiology 2013, 66, 253–260. Golder, S. & Loke, Y. K. The performance of adverse effects search filters in MEDLINE and EMBASE. Health Information and Libraries Journal 2012, 29, 141–151. Jadad, A. R., Cook, D. J., Jones, A., Klassen, T. P., Tugwell, P. & Moher, M. Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane

© 2014 Crown Copyright. Health Information & Libraries Journal, 31, pp. 92–105

59

60

61

62

reviews with articles published in paper-based journals. The Journal of the American Medical Association 1998, 280, 278–280. Egger, M., Davey Smith, G., Schneider, M. & Minder, C. Bias in meta-analysis detected by a simple, graphical test. British Medical Journal, 1997, 315, 629–634. Wen, J., Ren, Y., Wang, L., Li, Y., Liu, Y., Zhou, M., Liu, P., Ye, L., Li, Y. & Tian, W. The reporting quality of metaanalyses improves: a random sampling study. Journal of Clinical Epidemiology 2008, 61, 770–775. Centre for Reviews and Dissemination (CRD). About DARE. 2008. [cited 2008 23 October]. Accessible at: http://www. crd.york.ac.uk/crdweb/html/help.htm. Petticrew, M., Song, F., Wilson, P. & Wright, K. Quality-assessed reviews of health care interventions and the database of abstracts of reviews of effectiveness (DARE). International Journal of Technology Assessment in Health Care 1999, 15, 671–678.

Received 22 June 2012; Accepted 4 July 2013

105