Journal of Affective Disorders 174 (2015) 45–50
Contents lists available at ScienceDirect
Journal of Affective Disorders journal homepage: www.elsevier.com/locate/jad
Review
Rigour of development of clinical practice guidelines for the pharmacological treatment of bipolar disorder: Systematic review Arianna Castellani, Francesca Girlanda, Corrado Barbui n WHO Collaborating Centre for Research and Training in Mental Health and Service Evaluation, Department of Public Health and Community Medicine, Section of Psychiatry, University of Verona, Verona, Italy
art ic l e i nf o
a b s t r a c t
Article history: Received 17 September 2014 Received in revised form 17 November 2014 Accepted 17 November 2014 Available online 25 November 2014
Background: There is an increasing concern about the quality of clinical practice guidelines. Because no information is available on the rigour of development of clinical practice guidelines for bipolar disorder, we carried out a systematic review of those focusing on its pharmacological treatment. Methods: We searched the National Guideline Clearinghouse, MEDLINE, EMBASE, PsychINFO and CINHAL for guidelines published from 2003 to 2014. The quality of each guideline was assessed using the Appraisal of Guidelines for Research and Evaluation II (AGREE II). Results: Fourteen guidelines were appraised. The overall quality of included guidelines varied considerably, both within and across AGREE II domains. Overall, six guidelines were rated as “recommended”, two “recommended with modifications”, and six were not recommended according to AGREE II ratings. The mean score for rigour of development was 46.8% of the maximum possible score, with no guidelines scoring the maximum score in this domain. Guidelines with lower editorial independence scores also had lower rigour of development scores, whereas those with higher-quality domain scores scored high in both domains. Limitations: As current appraisal focused on guidelines for the pharmacological treatment of bipolar disorder, it will be important to critically assess the rigour of development of other guidelines for bipolar and other psychiatric disorders. Conclusions: Health care providers, policy makers, physicians and patients alike need to be aware of the variability in guideline quality and identify the high-quality guidelines that meet their needs. & 2014 Elsevier B.V. All rights reserved.
Keywords: Clinical practice guidelines Bipolar disorder Quality
Contents 1. 2.
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1. Data sources and search strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2. Guideline inclusion and exclusion criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3. Data extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4. Assessment of guideline quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3. Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1. Guideline selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2. Characteristics of included clinical practice guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3. Guideline quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4. Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1. Clinical implications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2. Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Role of funding source . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conflict of interest. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
n
Corresponding author. Tel.: þ 39 045 812 6418; fax: þ39 045 8027498.
http://dx.doi.org/10.1016/j.jad.2014.11.032 0165-0327/& 2014 Elsevier B.V. All rights reserved.
46 46 46 46 46 46 47 47 47 47 48 48 49 49 50 50
46
A. Castellani et al. / Journal of Affective Disorders 174 (2015) 45–50
Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 Appendix A. Supporting information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 Rerefences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
1. Introduction
2.2. Guideline inclusion and exclusion criteria
According to the principles of evidence-based medicine, research findings should guide doctors when taking decisions in daily clinical practice (Haynes and Haines, 1998; Sackett et al., 1996). However, not all research findings are attributed the same value, as randomised controlled trials and systematic reviews of randomised evidence are nowadays at the pinnacle of the evidence hierarchy (Cipriani and Barbui, 2006). In such a system a crucial issue is how the evidence base may effectively be translated into practice, considering that access and use of systematic reviews may not be straightforward for most doctors in most countries of the world (Barbui and Tansella, 2011b). In order to tackle this global issue, over the past 20 years clinical practice guidelines have become an increasingly popular tool for synthesis of clinical information so as to change clinical practice and improve quality of health care (Barbui and Cipriani, 2011a). Such a quantitative growth in the number of guidelines is, however, a source of concern. First, despite some increase in quality over time, the rigour of guideline development has remained moderate to low over the last two decades (AlonsoCoello et al., 2010). Second, although there are few studies describing financial conflicts of interest for guideline authors, the available data suggest that there is a high prevalence of nondisclosure among authors across a variety of clinical specialties, and a high percentage of authors with disclosures report conflicts of interest (Norris et al., 2011, 2012, 2013). Third, the quality of clinical practice guidelines developed by specialty societies has been shown to be highly unsatisfactory (Grilli et al., 2000). The present systematic review was therefore carried out to evaluate the quality of clinical practice guidelines for the pharmacological management of bipolar disorder. The case example of bipolar disorder is particularly interesting, as several active pharmacological treatments are available, a massive number of randomized trials and systematic reviews have been produced, and the overall quality of evidence remains a major concern. Specific aims were the following: (a) to determine whether the rigour of guideline development increased over the last decade; (b) to ascertain whether the rigour of guideline development was associated with less conflicts of interest; (c) to compare the quality of guidelines issued by specialty societies versus other agencies.
Two reviewers independently reviewed titles and abstracts to identify eligible guidelines. Clinical practice guidelines were eligible for inclusion if they provided recommendations on the pharmacological treatment of adolescents or adults with bipolar disorder, and were published between 2003 and 2014. Guidelines were excluded if they failed to meet the following definition: “statements that include recommendations intended to optimise patient care that are informed by a systematic review of evidence and an assessment of the benefits and harms of alternative care options” (Institute of Medicine, 2011). We in addition excluded guidelines with a focus on specific populations, such as pregnant women or individuals with co-morbidity medical conditions. When different editions of the same guideline were identified, only the most recent was considered, but we consulted all previous versions for retrieving as many details as possible on the methodology followed. Only guidelines in English, German, and Italian were considered. Conflicts between reviewers about eligibility were resolved by consensus. Third-party arbitration was available but not necessary. Eligible abstracts underwent full article review.
2. Methods 2.1. Data sources and search strategy We searched the National Guideline Clearinghouse, MEDLINE, EMBASE, PsychINFO and CINHAL databases for guidelines published between January 2003 and August 2014 using the following Medical Subject Headings search terms: “Bipolar Disorder”, “Bipolar”, “Manicdepression”, “Mania”, “Guidelines”, “Practice guideline”, “Recommendation”, “Algorithm”, “Pharmacological therapy”, “Pharmacological treatment”. The search was supplemented by hand-searching a number of additional web-based repositories of clinical practice guidelines (Supplement 1). We also hand-searched reference lists of review articles and selected papers on bipolar disorder.
2.3. Data extraction Two reviewers sequentially extracted relevant information from each eligible guideline: a first reviewer (AC) extracted the data, whereas a second reviewer (FG) checked the first reviewer's data abstraction forms for completeness and accuracy. Differences of opinion were resolved through consensus. Data extracted included information on year and country of publication, status of the guideline (new versus update), the producing organisation (scientific society versus other agencies), target audience, group members (number), and multidisciplinary of the panel (yes/no).
2.4. Assessment of guideline quality Guideline quality was assessed by using the Appraisal of Guidelines Research and Evaluation II (AGREE II) instrument (Brouwers et al., 2010a, 2010b, 2010c), which contains 23 items grouped in 6 domains: 1) scope and purpose; 2) stakeholder involvement; 3) rigour of development; 4) clarity and presentation; 5) applicability and 6) editorial independence. A final question evaluates the overall assessment of each guideline's quality by measuring the assessor's willingness to recommend the guideline for use in practice. Two reviewers independently rated each item on a 7-point Likert scale. Agreement between raters was investigated using Lin's (1989) concordance correlation coefficient. The AGREE II scores were standardized across domains by dividing the difference between the obtained score and the minimum possible score by the difference between the maximum and minimum possible scores. Domain scores were also averaged across guidelines to identify areas of strengths and weakness across guidelines. Mann–Whitney two-sample statistics and Spearman's rank correlation coefficients were used to analyse continuous data which were not normally distributed.
A. Castellani et al. / Journal of Affective Disorders 174 (2015) 45–50
3. Results 3.1. Guideline selection The search strategy identified 956 records, of which 893 were excluded because they were not relevant/pertinent. Of 63 records considered for full-text screening, 7 reports of clinical practice guidelines meeting the review entry criteria were included (Fig. 1). Web searching identified 13 additional reports of clinical practice guidelines, while hand-searching yielded 10 additional reports (Fig. 1). Overall, 30 reports, corresponding to 14 clinical practice guidelines, were included (references in Supplement 2). 3.2. Characteristics of included clinical practice guidelines The main characteristics of included guidelines are reported in Table 1. The majority were national guidelines, while three were international; nine were first editions, and five were updates. Eight guidelines were developed by specialty societies, while the others were developed by government agencies, universities or other organisations. The average size of guidelines development teams was 22 members, ranging from 7 to 55. In six cases, guideline development teams were multidisciplinary. In most cases specialists represented the target audience, although in some cases general practitioners and users were mentioned (Table 1). 3.3. Guideline quality Lin's concordance correlation coefficient showed very high (rho40.95) agreement between raters. The overall quality of included guidelines varied considerably, both within and across AGREE II domains (Table 2). Across guidelines, scores were highest for the domain clarity of presentation (mean 74% of the maximum possible score) and scope and purpose (mean 69.3% of the
Records identified through database searching (n=1153)
47
maximum possible score). Scores for stakeholder involvement (mean 50.5% of the maximum possible score) and editorial independence (mean 50.8 of the maximum possible score) were highly variable across guidelines. In particular, editorial independence had the highest variability within domain, with three guidelines scoring the maximum possible score and three the minimum (Table 2). The mean score for rigour of development was 46.8% of the maximum possible score, with no guidelines scoring the maximum score in this domain. The domain applicability received the lowest scores across guidelines (mean 33.4% of the maximum possible score), with four guidelines scoring 0% of the maximum possible score. Overall, six guidelines were rated as “recommended by reviewers”, two “recommended with modifications”, and six were “not recommended” according to AGREE II ratings. Rigour of guideline development was not associated with year of guideline publication (Spearman rho¼ 0.305, p ¼0.288) (Fig. 2). However, Fig. 3 demonstrates a linear relationship between domain 6 (editorial independence) and domain 3 (rigour of development) scores (Spearman rho¼ 0.842, p o0.001). In general, guidelines with lower editorial independence scores also had lower rigour of development scores, whereas those with higherquality domain scores scored high in both domains. Guidelines developed by specialty societies received lower scores as compared with guidelines developed by other agencies (Table 3). Differences reached statistical significance for stakeholder involvement (33.5% versus 73.2%, respectively; p ¼0.024), clarity of presentation (63.4% versus 88.5%; p ¼0.032) and applicability (15.9% versus 56.8%; p ¼0.011) (Table 3). Similarly, guidelines developed by multi-professional teams were of higher quality. Statistically significant differences were found for rigour of development (mean 76.3% versus 28.3%, respectively; p ¼0.005), clarity of presentation (mean 84.8% versus 73.3%, p ¼0.019) and applicability (mean 63.8% versus 14.2%, po 0.001).
Web searching in repositories of guidelines (listed in appendix)
Additional web searching of guidelines identified through manual inspection of references to included guidelines and review articles
Records screened after duplicates removed (n= 956) Records excluded (n=893)
Records included for fulltext assessment (n=63) Documents excluded (n= 26): - Full text unavalable (n=7) - Wrong focus (n=2) - Wrong methodology (n= 6) - Wrong lenguage (n=7) - Wrong diagnosis (n=4)
Documents included (n=7)
Documents selected (n=13)
Documents included in the review n=30 (corresponding to 14 guidelines) Fig. 1. Flow-chart of guideline identification and selection process.
Documents selected (n=10)
48
A. Castellani et al. / Journal of Affective Disorders 174 (2015) 45–50
Table 1 Main characteristics of guidelines for bipolar disorders published between 2003 and 2014. Short name
Full guideline name
Year
Country
Status
AACAP BAP CANMAT
American Academy of Child and Adolescent Psychiatry British Association for Psychopharmacology Canadian Network for Mood and Anxiety Treatments and ISBD International Society for Bipolar Disorders Deutsche Gesellshaft für Bipolare Störungen. DGPPN Deutsche Gesellshaft für Psychiatrie, Psychotherapie und Nervenheilkunde
2007 2009 2013
New Yes Update Yes Update Yes
2012
USA UK Canada/ world Germany
New
Yes
IPS
Indian Psychiatry Society
2005
India
New
Yes
MPA MALHI
2007 2009
Malaysia Australia
New New
Yes No
2006
UK
New
No
RANZCP
Malaysian Psychiatric Association Malhi GS, Adams D, Lampe L, Paton M, O'Connor N, Newton LA, Walter G, Taylor A, Porter R, Mulder RT, Berk M National Institute for Clinical Excellence/National Collaborating Centre for Mental Health Royal Australian and New Zealand College of Psychiatrists
2009
SING SMH
Scottish Intercollegiate Guideline Network Singapore Ministry of Health Guidelines
2005 2011
Australia/ Update Yes New Zealand Scotland New No Singapore New No
VA/DOD WFSBP WHO mhGAP
Department of Veterans Affaire, Department of Defence World Federation of Societies of Biological Psychiatry World Health Organization
2010 2012 2010
USA World World
DGBS
NICE
Scientific society
New No Update Yes New No
Target
Specialists
GPs, specialists, patients, carers Specialists, GPs Specialists
GPs, specialists Specialists, patients, carers Specialists GPs, specialists Specialists Specialists Health care providers
Group Multimembers professional 12 27 21
No No
23
Yes
14 36 11
No No
23
Yes
7
No
20 11
Yes Yes
22 55 23
Yes No Yes
GPs ¼general practitioners. Table 2 Quality assessment of guidelines for bipolar disorder using the AGREE II domain appraisal. Short guideline name
AACAP BAP CANMAT DGBS IPS MPA MALHI NICE RANZCP SING SMH VA/DOD WFSBP WHO mhGAP Mean (range)
a
AGREE domain D1 – Scope and purpose
D2 – Stakeholders involvement
D3 – Rigour of development
D4 – Clarity of presentation
D5 – Applicability
D6 – Editorial independence
Recommendeda
0.33 0.69 0.94 0.78 0.67 0.22 0.94 1.00 0.19 0.69 0.42 1.00 0.97 0.86 0.69 (0.19–1.00)
0.06 0.39 0.58 0.94 0.19 0.08 0.58 1.00 0.25 0.56 0.75 0.61 0.19 0.89 0.50 (0.06–1.00)
0.19 0.05 0.36 0.94 0.08 0.06 0.36 0.91 0.21 0.80 0.19 0.82 0.66 0.92 0.46 (0.05–0.94)
0.39 0.72 0.83 0.61 0.50 0.58 0.83 0.92 0.47 1.00 0.56 1.00 0.97 1.00 0.74 (0.39–1.00)
0.00 0.00 0.33 0.75 0.00 0.00 0.33 0.83 0.13 0.79 0.35 0.23 0.06 0.88 0.33 (0.00– 0.88)
0.33 0.08 0.67 0.83 0.00 0.00 0.67 0.50 0.29 0.79 0.00 1.00 0.96 1.00 0.50 (0.00–100)
N N N Y N N YM Y N Y Y Y YM Y
Y¼ yes; N¼ no; YM¼ yes with modifications.
4. Discussion 4.1. Clinical implications The present systematic review found variations in quality among clinical practice guidelines for the pharmacological treatment of bipolar disorder and, on an average, no improvement over the years. In Addition, specific areas of guideline development that need improvement have been highlighted. First, guidelines performed very low in terms of applicability, which is one of the key aspects that make guidelines useful in clinical practice (Gagliardi, 2012; Gagliardi et al., 2014). The content and format of guidelines may facilitate or impede their use, and current research has suggested that the most relevant
implementability elements, organised around an implementability framework, include adaptability, usability, relevance, validity, applicability, communicability, resource implication, implementation and evaluation (Gagliardi and Brouwers, 2012; Gagliardi, 2012; Gagliardi et al., 2014). The key idea is that addressing and taking into consideration these elements early in the initial phases of guideline development may significantly increase their applicability and uptake in practice. Second, overall guideline quality was poor with respect to rigour of the guideline development process, particularly in use of systematic methods to identify evidence. In addition, some guidelines were susceptible to bias because they lacked a description of editorial independence from funders and guideline developers failed to report potential conflicts of interest. Of note, these two
A. Castellani et al. / Journal of Affective Disorders 174 (2015) 45–50
aspects were highly correlated, as guidelines with lower editorial independence scores also had lower rigour of development scores. This finding is not new, as Bennett et al. (2012) showed a similar relationship in a sample of guidelines on oral medications for type 2 diabetes mellitus. Third, the quality of guidelines issued by specialty societies was lower as compared with guidelines issued by other agencies, especially in terms of applicability, clarity of presentation and stakeholders involvement. This finding is consistent with previous data showing that the quality of reporting of practice guidelines produced by specialty societies fell short of acceptable methodology up to mid 1998 (Grilli et al., 2000). 4.2. Limitations The present survey has limitations. First, our focus was very narrow, as we only included guidelines on medications for bipolar
D3 - Rigour of development, %
100
80
DGBS
WHO mhGAP
NICE SING
VA/DOD
WFSBP
60
40 MALHI
CANMAT
RANZCP
20 SMH
AACAP IPS
MPA
BAP
0 2004
2006
2008
2010
2012
2014
year Fig. 2. Relationship between the rigour of development domain summary score and year of guideline publication, by using the AGREE II instrument. WHO mhGAP
100
D6 – Editorial independence, %
WFSBP
VA/DOD
DGBS
80 SING MALHI CANMAT
60 NICE
40 AACAP RANZCP
20 BAP MPA IPS
0 0
SMH
20
49
disorder, leaving out guidelines on psychological and psychosocial interventions, as well as guidelines covering specific aspects of bipolar disorder, or guidelines on other psychiatric disorders. Hence, it is unclear whether the qualitative pattern described in our work can be generalised to other guidelines for bipolar disorder or other psychiatric or non-psychiatric disorders. A second limitation is that we did not discriminate between poor quality of reporting and limitation in the production of guidelines. Although we made every effort to access all available information on the guideline development process, without limiting the analysis to the main guideline report but always checking other supportive documentation, including supplementary web materials and previous editions of the same guideline, we acknowledge that our results could partly depend on the report itself and they might have been different if we had access to what was actually done. In Addition, even when all available information on the guideline development process is reported, the relationship between reporting and content quality is difficult to ascertain. For example, a formal fulfilment of the AGREE II items for rigour of development may imply that the methodology has been properly reported, but this reporting should be intended only as a proxy measure of the validity of the overall methodology. Similarly, for editorial independence the AGREE II requires an explicit statement that all group members have declared whether they have any conflict of interest. Although these statements are very relevant in terms of transparency, their practical implications remain unclear, as whether – and to what extent – any reported conflict of interest might have influenced the content or wording of a recommendation is almost impossible to judge. Third, we did not attempt to establish a relationship between rigour of development and guideline content. It would have been of interest to check, for example, whether high-quality guidelines, or guidelines with more editorial independence, produced different recommendations as compared with low-quality guidelines. We reasoned that this analysis would have been very weak, as other variables influencing the content of recommendations would have likely confounded the association. Year of guideline publication, for example, is one of these variables, as with new evidence accumulating over the years recent guidelines are expected to have different recommendations as compared with older guidelines. Finally, since no differences in quality of reporting related to language of publication have been demonstrated (Moher et al., 1996), we think that our focus only on guidelines published in English, Germany and Italian is unlikely to have caused any substantial sampling bias.
5. Conclusions 40
60
80
100
D3 - Rigour of development, % Fig. 3. Relationship between the rigour of development and editorial independence domain summary score, by using the AGREE II instrument.
Variability in quality and recommendations among clinical practice guidelines can lead to confusion and frustration on the part of health care providers, policy makers, and patients alike.
Table 3 Quality of guidelines for bipolar disorder developed by specialty societies versus other agencies. AGREE domain
Specialty society
p
No
D1 D2 D3 D4 D5 D6
– – – – – –
Scope and purpose Stakeholders involvement Rigour of development Clarity of presentation Applicability Editorial independence
Yes
Mean
SD (range)
Mean
SD (range)
0.81 0.73 0.66 0.88 0.56 0.66
0.22 0.18 0.31 0.17 0.29 0.37
0.59 0.33 0.31 0.63 0.15 0.39
0.31 0.29 0.32 0.19 0.26 0.38
(0.42–1.00) (0.56–1.00) (0.19–0.92) (0.56–1.00) (0.23–0.88) (0.00–1.00)
(0.19–0.97) (0.6–0.94) (0.5–0.94) (0.39–0.97) (0.00–0.75) (0.00–0.96)
0.120 0.024 0.093 0.032 0.011 0.217
50
A. Castellani et al. / Journal of Affective Disorders 174 (2015) 45–50
Rigour of development is critically important because guidelines are designed to promote effective and efficient health care and the implementation of guidelines based on an incomplete evidence base, or a biased synthesis and interpretation, can lead to suboptimal patient care and outcomes. For these reasons, our findings highlight that the quality of clinical practice guidelines for the pharmacological treatment of bipolar disorder needs to be improved. We suggest that an explicit, transparent, and unbiased process for translating the body of evidence in systematic reviews into clinical recommendations should be used. Examples of such formal processes include the Grading of Recommendations Assessment, Development and Evaluation (GRADE) tool (Barbui et al., 2010; Barbui and Cipriani, 2011a). This methodology, developed by an international network of methodologists with an interest for grading quality of evidence and strength of recommendations, may be employed to develop specific treatment recommendations for professionals working in a single mental health service or department, but also to develop recommendations for a wide range of interventions and disorders to be adopted at a district or regional level, or at a national or global level. Use of a tool such as GRADE would decrease variations in quality among clinical practice guidelines, especially in terms of rigour of development, group composition and stakeholders involvement. Until that time, users of clinical practice guidelines may consider the AGREE II instrument as a potentially useful framework for assessing the quality of clinical practice guidelines. This tool measures the confidence that the potential biases of guideline development have been addressed adequately and that the recommendations are both internally and externally valid, and are feasible for practice. Therefore, local, regional, national or International user groups may apply the AGREE II instrument to guidelines in any disease area including those for diagnosis, health promotion, treatment or interventions. On the basis of findings from this study, we suggest that users of guidelines for the pharmacological treatment of bipolar disorder need to critically appraise the guidelines they are considering to implement with a tool such as AGREE II. Role of funding source Nothing declared.
Conflict of interest None.
Acknowledgements None.
Appendix A. Supporting information Supplementary data associated with this article can be found in the online version at http://dx.doi.org/10.1016/j.jad.2014.11.032.
Rerefences Alonso-Coello, P., Irfan, A., Sola, I., Gich, I., gado-Noguera, M., Rigau, D., Tort, S., Bonfill, X., Burgers, J., Schunemann, H., 2010. The quality of clinical practice guidelines over the last two decades: a systematic review of guideline appraisal studies. Qual. Saf. Health Care 19, e58. Barbui, C., Dua, T., van Ommeren, M., Yasamy, M., Fleischmann, A., Clark, N., Thornicroft, G., Hill, S., Saxena, S., 2010. Challenges in developing evidencebased recommendations using the grade approach: the case of mental, neurological, and substance use disorders. PLoS Med. 7 (8), e1000322. Barbui, C., Cipriani, A., 2011a. What are evidence-based treatment recommendations? Epidemiol. Psychiatr. Sci. 20, 29–31. Barbui, C., Tansella, M., 2011b. Cochrane reviews impact on mental health policy and practice. Epidemiol. Psychiatr. Sci. 20, 211–214. Bennett, W.L., Odelola, O.A., Wilson, L.M., Bolen, S., Selvaraj, S., Robinson, K.A., Bass, E.B., Puhan, M.A., 2012. Evaluation of guideline recommendations on oral medications for type 2 diabetes mellitus: a systematic review. Ann Intern. Med. 156, 27–36. Brouwers, M.C., Kho, M.E., Browman, G.P., Burgers, J.S., Cluzeau, F., Feder, G., Fervers, B., Graham, I.D., Grimshaw, J., Hanna, S.E., Littlejohns, P., Makarski, J., Zitzelsberger, L., 2010a. AGREE II: advancing guideline development, reporting and evaluation in health care. J. Clin. Epidemiol. 63, 1308–1311. Brouwers, M.C., Kho, M.E., Browman, G.P., Burgers, J.S., Cluzeau, F., Feder, G., Fervers, B., Graham, I.D., Hanna, S.E., Makarski, J., 2010b. Development of the AGREE II, part 1: performance, usefulness and areas for improvement. CMAJ 182, 1045–1052. Brouwers, M.C., Kho, M.E., Browman, G.P., Burgers, J.S., Cluzeau, F., Feder, G., Fervers, B., Graham, I.D., Hanna, S.E., Makarski, J., 2010c. Development of the AGREE II, part 2: assessment of validity of items and tools to support application. CMAJ 182, E472–E478. Cipriani, A., Barbui, C., 2006. What is a systematic review? Epidemiologia e Psichiatria Sociale 15, 174–175. Gagliardi, A.R., 2012. Translating knowledge to practice: optimizing the use of guidelines. Epidemiol. Psychiatr. Sci. 21, 231–236. Gagliardi, A.R., Brouwers, M.C., 2012. Integrating guideline development and implementation: analysis of guideline development manual instructions for generating implementation advice. Implement. Sci. 7, 67. Gagliardi, A.R., Brouwers, M.C., Bhattacharyya, O.K., 2014. A framework of the desirable features of guideline implementation tools (GItools): Delphi survey and assessment of GItools. Implement. Sci. 9, 98. Grilli, R., Magrini, N., Penna, A., Mura, G., Liberati, A., 2000. Practice guidelines developed by specialty societies: the need for a critical appraisal. Lancet 355, 103–106. Haynes, B., Haines, A., 1998. Barriers and bridges to evidence based clinical practice. BMJ 317, 273–276. Institute of Medicine, 2011. Clinical practice guidelines we can trust. The National Academies Press, Washington, D.C. Lin, L.I., 1989. A concordance correlation coefficient to evaluate reproducibility. Biometrics 45, 255–268. Moher, D., Fortin, P., Jadad, A.R., Juni, P., Klassen, T., Le, L.J., Liberati, A., Linde, K., Penna, A., 1996. Completeness of reporting of trials published in languages other than English: implications for conduct and reporting of systematic reviews. Lancet 347, 363–366. Norris, S.L., Holmer, H.K., Ogden, L.A., Burda, B.U., 2011. Conflict of interest in clinical practice guideline development: a systematic review. PLoS ONE 6 (10), e25153. Norris, S.L., Holmer, H.K., Ogden, L.A., Burda, B.U., Fu, R., 2013. Conflicts of interest among authors of clinical practice guidelines for glycemic control in type 2 diabetes mellitus. PLoS ONE 8 (10), e75284. Norris, S.L., Holmer, H.K., Ogden, L.A., Selph, S.S., Fu, R., 2012. Conflict of interest disclosures for clinical practice guidelines in the national guideline clearinghouse. PLoS ONE 7 (11), e47343. Sackett, D.L., Rosenberg, W.M., Gray, J.A., Haynes, R.B., Richardson, W.S., 1996. Evidence based medicine: what it is and what it isn't. BMJ 312, 71–72.
Contents lists available at ScienceDirect
Journal of Affective Disorders journal homepage: www.elsevier.com/locate/jad
Review
Rigour of development of clinical practice guidelines for the pharmacological treatment of bipolar disorder: Systematic review Arianna Castellani, Francesca Girlanda, Corrado Barbui n WHO Collaborating Centre for Research and Training in Mental Health and Service Evaluation, Department of Public Health and Community Medicine, Section of Psychiatry, University of Verona, Verona, Italy
art ic l e i nf o
a b s t r a c t
Article history: Received 17 September 2014 Received in revised form 17 November 2014 Accepted 17 November 2014 Available online 25 November 2014
Background: There is an increasing concern about the quality of clinical practice guidelines. Because no information is available on the rigour of development of clinical practice guidelines for bipolar disorder, we carried out a systematic review of those focusing on its pharmacological treatment. Methods: We searched the National Guideline Clearinghouse, MEDLINE, EMBASE, PsychINFO and CINHAL for guidelines published from 2003 to 2014. The quality of each guideline was assessed using the Appraisal of Guidelines for Research and Evaluation II (AGREE II). Results: Fourteen guidelines were appraised. The overall quality of included guidelines varied considerably, both within and across AGREE II domains. Overall, six guidelines were rated as “recommended”, two “recommended with modifications”, and six were not recommended according to AGREE II ratings. The mean score for rigour of development was 46.8% of the maximum possible score, with no guidelines scoring the maximum score in this domain. Guidelines with lower editorial independence scores also had lower rigour of development scores, whereas those with higher-quality domain scores scored high in both domains. Limitations: As current appraisal focused on guidelines for the pharmacological treatment of bipolar disorder, it will be important to critically assess the rigour of development of other guidelines for bipolar and other psychiatric disorders. Conclusions: Health care providers, policy makers, physicians and patients alike need to be aware of the variability in guideline quality and identify the high-quality guidelines that meet their needs. & 2014 Elsevier B.V. All rights reserved.
Keywords: Clinical practice guidelines Bipolar disorder Quality
Contents 1. 2.
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.1. Data sources and search strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.2. Guideline inclusion and exclusion criteria . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.3. Data extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.4. Assessment of guideline quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3. Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.1. Guideline selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2. Characteristics of included clinical practice guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3. Guideline quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4. Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1. Clinical implications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2. Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Role of funding source . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conflict of interest. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
n
Corresponding author. Tel.: þ 39 045 812 6418; fax: þ39 045 8027498.
http://dx.doi.org/10.1016/j.jad.2014.11.032 0165-0327/& 2014 Elsevier B.V. All rights reserved.
46 46 46 46 46 46 47 47 47 47 48 48 49 49 50 50
46
A. Castellani et al. / Journal of Affective Disorders 174 (2015) 45–50
Acknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 Appendix A. Supporting information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 Rerefences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
1. Introduction
2.2. Guideline inclusion and exclusion criteria
According to the principles of evidence-based medicine, research findings should guide doctors when taking decisions in daily clinical practice (Haynes and Haines, 1998; Sackett et al., 1996). However, not all research findings are attributed the same value, as randomised controlled trials and systematic reviews of randomised evidence are nowadays at the pinnacle of the evidence hierarchy (Cipriani and Barbui, 2006). In such a system a crucial issue is how the evidence base may effectively be translated into practice, considering that access and use of systematic reviews may not be straightforward for most doctors in most countries of the world (Barbui and Tansella, 2011b). In order to tackle this global issue, over the past 20 years clinical practice guidelines have become an increasingly popular tool for synthesis of clinical information so as to change clinical practice and improve quality of health care (Barbui and Cipriani, 2011a). Such a quantitative growth in the number of guidelines is, however, a source of concern. First, despite some increase in quality over time, the rigour of guideline development has remained moderate to low over the last two decades (AlonsoCoello et al., 2010). Second, although there are few studies describing financial conflicts of interest for guideline authors, the available data suggest that there is a high prevalence of nondisclosure among authors across a variety of clinical specialties, and a high percentage of authors with disclosures report conflicts of interest (Norris et al., 2011, 2012, 2013). Third, the quality of clinical practice guidelines developed by specialty societies has been shown to be highly unsatisfactory (Grilli et al., 2000). The present systematic review was therefore carried out to evaluate the quality of clinical practice guidelines for the pharmacological management of bipolar disorder. The case example of bipolar disorder is particularly interesting, as several active pharmacological treatments are available, a massive number of randomized trials and systematic reviews have been produced, and the overall quality of evidence remains a major concern. Specific aims were the following: (a) to determine whether the rigour of guideline development increased over the last decade; (b) to ascertain whether the rigour of guideline development was associated with less conflicts of interest; (c) to compare the quality of guidelines issued by specialty societies versus other agencies.
Two reviewers independently reviewed titles and abstracts to identify eligible guidelines. Clinical practice guidelines were eligible for inclusion if they provided recommendations on the pharmacological treatment of adolescents or adults with bipolar disorder, and were published between 2003 and 2014. Guidelines were excluded if they failed to meet the following definition: “statements that include recommendations intended to optimise patient care that are informed by a systematic review of evidence and an assessment of the benefits and harms of alternative care options” (Institute of Medicine, 2011). We in addition excluded guidelines with a focus on specific populations, such as pregnant women or individuals with co-morbidity medical conditions. When different editions of the same guideline were identified, only the most recent was considered, but we consulted all previous versions for retrieving as many details as possible on the methodology followed. Only guidelines in English, German, and Italian were considered. Conflicts between reviewers about eligibility were resolved by consensus. Third-party arbitration was available but not necessary. Eligible abstracts underwent full article review.
2. Methods 2.1. Data sources and search strategy We searched the National Guideline Clearinghouse, MEDLINE, EMBASE, PsychINFO and CINHAL databases for guidelines published between January 2003 and August 2014 using the following Medical Subject Headings search terms: “Bipolar Disorder”, “Bipolar”, “Manicdepression”, “Mania”, “Guidelines”, “Practice guideline”, “Recommendation”, “Algorithm”, “Pharmacological therapy”, “Pharmacological treatment”. The search was supplemented by hand-searching a number of additional web-based repositories of clinical practice guidelines (Supplement 1). We also hand-searched reference lists of review articles and selected papers on bipolar disorder.
2.3. Data extraction Two reviewers sequentially extracted relevant information from each eligible guideline: a first reviewer (AC) extracted the data, whereas a second reviewer (FG) checked the first reviewer's data abstraction forms for completeness and accuracy. Differences of opinion were resolved through consensus. Data extracted included information on year and country of publication, status of the guideline (new versus update), the producing organisation (scientific society versus other agencies), target audience, group members (number), and multidisciplinary of the panel (yes/no).
2.4. Assessment of guideline quality Guideline quality was assessed by using the Appraisal of Guidelines Research and Evaluation II (AGREE II) instrument (Brouwers et al., 2010a, 2010b, 2010c), which contains 23 items grouped in 6 domains: 1) scope and purpose; 2) stakeholder involvement; 3) rigour of development; 4) clarity and presentation; 5) applicability and 6) editorial independence. A final question evaluates the overall assessment of each guideline's quality by measuring the assessor's willingness to recommend the guideline for use in practice. Two reviewers independently rated each item on a 7-point Likert scale. Agreement between raters was investigated using Lin's (1989) concordance correlation coefficient. The AGREE II scores were standardized across domains by dividing the difference between the obtained score and the minimum possible score by the difference between the maximum and minimum possible scores. Domain scores were also averaged across guidelines to identify areas of strengths and weakness across guidelines. Mann–Whitney two-sample statistics and Spearman's rank correlation coefficients were used to analyse continuous data which were not normally distributed.
A. Castellani et al. / Journal of Affective Disorders 174 (2015) 45–50
3. Results 3.1. Guideline selection The search strategy identified 956 records, of which 893 were excluded because they were not relevant/pertinent. Of 63 records considered for full-text screening, 7 reports of clinical practice guidelines meeting the review entry criteria were included (Fig. 1). Web searching identified 13 additional reports of clinical practice guidelines, while hand-searching yielded 10 additional reports (Fig. 1). Overall, 30 reports, corresponding to 14 clinical practice guidelines, were included (references in Supplement 2). 3.2. Characteristics of included clinical practice guidelines The main characteristics of included guidelines are reported in Table 1. The majority were national guidelines, while three were international; nine were first editions, and five were updates. Eight guidelines were developed by specialty societies, while the others were developed by government agencies, universities or other organisations. The average size of guidelines development teams was 22 members, ranging from 7 to 55. In six cases, guideline development teams were multidisciplinary. In most cases specialists represented the target audience, although in some cases general practitioners and users were mentioned (Table 1). 3.3. Guideline quality Lin's concordance correlation coefficient showed very high (rho40.95) agreement between raters. The overall quality of included guidelines varied considerably, both within and across AGREE II domains (Table 2). Across guidelines, scores were highest for the domain clarity of presentation (mean 74% of the maximum possible score) and scope and purpose (mean 69.3% of the
Records identified through database searching (n=1153)
47
maximum possible score). Scores for stakeholder involvement (mean 50.5% of the maximum possible score) and editorial independence (mean 50.8 of the maximum possible score) were highly variable across guidelines. In particular, editorial independence had the highest variability within domain, with three guidelines scoring the maximum possible score and three the minimum (Table 2). The mean score for rigour of development was 46.8% of the maximum possible score, with no guidelines scoring the maximum score in this domain. The domain applicability received the lowest scores across guidelines (mean 33.4% of the maximum possible score), with four guidelines scoring 0% of the maximum possible score. Overall, six guidelines were rated as “recommended by reviewers”, two “recommended with modifications”, and six were “not recommended” according to AGREE II ratings. Rigour of guideline development was not associated with year of guideline publication (Spearman rho¼ 0.305, p ¼0.288) (Fig. 2). However, Fig. 3 demonstrates a linear relationship between domain 6 (editorial independence) and domain 3 (rigour of development) scores (Spearman rho¼ 0.842, p o0.001). In general, guidelines with lower editorial independence scores also had lower rigour of development scores, whereas those with higherquality domain scores scored high in both domains. Guidelines developed by specialty societies received lower scores as compared with guidelines developed by other agencies (Table 3). Differences reached statistical significance for stakeholder involvement (33.5% versus 73.2%, respectively; p ¼0.024), clarity of presentation (63.4% versus 88.5%; p ¼0.032) and applicability (15.9% versus 56.8%; p ¼0.011) (Table 3). Similarly, guidelines developed by multi-professional teams were of higher quality. Statistically significant differences were found for rigour of development (mean 76.3% versus 28.3%, respectively; p ¼0.005), clarity of presentation (mean 84.8% versus 73.3%, p ¼0.019) and applicability (mean 63.8% versus 14.2%, po 0.001).
Web searching in repositories of guidelines (listed in appendix)
Additional web searching of guidelines identified through manual inspection of references to included guidelines and review articles
Records screened after duplicates removed (n= 956) Records excluded (n=893)
Records included for fulltext assessment (n=63) Documents excluded (n= 26): - Full text unavalable (n=7) - Wrong focus (n=2) - Wrong methodology (n= 6) - Wrong lenguage (n=7) - Wrong diagnosis (n=4)
Documents included (n=7)
Documents selected (n=13)
Documents included in the review n=30 (corresponding to 14 guidelines) Fig. 1. Flow-chart of guideline identification and selection process.
Documents selected (n=10)
48
A. Castellani et al. / Journal of Affective Disorders 174 (2015) 45–50
Table 1 Main characteristics of guidelines for bipolar disorders published between 2003 and 2014. Short name
Full guideline name
Year
Country
Status
AACAP BAP CANMAT
American Academy of Child and Adolescent Psychiatry British Association for Psychopharmacology Canadian Network for Mood and Anxiety Treatments and ISBD International Society for Bipolar Disorders Deutsche Gesellshaft für Bipolare Störungen. DGPPN Deutsche Gesellshaft für Psychiatrie, Psychotherapie und Nervenheilkunde
2007 2009 2013
New Yes Update Yes Update Yes
2012
USA UK Canada/ world Germany
New
Yes
IPS
Indian Psychiatry Society
2005
India
New
Yes
MPA MALHI
2007 2009
Malaysia Australia
New New
Yes No
2006
UK
New
No
RANZCP
Malaysian Psychiatric Association Malhi GS, Adams D, Lampe L, Paton M, O'Connor N, Newton LA, Walter G, Taylor A, Porter R, Mulder RT, Berk M National Institute for Clinical Excellence/National Collaborating Centre for Mental Health Royal Australian and New Zealand College of Psychiatrists
2009
SING SMH
Scottish Intercollegiate Guideline Network Singapore Ministry of Health Guidelines
2005 2011
Australia/ Update Yes New Zealand Scotland New No Singapore New No
VA/DOD WFSBP WHO mhGAP
Department of Veterans Affaire, Department of Defence World Federation of Societies of Biological Psychiatry World Health Organization
2010 2012 2010
USA World World
DGBS
NICE
Scientific society
New No Update Yes New No
Target
Specialists
GPs, specialists, patients, carers Specialists, GPs Specialists
GPs, specialists Specialists, patients, carers Specialists GPs, specialists Specialists Specialists Health care providers
Group Multimembers professional 12 27 21
No No
23
Yes
14 36 11
No No
23
Yes
7
No
20 11
Yes Yes
22 55 23
Yes No Yes
GPs ¼general practitioners. Table 2 Quality assessment of guidelines for bipolar disorder using the AGREE II domain appraisal. Short guideline name
AACAP BAP CANMAT DGBS IPS MPA MALHI NICE RANZCP SING SMH VA/DOD WFSBP WHO mhGAP Mean (range)
a
AGREE domain D1 – Scope and purpose
D2 – Stakeholders involvement
D3 – Rigour of development
D4 – Clarity of presentation
D5 – Applicability
D6 – Editorial independence
Recommendeda
0.33 0.69 0.94 0.78 0.67 0.22 0.94 1.00 0.19 0.69 0.42 1.00 0.97 0.86 0.69 (0.19–1.00)
0.06 0.39 0.58 0.94 0.19 0.08 0.58 1.00 0.25 0.56 0.75 0.61 0.19 0.89 0.50 (0.06–1.00)
0.19 0.05 0.36 0.94 0.08 0.06 0.36 0.91 0.21 0.80 0.19 0.82 0.66 0.92 0.46 (0.05–0.94)
0.39 0.72 0.83 0.61 0.50 0.58 0.83 0.92 0.47 1.00 0.56 1.00 0.97 1.00 0.74 (0.39–1.00)
0.00 0.00 0.33 0.75 0.00 0.00 0.33 0.83 0.13 0.79 0.35 0.23 0.06 0.88 0.33 (0.00– 0.88)
0.33 0.08 0.67 0.83 0.00 0.00 0.67 0.50 0.29 0.79 0.00 1.00 0.96 1.00 0.50 (0.00–100)
N N N Y N N YM Y N Y Y Y YM Y
Y¼ yes; N¼ no; YM¼ yes with modifications.
4. Discussion 4.1. Clinical implications The present systematic review found variations in quality among clinical practice guidelines for the pharmacological treatment of bipolar disorder and, on an average, no improvement over the years. In Addition, specific areas of guideline development that need improvement have been highlighted. First, guidelines performed very low in terms of applicability, which is one of the key aspects that make guidelines useful in clinical practice (Gagliardi, 2012; Gagliardi et al., 2014). The content and format of guidelines may facilitate or impede their use, and current research has suggested that the most relevant
implementability elements, organised around an implementability framework, include adaptability, usability, relevance, validity, applicability, communicability, resource implication, implementation and evaluation (Gagliardi and Brouwers, 2012; Gagliardi, 2012; Gagliardi et al., 2014). The key idea is that addressing and taking into consideration these elements early in the initial phases of guideline development may significantly increase their applicability and uptake in practice. Second, overall guideline quality was poor with respect to rigour of the guideline development process, particularly in use of systematic methods to identify evidence. In addition, some guidelines were susceptible to bias because they lacked a description of editorial independence from funders and guideline developers failed to report potential conflicts of interest. Of note, these two
A. Castellani et al. / Journal of Affective Disorders 174 (2015) 45–50
aspects were highly correlated, as guidelines with lower editorial independence scores also had lower rigour of development scores. This finding is not new, as Bennett et al. (2012) showed a similar relationship in a sample of guidelines on oral medications for type 2 diabetes mellitus. Third, the quality of guidelines issued by specialty societies was lower as compared with guidelines issued by other agencies, especially in terms of applicability, clarity of presentation and stakeholders involvement. This finding is consistent with previous data showing that the quality of reporting of practice guidelines produced by specialty societies fell short of acceptable methodology up to mid 1998 (Grilli et al., 2000). 4.2. Limitations The present survey has limitations. First, our focus was very narrow, as we only included guidelines on medications for bipolar
D3 - Rigour of development, %
100
80
DGBS
WHO mhGAP
NICE SING
VA/DOD
WFSBP
60
40 MALHI
CANMAT
RANZCP
20 SMH
AACAP IPS
MPA
BAP
0 2004
2006
2008
2010
2012
2014
year Fig. 2. Relationship between the rigour of development domain summary score and year of guideline publication, by using the AGREE II instrument. WHO mhGAP
100
D6 – Editorial independence, %
WFSBP
VA/DOD
DGBS
80 SING MALHI CANMAT
60 NICE
40 AACAP RANZCP
20 BAP MPA IPS
0 0
SMH
20
49
disorder, leaving out guidelines on psychological and psychosocial interventions, as well as guidelines covering specific aspects of bipolar disorder, or guidelines on other psychiatric disorders. Hence, it is unclear whether the qualitative pattern described in our work can be generalised to other guidelines for bipolar disorder or other psychiatric or non-psychiatric disorders. A second limitation is that we did not discriminate between poor quality of reporting and limitation in the production of guidelines. Although we made every effort to access all available information on the guideline development process, without limiting the analysis to the main guideline report but always checking other supportive documentation, including supplementary web materials and previous editions of the same guideline, we acknowledge that our results could partly depend on the report itself and they might have been different if we had access to what was actually done. In Addition, even when all available information on the guideline development process is reported, the relationship between reporting and content quality is difficult to ascertain. For example, a formal fulfilment of the AGREE II items for rigour of development may imply that the methodology has been properly reported, but this reporting should be intended only as a proxy measure of the validity of the overall methodology. Similarly, for editorial independence the AGREE II requires an explicit statement that all group members have declared whether they have any conflict of interest. Although these statements are very relevant in terms of transparency, their practical implications remain unclear, as whether – and to what extent – any reported conflict of interest might have influenced the content or wording of a recommendation is almost impossible to judge. Third, we did not attempt to establish a relationship between rigour of development and guideline content. It would have been of interest to check, for example, whether high-quality guidelines, or guidelines with more editorial independence, produced different recommendations as compared with low-quality guidelines. We reasoned that this analysis would have been very weak, as other variables influencing the content of recommendations would have likely confounded the association. Year of guideline publication, for example, is one of these variables, as with new evidence accumulating over the years recent guidelines are expected to have different recommendations as compared with older guidelines. Finally, since no differences in quality of reporting related to language of publication have been demonstrated (Moher et al., 1996), we think that our focus only on guidelines published in English, Germany and Italian is unlikely to have caused any substantial sampling bias.
5. Conclusions 40
60
80
100
D3 - Rigour of development, % Fig. 3. Relationship between the rigour of development and editorial independence domain summary score, by using the AGREE II instrument.
Variability in quality and recommendations among clinical practice guidelines can lead to confusion and frustration on the part of health care providers, policy makers, and patients alike.
Table 3 Quality of guidelines for bipolar disorder developed by specialty societies versus other agencies. AGREE domain
Specialty society
p
No
D1 D2 D3 D4 D5 D6
– – – – – –
Scope and purpose Stakeholders involvement Rigour of development Clarity of presentation Applicability Editorial independence
Yes
Mean
SD (range)
Mean
SD (range)
0.81 0.73 0.66 0.88 0.56 0.66
0.22 0.18 0.31 0.17 0.29 0.37
0.59 0.33 0.31 0.63 0.15 0.39
0.31 0.29 0.32 0.19 0.26 0.38
(0.42–1.00) (0.56–1.00) (0.19–0.92) (0.56–1.00) (0.23–0.88) (0.00–1.00)
(0.19–0.97) (0.6–0.94) (0.5–0.94) (0.39–0.97) (0.00–0.75) (0.00–0.96)
0.120 0.024 0.093 0.032 0.011 0.217
50
A. Castellani et al. / Journal of Affective Disorders 174 (2015) 45–50
Rigour of development is critically important because guidelines are designed to promote effective and efficient health care and the implementation of guidelines based on an incomplete evidence base, or a biased synthesis and interpretation, can lead to suboptimal patient care and outcomes. For these reasons, our findings highlight that the quality of clinical practice guidelines for the pharmacological treatment of bipolar disorder needs to be improved. We suggest that an explicit, transparent, and unbiased process for translating the body of evidence in systematic reviews into clinical recommendations should be used. Examples of such formal processes include the Grading of Recommendations Assessment, Development and Evaluation (GRADE) tool (Barbui et al., 2010; Barbui and Cipriani, 2011a). This methodology, developed by an international network of methodologists with an interest for grading quality of evidence and strength of recommendations, may be employed to develop specific treatment recommendations for professionals working in a single mental health service or department, but also to develop recommendations for a wide range of interventions and disorders to be adopted at a district or regional level, or at a national or global level. Use of a tool such as GRADE would decrease variations in quality among clinical practice guidelines, especially in terms of rigour of development, group composition and stakeholders involvement. Until that time, users of clinical practice guidelines may consider the AGREE II instrument as a potentially useful framework for assessing the quality of clinical practice guidelines. This tool measures the confidence that the potential biases of guideline development have been addressed adequately and that the recommendations are both internally and externally valid, and are feasible for practice. Therefore, local, regional, national or International user groups may apply the AGREE II instrument to guidelines in any disease area including those for diagnosis, health promotion, treatment or interventions. On the basis of findings from this study, we suggest that users of guidelines for the pharmacological treatment of bipolar disorder need to critically appraise the guidelines they are considering to implement with a tool such as AGREE II. Role of funding source Nothing declared.
Conflict of interest None.
Acknowledgements None.
Appendix A. Supporting information Supplementary data associated with this article can be found in the online version at http://dx.doi.org/10.1016/j.jad.2014.11.032.
Rerefences Alonso-Coello, P., Irfan, A., Sola, I., Gich, I., gado-Noguera, M., Rigau, D., Tort, S., Bonfill, X., Burgers, J., Schunemann, H., 2010. The quality of clinical practice guidelines over the last two decades: a systematic review of guideline appraisal studies. Qual. Saf. Health Care 19, e58. Barbui, C., Dua, T., van Ommeren, M., Yasamy, M., Fleischmann, A., Clark, N., Thornicroft, G., Hill, S., Saxena, S., 2010. Challenges in developing evidencebased recommendations using the grade approach: the case of mental, neurological, and substance use disorders. PLoS Med. 7 (8), e1000322. Barbui, C., Cipriani, A., 2011a. What are evidence-based treatment recommendations? Epidemiol. Psychiatr. Sci. 20, 29–31. Barbui, C., Tansella, M., 2011b. Cochrane reviews impact on mental health policy and practice. Epidemiol. Psychiatr. Sci. 20, 211–214. Bennett, W.L., Odelola, O.A., Wilson, L.M., Bolen, S., Selvaraj, S., Robinson, K.A., Bass, E.B., Puhan, M.A., 2012. Evaluation of guideline recommendations on oral medications for type 2 diabetes mellitus: a systematic review. Ann Intern. Med. 156, 27–36. Brouwers, M.C., Kho, M.E., Browman, G.P., Burgers, J.S., Cluzeau, F., Feder, G., Fervers, B., Graham, I.D., Grimshaw, J., Hanna, S.E., Littlejohns, P., Makarski, J., Zitzelsberger, L., 2010a. AGREE II: advancing guideline development, reporting and evaluation in health care. J. Clin. Epidemiol. 63, 1308–1311. Brouwers, M.C., Kho, M.E., Browman, G.P., Burgers, J.S., Cluzeau, F., Feder, G., Fervers, B., Graham, I.D., Hanna, S.E., Makarski, J., 2010b. Development of the AGREE II, part 1: performance, usefulness and areas for improvement. CMAJ 182, 1045–1052. Brouwers, M.C., Kho, M.E., Browman, G.P., Burgers, J.S., Cluzeau, F., Feder, G., Fervers, B., Graham, I.D., Hanna, S.E., Makarski, J., 2010c. Development of the AGREE II, part 2: assessment of validity of items and tools to support application. CMAJ 182, E472–E478. Cipriani, A., Barbui, C., 2006. What is a systematic review? Epidemiologia e Psichiatria Sociale 15, 174–175. Gagliardi, A.R., 2012. Translating knowledge to practice: optimizing the use of guidelines. Epidemiol. Psychiatr. Sci. 21, 231–236. Gagliardi, A.R., Brouwers, M.C., 2012. Integrating guideline development and implementation: analysis of guideline development manual instructions for generating implementation advice. Implement. Sci. 7, 67. Gagliardi, A.R., Brouwers, M.C., Bhattacharyya, O.K., 2014. A framework of the desirable features of guideline implementation tools (GItools): Delphi survey and assessment of GItools. Implement. Sci. 9, 98. Grilli, R., Magrini, N., Penna, A., Mura, G., Liberati, A., 2000. Practice guidelines developed by specialty societies: the need for a critical appraisal. Lancet 355, 103–106. Haynes, B., Haines, A., 1998. Barriers and bridges to evidence based clinical practice. BMJ 317, 273–276. Institute of Medicine, 2011. Clinical practice guidelines we can trust. The National Academies Press, Washington, D.C. Lin, L.I., 1989. A concordance correlation coefficient to evaluate reproducibility. Biometrics 45, 255–268. Moher, D., Fortin, P., Jadad, A.R., Juni, P., Klassen, T., Le, L.J., Liberati, A., Linde, K., Penna, A., 1996. Completeness of reporting of trials published in languages other than English: implications for conduct and reporting of systematic reviews. Lancet 347, 363–366. Norris, S.L., Holmer, H.K., Ogden, L.A., Burda, B.U., 2011. Conflict of interest in clinical practice guideline development: a systematic review. PLoS ONE 6 (10), e25153. Norris, S.L., Holmer, H.K., Ogden, L.A., Burda, B.U., Fu, R., 2013. Conflicts of interest among authors of clinical practice guidelines for glycemic control in type 2 diabetes mellitus. PLoS ONE 8 (10), e75284. Norris, S.L., Holmer, H.K., Ogden, L.A., Selph, S.S., Fu, R., 2012. Conflict of interest disclosures for clinical practice guidelines in the national guideline clearinghouse. PLoS ONE 7 (11), e47343. Sackett, D.L., Rosenberg, W.M., Gray, J.A., Haynes, R.B., Richardson, W.S., 1996. Evidence based medicine: what it is and what it isn't. BMJ 312, 71–72.
