[
Original Research Education, Research, and Quality Improvement
]
Quality Assessment of Clinical Practice Guidelines for Respiratory Diseases in China A Systematic Appraisal Mei Jiang, PhD; Li-yue Liao, MD; Xiao-qing Liu, MD; Wei-qun He, MD; Wei-jie Guan, PhD; Hao Chen, MD; and Yi-min Li, MD
There has been a significant increase in the publication of clinical practice guidelines (CPGs) for respiratory diseases in China. However, little is known about the quality and potential impacts of these CPGs. Our objective was to critically evaluate the quality of Chinese CPGs for respiratory diseases that were published in peer-reviewed medical journals. BACKGROUND:
A systematic search of scientific literature published between 1979 and 2013 was undertaken to identify and select CPGs that were related to respiratory diseases. Four Chinese databases (the Chinese Biomedical Literature database [CBM], the China National Knowledge Infrastructure [CNKI], the VIP database, and the WANFANG database) were used. The quality of eligible guidelines was assessed independently by four reviewers using the Appraisal of Guidelines for Research and Evaluation (AGREE) II instrument. The overall agreement among reviewers was evaluated using an intraclass correlation coefficient. METHODS:
A total of 109 guidelines published in 27 medical journals from 1979 to 2013 were evaluated. The overall agreement among reviewers was considered good (intraclass correlation coefficient, 0.838; 95% CI, 0.812-0.862). The scores of the six AGREE domains were low: 57.3% for scope and purpose (range, 4.2%-80.5%), 23.8% for stakeholder involvement (range, 2.8%-54.2%), 7.7% for rigor of development (range, 0%-27.1%), 59.8% for clarity and presentation (range, 22.2%-80.6%), 10.9% for applicability (range, 0%-22.9%), and 0.6% for editorial independence (range, 0%-16.7%). Scores for all guidelines were below 60%, and only three guidelines (2.8%) were recommended for clinical practice with modifications.
RESULTS:
The quality of the guidelines was low, and stakeholder involvement, rigor of development, applicability, and editorial independence should be considered in the future development of CPGs for respiratory diseases in China. CHEST 2015; 148(3):759-766
CONCLUSIONS:
Manuscript received December 22, 2014; revision accepted April 6, 2015; originally published Online First May 7, 2015. ABBREVIATIONS: AGREE 5 Appraisal of Guidelines for Research and Evaluation; CPG 5 clinical practice guideline AFFILIATIONS: From the State Key Laboratory of Respiratory Disease (Drs Jiang, Liao, Liu, He, Guan, and Li), National Clinical Research Center for Respiratory Disease, Guangzhou Institute of Respiratory Diseases, First Affiliated Hospital of Guangzhou Medical University, Guangzhou; and the Second Clinical College of Nanjing University of Chinese Medicine (Dr Chen), Nanjing, China. FUNDING/SUPPORT: This work was supported by the National Natural Science Foundation of China [Grants 81270125 and 81402772].
journal.publications.chestnet.org
CORRESPONDENCE TO: Yi-min Li, MD, State Key Laboratory of Respiratory Disease, National Clinical Research Center for Respiratory Disease, Guangzhou Institute of Respiratory Diseases, First Affiliated Hospital of Guangzhou Medical University, 151 Yanjiang Rd, Guangzhou, Guangdong 510120, China; e-mail: [email protected] © 2015 AMERICAN COLLEGE OF CHEST PHYSICIANS. Reproduction of this article is prohibited without written permission from the American College of Chest Physicians. See online for more details. DOI: 10.1378/chest.14-3201
759
Respiratory diseases are a major cause of mortality and morbidity in China. Despite improvement in health-care services, respiratory diseases are socioeconomic burdens and remain a major public health issue.1 In developing China, limited health-care resources and increased medical demands pose challenges to effective treatment, and substantial differences exist in clinical practices across regions. To standardize high-quality care in China, clinical practice guidelines (CPGs) must be developed to minimize the use of unnecessary, and sometimes even harmful, medical interventions. CPGs are developed by a systematic review of the existing evidence and an assessment of the pros and cons of alternative care options to provide recommendations that will optimize health care and integrate research and clinical practice.2,3 However, the potential benefits of
Materials and Methods Guideline Identification Guidelines for the management of respiratory diseases were identified by an exhaustive search of four major Chinese academic databases (the Chinese Biomedical Literature database [CBM], the China National Knowledge Infrastructure [CNKI], the VIP database, and the WANFANG database) for articles published between 1978 and 2013, using the following title search terms: “guideline,” “specification,” “consensus,” “statement,” “regulation,” “agreement,” and “recommendation.” Two of the authors (M. J. and L. L.) independently evaluated all search results to determine if they warranted inclusion in this study and then extracted the general characteristics of each guideline. Disagreements were resolved by consensus or by obtaining a third expert opinion. Guideline Selection We included all Chinese-language guidelines that were related to respiratory diseases and developed by mainland Chinese organizations. A document was considered to contain guidelines only if recommendations for clinical practice were provided. We excluded Chinese translations of international CPGs, adapted international CPGs, systematic reviews, editorials, critical pathways, short summaries, and other literature explaining or evaluating CPGs. All versions of a guideline were evaluated if that guideline was in the process of being updated. Multiple partially published guidelines from the same source over time were combined as a single set of guidelines in our assessment. Guideline Evaluation We used the Appraisal of Guidelines for Research and Evaluation (AGREE) II instrument7,8 to assess CPGs, because it has been extensively used previously to address guideline quality.9 AGREE II consists of 23 items in six domains, followed by two global rating items (“Overall Assessment”). Each domain identified a unique dimension of guideline quality.10 Each item was scored from 1 (strongly disagree) to 7 (strongly agree). Scores were derived as the percentage of the maximal possible score for each domain, using the following specific formula: (obtained
Results Literature Search
A total of 1,568 articles were identified through our comprehensive database search. Of these, 892 references 760 Original Research
CPGs depend primarily on the quality and rigor of the development process that assesses and assigns quality grades to the evidence underpinning recommendations.4 The need to assess the quality of CPGs based on rigorous evaluations has rendered quality improvement a crucial objective.5,6 Guidelines for the management of respiratory diseases in China have been produced and disseminated increasingly; however, the quality of these guidelines has not been evaluated systematically. We conducted this study to assess the methodologic quality of guidelines for the management of respiratory diseases. All the guidelines assessed in this study had been published previously in peer-reviewed Chinese medical journals. We used this assessment to provide recommendations for guideline development.
score 2 minimal possible score)/(maximal possible score 2 minimal possible score). The standardized scores ranged from 0% to 100%.10 The final overall guideline recommendation considered all domain items.10 We assigned double weight to the domains of rigor of development and applicability. A guideline would be “recommended” if overall scores were . 60%, “recommended with modifications” for scores between 30% and 60%, and “not recommended” for scores , 30%. Four independent physicians (L. L., X. L., W. H., and W. G.) who specialized in respiratory disease and were trained in CPG appraisal using the AGREE II instrument conducted an independent review of the quality of each eligible guideline. Statistical Analysis A database was compiled consisting of two components (general characteristics and the AGREE II content). For each document, the general characteristics included were the title, year of publication, organization/ institution, financial source, version, target populations, diseases addressed, number of references, year interval at which guidelines were updated, topics covered, and number of panel members for guideline development. Descriptive statistical analyses were performed for the standardized score of each domain. The descriptive values included median and range (or mean and SD) and percentage. Agreement among the four reviewers was measured by the intraclass correlation coefficient with a 95% CI.11 Subgroup analyses were performed according to the year of publication, type of article, version, purposes for recommendation (ie, diagnosis/ prevention, treatment, or diagnosis/prevention and treatment), objects, diseases addressed, developer, and whether CPGs were evidence based, using an independent t test and a one-way analysis of variance (or a Mann-Whitney and a Kruskal-Wallis test, as appropriate). All tests were two sided, and P values , .05 were considered statistically significant. Analyses were performed using SPSS, version 16.0 (IBM Corporation).
were excluded because of duplication, and the remaining 676 were screened by titles and abstracts. We excluded 365 of these because of irrelevance, and 311 were considered for full-text screening. Ultimately, 109 guidelines were included for assessment (Fig 1).
[
148#3 CHEST SEPTEMBER 2015
]
TABLE 1
] General Characteristics of Identified Guidelines (N 5 109)
Category
No. (%)
Version First
93 (85.3)
Updated
16 (14.7)
Year of publication 2009 or before
58 (53.2)
After 2009
51 (46.8)
Purposes for recommendation Diagnosis/prevention
12 (11.0)
Treatment
16 (14.7)
Diagnosis/prevention 1 treatment
81 (74.3)
Evidence-based CPGs Yes
16 (14.7)
No
93 (85.3)
Patients Children
25 (22.9)
Adults
84 (77.1)
Developers Medical specialty societies
78 (71.6)
Others
31 (28.4)
No. developed organizations or agencies
Figure 1 – Flowchart of guideline search and selection.
1
84 (77.1)
.1
25 (22.9)
Diseases addresseda
General Characteristics of Guidelines
The 109 eligible guidelines were published in 27 Chinese medical journals between 1979 and 2013. The earliest guideline included in this study was published in 1979; no other guidelines were published until 1997. Since 1997, the publication of guidelines has increased. The general characteristics of the guidelines are listed in Table 1. Only 16 guidelines (14.7%) were self-proclaimed to be evidence based, four (3.7%) described their methods for evidence search, 14 (12.8%) appraised the quality of evidence, and 13 (11.9%) graded the strength of recommendations. One guideline (0.9%) used the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system. Sixty-eight guidelines (62.4%) reported the number of panel members (median, 15; range, 2-187) involved in guideline development, but only three guidelines reported the involvement of methodologic experts. Two guidelines included a conflict of interest disclosure.
Pneumonia
12 (11.0)
Asthma
11 (10.1)
COPD
9 (8.3)
Type TCM
20 (18.3)
Western medicine
89 (81.7)
References Not reported
47 (43.1)
Reported
62 (56.9)
No. references, median (range)
26 (2-341)
No. development panel members, median (range)
15 (2-187)
Frequency of update, median (range), y
6 (2-12)
CPG 5 clinical practice guideline; TCM 5 traditional Chinese medicine. aOnly the data of the top three diseases addressed are presented.
Quality Assessment
95% CI, 0.812-0.862). After weighting of the overall scores, three guidelines (2.8%) were “recommended with modifications.” Most guidelines (97.2%) were “not recommended” (Table 2).
The overall agreement of the four appraisers was considered good (intraclass correlation coefficient, 0.838;
Table 3 shows the AGREE II scores of all guidelines. The scores for the scope and purpose (mean, 57.3%; range,
journal.publications.chestnet.org
761
TABLE 2
] Overall Assessment of Guidelines
Stratification of CPG Quality
Overall Assessment
No. (%)
Overall Score, Mean ⫾ SD
Recommended
0 (0)
…
Recommended with modifications
3 (2.8)
Not recommended
106 (97.2)
37.1 ⫾ 8.0 17.1 ⫾ 5.5
4.2%-80.6%) and clarity and presentation (mean, 55.2%; range, 5.6%-83.3%) domains were relatively high, with 57 guidelines (52.3%) and 38 guidelines (34.9%) scoring above 60%, respectively. The mean score for stakeholder involvement was 17.6% (range, 0%-52.8%), with only 11 guidelines scoring above 30%. Only three guidelines reported the involvement of methodologic experts, and no guideline involved patients or patient representatives during its drafting. The mean score for rigor of development was 10.2% (range, 2.6%-70.3%), with three guidelines scoring above 30%. One guideline clearly described the criteria for selecting evidence and formulating recommendations (score . 60%). No document described its procedure for updating guidelines. The mean score for applicability was 9.3% (range, 0.0%-37.5%). Overall, little information was reported regarding potential organizational barriers, cost implications, or tools for application. The mean score for editorial independence was 1.1% (range, 0.0%-50%), with 104 guidelines (95.4%) given a score of 0%. Only four guidelines reported their sources of funding, but they failed to report whether the views of funding organizations influenced the contents of the guidelines. Two guidelines declared potential conflicts of interest. TABLE 3
Table 4 presents the difference in domain quality scores among different subgroups. In the domain of clarity and presentation, CPG scores that were developed by medical societies were higher than those developed by other organizations (P , .05). In all AGREE II domains, evidence-based CPGs had higher quality scores than did CPGs that were not evidence based (P , .05). There was a slight increase in AGREE II scores over time during CPG development, especially in the stakeholder involvement domain (P , .01). CPGs based on traditional Chinese medicine had lower AGREE II scores than did guidelines based on Western medicine in the domains of scope and purpose, stakeholder involvement, clarity and presentation, and applicability (all P , .05). No significant differences were found with respect to the type of diseases (ie, pneumonia, asthma, or COPD) or between children and adults.
Discussion In this study, we demonstrated that the quality of current respiratory disease CPGs (Fig 2) in China was higher than that reported by Chen et al12 but much lower than the average international quality level,13 especially in the domains of stakeholder involvement, rigor of development, applicability, and editorial independence. The stakeholder involvement domain addressed the degree to which a guideline represented the views of its intended users.5 CPG development requires multidisciplinary panels that include physicians, methodologic experts, policy makers, and patient representatives.14 In our review of the literature, only three guidelines reported the involvement of methodologic experts, and no guideline included patient values and preferences. The rigor of development domain is considered to be the most crucial domain for the assessment of guideline
] AGREE II Scores of the Guidelines Score , 30%, No. (%)
30% ⱕ Score ⱕ 60%, No. (%)
Score . 60%, No. (%)
4.2-80.6
9 (8.3)
43 (39.4)
57 (52.3)
0.0-52.8
98 (89.9)
11 (10.1)
…
10.2 ⫾ 8.7
2.6-70.3
106 (97.2)
2 (1.8)
1 (0.9)
55.2 ⫾ 13.1
5.6-83.3
65 (59.6)
38 (34.9)
Mean ⫾ SD
Range
Scope and purpose
57.3 ⫾ 16.1
Stakeholder involvement
17.6 ⫾ 10.4
Rigor of development Clarity and presentation
Domain
6 (5.5)
Applicability
9.3 ⫾ 7.0
0.0-37.5
106 (97.2)
3 (2.8)
…
Editorial independence
1.1 ⫾ 5.7
0.0-50.0
108 (99.1)
1 (0.9)
…
AGREE 5 Appraisal of Guidelines for Research and Evaluation.
762 Original Research
[
148#3 CHEST SEPTEMBER 2015
]
journal.publications.chestnet.org
763
59.18 ⫾ 15.42
13.76 ⫾ 13.99 13.89 ⫾ 10.04
43.81 ⫾ 27.62 56.02 ⫾ 12.34
Asthma
COPD
9.41 ⫾ 8.87 10.37 ⫾ 8.71
12.75 ⫾ 12.69 20.59 ⫾ 11.81a
62.03 ⫾ 9.04a
Western medicine
8.10 ⫾ 3.26
6.77 ⫾ 1.66
17.66 ⫾ 20.25
57.39 ⫾ 12.15a
43.63 ⫾ 12.19
54.01 ⫾ 11.73
54.80 ⫾ 14.47
65.63 ⫾ 9.39
55.27 ⫾ 12.73 55.01 ⫾ 13.57
55.09 ⫾ 13.00
55.33 ⫾ 13.57
9.66 ⫾ 6.72
32.11 ⫾ 22.02
Data are presented as mean ⫾ SD. See Table 1 and 3 legends for expansion of abbreviations. aP , .05.
70.14 ⫾ 7.51 52.57 ⫾ 12.06a
10.81 ⫾ 10.47
9.63 ⫾ 8.01
12.12 ⫾ 10.51
TCM
Type
22.80 ⫾ 15.35
60.65 ⫾ 17.37
Pneumonia
Diseases addressed
14.97 ⫾ 8.19 20.59 ⫾ 11.81a
55.70 ⫾ 16.60
17.20 ⫾ 10.62
58.04 ⫾ 14.78
2009 or before
18.94 ⫾ 9.66
After 2009
Year of publication
Adult
Children
54.94 ⫾ 20.02
7.51 ⫾ 2.09a
No
Patients
25.85 ⫾ 14.38
25.09 ⫾ 12.22 16.31 ⫾ 9.53a
67.27 ⫾ 6.76 55.62 ⫾ 16.61a
Yes
Evidence-based CPGs
55.86 ⫾ 12.00a
60.42 ⫾ 10.58
11.59 ⫾ 5.76 10.47 ⫾ 9.58
21.88 ⫾ 6.24 16.84 ⫾ 11.18
64.15 ⫾ 6.68 57.13 ⫾ 16.28a
43.29 ⫾ 16.70
49.96 ⫾ 13.53a
57.21 ⫾ 12.37
64.58 ⫾ 8.10a
53.52 ⫾ 13.10
Clarity and Presentation
6.51 ⫾ 2.26
8.48 ⫾ 5.94
10.88 ⫾ 9.47
Treatment
17.01 ⫾ 8.12
16.22 ⫾ 9.07
56.82 ⫾ 14.87 49.54 ⫾ 20.43
18.14 ⫾ 10.87
Diagnosis/ prevention 1 treatment
Diagnosis/prevention
Purposes for recommendation
Others
Medical specialty societies
57.82 ⫾ 16.60
24.83 ⫾ 12.84a
Developers
9.53 ⫾ 6.61 14.31 ⫾ 15.88
16.35 ⫾ 9.44
55.69 ⫾ 16.67 66.84 ⫾ 6.63a
Rigor of Development
First
Stakeholder Involvement
Updated
Version
Scope and Purpose
] AGREE II Domain Scores of Guidelines According to Different Stratified Factors
Subgroup
TABLE 4
10.87 ⫾ 6.45a
1.10 ⫾ 2.98
10.53 ⫾ 4.31
12.22 ⫾ 14.40
10.24 ⫾ 6.36
8.56 ⫾ 6.84
9.88 ⫾ 7.17
9.37 ⫾ 6.28
8.87 ⫾ 9.24
8.73 ⫾ 7.24a
12.37 ⫾ 4.58
9.41 ⫾ 7.65
10.22 ⫾ 4.22
1.28 ⫾ 6.20
0.0 ⫾ 0.0
0.0 ⫾ 0.0
0.0 ⫾ 0.0
1.39 ⫾ 4.81
1.96 ⫾ 7.92
0.29 ⫾ 2.19
1.19 ⫾ 6.22
0.67 ⫾ 3.33
0.36 ⫾ 2.43a
5.21 ⫾ 13.22
0.82 ⫾ 3.63
3.13 ⫾ 12.50
0.0 ⫾ 0.0
1.08 ⫾ 4.16
7.86 ⫾ 5.61 6.94 ⫾ 5.15
1.07 ⫾ 6.21
2.08 ⫾ 5.69
0.90 ⫾ 5.69
Editorial Independence
9.82 ⫾ 7.47
15.62 ⫾ 7.19a
8.17 ⫾ 6.42
Applicability
Figure 2 – Comparison of the quality of clinical practice guidelines. AGREE 5 Appraisal of Guidelines for Research and Evaluation; CPG 5 clinical practice guideline.
development and may impact recommendations for clinical practice.5 The mean score for this domain was disturbingly low, perhaps because of a lack of methodologic consulting5 or unfamiliarity with the standards of guideline development, and poor reporting.15 Guidelines developed by expert consensus or using nonsystematic methodology may not reflect current practice, leading to bias.16 Hence, greater efforts are needed to improve the awareness of finding and using evidence when developing guidelines. Additionally, it is possible that some guideline developers may have used appropriate techniques but did not report in peer-reviewed journals15 because supplementary materials or background information may not have been reviewed favorably. It is recommended that guidelines be updated at 3-year intervals, because new evidence may result in substantial changes to the recommendations.17 However, we found that only 16 guidelines (14.7%) were updated at an average interval of 6 years. Furthermore, no guideline provided clear statements regarding their updating procedures. For the eligible guidelines, we observed that the scores of the stakeholder involvement and rigor of development domains were slightly higher for children than for adults, although these differences were statistically insignificant (Table 4). This difference may be explained by the fact that the guidelines for children (20.0%) were more frequently evidence based than were those for adults (13.1%). The score of the applicability domain was also low, similar to those reported by Chen et al.12 Many guidelines were developed without considering potential barriers to implementation,13 illustrating the importance of conducting pilot tests to ensure guideline feasibility prior to publication.12 Conflicts of interest are the most common source of bias in guideline development.18 To improve transpar764 Original Research
ency, all guidelines developers should explicitly disclose their conflicts of interest.19 In our analysis, the editorial independence domain yielded the lowest score, which was remarkably lower than those reported by Alonso-Coello et al13 and Acuña-Izcaray et al.5 These observations suggest that there are existing conflicts of interest between funding sources and guideline development panels or that reports about conflicts of interest are missing.15 The AGREE II instrument lacked clear instructions regarding the weighting of different domain scores when determining the optimal recommended guideline.6,13,20,21 The AGREE II instrument did not set minimum domain scores or patterns of scores across different domains that would permit differentiation between high- and low-quality guidelines. These decisions are left to the user’s discretion.10 Weights were added in some studies,5,22,23 which resulted in more reliable and reasonable evaluations. Therefore, to improve the determination of optimal guidelines, we assigned double weight to the rigor of development and applicability domains when deriving the overall scores. As expected, we found that more recently updated guidelines tended to have higher scores than earlier versions and that evidence-based guidelines scored consistently higher in all domains. This finding reinforced the fact that guidelines need to be updated as new evidence is provided and that the rigor of the methodology of guideline development must be improved. The benefits of this strategy are reflected in the finding that Chinese guidelines increased in quality over time. We did not survey the use of respiratory disease guidelines that were developed in China or elsewhere, and previous guidelines (ie, asthma guidelines developed in the early 1990s) were translated directly from international guidelines without significant modifications; however, we noted that governmental agencies and
[
148#3 CHEST SEPTEMBER 2015
]
medical societies have increasingly been acknowledging the importance of guidelines in health-care service. The publication of guidelines has been increasing since 1997, and CPG AGREE scores have also improved slightly over time (Table 4). Moreover, recently developed Chinese respiratory guidelines have already incorporated key recommendations that are particularly relevant to clinical practice. For instance, the 2008 Chinese asthma guideline24 explicitly outlined the optimal dose of theophylline for the treatment of asthma (6-10 mg/kg for the Chinese population vs 10-15 mg/kg for whites, because of ethnic differences).25 The 2006 guidelines for the management of community-acquired pneumonia in China26 did not recommend the use of macrolides as a first-line therapy because of the significant drug resistance of Streptococcus pneumoniae (approximately 60%), as documented by two Chinese studies.27,28 Alternatively, b-lactamase inhibitors were recommended as a first-line therapy for Chinese patients. Most likely, these ongoing efforts will lead to the production and dissemination of high-quality guidelines that will help incorporate evidence into clinical practice in China in the future. Our study possesses several strengths. First, we used AGREE II, the latest instrument for guideline assessment, to assess the methodologic quality of CPGs related to respiratory diseases. Second, our team consisted of clinical experts and methodologists with extensive experience in CPGs. Third, we conducted a systematic and thorough literature search and obtained consensus from four appraisers to ensure the reliability of our conclusions. Fourth, we weighted the different domains in
journal.publications.chestnet.org
decision-making guidelines according to their recommended use. Nevertheless, our study also possesses some limitations. First, we only assessed guidelines published in peerreviewed Chinese medical journals, which may not represent all of the guidelines for respiratory diseases in China. Guidelines published in other forms (ie, books, booklets, or government documents) may have been overlooked, which may have introduced bias into our assessment. Second, supplementary materials and background information on guidelines could not be searched and reviewed thoroughly. Therefore, we may have underestimated guideline quality in some instances. Third, our study was observational in nature, so we could not establish a cause-effect relationship between guideline quality and outcome. Finally, the AGREE II instrument was used only to assess the methods of formulating and reporting recommendations, and not to evaluate the content validity of recommendations. This limitation is a drawback in using this appraisal tool20 and may occasionally lead to conflicts between the methodologic quality of a guideline and its recommendation validity.29
Conclusions The overall quality of guidelines related to respiratory diseases is low in China. Guideline developers should pay more attention to stakeholder involvement, rigor of development, applicability, and editorial independence in the future. National efforts are needed to develop high-quality, evidence-based guidelines using sound methodologic frameworks in China.
765
Acknowledgments Author contributions: M. J. had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. M. J. and Y. L. contributed to the conception and design of the experiments; L. L., X. L., W. H., and W. G. contributed to the performing of the experiments; M. J. and L. L. contributed to the analysis of the data; W. H. and H. C. contributed to the reagents, materials, and analysis tools; M. J. and W. G. contributed to the writing of the paper; and M. J., L. L., X. L., W. H., W. G., H. C., and Y. L. contributed to the writing, review, and approval of the manuscript and made substantial contributions for important intellectual content. Financial/nonfinancial disclosures: The authors have reported to CHEST that no potential conflicts of interest exist with any companies/organizations whose products or services may be discussed in this article. Role of sponsors: The sponsor had no role in the design of the study, the collection and analysis of the data, or the preparation of the manuscript. Other contributions: We thank Hai-bo Zhang, PhD, Keenan Research Centre for Biomedical Science of St. Michael’s Hospital, Toronto, ON, Canada, for his linguistic assistance and insightful suggestions.
References 1. Wang C, Xiao F, Qiao R, Shen YH. Respiratory medicine in China: progress, challenges, and opportunities. Chest. 2013;143(6):1766-1773. 2. Institute of Medicine. Clinical practice guidelines we can trust. Institute of Medicine website. http://www.iom. edu/Reports/2011/Clinical-PracticeGuidelines-We-Can-Trust.aspx. Published 2011. Accessed October 2013. 3. Lo Vecchio A, Giannattasio A, Duggan C, et al. Evaluation of the quality of guidelines for acute gastroenteritis in children with the AGREE instrument. J Pediatr Gastroenterol Nutr. 2011;52(2):183-189. 4. Grol R, Cluzeau FA, Burgers JS. Clinical practice guidelines: towards better quality guidelines and increased international collaboration. Br J Cancer. 2003; 89(suppl 1):S4-S8. 5. Acuña-Izcaray A, Sánchez-Angarita E, Plaza V, et al. Quality assessment of asthma
766 Original Research
clinical practice guidelines: a systematic appraisal. Chest. 2013;144(2):390-397.
19. Norris SL, Holmer HK, Ogden LA, Burda BU. Conflict of interest in clinical practice guideline development: a systematic review. PLoS ONE. 2011;6(10):e25153.
6. Harpole LH, Kelley MJ, Schreiber G, Toloza EM, Kolimaga J, McCrory DC. Assessment of the scope and quality of clinical practice guidelines in lung cancer. Chest. 2003;123(1_suppl):7S-20S.
20. Vlayen J, Aertgeerts B, Hannes K, Sermeus W, Ramaekers D. A systematic review of appraisal tools for clinical practice guidelines: multiple similarities and one common deficit. Int J Qual Health Care. 2005;17(3):235-242.
7. Brouwers MC, Kho ME, Browman GP, et al; AGREE Next Steps Consortium. AGREE II: advancing guideline development, reporting and evaluation in health care. CMAJ. 2010;182(18):E839-E842.
21. Boluyt N, Lincke CR, Offringa M. Quality of evidence-based pediatric guidelines. Pediatrics. 2005;115(5): 1378-1391.
8. Burls A. AGREE II-improving the quality of clinical care. Lancet. 2010;376(9747): 1128-1129.
22. Lytras T, Bonovas S, Chronis C, et al. Occupational asthma guidelines: a systematic quality appraisal using the AGREE II instrument. Occup Environ Med. 2014;71(2):81-86.
9. AGREE Enterprise website. http://www. agreetrust.org. Accessed October 2013. 10. Next steps consortium. The AGREE II instrument. AGREE website. http://www. agreetrust.org. Published 2009. Accessed January 10, 2014.
23. Ríos E, Serón P, Lanas F, Bonfill X, Quigley EM, Alonso-Coello P. Evaluation of the quality of clinical practice guidelines for the management of esophageal or gastric variceal bleeding. Eur J Gastroenterol Hepatol. 2014;26(4):422-431.
11. Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979;86(2):420-428. 12. Chen YL, Yao L, Xiao XJ, et al. Quality assessment of clinical guidelines in China: 1993 - 2010. Chin Med J (Engl). 2012;125(20):3660-3664.
24. Asthma Workgroup; Chinese Thoracic Society; Chinese Society of General Practitioners. Chinese guideline for the prevention and management of bronchial asthma (Primary Health Care Version). J Thorac Dis. 2013;5(5):667-677.
13. Alonso-Coello P, Irfan A, Solà I, et al. The quality of clinical practice guidelines over the last two decades: a systematic review of guideline appraisal studies. Qual Saf Health Care. 2010;19(6):e58.
25. Zhong NS. Management of asthma in developing countries. In: Barnes PJ, Grunstein MM, Leff AR, Woolcock AJ, eds. Asthma. Vol 2. Philadephia, PA: Lippincott-Raven Publishers; 1997:1869-1882.
14. Cecamore C, Savino A, Salvatore R, et al. Clinical practice guidelines: what they are, why we need them and how they should be developed through rigorous evaluation. Eur J Pediatr. 2011;170(7):831-836.
26. Chinese Medical Society. Guidelines for the management of community-acquired pneumonia [in Chinese]. Zhonghua Jie He He Hu Xi Za Zhi. 2006;29:651-655.
15. Hu J, Chen R, Wu S, et al. The quality of clinical practice guidelines in China: a systematic assessment. J Eval Clin Pract. 2013;19(5):961-967.
27. Liu YN, Chen MJ, Zhao TM, et al. A multicentre study on the pathogenic agents in 665 adult patients with communityacquired pneumonia in cities of China [in Chinese]. Zhonghua Jie He He Hu Xi Za Zhi. 2006;29(1):3-8.
16. Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers TC. A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts. Treatments for myocardial infarction. JAMA. 1992;268(2):240-248.
28. Wang H, Yu YS, Liu Y, et al. Resistance surveillance of common community respiratory pathogens isolated in China, 2002 – 2003 [in Chinese]. Zhonghua Jie He He Hu Xi Za Zhi. 2004;27(3):155-160.
17. Shekelle PG, Ortiz E, Rhodes S, et al. Validity of the Agency for Healthcare Research and Quality clinical practice guidelines: how quickly do guidelines become outdated? JAMA. 2001;286(12):1461-1467.
29. Watine J, Friedberg B, Nagy E, et al. Conflict between guideline methodologic quality and recommendation validity: a potential problem for practitioners. Clin Chem. 2006;52(1):65-72.
18. Detsky AS. Sources of bias for authors of clinical practice guidelines [in English and French]. CMAJ. 2006;175(9):1033, 1035.
[
148#3 CHEST SEPTEMBER 2015
]
Original Research Education, Research, and Quality Improvement
]
Quality Assessment of Clinical Practice Guidelines for Respiratory Diseases in China A Systematic Appraisal Mei Jiang, PhD; Li-yue Liao, MD; Xiao-qing Liu, MD; Wei-qun He, MD; Wei-jie Guan, PhD; Hao Chen, MD; and Yi-min Li, MD
There has been a significant increase in the publication of clinical practice guidelines (CPGs) for respiratory diseases in China. However, little is known about the quality and potential impacts of these CPGs. Our objective was to critically evaluate the quality of Chinese CPGs for respiratory diseases that were published in peer-reviewed medical journals. BACKGROUND:
A systematic search of scientific literature published between 1979 and 2013 was undertaken to identify and select CPGs that were related to respiratory diseases. Four Chinese databases (the Chinese Biomedical Literature database [CBM], the China National Knowledge Infrastructure [CNKI], the VIP database, and the WANFANG database) were used. The quality of eligible guidelines was assessed independently by four reviewers using the Appraisal of Guidelines for Research and Evaluation (AGREE) II instrument. The overall agreement among reviewers was evaluated using an intraclass correlation coefficient. METHODS:
A total of 109 guidelines published in 27 medical journals from 1979 to 2013 were evaluated. The overall agreement among reviewers was considered good (intraclass correlation coefficient, 0.838; 95% CI, 0.812-0.862). The scores of the six AGREE domains were low: 57.3% for scope and purpose (range, 4.2%-80.5%), 23.8% for stakeholder involvement (range, 2.8%-54.2%), 7.7% for rigor of development (range, 0%-27.1%), 59.8% for clarity and presentation (range, 22.2%-80.6%), 10.9% for applicability (range, 0%-22.9%), and 0.6% for editorial independence (range, 0%-16.7%). Scores for all guidelines were below 60%, and only three guidelines (2.8%) were recommended for clinical practice with modifications.
RESULTS:
The quality of the guidelines was low, and stakeholder involvement, rigor of development, applicability, and editorial independence should be considered in the future development of CPGs for respiratory diseases in China. CHEST 2015; 148(3):759-766
CONCLUSIONS:
Manuscript received December 22, 2014; revision accepted April 6, 2015; originally published Online First May 7, 2015. ABBREVIATIONS: AGREE 5 Appraisal of Guidelines for Research and Evaluation; CPG 5 clinical practice guideline AFFILIATIONS: From the State Key Laboratory of Respiratory Disease (Drs Jiang, Liao, Liu, He, Guan, and Li), National Clinical Research Center for Respiratory Disease, Guangzhou Institute of Respiratory Diseases, First Affiliated Hospital of Guangzhou Medical University, Guangzhou; and the Second Clinical College of Nanjing University of Chinese Medicine (Dr Chen), Nanjing, China. FUNDING/SUPPORT: This work was supported by the National Natural Science Foundation of China [Grants 81270125 and 81402772].
journal.publications.chestnet.org
CORRESPONDENCE TO: Yi-min Li, MD, State Key Laboratory of Respiratory Disease, National Clinical Research Center for Respiratory Disease, Guangzhou Institute of Respiratory Diseases, First Affiliated Hospital of Guangzhou Medical University, 151 Yanjiang Rd, Guangzhou, Guangdong 510120, China; e-mail: [email protected] © 2015 AMERICAN COLLEGE OF CHEST PHYSICIANS. Reproduction of this article is prohibited without written permission from the American College of Chest Physicians. See online for more details. DOI: 10.1378/chest.14-3201
759
Respiratory diseases are a major cause of mortality and morbidity in China. Despite improvement in health-care services, respiratory diseases are socioeconomic burdens and remain a major public health issue.1 In developing China, limited health-care resources and increased medical demands pose challenges to effective treatment, and substantial differences exist in clinical practices across regions. To standardize high-quality care in China, clinical practice guidelines (CPGs) must be developed to minimize the use of unnecessary, and sometimes even harmful, medical interventions. CPGs are developed by a systematic review of the existing evidence and an assessment of the pros and cons of alternative care options to provide recommendations that will optimize health care and integrate research and clinical practice.2,3 However, the potential benefits of
Materials and Methods Guideline Identification Guidelines for the management of respiratory diseases were identified by an exhaustive search of four major Chinese academic databases (the Chinese Biomedical Literature database [CBM], the China National Knowledge Infrastructure [CNKI], the VIP database, and the WANFANG database) for articles published between 1978 and 2013, using the following title search terms: “guideline,” “specification,” “consensus,” “statement,” “regulation,” “agreement,” and “recommendation.” Two of the authors (M. J. and L. L.) independently evaluated all search results to determine if they warranted inclusion in this study and then extracted the general characteristics of each guideline. Disagreements were resolved by consensus or by obtaining a third expert opinion. Guideline Selection We included all Chinese-language guidelines that were related to respiratory diseases and developed by mainland Chinese organizations. A document was considered to contain guidelines only if recommendations for clinical practice were provided. We excluded Chinese translations of international CPGs, adapted international CPGs, systematic reviews, editorials, critical pathways, short summaries, and other literature explaining or evaluating CPGs. All versions of a guideline were evaluated if that guideline was in the process of being updated. Multiple partially published guidelines from the same source over time were combined as a single set of guidelines in our assessment. Guideline Evaluation We used the Appraisal of Guidelines for Research and Evaluation (AGREE) II instrument7,8 to assess CPGs, because it has been extensively used previously to address guideline quality.9 AGREE II consists of 23 items in six domains, followed by two global rating items (“Overall Assessment”). Each domain identified a unique dimension of guideline quality.10 Each item was scored from 1 (strongly disagree) to 7 (strongly agree). Scores were derived as the percentage of the maximal possible score for each domain, using the following specific formula: (obtained
Results Literature Search
A total of 1,568 articles were identified through our comprehensive database search. Of these, 892 references 760 Original Research
CPGs depend primarily on the quality and rigor of the development process that assesses and assigns quality grades to the evidence underpinning recommendations.4 The need to assess the quality of CPGs based on rigorous evaluations has rendered quality improvement a crucial objective.5,6 Guidelines for the management of respiratory diseases in China have been produced and disseminated increasingly; however, the quality of these guidelines has not been evaluated systematically. We conducted this study to assess the methodologic quality of guidelines for the management of respiratory diseases. All the guidelines assessed in this study had been published previously in peer-reviewed Chinese medical journals. We used this assessment to provide recommendations for guideline development.
score 2 minimal possible score)/(maximal possible score 2 minimal possible score). The standardized scores ranged from 0% to 100%.10 The final overall guideline recommendation considered all domain items.10 We assigned double weight to the domains of rigor of development and applicability. A guideline would be “recommended” if overall scores were . 60%, “recommended with modifications” for scores between 30% and 60%, and “not recommended” for scores , 30%. Four independent physicians (L. L., X. L., W. H., and W. G.) who specialized in respiratory disease and were trained in CPG appraisal using the AGREE II instrument conducted an independent review of the quality of each eligible guideline. Statistical Analysis A database was compiled consisting of two components (general characteristics and the AGREE II content). For each document, the general characteristics included were the title, year of publication, organization/ institution, financial source, version, target populations, diseases addressed, number of references, year interval at which guidelines were updated, topics covered, and number of panel members for guideline development. Descriptive statistical analyses were performed for the standardized score of each domain. The descriptive values included median and range (or mean and SD) and percentage. Agreement among the four reviewers was measured by the intraclass correlation coefficient with a 95% CI.11 Subgroup analyses were performed according to the year of publication, type of article, version, purposes for recommendation (ie, diagnosis/ prevention, treatment, or diagnosis/prevention and treatment), objects, diseases addressed, developer, and whether CPGs were evidence based, using an independent t test and a one-way analysis of variance (or a Mann-Whitney and a Kruskal-Wallis test, as appropriate). All tests were two sided, and P values , .05 were considered statistically significant. Analyses were performed using SPSS, version 16.0 (IBM Corporation).
were excluded because of duplication, and the remaining 676 were screened by titles and abstracts. We excluded 365 of these because of irrelevance, and 311 were considered for full-text screening. Ultimately, 109 guidelines were included for assessment (Fig 1).
[
148#3 CHEST SEPTEMBER 2015
]
TABLE 1
] General Characteristics of Identified Guidelines (N 5 109)
Category
No. (%)
Version First
93 (85.3)
Updated
16 (14.7)
Year of publication 2009 or before
58 (53.2)
After 2009
51 (46.8)
Purposes for recommendation Diagnosis/prevention
12 (11.0)
Treatment
16 (14.7)
Diagnosis/prevention 1 treatment
81 (74.3)
Evidence-based CPGs Yes
16 (14.7)
No
93 (85.3)
Patients Children
25 (22.9)
Adults
84 (77.1)
Developers Medical specialty societies
78 (71.6)
Others
31 (28.4)
No. developed organizations or agencies
Figure 1 – Flowchart of guideline search and selection.
1
84 (77.1)
.1
25 (22.9)
Diseases addresseda
General Characteristics of Guidelines
The 109 eligible guidelines were published in 27 Chinese medical journals between 1979 and 2013. The earliest guideline included in this study was published in 1979; no other guidelines were published until 1997. Since 1997, the publication of guidelines has increased. The general characteristics of the guidelines are listed in Table 1. Only 16 guidelines (14.7%) were self-proclaimed to be evidence based, four (3.7%) described their methods for evidence search, 14 (12.8%) appraised the quality of evidence, and 13 (11.9%) graded the strength of recommendations. One guideline (0.9%) used the Grading of Recommendations Assessment, Development and Evaluation (GRADE) system. Sixty-eight guidelines (62.4%) reported the number of panel members (median, 15; range, 2-187) involved in guideline development, but only three guidelines reported the involvement of methodologic experts. Two guidelines included a conflict of interest disclosure.
Pneumonia
12 (11.0)
Asthma
11 (10.1)
COPD
9 (8.3)
Type TCM
20 (18.3)
Western medicine
89 (81.7)
References Not reported
47 (43.1)
Reported
62 (56.9)
No. references, median (range)
26 (2-341)
No. development panel members, median (range)
15 (2-187)
Frequency of update, median (range), y
6 (2-12)
CPG 5 clinical practice guideline; TCM 5 traditional Chinese medicine. aOnly the data of the top three diseases addressed are presented.
Quality Assessment
95% CI, 0.812-0.862). After weighting of the overall scores, three guidelines (2.8%) were “recommended with modifications.” Most guidelines (97.2%) were “not recommended” (Table 2).
The overall agreement of the four appraisers was considered good (intraclass correlation coefficient, 0.838;
Table 3 shows the AGREE II scores of all guidelines. The scores for the scope and purpose (mean, 57.3%; range,
journal.publications.chestnet.org
761
TABLE 2
] Overall Assessment of Guidelines
Stratification of CPG Quality
Overall Assessment
No. (%)
Overall Score, Mean ⫾ SD
Recommended
0 (0)
…
Recommended with modifications
3 (2.8)
Not recommended
106 (97.2)
37.1 ⫾ 8.0 17.1 ⫾ 5.5
4.2%-80.6%) and clarity and presentation (mean, 55.2%; range, 5.6%-83.3%) domains were relatively high, with 57 guidelines (52.3%) and 38 guidelines (34.9%) scoring above 60%, respectively. The mean score for stakeholder involvement was 17.6% (range, 0%-52.8%), with only 11 guidelines scoring above 30%. Only three guidelines reported the involvement of methodologic experts, and no guideline involved patients or patient representatives during its drafting. The mean score for rigor of development was 10.2% (range, 2.6%-70.3%), with three guidelines scoring above 30%. One guideline clearly described the criteria for selecting evidence and formulating recommendations (score . 60%). No document described its procedure for updating guidelines. The mean score for applicability was 9.3% (range, 0.0%-37.5%). Overall, little information was reported regarding potential organizational barriers, cost implications, or tools for application. The mean score for editorial independence was 1.1% (range, 0.0%-50%), with 104 guidelines (95.4%) given a score of 0%. Only four guidelines reported their sources of funding, but they failed to report whether the views of funding organizations influenced the contents of the guidelines. Two guidelines declared potential conflicts of interest. TABLE 3
Table 4 presents the difference in domain quality scores among different subgroups. In the domain of clarity and presentation, CPG scores that were developed by medical societies were higher than those developed by other organizations (P , .05). In all AGREE II domains, evidence-based CPGs had higher quality scores than did CPGs that were not evidence based (P , .05). There was a slight increase in AGREE II scores over time during CPG development, especially in the stakeholder involvement domain (P , .01). CPGs based on traditional Chinese medicine had lower AGREE II scores than did guidelines based on Western medicine in the domains of scope and purpose, stakeholder involvement, clarity and presentation, and applicability (all P , .05). No significant differences were found with respect to the type of diseases (ie, pneumonia, asthma, or COPD) or between children and adults.
Discussion In this study, we demonstrated that the quality of current respiratory disease CPGs (Fig 2) in China was higher than that reported by Chen et al12 but much lower than the average international quality level,13 especially in the domains of stakeholder involvement, rigor of development, applicability, and editorial independence. The stakeholder involvement domain addressed the degree to which a guideline represented the views of its intended users.5 CPG development requires multidisciplinary panels that include physicians, methodologic experts, policy makers, and patient representatives.14 In our review of the literature, only three guidelines reported the involvement of methodologic experts, and no guideline included patient values and preferences. The rigor of development domain is considered to be the most crucial domain for the assessment of guideline
] AGREE II Scores of the Guidelines Score , 30%, No. (%)
30% ⱕ Score ⱕ 60%, No. (%)
Score . 60%, No. (%)
4.2-80.6
9 (8.3)
43 (39.4)
57 (52.3)
0.0-52.8
98 (89.9)
11 (10.1)
…
10.2 ⫾ 8.7
2.6-70.3
106 (97.2)
2 (1.8)
1 (0.9)
55.2 ⫾ 13.1
5.6-83.3
65 (59.6)
38 (34.9)
Mean ⫾ SD
Range
Scope and purpose
57.3 ⫾ 16.1
Stakeholder involvement
17.6 ⫾ 10.4
Rigor of development Clarity and presentation
Domain
6 (5.5)
Applicability
9.3 ⫾ 7.0
0.0-37.5
106 (97.2)
3 (2.8)
…
Editorial independence
1.1 ⫾ 5.7
0.0-50.0
108 (99.1)
1 (0.9)
…
AGREE 5 Appraisal of Guidelines for Research and Evaluation.
762 Original Research
[
148#3 CHEST SEPTEMBER 2015
]
journal.publications.chestnet.org
763
59.18 ⫾ 15.42
13.76 ⫾ 13.99 13.89 ⫾ 10.04
43.81 ⫾ 27.62 56.02 ⫾ 12.34
Asthma
COPD
9.41 ⫾ 8.87 10.37 ⫾ 8.71
12.75 ⫾ 12.69 20.59 ⫾ 11.81a
62.03 ⫾ 9.04a
Western medicine
8.10 ⫾ 3.26
6.77 ⫾ 1.66
17.66 ⫾ 20.25
57.39 ⫾ 12.15a
43.63 ⫾ 12.19
54.01 ⫾ 11.73
54.80 ⫾ 14.47
65.63 ⫾ 9.39
55.27 ⫾ 12.73 55.01 ⫾ 13.57
55.09 ⫾ 13.00
55.33 ⫾ 13.57
9.66 ⫾ 6.72
32.11 ⫾ 22.02
Data are presented as mean ⫾ SD. See Table 1 and 3 legends for expansion of abbreviations. aP , .05.
70.14 ⫾ 7.51 52.57 ⫾ 12.06a
10.81 ⫾ 10.47
9.63 ⫾ 8.01
12.12 ⫾ 10.51
TCM
Type
22.80 ⫾ 15.35
60.65 ⫾ 17.37
Pneumonia
Diseases addressed
14.97 ⫾ 8.19 20.59 ⫾ 11.81a
55.70 ⫾ 16.60
17.20 ⫾ 10.62
58.04 ⫾ 14.78
2009 or before
18.94 ⫾ 9.66
After 2009
Year of publication
Adult
Children
54.94 ⫾ 20.02
7.51 ⫾ 2.09a
No
Patients
25.85 ⫾ 14.38
25.09 ⫾ 12.22 16.31 ⫾ 9.53a
67.27 ⫾ 6.76 55.62 ⫾ 16.61a
Yes
Evidence-based CPGs
55.86 ⫾ 12.00a
60.42 ⫾ 10.58
11.59 ⫾ 5.76 10.47 ⫾ 9.58
21.88 ⫾ 6.24 16.84 ⫾ 11.18
64.15 ⫾ 6.68 57.13 ⫾ 16.28a
43.29 ⫾ 16.70
49.96 ⫾ 13.53a
57.21 ⫾ 12.37
64.58 ⫾ 8.10a
53.52 ⫾ 13.10
Clarity and Presentation
6.51 ⫾ 2.26
8.48 ⫾ 5.94
10.88 ⫾ 9.47
Treatment
17.01 ⫾ 8.12
16.22 ⫾ 9.07
56.82 ⫾ 14.87 49.54 ⫾ 20.43
18.14 ⫾ 10.87
Diagnosis/ prevention 1 treatment
Diagnosis/prevention
Purposes for recommendation
Others
Medical specialty societies
57.82 ⫾ 16.60
24.83 ⫾ 12.84a
Developers
9.53 ⫾ 6.61 14.31 ⫾ 15.88
16.35 ⫾ 9.44
55.69 ⫾ 16.67 66.84 ⫾ 6.63a
Rigor of Development
First
Stakeholder Involvement
Updated
Version
Scope and Purpose
] AGREE II Domain Scores of Guidelines According to Different Stratified Factors
Subgroup
TABLE 4
10.87 ⫾ 6.45a
1.10 ⫾ 2.98
10.53 ⫾ 4.31
12.22 ⫾ 14.40
10.24 ⫾ 6.36
8.56 ⫾ 6.84
9.88 ⫾ 7.17
9.37 ⫾ 6.28
8.87 ⫾ 9.24
8.73 ⫾ 7.24a
12.37 ⫾ 4.58
9.41 ⫾ 7.65
10.22 ⫾ 4.22
1.28 ⫾ 6.20
0.0 ⫾ 0.0
0.0 ⫾ 0.0
0.0 ⫾ 0.0
1.39 ⫾ 4.81
1.96 ⫾ 7.92
0.29 ⫾ 2.19
1.19 ⫾ 6.22
0.67 ⫾ 3.33
0.36 ⫾ 2.43a
5.21 ⫾ 13.22
0.82 ⫾ 3.63
3.13 ⫾ 12.50
0.0 ⫾ 0.0
1.08 ⫾ 4.16
7.86 ⫾ 5.61 6.94 ⫾ 5.15
1.07 ⫾ 6.21
2.08 ⫾ 5.69
0.90 ⫾ 5.69
Editorial Independence
9.82 ⫾ 7.47
15.62 ⫾ 7.19a
8.17 ⫾ 6.42
Applicability
Figure 2 – Comparison of the quality of clinical practice guidelines. AGREE 5 Appraisal of Guidelines for Research and Evaluation; CPG 5 clinical practice guideline.
development and may impact recommendations for clinical practice.5 The mean score for this domain was disturbingly low, perhaps because of a lack of methodologic consulting5 or unfamiliarity with the standards of guideline development, and poor reporting.15 Guidelines developed by expert consensus or using nonsystematic methodology may not reflect current practice, leading to bias.16 Hence, greater efforts are needed to improve the awareness of finding and using evidence when developing guidelines. Additionally, it is possible that some guideline developers may have used appropriate techniques but did not report in peer-reviewed journals15 because supplementary materials or background information may not have been reviewed favorably. It is recommended that guidelines be updated at 3-year intervals, because new evidence may result in substantial changes to the recommendations.17 However, we found that only 16 guidelines (14.7%) were updated at an average interval of 6 years. Furthermore, no guideline provided clear statements regarding their updating procedures. For the eligible guidelines, we observed that the scores of the stakeholder involvement and rigor of development domains were slightly higher for children than for adults, although these differences were statistically insignificant (Table 4). This difference may be explained by the fact that the guidelines for children (20.0%) were more frequently evidence based than were those for adults (13.1%). The score of the applicability domain was also low, similar to those reported by Chen et al.12 Many guidelines were developed without considering potential barriers to implementation,13 illustrating the importance of conducting pilot tests to ensure guideline feasibility prior to publication.12 Conflicts of interest are the most common source of bias in guideline development.18 To improve transpar764 Original Research
ency, all guidelines developers should explicitly disclose their conflicts of interest.19 In our analysis, the editorial independence domain yielded the lowest score, which was remarkably lower than those reported by Alonso-Coello et al13 and Acuña-Izcaray et al.5 These observations suggest that there are existing conflicts of interest between funding sources and guideline development panels or that reports about conflicts of interest are missing.15 The AGREE II instrument lacked clear instructions regarding the weighting of different domain scores when determining the optimal recommended guideline.6,13,20,21 The AGREE II instrument did not set minimum domain scores or patterns of scores across different domains that would permit differentiation between high- and low-quality guidelines. These decisions are left to the user’s discretion.10 Weights were added in some studies,5,22,23 which resulted in more reliable and reasonable evaluations. Therefore, to improve the determination of optimal guidelines, we assigned double weight to the rigor of development and applicability domains when deriving the overall scores. As expected, we found that more recently updated guidelines tended to have higher scores than earlier versions and that evidence-based guidelines scored consistently higher in all domains. This finding reinforced the fact that guidelines need to be updated as new evidence is provided and that the rigor of the methodology of guideline development must be improved. The benefits of this strategy are reflected in the finding that Chinese guidelines increased in quality over time. We did not survey the use of respiratory disease guidelines that were developed in China or elsewhere, and previous guidelines (ie, asthma guidelines developed in the early 1990s) were translated directly from international guidelines without significant modifications; however, we noted that governmental agencies and
[
148#3 CHEST SEPTEMBER 2015
]
medical societies have increasingly been acknowledging the importance of guidelines in health-care service. The publication of guidelines has been increasing since 1997, and CPG AGREE scores have also improved slightly over time (Table 4). Moreover, recently developed Chinese respiratory guidelines have already incorporated key recommendations that are particularly relevant to clinical practice. For instance, the 2008 Chinese asthma guideline24 explicitly outlined the optimal dose of theophylline for the treatment of asthma (6-10 mg/kg for the Chinese population vs 10-15 mg/kg for whites, because of ethnic differences).25 The 2006 guidelines for the management of community-acquired pneumonia in China26 did not recommend the use of macrolides as a first-line therapy because of the significant drug resistance of Streptococcus pneumoniae (approximately 60%), as documented by two Chinese studies.27,28 Alternatively, b-lactamase inhibitors were recommended as a first-line therapy for Chinese patients. Most likely, these ongoing efforts will lead to the production and dissemination of high-quality guidelines that will help incorporate evidence into clinical practice in China in the future. Our study possesses several strengths. First, we used AGREE II, the latest instrument for guideline assessment, to assess the methodologic quality of CPGs related to respiratory diseases. Second, our team consisted of clinical experts and methodologists with extensive experience in CPGs. Third, we conducted a systematic and thorough literature search and obtained consensus from four appraisers to ensure the reliability of our conclusions. Fourth, we weighted the different domains in
journal.publications.chestnet.org
decision-making guidelines according to their recommended use. Nevertheless, our study also possesses some limitations. First, we only assessed guidelines published in peerreviewed Chinese medical journals, which may not represent all of the guidelines for respiratory diseases in China. Guidelines published in other forms (ie, books, booklets, or government documents) may have been overlooked, which may have introduced bias into our assessment. Second, supplementary materials and background information on guidelines could not be searched and reviewed thoroughly. Therefore, we may have underestimated guideline quality in some instances. Third, our study was observational in nature, so we could not establish a cause-effect relationship between guideline quality and outcome. Finally, the AGREE II instrument was used only to assess the methods of formulating and reporting recommendations, and not to evaluate the content validity of recommendations. This limitation is a drawback in using this appraisal tool20 and may occasionally lead to conflicts between the methodologic quality of a guideline and its recommendation validity.29
Conclusions The overall quality of guidelines related to respiratory diseases is low in China. Guideline developers should pay more attention to stakeholder involvement, rigor of development, applicability, and editorial independence in the future. National efforts are needed to develop high-quality, evidence-based guidelines using sound methodologic frameworks in China.
765
Acknowledgments Author contributions: M. J. had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. M. J. and Y. L. contributed to the conception and design of the experiments; L. L., X. L., W. H., and W. G. contributed to the performing of the experiments; M. J. and L. L. contributed to the analysis of the data; W. H. and H. C. contributed to the reagents, materials, and analysis tools; M. J. and W. G. contributed to the writing of the paper; and M. J., L. L., X. L., W. H., W. G., H. C., and Y. L. contributed to the writing, review, and approval of the manuscript and made substantial contributions for important intellectual content. Financial/nonfinancial disclosures: The authors have reported to CHEST that no potential conflicts of interest exist with any companies/organizations whose products or services may be discussed in this article. Role of sponsors: The sponsor had no role in the design of the study, the collection and analysis of the data, or the preparation of the manuscript. Other contributions: We thank Hai-bo Zhang, PhD, Keenan Research Centre for Biomedical Science of St. Michael’s Hospital, Toronto, ON, Canada, for his linguistic assistance and insightful suggestions.
References 1. Wang C, Xiao F, Qiao R, Shen YH. Respiratory medicine in China: progress, challenges, and opportunities. Chest. 2013;143(6):1766-1773. 2. Institute of Medicine. Clinical practice guidelines we can trust. Institute of Medicine website. http://www.iom. edu/Reports/2011/Clinical-PracticeGuidelines-We-Can-Trust.aspx. Published 2011. Accessed October 2013. 3. Lo Vecchio A, Giannattasio A, Duggan C, et al. Evaluation of the quality of guidelines for acute gastroenteritis in children with the AGREE instrument. J Pediatr Gastroenterol Nutr. 2011;52(2):183-189. 4. Grol R, Cluzeau FA, Burgers JS. Clinical practice guidelines: towards better quality guidelines and increased international collaboration. Br J Cancer. 2003; 89(suppl 1):S4-S8. 5. Acuña-Izcaray A, Sánchez-Angarita E, Plaza V, et al. Quality assessment of asthma
766 Original Research
clinical practice guidelines: a systematic appraisal. Chest. 2013;144(2):390-397.
19. Norris SL, Holmer HK, Ogden LA, Burda BU. Conflict of interest in clinical practice guideline development: a systematic review. PLoS ONE. 2011;6(10):e25153.
6. Harpole LH, Kelley MJ, Schreiber G, Toloza EM, Kolimaga J, McCrory DC. Assessment of the scope and quality of clinical practice guidelines in lung cancer. Chest. 2003;123(1_suppl):7S-20S.
20. Vlayen J, Aertgeerts B, Hannes K, Sermeus W, Ramaekers D. A systematic review of appraisal tools for clinical practice guidelines: multiple similarities and one common deficit. Int J Qual Health Care. 2005;17(3):235-242.
7. Brouwers MC, Kho ME, Browman GP, et al; AGREE Next Steps Consortium. AGREE II: advancing guideline development, reporting and evaluation in health care. CMAJ. 2010;182(18):E839-E842.
21. Boluyt N, Lincke CR, Offringa M. Quality of evidence-based pediatric guidelines. Pediatrics. 2005;115(5): 1378-1391.
8. Burls A. AGREE II-improving the quality of clinical care. Lancet. 2010;376(9747): 1128-1129.
22. Lytras T, Bonovas S, Chronis C, et al. Occupational asthma guidelines: a systematic quality appraisal using the AGREE II instrument. Occup Environ Med. 2014;71(2):81-86.
9. AGREE Enterprise website. http://www. agreetrust.org. Accessed October 2013. 10. Next steps consortium. The AGREE II instrument. AGREE website. http://www. agreetrust.org. Published 2009. Accessed January 10, 2014.
23. Ríos E, Serón P, Lanas F, Bonfill X, Quigley EM, Alonso-Coello P. Evaluation of the quality of clinical practice guidelines for the management of esophageal or gastric variceal bleeding. Eur J Gastroenterol Hepatol. 2014;26(4):422-431.
11. Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979;86(2):420-428. 12. Chen YL, Yao L, Xiao XJ, et al. Quality assessment of clinical guidelines in China: 1993 - 2010. Chin Med J (Engl). 2012;125(20):3660-3664.
24. Asthma Workgroup; Chinese Thoracic Society; Chinese Society of General Practitioners. Chinese guideline for the prevention and management of bronchial asthma (Primary Health Care Version). J Thorac Dis. 2013;5(5):667-677.
13. Alonso-Coello P, Irfan A, Solà I, et al. The quality of clinical practice guidelines over the last two decades: a systematic review of guideline appraisal studies. Qual Saf Health Care. 2010;19(6):e58.
25. Zhong NS. Management of asthma in developing countries. In: Barnes PJ, Grunstein MM, Leff AR, Woolcock AJ, eds. Asthma. Vol 2. Philadephia, PA: Lippincott-Raven Publishers; 1997:1869-1882.
14. Cecamore C, Savino A, Salvatore R, et al. Clinical practice guidelines: what they are, why we need them and how they should be developed through rigorous evaluation. Eur J Pediatr. 2011;170(7):831-836.
26. Chinese Medical Society. Guidelines for the management of community-acquired pneumonia [in Chinese]. Zhonghua Jie He He Hu Xi Za Zhi. 2006;29:651-655.
15. Hu J, Chen R, Wu S, et al. The quality of clinical practice guidelines in China: a systematic assessment. J Eval Clin Pract. 2013;19(5):961-967.
27. Liu YN, Chen MJ, Zhao TM, et al. A multicentre study on the pathogenic agents in 665 adult patients with communityacquired pneumonia in cities of China [in Chinese]. Zhonghua Jie He He Hu Xi Za Zhi. 2006;29(1):3-8.
16. Antman EM, Lau J, Kupelnick B, Mosteller F, Chalmers TC. A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts. Treatments for myocardial infarction. JAMA. 1992;268(2):240-248.
28. Wang H, Yu YS, Liu Y, et al. Resistance surveillance of common community respiratory pathogens isolated in China, 2002 – 2003 [in Chinese]. Zhonghua Jie He He Hu Xi Za Zhi. 2004;27(3):155-160.
17. Shekelle PG, Ortiz E, Rhodes S, et al. Validity of the Agency for Healthcare Research and Quality clinical practice guidelines: how quickly do guidelines become outdated? JAMA. 2001;286(12):1461-1467.
29. Watine J, Friedberg B, Nagy E, et al. Conflict between guideline methodologic quality and recommendation validity: a potential problem for practitioners. Clin Chem. 2006;52(1):65-72.
18. Detsky AS. Sources of bias for authors of clinical practice guidelines [in English and French]. CMAJ. 2006;175(9):1033, 1035.
[
148#3 CHEST SEPTEMBER 2015
]
