Oncology: Prostate/Testis/Penis/Urethra
Clinical Practice Guidelines on Prostate Cancer: A Critical Appraisal Mohit Gupta, John McCauley, Amy Farkas, Ahmet Gudeloglu, Molly M. Neuberger, Yen-Yi Ho, Lawrence Yeung, Johannes Vieweg* and Philipp Dahm† From the Department of Urology, University of Florida (MG, JM, AF, AG, MMN, LY, JV) and Malcom Randall Veterans Affairs Medical Center, Gainesville (PD), Florida, and Department of Urology (PD) and Division of Biostatistics, School of Public Health (YYH), University of Minnesota and Urology Section, Minneapolis Veterans Affairs Health Care System (MMN, PD), Minneapolis, Minnesota
Purpose: Clinical practice guidelines are increasingly being used by leading organizations to promote high quality evidence-based patient care. However, the methodological quality of clinical practice guidelines developed by different organizations varies considerably. We assessed published clinical practice guidelines on the treatment of localized prostate cancer to evaluate the rigor, applicability and transparency of their recommendations. Materials and Methods: We searched for English based clinical practice guidelines on treatment of localized prostate cancer from leading organizations in the 15-year period from 1999 to 2014. Clinical practice guidelines limited to early detection, screening, staging and/or diagnosis of prostate cancer were excluded from analysis. Four independent reviewers used the validated AGREE II instrument to assess the quality of clinical practice guidelines in 6 domains, including 1) scope and purpose, 2) stakeholder involvement, 3) rigor of development, 4) clarity of presentation, 5) applicability and 6) editorial independence. Results: A total of 13 clinical practice guidelines met inclusion criteria. Overall the highest median scores were in the AGREE II domains of clarity of presentation, editorial independence, and scope and purpose. The lowest median score was for applicability (28.1%). Although the median score of editorial independence was high (85.4%), variability was also substantial (IQR 12.5e100). NICE and AUA clinical practice guidelines consistently scored well in most domains. Conclusions: Clinical practice guidelines from different organizations on treatment of localized prostate cancer are of variable quality and fall short of current standards in certain areas, especially in applicability and stakeholder involvement. Improvements in these key domains can enhance the impact and implementation of clinical practice guidelines.
Abbreviations and Acronyms ABS ¼ American Brachytherapy Society AGREE ¼ Appraisal of Guidelines for Research and Evaluation AUA ¼ American Urological Association CPG ¼ clinical practice guideline IOM ¼ Institute of Medicine NCCN ¼ National Comprehensive Cancer NetworkÒ NICE ¼ National Institute for Health and Care Excellence Accepted for publication October 29, 2014. * Financial interest and/or other relationship with American Urological Association. † Correspondence: Department of Urology, University of Minnesota, Minneapolis Veterans Affairs Health Care System, Urology Section 112D, One Veterans Dr., Minneapolis, Minnesota 55417 (telephone: 612-467-3532; FAX: 612-4672232; e-mail: [email protected]).
For another article on a related topic see page 1382.
Key Words: prostatic neoplasms, practice guidelines as topic, evidence-based medicine, government, Florida
CLINICAL practice guidelines are important tools to help clinicians and patients reach evidence-based decisions about health care. The development of CPGs has been central
to promoting high quality, evidencebased and safe patient care. They hold promise for improving the quality, appropriateness and costeffectiveness of medical therapies.
0022-5347/15/1934-1153/0 THE JOURNAL OF UROLOGY® © 2015 by AMERICAN UROLOGICAL ASSOCIATION EDUCATION AND RESEARCH, INC.
http://dx.doi.org/10.1016/j.juro.2014.10.105 Vol. 193, 1153-1158, April 2015 Printed in U.S.A.
www.jurology.com
j
1153
1154
CLINICAL PRACTICE GUIDELINES ON PROSTATE CANCER
Thus, leading organizations in the field of urology are increasingly recognizing the importance of CPGs and dedicating considerable resources toward developing and disseminating them. CPGs differ from systematic reviews, cost analyses and decision models by making explicit recommendations aimed at directly influencing patient, clinician and policy maker decision making. They are also becoming the basis of quality of care measures that are likely to affect urologist reimbursements with pay for performance measures on the horizon.1,2 Ideally CPGs from different professional organizations would use consistent, high quality methodology to reach similar clinical recommendations. Unfortunately the methodological quality of CPGs developed by different organizations varies considerably. These differences reflect the specific mission, size, financial resources, membership and target audience of each organization. Therefore, before specific recommendations from CPGs are implemented into clinical practice their underlying methodology and quality of evidence should be critically reviewed. Accordingly we appraised published CPGs from leading organizations on the treatment of prostate cancer. Our immediate goal was to guide efforts of the Florida Prostate Cancer Advisory Council (http://prostatecanceradvisorycouncil.org) to develop a state legislature commissioned system of care for Florida. Using the AGREE II instrument we assessed the methodological rigor and transparency of those CPGs as well as the variability among them.
reviewers with prior evidence-based medicine training assessed methodological quality using the validated AGREE II instrument.3,4 It includes 23 items that map to 6 domains, including 1) scope and purpose (3 items), 2) stakeholder involvement (3 items), 3) rigor of development (8 items), 4) clarity of presentation (3 items), 5) applicability (4 items) and 6) editorial independence (2 items) (supplementary table, http://jurology.com/). The 4 reviewers completed the user training recommended by the AGREE II developers as well as 2 training rounds of CPG assessment using bladder cancer guidelines. They independently scored CPGs on a scale of 0 to 7 on each item per AGREE II instrument recommendations, quantifying the extent that criteria were met. Reviewer scores were then expressed as standardized domain scores on a percent scale of 0% to 100%. We calculated domain scores by adding all scores of individual items in a domain and scaling the total as a percent of the maximum possible score for that domain. All assessments were based on the published full text versions of the CPGs and on any supporting documentation as referenced. Discrepancies were resolved by consensus after discussion among reviewers. If several versions of a CPG from an organization were available, we formally reviewed only the most recently published version. To test our hypothesis we performed descriptive statistics and nonparametric tests using SPSSÒ, version 21. We calculated the intraclass correlation and the within question variation for each of the 23 AGREE II questions as a measure of interrater reliability. Intraclass correlation was considered poor, fair, good and excellent for values in the range of less than 0.4, 0.40 to 0.59, 0.60 to 0.74 and 0.75 to 1.0, respectively.5
RESULTS MATERIALS AND METHODS We searched for CPGs on the therapeutic management of prostate cancer using 3 databases, including 1) the National Guideline Clearinghouse (http://www.guideline. gov), a public resource of AHRQ (Agency for Healthcare Research and Quality), 2) the guideline database of G-I-N (Guidelines International Network, http://www.g-i-n.net), an international nonprofit organization devoted to the development and dissemination of CPGs, and 3) PubMedÒ (http://www.ncbi.nlm.nih.gov/pubmed), which searches the United States NLM (National Library of Medicine). For each of those databases we used broad, sensitive search strategies to identify relevant CPGs from leading organizations during the 15-year study period of 1999 through 2014. We included the most recent updates of previously published guidelines. CPGs limited to early detection, screening, staging and/or diagnosis were excluded from analysis. We also excluded publications (eg editorials and letters) that simply discussed guidelines. We limited our study to CPGs published in English. To assess the quality of the CPGs in our study we applied structured data abstraction. Four independent
Ultimately 13 CPGs met our study inclusion criteria (supplementary Appendix, http://jurology.com/). Six CPGs were from organizations originating in the United States and the other 7 were from international government entities. Overall the highest median scores were in 3 AGREE II domains, including domain 4dclarity of presentation (87.5%), domain 6deditorial independence (85.4%) and domain 1dscope and purpose (84.7%) (see table and figure). The lowest median score of 28.1% was for applicability (domain 5). Although the median score of editorial independence (domain 6) was high at 85.4%, variability was also substantial with an IQR of 12.5% to 100%. To better understand the observed AGREE II scores, especially for domains with low scores, we analyzed the responses that contributed to each domain (supplementary table, http://jurology.com/). Scores of applicability (domain 5) were low due to the low median scores (less than 50%) on questions on the presentation of monitoring and/or auditing criteria (17.9%) and on the consideration of resource implication (42.9%). In regard to stakeholder
CLINICAL PRACTICE GUIDELINES ON PROSTATE CANCER
1155
Mean AGREE II domain scores of 13 prostate cancer guidelines % Domain (type) Guideline
1 (scope þ purpose)
2 (stakeholder involvement)
3 (development rigor)
4 (presentation clarity)
5 (applicability)
6 (editorial independence)
ABS Aragon Institute of Health Sciences ASCOÒ AUA AUA/American Society for Radiation Oncology Cancer Council Australia British Association of Urological Surgeons European Association of Urology NCCN NICE New Zealand Ministry of Health Royal College of Surgeons in Ireland Society of Urologic Oncology
81.94 95.83 93.06 100.00 84.72 81.94 58.33 68.06 80.56 98.61 73.61 90.74 91.67
56.94 68.06 38.89 81.94 44.44 90.28 25.00 63.89 58.33 100.00 59.72 53.70 52.78
54.17 88.02 63.54 82.81 25.00 93.23 18.23 66.15 57.29 87.50 22.40 50.00 50.52
84.72 94.44 81.94 93.06 62.50 98.61 73.61 98.61 93.06 87.50 81.94 94.44 62.50
13.54 54.17 31.25 11.46 8.33 50.00 21.88 64.58 52.08 77.08 28.13 4.17 26.04
33.33 89.58 93.75 89.58 16.67 97.92 12.50 100.00 85.42 97.92 22.92 33.33 83.33
84.72
58.33
57.29
87.50
28.13
85.42
Medians
involvement (domain 2) the views and preferences of the target population were rarely sought (45.7%). To assess the quality of the CPGs we calculated the domain scores of individual guidelines (see table). Overall the United Kingdom NICE CPG achieved the highest scores of greater than 80% for all domains except applicability (domain 5), with a score that was still high at 77.1%. The AUA CPG also scored consistently above 80% but the score for applicability (domain 5) was only 11.5%. The average intraclass correlation of the 23 AGREE II questions was 0.54 (95% CI 0.29e0.79) with a within question variation of 1.77. There was no apparent association between the average AGREE II score and the intraclass correlation.
AGREE II scores of 13 prostate cancer guidelines by domain
DISCUSSION IOM defined CPGs as “statements that include recommendations intended to optimize patient care that are informed by a systematic review of evidence and an assessment of the benefits and harms of alternative care options.”6 There is increasing interest in using CPGs as a tool to assimilate the best available evidence for specific clinical questions to guide evidence-based clinical practice and thereby improve quality of care, decrease variation and improve cost-effectiveness of health care delivery. Consequently IOM identified 8 standards to help develop rigorous, trustworthy CPGs. They include transparency and disclosure of conflicts of interest, identification of group/author composition and evidence supporting recommendations. These criteria closely mirror those of the AGREE II survey, which may serve as an instrument for clinicians to identify high quality CPGs. We critically assessed the methodological quality of 13 CPGs for prostate cancer. We had 2 key findings. 1) We found a large degree of heterogeneity in the methodological quality of CPGs developed by different organizations, underscoring the need for users to critically appraise such documents before applying them to decisions about individual patient care or health policy.1 2) We found major shortcomings in the domains of stakeholder involvement and applicability that potentially undermine the validity of CPGs, raising concerns about their dissemination and impact. Clinician use of CPGs represents a final translation hurdle of applying scientific research to patient care. The inconsistencies and lack of methodological rigor identified in our study, which are recognized as part of IOM standards, threaten to undermine the common goal of advancing high
1156
CLINICAL PRACTICE GUIDELINES ON PROSTATE CANCER
quality patient care. For example, in the domain of applicability (domain 5) our evaluation demonstrated that most guidelines did not address the barriers and facilitators to implementing recommendations and did not provide ways to audit effective implementation. As a result, clinicians may have difficulty enacting the clinical recommendations set forth in CPGs and organizations that propose these guidelines may have limited means to monitor how their recommendations are put into practice. As another example the IOM identified transparency and disclosure as a key element in developing trustworthy CPGs. In domain 6 the guideline scores were based largely on the degree to which the developers provided explicit disclosures regarding conflicts of interest (external funding, financial conflicts among authors, etc). Guidelines that scored high in this domain disclosed financial conflicts of interest and maintained explicit statements to provide assurance that the views and interests of the funding body behind the CPG did not unduly influence the final recommendations. Most guidelines recognized the importance of these conflicts of interest on editorial independence and clearly disclosed them. The mean score of this domain was 85%, although there was significant variability among guidelines. Few studies to date have specifically addressed the methodological quality of CPGs on prostate cancer. Prior studies from our group provided a critical overview of such CPGs by major professional organizations but we focused on the lack of methodological rigor of development. We did not apply the AGREE II instrument until the current study.7,8 An assessment similar to our current study was performed by the CEP (Capacity Enhancement Program) of the Cancer Guidelines Advisory Group in Canada with the objective of assessing the quality and comprehensiveness of prostate cancer CPGs.9 That study used the AGREE II instrument to systematically evaluate 46 English language CPGs published from 2003 through 2008. In contrast to our study, the Canadian study had a broader scope. It included questions on prevention and promotion, screening, diagnostic assessment and staging, which are questions that we excluded. The Canadian study also included several guideline documents per organization if more than one was available. Its mean domain scores were 60% (range 14% to 94%) for scope and purpose, 40% (range 0% to 83%) for stakeholder involvement, 42% (range 5% to 98%) for rigor of development, 67% (range 27% to 97%) for clarity of presentation, 25% (range 4% to 73%) for applicability and 31% (range 0% to 94%) for editorial independence. Briefly, scores in the Canadian study reflected major variability between CPGs
and a particular need for improvement in the domain of applicability, which had the lowest score. As in our study, the Canadian study showed that NICE CPGs consistently had the highest scores across domains. In a similar study Qaseem et al focused exclusively on prostate cancer screening CPGs by 4 major organizations, including ACS (American Cancer SocietyÒ), AUA, USPSTF (United States Preventive Services Task Force) and ACPM (American College of Preventive Medicine).10 They found the best scores for scope and purpose (domain 1) and clarity of presentation (domain 4), and the lowest score for stakeholder involvement (domain 2). Lastly our findings are consistent with those of a systematic review of all studies that used the AGREE instrument to assess CPG quality.11 That systematic review showed that despite some improvement with time quality had to further improve. Enhanced international collaboration was suggested as an approach to help increase the efficiency of the CPG development process. Given that CPGs are designed to guide clinician behavior and provide explicit recommendations for the treatment of typical index patients based on the best available research evidence, our study highlights the importance of urologists carefully reviewing the quality of CPGs before using them.1 As our study demonstrates, the methodological quality developed by different organizations varies considerably, impacting the confidence that we can place in their recommendations. Therefore, urologists should be familiar with the defining characteristics of clinical guidelines that deserve the evidence-based label. Meanwhile guideline developers in urology should strive to raise the bar by adopting a transparent, methodologically rigorous and ideally unified framework to rate the quality of evidence and move from evidence to recommendations.12 We recognize several limitations to our study. 1) We relied exclusively on published full text versions of the CPGs and on any supporting documentation as referenced. We did not reach out to CPG authors or professional organizations to obtain additional information. However, given that the published CPG remains the only practical source of information for most users, our findings are relevant and important. 2) Our study was limited by the inherent subjectivity of any assessment of methodological quality. However, AGREE II was shown to be a reliable assessment tool and our evaluation was performed by 4 (rather than a minimum of 2) independent reviewers.3,4 All 4 reviewers completed online training modules specific to the AGREE II instrument and successfully participated in the
CLINICAL PRACTICE GUIDELINES ON PROSTATE CANCER
pilot testing of our data abstraction form. Despite the involvement of a larger number of independent reviewers interrater reliability was acceptable. Our study findings are important for organizations such as the Florida Prostate Cancer Advisory Council and for payers such as BlueCrossÒ BlueShieldÒ as they seek to develop measures of accountability to assess and monitor quality of care, and for guiding improvement initiatives.13,14 Quality of care measures should be based on strong recommendations15 developed using rigorous methodology, as reflected by high AGREE II scores, ideally across all domains. However, the quality of evidence supporting many CPGs has not been systematically investigated. For example, Poonacha et al found that recommendation issues in guidelines by NCCN were largely developed based on lower levels of evidence.16 Given the relatively low scores achieved by the NCCN CPG in the domains of stakeholder involvement, rigor of development and applicability, its prominent role in advising drug coverage decisions for the Centers for Medicare and Medicaid Services appears questionable.17 In contrast, high standards in the form of high AGREE II scores were met by the NICE CPG, which provides additional cost-effectiveness analyses that routinely help inform health policy decisions for the United Kingdom NHS (National Health Service). NICE has tremendous resources with an annual budget equivalent to more than $70 million. However, its CPG is specific to the United Kingdom health care setting and, thus, it may have limited applicability to other countries.18 The AUA CPG comes close to meeting the AGREE II standards except for the applicability domain. Ongoing efforts to standardize reporting and facilitate the integration of recommendations
1157
into electronic medical records systems with tools such as BRIDGE-Wiz are expected to increase the applicability of CPGs, enhancing their impact and promoting their implementation.19,20 Formal consideration of resource utilization may be a more elusive goal. In a recent study Schwartz and Pearson found that only about 50% of all major professional organizations considered costs in their CPGs and only about 25% used a formal grading system.2 Barriers to increased consideration of costs include patient concerns over bed side rationing. The Choosing WiselyÒ initiative of the ABIM (American Board of Internal MedicineÒ) Foundation (http://www.choosingwisely.org/doctor-patientlists/) represents a step in the right direction since it identifies medical tests and procedures that likely provide little benefit, raise costs and in some cases cause harm.21
CONCLUSIONS Our study reflects growing awareness of the need for improving CPG methodology and quality. In our systematic appraisal we found that published CPGs on the treatment of prostate cancer are of variable methodological quality and frequently fell short of current standards. We noted significant deficiencies in the domains of applicability and stakeholder involvement as well as a large degree of heterogeneity in the methodological quality of CPGs developed by different organizations. These shortcomings limit the effectiveness of CPGs for use by policy makers and health care providers.
ACKNOWLEDGMENT Dr. Mary E. Knatterud, Department of Urology, University of Minnesota, provided editorial assistance.
REFERENCES 1. Dahm P, Yeung LL, Gallucci M et al: How to use a clinical practice guideline. J Urol 2009; 181: 472.
5. Hallgren KA: Computing inter-rater reliability for observational data: an overview and tutorial. Tutor Quant Methods Psychol 2012; 8: 23.
2. Schwartz JA and Pearson SD: Cost consideration in the clinical guidance documents of physician specialty societies in the United States. JAMA Intern Med 2013; 173: 1091.
6. Institute of Medicine (U.S.). Committee on Standards for Developing Trustworthy Clinical Practice Guidelines: Clinical Practice Guidelines We Can Trust. Edited by R Graham, M Mancher, D Miller Wolman et al. Washington, D.C.: National Academies Press 2011; pp 4 and 266.
3. Brouwers MC, Kho ME, Browman GP et al: Development of the AGREE II, part 1: performance, usefulness and areas for improvement. CMAJ 2010; 182: 1045. 4. Brouwers MC, Kho ME, Browman GP et al: Development of the AGREE II, part 2: assessment of validity of items and tools to support application. CMAJ 2010; 182: E472.
7. Dahm P, Kunz R and Sch€unemann H: Evidence-based clinical practice guidelines for prostate cancer: the need for a unified approach. Curr Opin Urol 2007; 17: 200. 8. Dahm P, Yeung LL, Chang SS et al: A critical review of clinical practice guidelines for the management of clinically localized prostate cancer. J Urol 2008; 180: 451.
9. Cancer Practice Guidelines Status Report: Prostate Cancer. Toronto: Capacity Enhancement Program of the Cancer Guidelines Advisory Group. 10. Qaseem A, Barry MJ, Denberg TD et al: Screening for prostate cancer: a guidance statement from the Clinical Guidelines Committee of the American College of Physicians. Ann Intern Med 2013; 158: 761. 11. Alonso-Coello P, Irfan A, Sola I et al: The quality of clinical practice guidelines over the last two decades: a systematic review of guideline appraisal studies. Qual Saf Health Care 2010; 19: e58. 12. Dahm P, Chapple CR, Konety BR et al: The future of clinical practice guidelines in urology. Eur Urol 2011; 60: 72.
1158
CLINICAL PRACTICE GUIDELINES ON PROSTATE CANCER
13. Miller DC, Murtagh DS, Suh RS et al: Establishment of a urological surgery quality collaborative. J Urol 2010; 184: 2485. 14. Miller DC and Saigal CS: Quality of care indicators for prostate cancer: progress toward consensus. Urol Oncol 2009; 27: 427. 15. Guyatt GH, Oxman AD, Vist GE et al: GRADE: what is “quality of evidence” and why is it important to clinicians? BMJ 2008; 336: 995. 16. Poonacha TK and Go RS: Level of scientific evidence underlying recommendations arising from the National Comprehensive Cancer
Network clinical practice guidelines. J Clin Oncol 2011; 29: 186. 17. Abernethy AP, Raman G, Balk EM et al: Systematic review: reliability of compendia methods for off-label oncology indications. Ann Intern Med 2009; 150: 336. 18. Chalkidou K: Comparative effectiveness review within the U.K.’s National Institute for Health and Clinical Excellence. Issue Brief (Commonw Fund) 2009; 59: 1. 19. Shiffman RN, Michel G, Rosenfeld RM et al: Building better guidelines with BRIDGE-Wiz:
development and evaluation of a software assistant to promote clarity, transparency, and implementability. J Am Med Inform Assoc 2012; 19: 94. 20. Shiffman RN, Shekelle P, Overhage JM et al: Standardized reporting of clinical practice guidelines: a proposal from the Conference on Guideline Standardization. Ann Intern Med 2003; 139: 493. 21. Gliwa C and Pearson SD: Evidentiary rationales for the Choosing Wisely Top 5 lists. JAMA 2014; 311: 1443.
Clinical Practice Guidelines on Prostate Cancer: A Critical Appraisal Mohit Gupta, John McCauley, Amy Farkas, Ahmet Gudeloglu, Molly M. Neuberger, Yen-Yi Ho, Lawrence Yeung, Johannes Vieweg* and Philipp Dahm† From the Department of Urology, University of Florida (MG, JM, AF, AG, MMN, LY, JV) and Malcom Randall Veterans Affairs Medical Center, Gainesville (PD), Florida, and Department of Urology (PD) and Division of Biostatistics, School of Public Health (YYH), University of Minnesota and Urology Section, Minneapolis Veterans Affairs Health Care System (MMN, PD), Minneapolis, Minnesota
Purpose: Clinical practice guidelines are increasingly being used by leading organizations to promote high quality evidence-based patient care. However, the methodological quality of clinical practice guidelines developed by different organizations varies considerably. We assessed published clinical practice guidelines on the treatment of localized prostate cancer to evaluate the rigor, applicability and transparency of their recommendations. Materials and Methods: We searched for English based clinical practice guidelines on treatment of localized prostate cancer from leading organizations in the 15-year period from 1999 to 2014. Clinical practice guidelines limited to early detection, screening, staging and/or diagnosis of prostate cancer were excluded from analysis. Four independent reviewers used the validated AGREE II instrument to assess the quality of clinical practice guidelines in 6 domains, including 1) scope and purpose, 2) stakeholder involvement, 3) rigor of development, 4) clarity of presentation, 5) applicability and 6) editorial independence. Results: A total of 13 clinical practice guidelines met inclusion criteria. Overall the highest median scores were in the AGREE II domains of clarity of presentation, editorial independence, and scope and purpose. The lowest median score was for applicability (28.1%). Although the median score of editorial independence was high (85.4%), variability was also substantial (IQR 12.5e100). NICE and AUA clinical practice guidelines consistently scored well in most domains. Conclusions: Clinical practice guidelines from different organizations on treatment of localized prostate cancer are of variable quality and fall short of current standards in certain areas, especially in applicability and stakeholder involvement. Improvements in these key domains can enhance the impact and implementation of clinical practice guidelines.
Abbreviations and Acronyms ABS ¼ American Brachytherapy Society AGREE ¼ Appraisal of Guidelines for Research and Evaluation AUA ¼ American Urological Association CPG ¼ clinical practice guideline IOM ¼ Institute of Medicine NCCN ¼ National Comprehensive Cancer NetworkÒ NICE ¼ National Institute for Health and Care Excellence Accepted for publication October 29, 2014. * Financial interest and/or other relationship with American Urological Association. † Correspondence: Department of Urology, University of Minnesota, Minneapolis Veterans Affairs Health Care System, Urology Section 112D, One Veterans Dr., Minneapolis, Minnesota 55417 (telephone: 612-467-3532; FAX: 612-4672232; e-mail: [email protected]).
For another article on a related topic see page 1382.
Key Words: prostatic neoplasms, practice guidelines as topic, evidence-based medicine, government, Florida
CLINICAL practice guidelines are important tools to help clinicians and patients reach evidence-based decisions about health care. The development of CPGs has been central
to promoting high quality, evidencebased and safe patient care. They hold promise for improving the quality, appropriateness and costeffectiveness of medical therapies.
0022-5347/15/1934-1153/0 THE JOURNAL OF UROLOGY® © 2015 by AMERICAN UROLOGICAL ASSOCIATION EDUCATION AND RESEARCH, INC.
http://dx.doi.org/10.1016/j.juro.2014.10.105 Vol. 193, 1153-1158, April 2015 Printed in U.S.A.
www.jurology.com
j
1153
1154
CLINICAL PRACTICE GUIDELINES ON PROSTATE CANCER
Thus, leading organizations in the field of urology are increasingly recognizing the importance of CPGs and dedicating considerable resources toward developing and disseminating them. CPGs differ from systematic reviews, cost analyses and decision models by making explicit recommendations aimed at directly influencing patient, clinician and policy maker decision making. They are also becoming the basis of quality of care measures that are likely to affect urologist reimbursements with pay for performance measures on the horizon.1,2 Ideally CPGs from different professional organizations would use consistent, high quality methodology to reach similar clinical recommendations. Unfortunately the methodological quality of CPGs developed by different organizations varies considerably. These differences reflect the specific mission, size, financial resources, membership and target audience of each organization. Therefore, before specific recommendations from CPGs are implemented into clinical practice their underlying methodology and quality of evidence should be critically reviewed. Accordingly we appraised published CPGs from leading organizations on the treatment of prostate cancer. Our immediate goal was to guide efforts of the Florida Prostate Cancer Advisory Council (http://prostatecanceradvisorycouncil.org) to develop a state legislature commissioned system of care for Florida. Using the AGREE II instrument we assessed the methodological rigor and transparency of those CPGs as well as the variability among them.
reviewers with prior evidence-based medicine training assessed methodological quality using the validated AGREE II instrument.3,4 It includes 23 items that map to 6 domains, including 1) scope and purpose (3 items), 2) stakeholder involvement (3 items), 3) rigor of development (8 items), 4) clarity of presentation (3 items), 5) applicability (4 items) and 6) editorial independence (2 items) (supplementary table, http://jurology.com/). The 4 reviewers completed the user training recommended by the AGREE II developers as well as 2 training rounds of CPG assessment using bladder cancer guidelines. They independently scored CPGs on a scale of 0 to 7 on each item per AGREE II instrument recommendations, quantifying the extent that criteria were met. Reviewer scores were then expressed as standardized domain scores on a percent scale of 0% to 100%. We calculated domain scores by adding all scores of individual items in a domain and scaling the total as a percent of the maximum possible score for that domain. All assessments were based on the published full text versions of the CPGs and on any supporting documentation as referenced. Discrepancies were resolved by consensus after discussion among reviewers. If several versions of a CPG from an organization were available, we formally reviewed only the most recently published version. To test our hypothesis we performed descriptive statistics and nonparametric tests using SPSSÒ, version 21. We calculated the intraclass correlation and the within question variation for each of the 23 AGREE II questions as a measure of interrater reliability. Intraclass correlation was considered poor, fair, good and excellent for values in the range of less than 0.4, 0.40 to 0.59, 0.60 to 0.74 and 0.75 to 1.0, respectively.5
RESULTS MATERIALS AND METHODS We searched for CPGs on the therapeutic management of prostate cancer using 3 databases, including 1) the National Guideline Clearinghouse (http://www.guideline. gov), a public resource of AHRQ (Agency for Healthcare Research and Quality), 2) the guideline database of G-I-N (Guidelines International Network, http://www.g-i-n.net), an international nonprofit organization devoted to the development and dissemination of CPGs, and 3) PubMedÒ (http://www.ncbi.nlm.nih.gov/pubmed), which searches the United States NLM (National Library of Medicine). For each of those databases we used broad, sensitive search strategies to identify relevant CPGs from leading organizations during the 15-year study period of 1999 through 2014. We included the most recent updates of previously published guidelines. CPGs limited to early detection, screening, staging and/or diagnosis were excluded from analysis. We also excluded publications (eg editorials and letters) that simply discussed guidelines. We limited our study to CPGs published in English. To assess the quality of the CPGs in our study we applied structured data abstraction. Four independent
Ultimately 13 CPGs met our study inclusion criteria (supplementary Appendix, http://jurology.com/). Six CPGs were from organizations originating in the United States and the other 7 were from international government entities. Overall the highest median scores were in 3 AGREE II domains, including domain 4dclarity of presentation (87.5%), domain 6deditorial independence (85.4%) and domain 1dscope and purpose (84.7%) (see table and figure). The lowest median score of 28.1% was for applicability (domain 5). Although the median score of editorial independence (domain 6) was high at 85.4%, variability was also substantial with an IQR of 12.5% to 100%. To better understand the observed AGREE II scores, especially for domains with low scores, we analyzed the responses that contributed to each domain (supplementary table, http://jurology.com/). Scores of applicability (domain 5) were low due to the low median scores (less than 50%) on questions on the presentation of monitoring and/or auditing criteria (17.9%) and on the consideration of resource implication (42.9%). In regard to stakeholder
CLINICAL PRACTICE GUIDELINES ON PROSTATE CANCER
1155
Mean AGREE II domain scores of 13 prostate cancer guidelines % Domain (type) Guideline
1 (scope þ purpose)
2 (stakeholder involvement)
3 (development rigor)
4 (presentation clarity)
5 (applicability)
6 (editorial independence)
ABS Aragon Institute of Health Sciences ASCOÒ AUA AUA/American Society for Radiation Oncology Cancer Council Australia British Association of Urological Surgeons European Association of Urology NCCN NICE New Zealand Ministry of Health Royal College of Surgeons in Ireland Society of Urologic Oncology
81.94 95.83 93.06 100.00 84.72 81.94 58.33 68.06 80.56 98.61 73.61 90.74 91.67
56.94 68.06 38.89 81.94 44.44 90.28 25.00 63.89 58.33 100.00 59.72 53.70 52.78
54.17 88.02 63.54 82.81 25.00 93.23 18.23 66.15 57.29 87.50 22.40 50.00 50.52
84.72 94.44 81.94 93.06 62.50 98.61 73.61 98.61 93.06 87.50 81.94 94.44 62.50
13.54 54.17 31.25 11.46 8.33 50.00 21.88 64.58 52.08 77.08 28.13 4.17 26.04
33.33 89.58 93.75 89.58 16.67 97.92 12.50 100.00 85.42 97.92 22.92 33.33 83.33
84.72
58.33
57.29
87.50
28.13
85.42
Medians
involvement (domain 2) the views and preferences of the target population were rarely sought (45.7%). To assess the quality of the CPGs we calculated the domain scores of individual guidelines (see table). Overall the United Kingdom NICE CPG achieved the highest scores of greater than 80% for all domains except applicability (domain 5), with a score that was still high at 77.1%. The AUA CPG also scored consistently above 80% but the score for applicability (domain 5) was only 11.5%. The average intraclass correlation of the 23 AGREE II questions was 0.54 (95% CI 0.29e0.79) with a within question variation of 1.77. There was no apparent association between the average AGREE II score and the intraclass correlation.
AGREE II scores of 13 prostate cancer guidelines by domain
DISCUSSION IOM defined CPGs as “statements that include recommendations intended to optimize patient care that are informed by a systematic review of evidence and an assessment of the benefits and harms of alternative care options.”6 There is increasing interest in using CPGs as a tool to assimilate the best available evidence for specific clinical questions to guide evidence-based clinical practice and thereby improve quality of care, decrease variation and improve cost-effectiveness of health care delivery. Consequently IOM identified 8 standards to help develop rigorous, trustworthy CPGs. They include transparency and disclosure of conflicts of interest, identification of group/author composition and evidence supporting recommendations. These criteria closely mirror those of the AGREE II survey, which may serve as an instrument for clinicians to identify high quality CPGs. We critically assessed the methodological quality of 13 CPGs for prostate cancer. We had 2 key findings. 1) We found a large degree of heterogeneity in the methodological quality of CPGs developed by different organizations, underscoring the need for users to critically appraise such documents before applying them to decisions about individual patient care or health policy.1 2) We found major shortcomings in the domains of stakeholder involvement and applicability that potentially undermine the validity of CPGs, raising concerns about their dissemination and impact. Clinician use of CPGs represents a final translation hurdle of applying scientific research to patient care. The inconsistencies and lack of methodological rigor identified in our study, which are recognized as part of IOM standards, threaten to undermine the common goal of advancing high
1156
CLINICAL PRACTICE GUIDELINES ON PROSTATE CANCER
quality patient care. For example, in the domain of applicability (domain 5) our evaluation demonstrated that most guidelines did not address the barriers and facilitators to implementing recommendations and did not provide ways to audit effective implementation. As a result, clinicians may have difficulty enacting the clinical recommendations set forth in CPGs and organizations that propose these guidelines may have limited means to monitor how their recommendations are put into practice. As another example the IOM identified transparency and disclosure as a key element in developing trustworthy CPGs. In domain 6 the guideline scores were based largely on the degree to which the developers provided explicit disclosures regarding conflicts of interest (external funding, financial conflicts among authors, etc). Guidelines that scored high in this domain disclosed financial conflicts of interest and maintained explicit statements to provide assurance that the views and interests of the funding body behind the CPG did not unduly influence the final recommendations. Most guidelines recognized the importance of these conflicts of interest on editorial independence and clearly disclosed them. The mean score of this domain was 85%, although there was significant variability among guidelines. Few studies to date have specifically addressed the methodological quality of CPGs on prostate cancer. Prior studies from our group provided a critical overview of such CPGs by major professional organizations but we focused on the lack of methodological rigor of development. We did not apply the AGREE II instrument until the current study.7,8 An assessment similar to our current study was performed by the CEP (Capacity Enhancement Program) of the Cancer Guidelines Advisory Group in Canada with the objective of assessing the quality and comprehensiveness of prostate cancer CPGs.9 That study used the AGREE II instrument to systematically evaluate 46 English language CPGs published from 2003 through 2008. In contrast to our study, the Canadian study had a broader scope. It included questions on prevention and promotion, screening, diagnostic assessment and staging, which are questions that we excluded. The Canadian study also included several guideline documents per organization if more than one was available. Its mean domain scores were 60% (range 14% to 94%) for scope and purpose, 40% (range 0% to 83%) for stakeholder involvement, 42% (range 5% to 98%) for rigor of development, 67% (range 27% to 97%) for clarity of presentation, 25% (range 4% to 73%) for applicability and 31% (range 0% to 94%) for editorial independence. Briefly, scores in the Canadian study reflected major variability between CPGs
and a particular need for improvement in the domain of applicability, which had the lowest score. As in our study, the Canadian study showed that NICE CPGs consistently had the highest scores across domains. In a similar study Qaseem et al focused exclusively on prostate cancer screening CPGs by 4 major organizations, including ACS (American Cancer SocietyÒ), AUA, USPSTF (United States Preventive Services Task Force) and ACPM (American College of Preventive Medicine).10 They found the best scores for scope and purpose (domain 1) and clarity of presentation (domain 4), and the lowest score for stakeholder involvement (domain 2). Lastly our findings are consistent with those of a systematic review of all studies that used the AGREE instrument to assess CPG quality.11 That systematic review showed that despite some improvement with time quality had to further improve. Enhanced international collaboration was suggested as an approach to help increase the efficiency of the CPG development process. Given that CPGs are designed to guide clinician behavior and provide explicit recommendations for the treatment of typical index patients based on the best available research evidence, our study highlights the importance of urologists carefully reviewing the quality of CPGs before using them.1 As our study demonstrates, the methodological quality developed by different organizations varies considerably, impacting the confidence that we can place in their recommendations. Therefore, urologists should be familiar with the defining characteristics of clinical guidelines that deserve the evidence-based label. Meanwhile guideline developers in urology should strive to raise the bar by adopting a transparent, methodologically rigorous and ideally unified framework to rate the quality of evidence and move from evidence to recommendations.12 We recognize several limitations to our study. 1) We relied exclusively on published full text versions of the CPGs and on any supporting documentation as referenced. We did not reach out to CPG authors or professional organizations to obtain additional information. However, given that the published CPG remains the only practical source of information for most users, our findings are relevant and important. 2) Our study was limited by the inherent subjectivity of any assessment of methodological quality. However, AGREE II was shown to be a reliable assessment tool and our evaluation was performed by 4 (rather than a minimum of 2) independent reviewers.3,4 All 4 reviewers completed online training modules specific to the AGREE II instrument and successfully participated in the
CLINICAL PRACTICE GUIDELINES ON PROSTATE CANCER
pilot testing of our data abstraction form. Despite the involvement of a larger number of independent reviewers interrater reliability was acceptable. Our study findings are important for organizations such as the Florida Prostate Cancer Advisory Council and for payers such as BlueCrossÒ BlueShieldÒ as they seek to develop measures of accountability to assess and monitor quality of care, and for guiding improvement initiatives.13,14 Quality of care measures should be based on strong recommendations15 developed using rigorous methodology, as reflected by high AGREE II scores, ideally across all domains. However, the quality of evidence supporting many CPGs has not been systematically investigated. For example, Poonacha et al found that recommendation issues in guidelines by NCCN were largely developed based on lower levels of evidence.16 Given the relatively low scores achieved by the NCCN CPG in the domains of stakeholder involvement, rigor of development and applicability, its prominent role in advising drug coverage decisions for the Centers for Medicare and Medicaid Services appears questionable.17 In contrast, high standards in the form of high AGREE II scores were met by the NICE CPG, which provides additional cost-effectiveness analyses that routinely help inform health policy decisions for the United Kingdom NHS (National Health Service). NICE has tremendous resources with an annual budget equivalent to more than $70 million. However, its CPG is specific to the United Kingdom health care setting and, thus, it may have limited applicability to other countries.18 The AUA CPG comes close to meeting the AGREE II standards except for the applicability domain. Ongoing efforts to standardize reporting and facilitate the integration of recommendations
1157
into electronic medical records systems with tools such as BRIDGE-Wiz are expected to increase the applicability of CPGs, enhancing their impact and promoting their implementation.19,20 Formal consideration of resource utilization may be a more elusive goal. In a recent study Schwartz and Pearson found that only about 50% of all major professional organizations considered costs in their CPGs and only about 25% used a formal grading system.2 Barriers to increased consideration of costs include patient concerns over bed side rationing. The Choosing WiselyÒ initiative of the ABIM (American Board of Internal MedicineÒ) Foundation (http://www.choosingwisely.org/doctor-patientlists/) represents a step in the right direction since it identifies medical tests and procedures that likely provide little benefit, raise costs and in some cases cause harm.21
CONCLUSIONS Our study reflects growing awareness of the need for improving CPG methodology and quality. In our systematic appraisal we found that published CPGs on the treatment of prostate cancer are of variable methodological quality and frequently fell short of current standards. We noted significant deficiencies in the domains of applicability and stakeholder involvement as well as a large degree of heterogeneity in the methodological quality of CPGs developed by different organizations. These shortcomings limit the effectiveness of CPGs for use by policy makers and health care providers.
ACKNOWLEDGMENT Dr. Mary E. Knatterud, Department of Urology, University of Minnesota, provided editorial assistance.
REFERENCES 1. Dahm P, Yeung LL, Gallucci M et al: How to use a clinical practice guideline. J Urol 2009; 181: 472.
5. Hallgren KA: Computing inter-rater reliability for observational data: an overview and tutorial. Tutor Quant Methods Psychol 2012; 8: 23.
2. Schwartz JA and Pearson SD: Cost consideration in the clinical guidance documents of physician specialty societies in the United States. JAMA Intern Med 2013; 173: 1091.
6. Institute of Medicine (U.S.). Committee on Standards for Developing Trustworthy Clinical Practice Guidelines: Clinical Practice Guidelines We Can Trust. Edited by R Graham, M Mancher, D Miller Wolman et al. Washington, D.C.: National Academies Press 2011; pp 4 and 266.
3. Brouwers MC, Kho ME, Browman GP et al: Development of the AGREE II, part 1: performance, usefulness and areas for improvement. CMAJ 2010; 182: 1045. 4. Brouwers MC, Kho ME, Browman GP et al: Development of the AGREE II, part 2: assessment of validity of items and tools to support application. CMAJ 2010; 182: E472.
7. Dahm P, Kunz R and Sch€unemann H: Evidence-based clinical practice guidelines for prostate cancer: the need for a unified approach. Curr Opin Urol 2007; 17: 200. 8. Dahm P, Yeung LL, Chang SS et al: A critical review of clinical practice guidelines for the management of clinically localized prostate cancer. J Urol 2008; 180: 451.
9. Cancer Practice Guidelines Status Report: Prostate Cancer. Toronto: Capacity Enhancement Program of the Cancer Guidelines Advisory Group. 10. Qaseem A, Barry MJ, Denberg TD et al: Screening for prostate cancer: a guidance statement from the Clinical Guidelines Committee of the American College of Physicians. Ann Intern Med 2013; 158: 761. 11. Alonso-Coello P, Irfan A, Sola I et al: The quality of clinical practice guidelines over the last two decades: a systematic review of guideline appraisal studies. Qual Saf Health Care 2010; 19: e58. 12. Dahm P, Chapple CR, Konety BR et al: The future of clinical practice guidelines in urology. Eur Urol 2011; 60: 72.
1158
CLINICAL PRACTICE GUIDELINES ON PROSTATE CANCER
13. Miller DC, Murtagh DS, Suh RS et al: Establishment of a urological surgery quality collaborative. J Urol 2010; 184: 2485. 14. Miller DC and Saigal CS: Quality of care indicators for prostate cancer: progress toward consensus. Urol Oncol 2009; 27: 427. 15. Guyatt GH, Oxman AD, Vist GE et al: GRADE: what is “quality of evidence” and why is it important to clinicians? BMJ 2008; 336: 995. 16. Poonacha TK and Go RS: Level of scientific evidence underlying recommendations arising from the National Comprehensive Cancer
Network clinical practice guidelines. J Clin Oncol 2011; 29: 186. 17. Abernethy AP, Raman G, Balk EM et al: Systematic review: reliability of compendia methods for off-label oncology indications. Ann Intern Med 2009; 150: 336. 18. Chalkidou K: Comparative effectiveness review within the U.K.’s National Institute for Health and Clinical Excellence. Issue Brief (Commonw Fund) 2009; 59: 1. 19. Shiffman RN, Michel G, Rosenfeld RM et al: Building better guidelines with BRIDGE-Wiz:
development and evaluation of a software assistant to promote clarity, transparency, and implementability. J Am Med Inform Assoc 2012; 19: 94. 20. Shiffman RN, Shekelle P, Overhage JM et al: Standardized reporting of clinical practice guidelines: a proposal from the Conference on Guideline Standardization. Ann Intern Med 2003; 139: 493. 21. Gliwa C and Pearson SD: Evidentiary rationales for the Choosing Wisely Top 5 lists. JAMA 2014; 311: 1443.
