539728
research-article2014
AJMXXX10.1177/1062860614539728American Journal of Medical QualityYoung et al
Commentary
Are Clinical Practice Guidelines for Cataract and Glaucoma Trustworthy?
American Journal of Medical Quality 2015, Vol. 30(2) 188–190 © 2014 by the American College of Medical Quality Reprints and permissions: sagepub.com/journalsPermissions.nav DOI: 10.1177/1062860614539728 ajmq.sagepub.com
Benjamin K. Young, MS1,2,3, Connie M. Wu, ScB2,3, Annie M. Wu, AB2,3, Curtis E. Margo, MD, MPH4, and Paul B. Greenberg, MD1,2,3 Surgical procedures to restore vision from cataract and to preserve vision from primary open-angle glaucoma (POAG) are frequently performed in the United States. Given their importance in preventing blindness and maximizing sight, the American Academy of Ophthalmology (AAO) has devoted considerable resources to developing Cataract in the Adult Eye1 and Primary Open-Angle Glaucoma2 Preferred Practice Patterns (PPPs). Each document outlines the best practices for diagnosis and management of cataract and POAG, respectively.3 However, concerns have been raised about the quality of clinical practice guidelines (CPGs) in general, and in their trustworthiness in particular.4-6 In response, the Institute of Medicine (IOM) published a set of 8 standards required for a CPG to be considered trustworthy.7 This commentary describes the adherence of PPPs for adult cataract and POAG based on the IOM trustworthiness standards and addresses the role that such critiques hold for future improvement of practice guidelines. The analysis was performed by 4 of the authors (BKY, CMW, AMW, PBG) using the methods prescribed by the IOM standards.7 We required all substandards to be met for a standard to be considered passed. Discrepancies were resolved by discussion among the authors. Table 1 summarizes the results. The adult cataract PPP passed 25% of the IOM standards; the POAG PPP passed 50%. Transparency was not fully established in both PPPs because there was insufficient information to evaluate whether the documents met IOM standards for Conflict of Interest (COI), Development Group Composition, and External Review. Contrary to IOM recommendations, the chair of the PPP panels had COIs, and there was no explanation regarding the potential influence of panel members’ COIs on the guideline development process. It was unclear whether patients or patient advocates were included as developers or external reviewers. Furthermore, because the expertise of panel members was not stated, it was not clear whether or not they comprised a multidisciplinary group. Although the POAG PPP stated that it was externally reviewed by “experts and stakeholders before publication,” it did not specify who these individuals were.1,2 Evidence foundations and clarity of recommendations were problematic in the adult cataract PPP. The PPP noted
“All studies used to form a recommendation for care are graded for strength of evidence individually.” However, aside from the highlighted recommendations and summary benchmarks, no other statements were graded for evidence making it unclear which statements were recommendations. The main text of the adult cataract PPP was devoid of graded study citations, implying statements such as “applying 5% povidone iodine to the conjunctival cul de sac” were not “recommendation[s] for care.” Additionally, the highlighted recommendations and the summary benchmarks used different grading systems. Finally, some of the highlighted recommendations were not clinically useful; for example, “Cataract surgery is a procedure appropriately utilized in the United States.” We suggest the following changes to improve the 2 AAO PPPs: •• Ensure that the chair and no more than half of the CPG development group members have COIs •• Disclose the composition of the CPG development group to ensure that it includes a diverse representation of clinical, scientific, and methodological experts •• Record the inclusion of patients or patient representatives in CPG development •• Ensure that patients, patient organizations, and interested members of the public have an opportunity to review the CPG and describe how their comments were addressed Using the POAG PPP as a model, the adult cataract PPP also should apply an evidence rating system to the recommendations in the main text. We acknowledge limitations to this analysis. First, it may have been prohibitively strict to require all 1
Providence VA Medical Center, Providence, RI Warren Alpert Medical School of Brown University, Providence, RI 3 Rhode Island Hospital, Providence, RI 4 Morsani College of Medicine at the University of South Florida, Tampa, FL 2
Corresponding Author: Paul B. Greenberg, MD, Section of Ophthalmology, Providence VA Medical Center, 830 Chalkstone Ave, Providence, RI 02908. Email: [email protected]
Downloaded from ajm.sagepub.com at The University of Auckland Library on April 21, 2015
189
Young et al Table 1. Overview of Adherence of the American Academy of Ophthalmology Adult Cataract and Primary Open-Angle Glaucoma Best Practice Guidelines With Institute of Medicine Trustworthiness Standards. IOM Standard7
Cataract in the Adult Eye PPP1
Primary Open-Angle Glaucoma PPP2
1. Establishing Transparency
Fail: The methodology by which recommendations were formulated was not specified (eg, voting procedures, how consensus was achieved). Explanation of committee members’ areas of clinical expertise, potential influence of COIs on guideline development, and clear documentation of the external review process were lacking.
2. Management of COI
Fail: No information on how the COIs influenced PPP development. The chair was not free of COIs.
3. GDG Composition
Fail: Uncertain if the members of the PPP Committee or Panel represented a multidisciplinary group. Members’ expertise was not explicitly detailed. No representatives of the affected population, current patients, patient advocates, or patient organizations appeared involved in the development of the PPP.
4. Clinical Practice Guideline– Systematic Review Intersection
Pass: The review team (Hoskins Center Staff) used a systematic method for obtaining evidence and regularly interacted with the PPP Panel when necessary.
5. Establishing evidence foundations for and rating strength of recommendations
Fail: A rating system of level of confidence and strength of recommendations was only used for the highlighted recommendations and the summary benchmarks. Other recommendations within the main text had no rating system.
Pass: All recommendations included a rating of quality and consistency of evidence.
6. Articulation of Recommendation
Fail: Only highlighted recommendations and summary benchmarks had evidence ratings; uncertain which statements in the main text were recommendations. Furthermore, 6 of the 11 highlighted recommendations fail because they cannot be evaluated discretely.
Pass: All recommendations were properly articulated and were written such that they could be acted upon.
7. External Review
Fail: Uncertain if any patient, former patients, or patient advocates provided external review of the PPP document. The PPP did not specify who was in the “council” review group. Unclear if a record of external review comments was kept. The PPP panel did not release a review draft to the public before publication.
8. Updating
Pass: The process for updating the PPP based on discovery of new evidence and the schedules for revision was explicitly detailed online.
Fail: The chair and a majority (6 of 8) of the PPP Committee Members had COIs, and a majority (6 of 8) of the Glaucoma Panel Members had COIs. These COIs were not divested.
Abbreviations: COI, conflict of interest; GDG, Guideline Development Group; IOM, Institute of Medicine; PPP, preferred practice pattern.
substandards to pass for a main standard to be considered compliant. However, each failed main standard in the 2 PPPs still had a substandard failure rate of at least 50%. Second, the judgments were made on a pass/fail basis. However, since a CPG would be considered untrustworthy if any one standard failed, it is unlikely that a more complicated adjudication system would have changed the study results. Third, the IOM standards were not available at the time the AAO published the PPPs. However, our primary purpose was to investigate the quality of current practice guidelines using the IOM instrument with an eye toward improving future versions.
Although the validity and reliability of IOM standards have not been demonstrated, the IOM serves as an independent, nonprofit resource dedicated to improving the US health care system and plays a key role in shaping trust between patients and providers.8 Although its recommendations carry considerable weight and provide a meaningful tool to assess CPGs, other methods exist to critique practice guidelines.9 In a recent review of 130 CPGs, the median percentage of IOM standards passed was 44%.10 We assert that the AAO PPPs for adult cataract and POAG can be improved to meet all the IOM standards of trustworthiness. More
Downloaded from ajm.sagepub.com at The University of Auckland Library on April 21, 2015
190
American Journal of Medical Quality 30(2)
fundamentally, we wish to underscore the role of continuous improvement in the formulation of CPGs: for this process to occur, PPPs must be reviewed rigorously from a variety of perspectives and with an assortment of tools. The IOM standards are not the ultimate arbiter of trustworthiness but they are a reminder that the development of CPGs is an ongoing process. Authors’ Note The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs or the United States government.
Declaration of Conflicting Interests The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Dr Greenberg has received a United States Veterans Health Administration Health Services Research & Development (HSR&D) Veterans Integrated Service Network (VISN) 1 Career Development Award. The funding organization had no role in the design or conduct of this research.
References 1. American Academy of Ophthalmology; One Network. Cataract in the adult eye PPP—2011. http://one.aao.org/ preferred-practice-pattern/cataract-in-adult-eye-ppp–october-2011. Accessed May 22, 2014.
2. American Academy of Ophthalmology. Primary open-angle glaucoma PPP—2010. http://one.aao.org/ preferred-practice-pattern/primary-openangle-glaucomappp–october-2010. Accessed May 22, 2014. 3. NIH Medline Plus. Leading causes of blindness. http:// www.nlm.nih.gov/medlineplus/magazine/issues/summer08/articles/summer08pg14-15.html. Published 2008. Accessed January 2, 2014. 4. Ransohoff DF, Pignone M, Sox HC. How to decide whether a clinical practice guideline is trustworthy. JAMA. 2013;309:139-140. 5. Shaneyfelt TM, Centor RM. Reassessment of clinical practice guidelines: go gently into that good night. JAMA. 2009;301:868-869. 6. Guyatt G, Akl EA, Hirsh J, et al. The vexing problem of guidelines and conflict of interest: a potential solution. Ann Intern Med. 2010;152:738-741. 7. Graham R, Mancher M, Wolman DM, Greenfield S, Steinberg E, eds. Clinical Practice Guidelines We Can Trust. Washington, DC: National Academies Press; 2011. 8. Institute of Medicine. Informing the Future: Critical Issues in Health. 7th ed. http://www.iom.edu/About-IOM/~/ media/Files/About%20the%20IOM/ITF_seventh.pdf. Published 2013. Accessed January 2, 2014. 9. The AGREE Next Steps Consortium. Appraisal of Guidelines for Research and Evaluation II instrument. http:// www.agreetrust.org/wp-content/uploads/2013/03/1397_ AGREE+II+Users+Manual+and+23-item+Instrument+ENGLISH1.pdf. Published May 2009. Accessed January 2, 2014. 10. Kung J, Miller RR, Mackowiak PA. Failure of clinical practice guidelines to meet Institute of Medicine standards: two more decades of little, if any, progress. Arch Intern Med. 2012;172:1628-1633.
Downloaded from ajm.sagepub.com at The University of Auckland Library on April 21, 2015
research-article2014
AJMXXX10.1177/1062860614539728American Journal of Medical QualityYoung et al
Commentary
Are Clinical Practice Guidelines for Cataract and Glaucoma Trustworthy?
American Journal of Medical Quality 2015, Vol. 30(2) 188–190 © 2014 by the American College of Medical Quality Reprints and permissions: sagepub.com/journalsPermissions.nav DOI: 10.1177/1062860614539728 ajmq.sagepub.com
Benjamin K. Young, MS1,2,3, Connie M. Wu, ScB2,3, Annie M. Wu, AB2,3, Curtis E. Margo, MD, MPH4, and Paul B. Greenberg, MD1,2,3 Surgical procedures to restore vision from cataract and to preserve vision from primary open-angle glaucoma (POAG) are frequently performed in the United States. Given their importance in preventing blindness and maximizing sight, the American Academy of Ophthalmology (AAO) has devoted considerable resources to developing Cataract in the Adult Eye1 and Primary Open-Angle Glaucoma2 Preferred Practice Patterns (PPPs). Each document outlines the best practices for diagnosis and management of cataract and POAG, respectively.3 However, concerns have been raised about the quality of clinical practice guidelines (CPGs) in general, and in their trustworthiness in particular.4-6 In response, the Institute of Medicine (IOM) published a set of 8 standards required for a CPG to be considered trustworthy.7 This commentary describes the adherence of PPPs for adult cataract and POAG based on the IOM trustworthiness standards and addresses the role that such critiques hold for future improvement of practice guidelines. The analysis was performed by 4 of the authors (BKY, CMW, AMW, PBG) using the methods prescribed by the IOM standards.7 We required all substandards to be met for a standard to be considered passed. Discrepancies were resolved by discussion among the authors. Table 1 summarizes the results. The adult cataract PPP passed 25% of the IOM standards; the POAG PPP passed 50%. Transparency was not fully established in both PPPs because there was insufficient information to evaluate whether the documents met IOM standards for Conflict of Interest (COI), Development Group Composition, and External Review. Contrary to IOM recommendations, the chair of the PPP panels had COIs, and there was no explanation regarding the potential influence of panel members’ COIs on the guideline development process. It was unclear whether patients or patient advocates were included as developers or external reviewers. Furthermore, because the expertise of panel members was not stated, it was not clear whether or not they comprised a multidisciplinary group. Although the POAG PPP stated that it was externally reviewed by “experts and stakeholders before publication,” it did not specify who these individuals were.1,2 Evidence foundations and clarity of recommendations were problematic in the adult cataract PPP. The PPP noted
“All studies used to form a recommendation for care are graded for strength of evidence individually.” However, aside from the highlighted recommendations and summary benchmarks, no other statements were graded for evidence making it unclear which statements were recommendations. The main text of the adult cataract PPP was devoid of graded study citations, implying statements such as “applying 5% povidone iodine to the conjunctival cul de sac” were not “recommendation[s] for care.” Additionally, the highlighted recommendations and the summary benchmarks used different grading systems. Finally, some of the highlighted recommendations were not clinically useful; for example, “Cataract surgery is a procedure appropriately utilized in the United States.” We suggest the following changes to improve the 2 AAO PPPs: •• Ensure that the chair and no more than half of the CPG development group members have COIs •• Disclose the composition of the CPG development group to ensure that it includes a diverse representation of clinical, scientific, and methodological experts •• Record the inclusion of patients or patient representatives in CPG development •• Ensure that patients, patient organizations, and interested members of the public have an opportunity to review the CPG and describe how their comments were addressed Using the POAG PPP as a model, the adult cataract PPP also should apply an evidence rating system to the recommendations in the main text. We acknowledge limitations to this analysis. First, it may have been prohibitively strict to require all 1
Providence VA Medical Center, Providence, RI Warren Alpert Medical School of Brown University, Providence, RI 3 Rhode Island Hospital, Providence, RI 4 Morsani College of Medicine at the University of South Florida, Tampa, FL 2
Corresponding Author: Paul B. Greenberg, MD, Section of Ophthalmology, Providence VA Medical Center, 830 Chalkstone Ave, Providence, RI 02908. Email: [email protected]
Downloaded from ajm.sagepub.com at The University of Auckland Library on April 21, 2015
189
Young et al Table 1. Overview of Adherence of the American Academy of Ophthalmology Adult Cataract and Primary Open-Angle Glaucoma Best Practice Guidelines With Institute of Medicine Trustworthiness Standards. IOM Standard7
Cataract in the Adult Eye PPP1
Primary Open-Angle Glaucoma PPP2
1. Establishing Transparency
Fail: The methodology by which recommendations were formulated was not specified (eg, voting procedures, how consensus was achieved). Explanation of committee members’ areas of clinical expertise, potential influence of COIs on guideline development, and clear documentation of the external review process were lacking.
2. Management of COI
Fail: No information on how the COIs influenced PPP development. The chair was not free of COIs.
3. GDG Composition
Fail: Uncertain if the members of the PPP Committee or Panel represented a multidisciplinary group. Members’ expertise was not explicitly detailed. No representatives of the affected population, current patients, patient advocates, or patient organizations appeared involved in the development of the PPP.
4. Clinical Practice Guideline– Systematic Review Intersection
Pass: The review team (Hoskins Center Staff) used a systematic method for obtaining evidence and regularly interacted with the PPP Panel when necessary.
5. Establishing evidence foundations for and rating strength of recommendations
Fail: A rating system of level of confidence and strength of recommendations was only used for the highlighted recommendations and the summary benchmarks. Other recommendations within the main text had no rating system.
Pass: All recommendations included a rating of quality and consistency of evidence.
6. Articulation of Recommendation
Fail: Only highlighted recommendations and summary benchmarks had evidence ratings; uncertain which statements in the main text were recommendations. Furthermore, 6 of the 11 highlighted recommendations fail because they cannot be evaluated discretely.
Pass: All recommendations were properly articulated and were written such that they could be acted upon.
7. External Review
Fail: Uncertain if any patient, former patients, or patient advocates provided external review of the PPP document. The PPP did not specify who was in the “council” review group. Unclear if a record of external review comments was kept. The PPP panel did not release a review draft to the public before publication.
8. Updating
Pass: The process for updating the PPP based on discovery of new evidence and the schedules for revision was explicitly detailed online.
Fail: The chair and a majority (6 of 8) of the PPP Committee Members had COIs, and a majority (6 of 8) of the Glaucoma Panel Members had COIs. These COIs were not divested.
Abbreviations: COI, conflict of interest; GDG, Guideline Development Group; IOM, Institute of Medicine; PPP, preferred practice pattern.
substandards to pass for a main standard to be considered compliant. However, each failed main standard in the 2 PPPs still had a substandard failure rate of at least 50%. Second, the judgments were made on a pass/fail basis. However, since a CPG would be considered untrustworthy if any one standard failed, it is unlikely that a more complicated adjudication system would have changed the study results. Third, the IOM standards were not available at the time the AAO published the PPPs. However, our primary purpose was to investigate the quality of current practice guidelines using the IOM instrument with an eye toward improving future versions.
Although the validity and reliability of IOM standards have not been demonstrated, the IOM serves as an independent, nonprofit resource dedicated to improving the US health care system and plays a key role in shaping trust between patients and providers.8 Although its recommendations carry considerable weight and provide a meaningful tool to assess CPGs, other methods exist to critique practice guidelines.9 In a recent review of 130 CPGs, the median percentage of IOM standards passed was 44%.10 We assert that the AAO PPPs for adult cataract and POAG can be improved to meet all the IOM standards of trustworthiness. More
Downloaded from ajm.sagepub.com at The University of Auckland Library on April 21, 2015
190
American Journal of Medical Quality 30(2)
fundamentally, we wish to underscore the role of continuous improvement in the formulation of CPGs: for this process to occur, PPPs must be reviewed rigorously from a variety of perspectives and with an assortment of tools. The IOM standards are not the ultimate arbiter of trustworthiness but they are a reminder that the development of CPGs is an ongoing process. Authors’ Note The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs or the United States government.
Declaration of Conflicting Interests The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Dr Greenberg has received a United States Veterans Health Administration Health Services Research & Development (HSR&D) Veterans Integrated Service Network (VISN) 1 Career Development Award. The funding organization had no role in the design or conduct of this research.
References 1. American Academy of Ophthalmology; One Network. Cataract in the adult eye PPP—2011. http://one.aao.org/ preferred-practice-pattern/cataract-in-adult-eye-ppp–october-2011. Accessed May 22, 2014.
2. American Academy of Ophthalmology. Primary open-angle glaucoma PPP—2010. http://one.aao.org/ preferred-practice-pattern/primary-openangle-glaucomappp–october-2010. Accessed May 22, 2014. 3. NIH Medline Plus. Leading causes of blindness. http:// www.nlm.nih.gov/medlineplus/magazine/issues/summer08/articles/summer08pg14-15.html. Published 2008. Accessed January 2, 2014. 4. Ransohoff DF, Pignone M, Sox HC. How to decide whether a clinical practice guideline is trustworthy. JAMA. 2013;309:139-140. 5. Shaneyfelt TM, Centor RM. Reassessment of clinical practice guidelines: go gently into that good night. JAMA. 2009;301:868-869. 6. Guyatt G, Akl EA, Hirsh J, et al. The vexing problem of guidelines and conflict of interest: a potential solution. Ann Intern Med. 2010;152:738-741. 7. Graham R, Mancher M, Wolman DM, Greenfield S, Steinberg E, eds. Clinical Practice Guidelines We Can Trust. Washington, DC: National Academies Press; 2011. 8. Institute of Medicine. Informing the Future: Critical Issues in Health. 7th ed. http://www.iom.edu/About-IOM/~/ media/Files/About%20the%20IOM/ITF_seventh.pdf. Published 2013. Accessed January 2, 2014. 9. The AGREE Next Steps Consortium. Appraisal of Guidelines for Research and Evaluation II instrument. http:// www.agreetrust.org/wp-content/uploads/2013/03/1397_ AGREE+II+Users+Manual+and+23-item+Instrument+ENGLISH1.pdf. Published May 2009. Accessed January 2, 2014. 10. Kung J, Miller RR, Mackowiak PA. Failure of clinical practice guidelines to meet Institute of Medicine standards: two more decades of little, if any, progress. Arch Intern Med. 2012;172:1628-1633.
Downloaded from ajm.sagepub.com at The University of Auckland Library on April 21, 2015
