EDITORIAL
Improving the Transparency and Trustworthiness of Subspecialty-Based Clinical Practice Guidelines
T
he growth of clinical practice guideline development programs worldwide has resulted in what appears to be an uncontrolled proliferation of recommendations on providing medical care to patients with acute and chronic illnesses. In turn, there has been substantial interest and scrutiny regarding the quality and transparency of the processes used to craft these important documents. Recent discussions and guidances issued by the Institute of Medicine (IOM) on what practices and procedures should be used to make clinical practice guidelines trustworthy underscore this observation. Although most current guidelines focus on common conditions seen in primary and specialty care settings, an increase in clinical practice guidelines developed by subspecialty societies (eg, focused on interventions such as medical procedures) is being recognized. Notably, there has been little discourse on the quality and transparency of these guidelines to date. To address these factors, in the current issue of the Mayo Clinic Proceedings, Feuerstein et al1 report a systematic review of medical subspecialty guidelines focused on procedural interventions to evaluate the evidence quality, recommendations, and potential bias associated with author conflicts of interest (COI) within the development process. The authors specifically examined clinical practice guidelines related to procedural interventions developed by 4 specific professional societies, the American Society for Gastrointestinal Endoscopy, the American Association for Bronchology and Interventional Pulmonology, the American Society of Diagnostic and Interventional Nephrology, and the Society for Cardiovascular Angiography and Interventions. The specific details for analysis included (1) the type of grading system, (2) the level of evidence supporting the recommendations, (3) the age of the guidelines, including planned updates to the current guidelines, and (4) the presence and extent of COI disclosures from the writing groups. When multiple systems for grading the level of evidence were used by societies, the authors merged the grading systems into a standard
ABC grading system for the analysis. (Details of the ABC classification of evidence include: Grade A, conclusions based on randomized controlled trials or meta-analysis; Grade B, conclusions based on a single randomized controlled trial or nonrandomized trials; Grade C, conclusions based on expert opinion, case studies, and standards of care.1). Finally, all guidelines were reviewed to document the presence and extent of COI disclosures. Among the 149 interventional-based guidelines eligible for evaluation, only 69 documents (46%) incorporated a grading or classification system for their recommendations. The proportion of guidelines with grading systems produced by individual societies ranged from 0% to 71%. In total, there were 3425 recommendations with supporting levels of evidence generated by the guidelines: 364 recommendations (11%) were designated with level A (greatest) evidence, 1432 (42%) with level B evidence, and 1629 (48%) with level C (least) evidence. With respect to updated guidelines, the mean age of evaluated guidelines (including any online updates on the society websites) was 5.2 years. In terms of COI, a total of 1827 disclosures were reported, with 45% of all authors involved in writing the guidelines reporting at least 5 or more disclosures. Notably, an estimated 62% of the evaluated guidelines failed to comment on COI disclosures entirely. The proportion of authors disclosing COI within guidelines was quite varied when stratified by professional society (0% to 54%). The strengths of the study by Feuerstein et al include (1) a focus on guidelines related to commonly performed medical procedures, (2) an independent blinded review of data elements by multiple authors, and (3) the use of accepted standards in the guideline development process to guide data collection and analysis. Limitations of the analysis include (1) lack of a standardized grading system used by all guidelines, (2) the exclusion of interventional guidelines not published on society websites or parent professional societies, and (3) limited detail on potential COI from
Mayo Clin Proc. n January 2014;89(1):5-7 n http://dx.doi.org/10.1016/j.mayocp.2013.11.008 www.mayoclinicproceedings.org n ª 2014 Mayo Foundation for Medical Education and Research
See also page 16
5
MAYO CLINIC PROCEEDINGS
authors that might be irrelevant for the actual guidelines in question. Through the Medicare Improvements for Patients and Providers Act of 2008, the IOM developed 8 standards for developing rigorous, trustworthy clinical practice guidelines.2 These standards center around (1) establishing transparency for guideline development processes, (2) actively managing potential COI, (3) striving toward multidisciplinary and balanced guideline development group composition, (4) using systematic reviews adhering to IOM standards to inform guidelines, (5) establishing evidence foundations for rating strength of recommendations, (6) standardizing approaches for explicit articulation of recommendations, (7) performing external guideline review by the full spectrum of affected stakeholders, and (8) focusing on updating guidelines in a timely manner. Several of these IOM criteria warrant further elaboration. After publication of the IOM criteria, several studies have examined the extent to which existing guidelines meet these standards. Tricoci et al3 described the evolution of recommendations in American College of Cardiology/American Heart Association cardiovascular guidelines and the distribution across classes of recommendations and levels of evidence. Among guidelines with at least one revision or update, the number of recommendations increased from 1330 to 1973 (48%) from the first to the current version. The median percentage of recommendations with level A of evidence was 11%. while 48% of recommendations were supported by level C evidence. Although the accessed literature was obtained by specialists not directly involved with the writing of the guidelines, the processes used for assessing the level of evidence were not explicit. Similar findings have been reported from other specialty societies.4 These findings, along with those of Feuerstein et al,1 confirm the observations by the IOM concerning the need for developing standards. The historical lack of attention to utilizing systematic reviews to identify the salient literature in formulating high-quality clinical practice guidelines is also noted in the current and previously published studies.1,3,4 Remarkably, only a small proportion of trials in the published literature have been synthesized into systematic reviews to date.5 However, the growing number of systematic reviews in the literature is being recognized by guideline development panels 6
Mayo Clin Proc.
n
that include members with methodological expertise in evidence-based medicine. In turn, the inclusion or creation of systematic reviews to inform guidelines is increasing but far from universal. The increasing adoption of the GRADE (Grading of Recommendations Assessment, Development and Evaluation) classification system6 by development groups worldwide is expected to further galvanize the incorporation of high-quality systematic reviews and meta-analyses into clinical practice guidelines.5,6 Even with greater adherence to methods that improve the identification and grading of evidence to support explicit recommendations, the lack of attention to individual and organizational COI may be the greatest threat to creating trustworthy clinical practice guidelines.6,7 As observed in the current study,1 there is a growing body of literature documenting the existence of one or more potential COI reported for individual authors and members within guideline development panels.7 As a result, the influence of external activities such as consulting or speaker’s fees, research grant funding, and stock ownership from a commercial entity have the potential to create considerable bias and uncertainty for issued recommendations.8 This is especially important given that more than 40% of recommendations from most guidelines are supported by level C (or low-quality) evidence driven by expert opinion or writing group consensus. Efforts to minimize the bias from COI such as balancing guideline writing group panels with members having no COI (including the writing group chair) make sense,2 yet the dichotomy in seeking “experts” who participate in guidelines being the same individuals who are typically sought for industry-based collaborations will continue to be debated. Ultimately, the full spectrum of stakeholders vested in the development of clinical practice guidelines seek the same ideal: an unbiased and transparent assessment of the literature and subsequent recommendations to guide safe and effective medical practice. Notably, the perspective taken by guideline developers has traditionally focused on health care professionals and payers rather than the patients and their preferences for guidelinebased care. Patient preferences refer to beliefs, expectations, and goals for health that individuals use in considering the potential benefits, harms, and inconveniences related to
January 2014;89(1):5-7
n
http://dx.doi.org/10.1016/j.mayocp.2013.11.008 www.mayoclinicproceedings.org
EDITORIAL
management options for their conditions. In turn, there will be informed patients (and physicians) who may choose not to follow a guideline that does not incorporate their preferences,9 reflecting the incomplete nature of guidelines per se. Increased uptake and full incorporation of the GRADE system, for example, will improve the frequency with which developers seek information on patient preferences as well as resource utilization. Clinical practice guidelines are clearly a part of the fabric defined by evidence-based medicine and the pursuit to achieve safe, efficient, and patient-centered delivery of medical care. The study by Feuerstein et al further highlights that existing guidelines are highly variable with respect to high quality and transparency, but these problems can be overcome with further education and adherence to emerging standards in the field. Jayant A. Talwalkar, MD, MPH Division of Gastroenterology and Hepatology Center for the Science of Health Care Delivery Mayo Clinic Rochester, MN Correspondence: Address to Jayant A. Talwalkar, MD, MPH, Center for the Science of Health Care Delivery, Mayo Clinic, 200 First St SW, Rochester, MN 55905 ([email protected]).
Mayo Clin Proc. n January 2014;89(1):5-7 www.mayoclinicproceedings.org
n
REFERENCES 1. Feuerstein JD, Akbari M, Gifford AE, et al. Systematic analysis underlying the quality of the scientific evidence and conflicts of interest in interventional medicine subspecialty guidelines. Mayo Clin Proc. 2014;89(1):16-24. 2. Institute of Medicine. Practice Guidelines We Can Trust. Washington, DC: National Academies Press; 2011. http://iom.edu/ Reports/2011/Clinical-Practice-Guidelines-We-Can-Trust.aspx. Accessed November 2, 2013. 3. Tricoci P, Allen JM, Kramer JM, Califf RM, Smith SC Jr. Scientific evidence underlying the ACC/AHA clinical practice guidelines [published correction appears in JAMA. 2009;301(15):1544]. JAMA. 2009;301(8):831-841. 4. Koh C, Zhao X, Samala N, Sakiani S, Liang TJ, Talwalkar JA. AASLD clinical practice guidelines: a critical review of scientific evidence and evolving recommendations [published online ahead of print June 14, 2013]. Hepatology. http://dx.doi.org/10. 1002/hep.26578. 5. Murad MH, Montori VM. Synthesizing evidence: shifting the focus from individual studies to the body of evidence. JAMA. 2013;309(21):2217-2218. 6. Guyatt GH, Oxman AD, Vist GE, et al; GRADE Working Group. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650): 924-926. 7. Cosgrove L, Bursztajn HJ, Erlich DR, Wheeler EE, Shaughnessy AF. Conflicts of interest and the quality of recommendations in clinical guidelines. J Eval Clin Pract. 2013;19(4): 674-681. 8. Lenzer J, Hoffman JR, Furberg CD, Ioannidis JP; Guideline Panel Review Working Group. Ensuring the integrity of clinical practice guidelines: a tool for protecting patients. BMJ. 2013;347: f5535. 9. Montori VM, Brito JP, Murad MH. The optimal practice of evidence-based medicine: incorporating patient preferences in practice guidelines [published online ahead of print October 28, 2013]. JAMA. http://dx.doi.org/10.1001/jama. 2013.281422.
http://dx.doi.org/10.1016/j.mayocp.2013.11.008
7
Improving the Transparency and Trustworthiness of Subspecialty-Based Clinical Practice Guidelines
T
he growth of clinical practice guideline development programs worldwide has resulted in what appears to be an uncontrolled proliferation of recommendations on providing medical care to patients with acute and chronic illnesses. In turn, there has been substantial interest and scrutiny regarding the quality and transparency of the processes used to craft these important documents. Recent discussions and guidances issued by the Institute of Medicine (IOM) on what practices and procedures should be used to make clinical practice guidelines trustworthy underscore this observation. Although most current guidelines focus on common conditions seen in primary and specialty care settings, an increase in clinical practice guidelines developed by subspecialty societies (eg, focused on interventions such as medical procedures) is being recognized. Notably, there has been little discourse on the quality and transparency of these guidelines to date. To address these factors, in the current issue of the Mayo Clinic Proceedings, Feuerstein et al1 report a systematic review of medical subspecialty guidelines focused on procedural interventions to evaluate the evidence quality, recommendations, and potential bias associated with author conflicts of interest (COI) within the development process. The authors specifically examined clinical practice guidelines related to procedural interventions developed by 4 specific professional societies, the American Society for Gastrointestinal Endoscopy, the American Association for Bronchology and Interventional Pulmonology, the American Society of Diagnostic and Interventional Nephrology, and the Society for Cardiovascular Angiography and Interventions. The specific details for analysis included (1) the type of grading system, (2) the level of evidence supporting the recommendations, (3) the age of the guidelines, including planned updates to the current guidelines, and (4) the presence and extent of COI disclosures from the writing groups. When multiple systems for grading the level of evidence were used by societies, the authors merged the grading systems into a standard
ABC grading system for the analysis. (Details of the ABC classification of evidence include: Grade A, conclusions based on randomized controlled trials or meta-analysis; Grade B, conclusions based on a single randomized controlled trial or nonrandomized trials; Grade C, conclusions based on expert opinion, case studies, and standards of care.1). Finally, all guidelines were reviewed to document the presence and extent of COI disclosures. Among the 149 interventional-based guidelines eligible for evaluation, only 69 documents (46%) incorporated a grading or classification system for their recommendations. The proportion of guidelines with grading systems produced by individual societies ranged from 0% to 71%. In total, there were 3425 recommendations with supporting levels of evidence generated by the guidelines: 364 recommendations (11%) were designated with level A (greatest) evidence, 1432 (42%) with level B evidence, and 1629 (48%) with level C (least) evidence. With respect to updated guidelines, the mean age of evaluated guidelines (including any online updates on the society websites) was 5.2 years. In terms of COI, a total of 1827 disclosures were reported, with 45% of all authors involved in writing the guidelines reporting at least 5 or more disclosures. Notably, an estimated 62% of the evaluated guidelines failed to comment on COI disclosures entirely. The proportion of authors disclosing COI within guidelines was quite varied when stratified by professional society (0% to 54%). The strengths of the study by Feuerstein et al include (1) a focus on guidelines related to commonly performed medical procedures, (2) an independent blinded review of data elements by multiple authors, and (3) the use of accepted standards in the guideline development process to guide data collection and analysis. Limitations of the analysis include (1) lack of a standardized grading system used by all guidelines, (2) the exclusion of interventional guidelines not published on society websites or parent professional societies, and (3) limited detail on potential COI from
Mayo Clin Proc. n January 2014;89(1):5-7 n http://dx.doi.org/10.1016/j.mayocp.2013.11.008 www.mayoclinicproceedings.org n ª 2014 Mayo Foundation for Medical Education and Research
See also page 16
5
MAYO CLINIC PROCEEDINGS
authors that might be irrelevant for the actual guidelines in question. Through the Medicare Improvements for Patients and Providers Act of 2008, the IOM developed 8 standards for developing rigorous, trustworthy clinical practice guidelines.2 These standards center around (1) establishing transparency for guideline development processes, (2) actively managing potential COI, (3) striving toward multidisciplinary and balanced guideline development group composition, (4) using systematic reviews adhering to IOM standards to inform guidelines, (5) establishing evidence foundations for rating strength of recommendations, (6) standardizing approaches for explicit articulation of recommendations, (7) performing external guideline review by the full spectrum of affected stakeholders, and (8) focusing on updating guidelines in a timely manner. Several of these IOM criteria warrant further elaboration. After publication of the IOM criteria, several studies have examined the extent to which existing guidelines meet these standards. Tricoci et al3 described the evolution of recommendations in American College of Cardiology/American Heart Association cardiovascular guidelines and the distribution across classes of recommendations and levels of evidence. Among guidelines with at least one revision or update, the number of recommendations increased from 1330 to 1973 (48%) from the first to the current version. The median percentage of recommendations with level A of evidence was 11%. while 48% of recommendations were supported by level C evidence. Although the accessed literature was obtained by specialists not directly involved with the writing of the guidelines, the processes used for assessing the level of evidence were not explicit. Similar findings have been reported from other specialty societies.4 These findings, along with those of Feuerstein et al,1 confirm the observations by the IOM concerning the need for developing standards. The historical lack of attention to utilizing systematic reviews to identify the salient literature in formulating high-quality clinical practice guidelines is also noted in the current and previously published studies.1,3,4 Remarkably, only a small proportion of trials in the published literature have been synthesized into systematic reviews to date.5 However, the growing number of systematic reviews in the literature is being recognized by guideline development panels 6
Mayo Clin Proc.
n
that include members with methodological expertise in evidence-based medicine. In turn, the inclusion or creation of systematic reviews to inform guidelines is increasing but far from universal. The increasing adoption of the GRADE (Grading of Recommendations Assessment, Development and Evaluation) classification system6 by development groups worldwide is expected to further galvanize the incorporation of high-quality systematic reviews and meta-analyses into clinical practice guidelines.5,6 Even with greater adherence to methods that improve the identification and grading of evidence to support explicit recommendations, the lack of attention to individual and organizational COI may be the greatest threat to creating trustworthy clinical practice guidelines.6,7 As observed in the current study,1 there is a growing body of literature documenting the existence of one or more potential COI reported for individual authors and members within guideline development panels.7 As a result, the influence of external activities such as consulting or speaker’s fees, research grant funding, and stock ownership from a commercial entity have the potential to create considerable bias and uncertainty for issued recommendations.8 This is especially important given that more than 40% of recommendations from most guidelines are supported by level C (or low-quality) evidence driven by expert opinion or writing group consensus. Efforts to minimize the bias from COI such as balancing guideline writing group panels with members having no COI (including the writing group chair) make sense,2 yet the dichotomy in seeking “experts” who participate in guidelines being the same individuals who are typically sought for industry-based collaborations will continue to be debated. Ultimately, the full spectrum of stakeholders vested in the development of clinical practice guidelines seek the same ideal: an unbiased and transparent assessment of the literature and subsequent recommendations to guide safe and effective medical practice. Notably, the perspective taken by guideline developers has traditionally focused on health care professionals and payers rather than the patients and their preferences for guidelinebased care. Patient preferences refer to beliefs, expectations, and goals for health that individuals use in considering the potential benefits, harms, and inconveniences related to
January 2014;89(1):5-7
n
http://dx.doi.org/10.1016/j.mayocp.2013.11.008 www.mayoclinicproceedings.org
EDITORIAL
management options for their conditions. In turn, there will be informed patients (and physicians) who may choose not to follow a guideline that does not incorporate their preferences,9 reflecting the incomplete nature of guidelines per se. Increased uptake and full incorporation of the GRADE system, for example, will improve the frequency with which developers seek information on patient preferences as well as resource utilization. Clinical practice guidelines are clearly a part of the fabric defined by evidence-based medicine and the pursuit to achieve safe, efficient, and patient-centered delivery of medical care. The study by Feuerstein et al further highlights that existing guidelines are highly variable with respect to high quality and transparency, but these problems can be overcome with further education and adherence to emerging standards in the field. Jayant A. Talwalkar, MD, MPH Division of Gastroenterology and Hepatology Center for the Science of Health Care Delivery Mayo Clinic Rochester, MN Correspondence: Address to Jayant A. Talwalkar, MD, MPH, Center for the Science of Health Care Delivery, Mayo Clinic, 200 First St SW, Rochester, MN 55905 ([email protected]).
Mayo Clin Proc. n January 2014;89(1):5-7 www.mayoclinicproceedings.org
n
REFERENCES 1. Feuerstein JD, Akbari M, Gifford AE, et al. Systematic analysis underlying the quality of the scientific evidence and conflicts of interest in interventional medicine subspecialty guidelines. Mayo Clin Proc. 2014;89(1):16-24. 2. Institute of Medicine. Practice Guidelines We Can Trust. Washington, DC: National Academies Press; 2011. http://iom.edu/ Reports/2011/Clinical-Practice-Guidelines-We-Can-Trust.aspx. Accessed November 2, 2013. 3. Tricoci P, Allen JM, Kramer JM, Califf RM, Smith SC Jr. Scientific evidence underlying the ACC/AHA clinical practice guidelines [published correction appears in JAMA. 2009;301(15):1544]. JAMA. 2009;301(8):831-841. 4. Koh C, Zhao X, Samala N, Sakiani S, Liang TJ, Talwalkar JA. AASLD clinical practice guidelines: a critical review of scientific evidence and evolving recommendations [published online ahead of print June 14, 2013]. Hepatology. http://dx.doi.org/10. 1002/hep.26578. 5. Murad MH, Montori VM. Synthesizing evidence: shifting the focus from individual studies to the body of evidence. JAMA. 2013;309(21):2217-2218. 6. Guyatt GH, Oxman AD, Vist GE, et al; GRADE Working Group. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ. 2008;336(7650): 924-926. 7. Cosgrove L, Bursztajn HJ, Erlich DR, Wheeler EE, Shaughnessy AF. Conflicts of interest and the quality of recommendations in clinical guidelines. J Eval Clin Pract. 2013;19(4): 674-681. 8. Lenzer J, Hoffman JR, Furberg CD, Ioannidis JP; Guideline Panel Review Working Group. Ensuring the integrity of clinical practice guidelines: a tool for protecting patients. BMJ. 2013;347: f5535. 9. Montori VM, Brito JP, Murad MH. The optimal practice of evidence-based medicine: incorporating patient preferences in practice guidelines [published online ahead of print October 28, 2013]. JAMA. http://dx.doi.org/10.1001/jama. 2013.281422.
http://dx.doi.org/10.1016/j.mayocp.2013.11.008
7
