Letters to the Editor / Journal of Clinical Epidemiology 67 (2014) 236e240
Faculty of Medicine University of Cologne, Cologne Germany
Mandy Kromp Corinna Kiefer Sibylle Sturtz Department of Medical Biometry Institute for Quality and Efficiency in Health Care Im Mediapark 8, De50670 Cologne Germany * Corresponding author. Tel.: þ49-221-35685-451; fax: þ49-221-35685-10. E-mail address: [email protected]
References [1] Girerd N, Rabilloud M, Duarte K, Roy P. Number needed to treat from absolute risk and incidence rate: How to make apples and oranges comparable? J Clin Epidemiol 2014;67:236e8. [in this issue]. [2] Bender R, Kromp M, Kiefer C, Sturtz S. Absolute risks rather than incidence rates should be used to estimate the number needed to treat from time-to-event data. J Clin Epidemiol 2013;66:1038e44. [3] Stang A, Poole C, Bender R. Common problems related to the use of number needed to treat. J Clin Epidemiol 2010;63:820e5. [4] Suissa D, Brassard P, Smiechowski B, Suissa S. Number needed to treat is incorrect without proper time-related considerations. J Clin Epidemiol 2012;65:42e6. [5] Lubsen J, Hoes A, Grobbee D. Implications of trial results: the potentially misleading notations of number needed to treat and average duration life gained. Lancet 2000;356:1757e9. http://dx.doi.org/10.1016/j.jclinepi.2013.08.009
The GRADE recommendations. Weak recommendations are unhelpful to decision makers To the Editor: The 14th and 15th publications of the ‘‘GRADE Series’’ have now appeared [1,2], and because of the sheer bulk of this series and the prestige of the authors, these publications will have a significant influence in the field of evaluation, irrespective of their intrinsic value. However, the principal assumption on which the latter two publications are based, namely that in the absence of convincing evidence to support a recommendation, a weak recommendation should be supplied, should be questioned. Developers of Clinical Guidelines or Health Technology Assessments (referred to by the GRADE authors as ‘‘panels’’) develop recommendations to assist decision makers (health administrators or individual patients and their physicians) to decide whether to undertake a particular action. Conflict of interest: None.
239
To arrive at their recommendations, panels estimate the changes in relevant outcomes to be expected from the intervention and compare them to one or more alternative intervention. Recommendations are then based on the size of the estimated gains (health benefits and health resources conserved) compared with the losses (unwanted side effects and increased resource use) of the intervention under consideration. When the evidence for each outcome is of adequate quality and quantity and when the expected gains clearly exceed the expected losses, panels recommend that the action in question be taken, and when they are not, they do not. However, the evidence on outcomes is frequently inadequate in quality or quantity or the estimated benefits and losses may be too evenly balanced. When this occurs, the GRADE authors recommend that a weak recommendation should be made because not to do so would ‘‘leave those looking for answers from guidelines without the guidance they are seeking.’’ To supply advice in the absence of good supporting evidence, simply because someone wants it, is questionable, and a weak recommendation is unlikely to be of much help to decision makers. However, having decided to make recommendations in spite of the lack of evidence, the GRADE authors give advice on how they should be nuanced. They advise that ‘‘recommendations in the passive voice may lack clarity’’ and that organizations use ‘‘we suggest’’ rather than ‘‘we recommend’’ for weak recommendations. Furthermore, symbols may be less confusing than numbers or letters, and the words ‘‘strong’’ and ‘‘weak’’ should be replaced by ‘‘[[’’ and ‘‘[?’’ respectively! There is no evidence to support such advice, which many would find questionable. By contrast with this GRADE advice, I suggest that a weak recommendation, however nuanced, is unlikely to be helpful to those who have to make decisions. Rather, when the evidence is inadequate, or the balance between gains and losses too close to support a recommendation, I believe it would be preferable for panels to simply report this fact and make their review of the evidence and their reasoning available to decision makers. This would be both simpler and possibly more helpful than supplying a weak recommendation. Maurice McGregor Royal Victoria Hospital 687 Pine Avenue West Room H7-08, Montreal, Quebec Canada H3A1A1 * Corresponding author. Tel.: þ1-514-934-1934. E-mail address: [email protected]
References [1] Andrews JC, Guyatt G, Oxman A, Alderson P, Dahm P, Falk-Ytter Y, et al. GRADE guidelines: 14. Going from evidence to
240
Letters to the Editor / Journal of Clinical Epidemiology 67 (2014) 236e240
recommendations: the significance and presentation of recommendations. J Clin Epidemiol 2013;66:719e25. [2] Andrews JC, Schunemann HJ, Oxman A, Pottie K, Meerpohl JJ, Coello PA, et al. Grade guidelines:15. Going from evidence to recommendationddeterminants of a recommendation’s direction and strength. J Clin Epidemiol 2013;66:726e33. http://dx.doi.org/10.1016/j.jclinepi.2013.09.017
Letter reply to GRADE guidelines articles 14 and 15
remains in some question. Nevertheless, reasons to consider a structured approach to formulating weak recommendations are (1) for physicians who would appreciate guidance, providing that guidance; (2) specifying that a recommendation is weak will reduce the likelihood that it will be inappropriately considered as a measure of performance or quality of care; (3) signaling clinicians that patient values and preferences are likely to play a large role in these instances; and (4) helping in the justification for the research community of further research.
In reply: Dr. McGregor states that there is no evidence to support GRADE formatting recommendations. A randomized trial compared different ways of formulating and presenting recommendations (including weak recommendations and symbols) that suggested clinicians had better understanding of recommendations using GRADE than other systems, including National Institute of Clinical Evidence, the Centre for Evidence Based Medicine, and the Scottish Intercollegiate Guideline Network [1]. Furthermore, published evidence suggests that adding symbols or letters enhances understanding of recommendations [2]. A large, international, GRADE-associated project funded by the European Union, DECIDE [3], has the express purpose to determine the optimal presentation of recommendations and the associated underlying evidence. This project will greatly increase the currently limited evidence regarding optimal presentation. The project will include exploration of recommendations based on low confidence (low quality of evidence) in the area of rare diseases (http://www.rarebestpractices.eu/). Thus, in the not too distant future, we will have a substantial evidence base supporting, or refuting, current GRADE formatting suggestions. Dr. McGregor contends that clinicians do not find weak recommendations helpful. Available evidence, including the study by Cuello Garcia et al. [1], suggests that this is not case. Additional evidence includes the popularity of guidelines that provide recommendations very often based on low-quality evidence. These include the very widely used electronic textbook UpToDate in which marketing research has suggested one reason for success is that recommendations are offered even when there is low-quality evidence (and these recommendations, following GRADE, are usually labeled as ‘‘weak’’). Further evidence addressing clinicians’ perceived value of weak recommendations would be desirabledindeed, we are planning a study addressing this issue. Until such evidence is available, the utility of weak recommendations
Conflict of interest: G.G. is a paid consultant of UpToDate and D.R. is an UpToDate employee.
Gordon Guyatt* Victor Montori Holger Schunemann Pablo Alonso-Coello Philipp Dahm Juan Pablo Brito Campana Jan Brozek Mona Nasser Joerg Meerpohl David Rind Roman Jaeschke Ynge Fack-Ytter Susan Norris * Corresponding author. Clinical Epidemiology and Biostatistics Faculty of Health Sciences, McMaster University 1280 Main Street West Room 2C12, Hamilton Ontario L8S 4K1, Canada Tel.: þ1-905-525-9140; fax: þ1-905-524-3841 E-mail address: [email protected]
References [1] Cuello Garcıa C, Pacheco Alvarado K, Perez Gaxiola G. Grading recommendations in clinical practice guidelines: randomised experimental evaluation of four different systems. Arch Dis Child 2011;96:723e8. [2] Akl EA, Maroun N, Guyatt GH, Oxman AD, Alonso-Coello P, Vist GE, et al. Symbols were superior to numbers for presenting strength of recommendations to health care consumers: a randomized trial. J Clin Epidemiol 2007;60:1298e305. [3] Treweek S, Oxman AD, Alderson P, Bossuyt PM, Brandt L, Bro_zek J, et al, DECIDE Consortium. Developing and evaluating communication strategies to support informed decisions and practice based on evidence (DECIDE): protocol and preliminary results. Implement Sci 2013;8:6. http://dx.doi.org/10.1016/j.jclinepi.2013.09.016
Faculty of Medicine University of Cologne, Cologne Germany
Mandy Kromp Corinna Kiefer Sibylle Sturtz Department of Medical Biometry Institute for Quality and Efficiency in Health Care Im Mediapark 8, De50670 Cologne Germany * Corresponding author. Tel.: þ49-221-35685-451; fax: þ49-221-35685-10. E-mail address: [email protected]
References [1] Girerd N, Rabilloud M, Duarte K, Roy P. Number needed to treat from absolute risk and incidence rate: How to make apples and oranges comparable? J Clin Epidemiol 2014;67:236e8. [in this issue]. [2] Bender R, Kromp M, Kiefer C, Sturtz S. Absolute risks rather than incidence rates should be used to estimate the number needed to treat from time-to-event data. J Clin Epidemiol 2013;66:1038e44. [3] Stang A, Poole C, Bender R. Common problems related to the use of number needed to treat. J Clin Epidemiol 2010;63:820e5. [4] Suissa D, Brassard P, Smiechowski B, Suissa S. Number needed to treat is incorrect without proper time-related considerations. J Clin Epidemiol 2012;65:42e6. [5] Lubsen J, Hoes A, Grobbee D. Implications of trial results: the potentially misleading notations of number needed to treat and average duration life gained. Lancet 2000;356:1757e9. http://dx.doi.org/10.1016/j.jclinepi.2013.08.009
The GRADE recommendations. Weak recommendations are unhelpful to decision makers To the Editor: The 14th and 15th publications of the ‘‘GRADE Series’’ have now appeared [1,2], and because of the sheer bulk of this series and the prestige of the authors, these publications will have a significant influence in the field of evaluation, irrespective of their intrinsic value. However, the principal assumption on which the latter two publications are based, namely that in the absence of convincing evidence to support a recommendation, a weak recommendation should be supplied, should be questioned. Developers of Clinical Guidelines or Health Technology Assessments (referred to by the GRADE authors as ‘‘panels’’) develop recommendations to assist decision makers (health administrators or individual patients and their physicians) to decide whether to undertake a particular action. Conflict of interest: None.
239
To arrive at their recommendations, panels estimate the changes in relevant outcomes to be expected from the intervention and compare them to one or more alternative intervention. Recommendations are then based on the size of the estimated gains (health benefits and health resources conserved) compared with the losses (unwanted side effects and increased resource use) of the intervention under consideration. When the evidence for each outcome is of adequate quality and quantity and when the expected gains clearly exceed the expected losses, panels recommend that the action in question be taken, and when they are not, they do not. However, the evidence on outcomes is frequently inadequate in quality or quantity or the estimated benefits and losses may be too evenly balanced. When this occurs, the GRADE authors recommend that a weak recommendation should be made because not to do so would ‘‘leave those looking for answers from guidelines without the guidance they are seeking.’’ To supply advice in the absence of good supporting evidence, simply because someone wants it, is questionable, and a weak recommendation is unlikely to be of much help to decision makers. However, having decided to make recommendations in spite of the lack of evidence, the GRADE authors give advice on how they should be nuanced. They advise that ‘‘recommendations in the passive voice may lack clarity’’ and that organizations use ‘‘we suggest’’ rather than ‘‘we recommend’’ for weak recommendations. Furthermore, symbols may be less confusing than numbers or letters, and the words ‘‘strong’’ and ‘‘weak’’ should be replaced by ‘‘[[’’ and ‘‘[?’’ respectively! There is no evidence to support such advice, which many would find questionable. By contrast with this GRADE advice, I suggest that a weak recommendation, however nuanced, is unlikely to be helpful to those who have to make decisions. Rather, when the evidence is inadequate, or the balance between gains and losses too close to support a recommendation, I believe it would be preferable for panels to simply report this fact and make their review of the evidence and their reasoning available to decision makers. This would be both simpler and possibly more helpful than supplying a weak recommendation. Maurice McGregor Royal Victoria Hospital 687 Pine Avenue West Room H7-08, Montreal, Quebec Canada H3A1A1 * Corresponding author. Tel.: þ1-514-934-1934. E-mail address: [email protected]
References [1] Andrews JC, Guyatt G, Oxman A, Alderson P, Dahm P, Falk-Ytter Y, et al. GRADE guidelines: 14. Going from evidence to
240
Letters to the Editor / Journal of Clinical Epidemiology 67 (2014) 236e240
recommendations: the significance and presentation of recommendations. J Clin Epidemiol 2013;66:719e25. [2] Andrews JC, Schunemann HJ, Oxman A, Pottie K, Meerpohl JJ, Coello PA, et al. Grade guidelines:15. Going from evidence to recommendationddeterminants of a recommendation’s direction and strength. J Clin Epidemiol 2013;66:726e33. http://dx.doi.org/10.1016/j.jclinepi.2013.09.017
Letter reply to GRADE guidelines articles 14 and 15
remains in some question. Nevertheless, reasons to consider a structured approach to formulating weak recommendations are (1) for physicians who would appreciate guidance, providing that guidance; (2) specifying that a recommendation is weak will reduce the likelihood that it will be inappropriately considered as a measure of performance or quality of care; (3) signaling clinicians that patient values and preferences are likely to play a large role in these instances; and (4) helping in the justification for the research community of further research.
In reply: Dr. McGregor states that there is no evidence to support GRADE formatting recommendations. A randomized trial compared different ways of formulating and presenting recommendations (including weak recommendations and symbols) that suggested clinicians had better understanding of recommendations using GRADE than other systems, including National Institute of Clinical Evidence, the Centre for Evidence Based Medicine, and the Scottish Intercollegiate Guideline Network [1]. Furthermore, published evidence suggests that adding symbols or letters enhances understanding of recommendations [2]. A large, international, GRADE-associated project funded by the European Union, DECIDE [3], has the express purpose to determine the optimal presentation of recommendations and the associated underlying evidence. This project will greatly increase the currently limited evidence regarding optimal presentation. The project will include exploration of recommendations based on low confidence (low quality of evidence) in the area of rare diseases (http://www.rarebestpractices.eu/). Thus, in the not too distant future, we will have a substantial evidence base supporting, or refuting, current GRADE formatting suggestions. Dr. McGregor contends that clinicians do not find weak recommendations helpful. Available evidence, including the study by Cuello Garcia et al. [1], suggests that this is not case. Additional evidence includes the popularity of guidelines that provide recommendations very often based on low-quality evidence. These include the very widely used electronic textbook UpToDate in which marketing research has suggested one reason for success is that recommendations are offered even when there is low-quality evidence (and these recommendations, following GRADE, are usually labeled as ‘‘weak’’). Further evidence addressing clinicians’ perceived value of weak recommendations would be desirabledindeed, we are planning a study addressing this issue. Until such evidence is available, the utility of weak recommendations
Conflict of interest: G.G. is a paid consultant of UpToDate and D.R. is an UpToDate employee.
Gordon Guyatt* Victor Montori Holger Schunemann Pablo Alonso-Coello Philipp Dahm Juan Pablo Brito Campana Jan Brozek Mona Nasser Joerg Meerpohl David Rind Roman Jaeschke Ynge Fack-Ytter Susan Norris * Corresponding author. Clinical Epidemiology and Biostatistics Faculty of Health Sciences, McMaster University 1280 Main Street West Room 2C12, Hamilton Ontario L8S 4K1, Canada Tel.: þ1-905-525-9140; fax: þ1-905-524-3841 E-mail address: [email protected]
References [1] Cuello Garcıa C, Pacheco Alvarado K, Perez Gaxiola G. Grading recommendations in clinical practice guidelines: randomised experimental evaluation of four different systems. Arch Dis Child 2011;96:723e8. [2] Akl EA, Maroun N, Guyatt GH, Oxman AD, Alonso-Coello P, Vist GE, et al. Symbols were superior to numbers for presenting strength of recommendations to health care consumers: a randomized trial. J Clin Epidemiol 2007;60:1298e305. [3] Treweek S, Oxman AD, Alderson P, Bossuyt PM, Brandt L, Bro_zek J, et al, DECIDE Consortium. Developing and evaluating communication strategies to support informed decisions and practice based on evidence (DECIDE): protocol and preliminary results. Implement Sci 2013;8:6. http://dx.doi.org/10.1016/j.jclinepi.2013.09.016
![[GRADE guidelines: 14. Going from evidence to recommendations: the significance and presentation of recommendations].](/assets/img/hold.png)
