Review
513
Methodological quality of guidelines in gastroenterology
Authors
Rui Malheiro1, Matilde de Monteiro-Soares1, Cesare Hassan2, Mário Dinis-Ribeiro1
Institutions
1
Bibliography DOI http://dx.doi.org/ 10.1055/s-0034-1365394 Published online: 23.4.2014 Endoscopy 2014; 46: 513–525 © Georg Thieme Verlag KG Stuttgart · New York ISSN 0013-726X Corresponding author Mário Dinis-Ribeiro, MD CINTESIS/CIDES – Health Information and Decision Sciences Department Oporto Faculty of Medicine Rua Dr. Plácido da Costa, s/n 4200-450 Oporto Portugal Fax: +351-22-5513623 [email protected]
CINTESIS/CIDES – Health Information and Decision Sciences Department, Oporto Faculty of Medicine, Oporto, Portugal Gastroenterology and Digestive Endoscopy, Nuovo Regina Margherita Hospital, Rome, Italy
Background and study aims: Clinical guidelines are a common feature in modern endoscopy practice and they are being produced faster than ever. However, their methodological quality is rarely assessed. This study evaluated the methodological quality of current clinical guidelines in the field of gastroenterology, with an emphasis on endoscopy. Materials and methods: Practice guidelines published by the American College of Gastroenterology (ACG), American Gastroenterological Association (AGA), American Society for Gastrointestinal Endoscopy (ASGE), European Society of Gastrointestinal Endoscopy (ESGE), British Society of Gastroenterology (BSG), National Institute for Health and Care Excellence (NICE), and the Scottish Intercollegiate Guidelines Network (SIGN) were searched between September and October 2012 and evaluated using the AGREE II (Appraisal of Guide-
lines for Research and Evaluation) instrument (23 items, scores 1 – 7 for each item; higher scores mean better quality). Results: A total of 100 guidelines were assessed. The mean number of items scoring 6 or 7 per guideline was 9.2 (out of 23 items). Overall, 99 % of guidelines failed to include the target population in the development process, and 96 % did not report facilitators and barriers to guideline application. In addition, 86 % did not include advice or tools, and 94 % did not present monitoring or auditing criteria. Conclusion: The global methodological quality of clinical guidelines in the field of gastroenterology is poor, particularly regarding involvement of the target population in the development of guidelines and in the provision of clear suggestions to practitioners.
Introduction
The use of clinical guidelines is critical to maximize the benefit:risk ratio when implementing endoscopic technologies and new treatments in gastroenterology, and thus, guidelines help to protect patients from an under- or overuse of such innovations. The primary aim of this study was to assess the methodological quality of current clinical guidelines within the gastroenterology area, with particular reference to endoscopy, using the AGREE II instrument. The gaps in developmental methodology that were identified were used to recommend areas for further improvement.
!
Scan this QR-Code to watch the video comment.
Guidelines are a common feature of modern clinical practice [1]. As their number grows internationally, recognized standards to improve the development and reporting of clinical guidelines have been established [2]. Although guideline development is monitored by some of the best experts in the respective fields and they are published by some of the most respected international organizations, their scientific validity, use, and reliability are rarely assessed [1]. This is critical when considering that guidelines are mainly intended to be clinical tools for the decision-making process, and as such are characterized by benefit and risks. In order to face such reality, the Appraisal of Guidelines for Research and Evaluation (AGREE) instrument, which was published in 2003 and updated in 2010, was created as an international instrument to assess the quality of the process and reporting of clinical practice guideline development [1].
Materials and methods !
Selection of guidelines In order to retrieve the most recent guidelines on gastroenterology, and in particular endoscopy, the websites of the following large European and US societies were searched between September
Malheiro Rui et al. Methodological quality of guidelines in gastroenterology … Endoscopy 2014; 46: 513–525
This document was downloaded for personal use only. Unauthorized distribution is strictly prohibited.
2
Review
and October 2012: American College of Gastroenterology (ACG), American Gastroenterological Association (AGA), American Society for Gastrointestinal Endoscopy (ASGE), European Society of Gastrointestinal Endoscopy (ESGE), British Society of Gastroenterology (BSG), National Institute for Health and Care Excellence (NICE), and the Scottish Intercollegiate Guidelines Network (SIGN). All available guidelines were evaluated and those related to gastrointestinal endoscopy were selected. All selected guidelines were in English. Position statements and technical reviews were not included.
Procedures and quality of guidelines The AGREE II instrument [1] comprises a checklist of 23 items grouped into 6 domains for use in clinical practice guideline de" Table 1). In the current study, the AGREE velopment reporting (● II instrument was used to evaluate the methodological quality of the retrieved guidelines. The AGREE II scoring was applied as recommended using a 7-point scale, where a score of 7 applied to an item for which the quality of reporting was exceptional and where the full criteria and considerations articulated in the User’s Manual had been met, and a score of 1 applied to items with no relevant information according to AGREE II or where the concept was very poorly reported [1]. The ratings used in the current study were based on these principles: explicit adherence to items (score 6 or 7); suboptimal adherence (score 4 or 5); deficient adherence (score 1 – 3). All guidelines were evaluated by one reviewer (R. M.), who read the complete printed format (title, abstract, main text, and tables), and used only this information to analyze and score each guideline according to the AGREE II instrument. There was no search for supplementary data in any internet files or appendices.
Item #
The following items were interpreted differently from the suggestions in the AGREE II User Manual [1]. ▶ Item 4: The guideline development group includes individuals from all relevant professional groups. As the definition of all relevant professional groups is rather subjective, the guidelines were classified in the current study by the extent of information given about the authors (i. e. name, area of expertise, institution, geographical location, and the member’s role in the guideline development group). If guidelines presented all this information, they scored 7, even if no professional group was identified. ▶ Item 7: Systematic methods were used to search for evidence. The methods were classified according to whether or not they were explicit, and not whether or not they were systematic. ▶ Item 18: The guideline describes facilitators and barriers to its application. Clarity on the facilitators and barriers to guideline application were scored according to whether or not there was an obvious section describing these issues. The classification “Not clear” was used if the information was omitted or somewhat dispersed throughout the article. ▶ Item 22: The views of the funding body have not influenced the content of the guideline. For the current study, this item was classified according to whether or not the funding body was named and recorded, as it was considered that an evaluation of influence over the final guideline content would result in questionable classifications due to subjectivity and difficulty in interpretation.
Item description
Domain 1 Scope and purpose Item #1
The overall objective(s) of the guideline is (are) specifically described
Item #2
The health question(s) covered by the guideline is (are) specifically described
Item #3
The population (patients, public, etc.) to whom the guideline is meant to apply is specifically described
Domain 2 Stakeholder involvement Item #4
The guideline development group includes individuals from all relevant professional groups
Item #5
The views and preferences of the target population (patients, public, etc.) have been sought
Item #6
The target users of the guideline are clearly defined
Domain 3 Rigor of development Item #7
Systematic methods were used to search for evidence
Item #8
The criteria for selecting the evidence are clearly described
Item #9
The strengths and limitations of the body of evidence are clearly described
Item #10
The methods for formulating the recommendations are clearly described
Item #11
The health benefits, side effects, and risks have been considered in formulating the recommendations
Item #12
There is an explicit link between the recommendations and the supporting evidence
Item #13
The guideline has been externally reviewed by experts prior to its publication
Item #14
A procedure for updating the guideline is provided
Domain 4 Clarity of presentation Item #15
The recommendations are specific and unambiguous
Item #16
The different options for management of the condition or health issue are clearly presented
Item #17
Key recommendations are easily identifiable
Domain 5 Applicability Item #18
The guideline describes facilitators and barriers to its application
Item #19
The guideline provides advice and/or tools on how the recommendations can be put into practice
Item #20
The potential resource implications of applying the recommendations have been considered
Item #21
The guideline presents monitoring and/or auditing criteria
Domain 6 Editorial independence Item #22
The views of the funding body have not influenced the content of the guideline
Item #23
Competing interests of guideline development group members have been recorded and addressed
Malheiro Rui et al. Methodological quality of guidelines in gastroenterology … Endoscopy 2014; 46: 513–525
Table 1 The Appraisal of Guidelines for Research and Evaluation (AGREE II) instrument checklist used for the quality assessment of methodology in retrieved guidelines.
This document was downloaded for personal use only. Unauthorized distribution is strictly prohibited.
514
Review
Year
Endoscopy 2001
Table 2 topic.
Topics [reference] IBD
Oncology
[9, 10]
2003
[11]
2005
[14, 15]
2006
[21, 22]
2007
[28, 29]
2008
[32 – 37]
2009
[38]
[12, 13] [16] [23]
[24 – 26]
[39 – 42] [43 – 45]
[46 – 48]
2012
[49 – 51]
[52]
1996
[53]
1997
[54, 55] [56]
1999
[57]
2004
[60]
[62]
2006
[63] [64]
2008 [69]
[70]
[72]
[73, 74]
[71]
2012 2001
[4]
2007
[75, 76]
2008
[77, 78]
2010
[79 – 81]
2011
[82, 83]
2012
[84 – 86]
2008
[90, 91]
2009
[92]
[87]
2010
[93]
[94] [98]
2012
[99]
[100]
2003
[101]
2011
[95]
[102]
2012 NICE SIGN
[88, 89]
[96, 97]
2011
AGA
[65 – 67] [68]
2009 2010
[58, 59] [61]
2005 2007
BSG
[27]
2011
1998
ESGE
[17, 20] [30, 31]
2010
ACG
Other
[8]
2002
ASGE
Guidelines according to
[103]
2004
[104]
2011
[105]
2008
[3]
ACG, American College of Gastroenterology; AGA, American Gastroenterological Association; ASGE, American Society for Gastrointestinal Endoscopy; ESGE, European Society of Gastrointestinal Endoscopy; BSG, British Society of Gastroenterology; IBD, inflammatory bowel disease; NICE, National Institute for Health and Care Excellence; SIGN, Scottish Intercollegiate Guidelines Network. * Guidelines are ordered, within each society, by year of publication and AGREE II final score.
For each guideline, a final score was calculated using the following formula: obtained score – minimum possible score maximum possible score – minimum possible score The maximum possible score was calculated as: 7 × number of items × the number of evaluators and the minimum possible score was calculated as: 1 × number of items × the number of evaluators The total score was then converted to a percentage ( × 100), and could vary from 0 (the obtained score was equal to the minimum score) to 100. In this study, as only one of the authors conducted the evaluation of the guidelines, the formula was simplified to:
obtained score – number of items (7 × number of items) – number of items [1]. The mean and median scores for items were calculated for each guideline. In addition to this global score, for each guideline the proportion of explicit (score 6 or 7), suboptimal (score 4 or 5), deficient or absent (score 1 – 3) reporting was calculated. A summary measure per domain was also included that represented at least a 50 % completeness of the items (score ≥ 6). Data are presented on a per society basis and ordered by year and AGREE II mean score.
Malheiro Rui et al. Methodological quality of guidelines in gastroenterology … Endoscopy 2014; 46: 513–525
This document was downloaded for personal use only. Unauthorized distribution is strictly prohibited.
Society*
515
Review
Other data In addition to the information required to evaluate the guidelines, the year of publication, the specific society, and the overall topic were recorded for each guideline. Microsoft Excel was used to aggregate all the collected data.
Results !
General results A total of 100 guidelines were retrieved. These were published " Table 2). between 1996 and 2012 (● Of the 23 items in the AGREE II checklist, the mean number of items scoring 6 or 7 for each guideline was 9.2. The overall mean (median) score was 54 % (55 %), ranging from 90 % [3] down to 28 " Fig. 1). Approximately a third (34 %) of the guidelines % [4] (● scored below 50 %. Overall, 99 % of the guidelines had a score of 3 or less for item #5 (views and preferences of the target population), 84 % did not provide a procedure for updating the guideline (item #14), and two-thirds had a “not clear” classification for item #13, as there was no explicit information about external review prior to publi" Fig. 1). The worst classified domain was “Applicability” cation (● (items #18 – 21): 97 % of guidelines did not provide a clear description of facilitators and barriers to guideline application (item #18); 87 % did not provide advice and/or tools in order to apply the recommendations (item #19), scoring 3 or less; and 94 % got the lowest score possible (score 1) on item #21, as they did not present monitoring or auditing criteria. The best scored items were: item #11 (93 % scored 6 or higher), regarding the health benefits, side effects, and risks; item #15, regarding the specificity and ambiguity (76 %); and item #22, considering the influence of the funding body over the content of the guideline (98 %).
Results by society For all societies, ≥ 50 % of the items in the domains of “Editorial independence” and “Clarity of presentation” scored 6 or 7 " Fig. 1). (●
ASGE The mean number of items scoring 6 or 7 in ASGE guidelines was 7.8. The mean score was 50 % and the median value was 48 %. Item #2 (health questions), item #11 (health benefits, side effects, and risks), items #15 – 17 (specificity and ambiguity of recommendations, different options for managing a condition, and key recommendations), and item #22 (influence of the funding body) typi" Fig. 1). The most poorly described cally had a score of 6 or 7 (● items (score ≤ 3) were: item #3 (target population), item #5 (patients’ views and preferences), item #6 (target users), item #8 (criteria for selecting the evidence), item #14 (procedure for updating the guideline), item #19 (advice or tools for application of recommendations), item #21 (monitoring or auditing criteria), and item #23 (conflict of interest).
ACG In ACG guidelines, the mean number of items scoring 6 or 7 was 10.4. The mean and median scores were both 54 %. The items that scored highest were item #2 (health questions), item #11 (health benefits, side effects, and risks), item #16 (different options for managing a condition), item #20 (potential resource implications of applying the recommendations), and item #22 (influence of
" Fig. 1). The items that were poorly reportthe funding body) (● ed were the same as for the ASGE, except for item #3, which scored ≥ 6 in nearly half of the guidelines.
ESGE The mean number of items scoring 6 or 7 in ESGE guidelines was 11.6. The mean score was 62 % and the median score was 67 %. The most adequately reported items were item #1 (objective of the guideline), item #2 (health questions), item #4 (relevant professional groups), item #11 (health benefits, side effects, and risks), item #15 (specificity and ambiguity of recommendations), item #16 (different options for managing a condition), and item " Fig. 1). The weakest re#22 (influence of the funding body) (● ported items were item #5 (patients’ views and preferences) and item #21 (monitoring or auditing criteria).
BSG The mean number of items scoring of 6 or 7 in BSG guidelines was 12.3. The mean score was 64 % and the median score was 63 %. The items with the highest score were item #4 (relevant professional groups), item #10 (methods for formulating recommendations), item #15 (specificity and ambiguity of recommendations), item #16 (different options for managing a condition), item #22 (influence of the funding body), and item #23 (conflict " Fig. 1). The poorest items were item #5 (patients’ of interest) (● views and preferences), item #8 (criteria for selection evidence), item #14 (procedure for updating the guideline), item #19 (advice or tools for application of recommendations), and item #21 (monitoring or auditing criteria).
AGA Three AGA guidelines were retrieved (2003, 2011, and 2012), which scored 50 %, 61 %, and 70 %, respectively. All three guidelines scored over 6 on item #11 (health benefits, adverse effects, and risks), item #16 (different options for managing a condition), and item #22 (influence of the funding body), and below 3 on item #5 (patients’ views and preferences), item #14 (procedure for updating the guideline), item #19 (advice or tools for application of recommendations), and item #21 (monitoring or auditing criteria).
NICE Two guidelines from NICE (from 2004 and 2011) were reviewed, and scored 73 % and 70 %, respectively. Both of them scored over 6 on item #4 (relevant professional groups), items #9 – 11 (strengths and limitations of evidence; methods for formulating recommendations; health benefits, adverse effects, and risks), items #14 – 16 (procedure for updating the guideline; specificity and ambiguity; different options for managing a condition), item #19 (advice or tools for application of recommendations), item #20 (potential resource implications), item #22 (influence of the funding body), and item #23 (conflict of interest).
SIGN Finally, only a single guideline was published by the SIGN group (2008) and this achieved the best overall score (90 %). It is notable that this guideline had only one negative evaluation, which was for item #8 (criteria for selecting evidence).
Malheiro Rui et al. Methodological quality of guidelines in gastroenterology … Endoscopy 2014; 46: 513–525
This document was downloaded for personal use only. Unauthorized distribution is strictly prohibited.
516
[33]
[36]
Antibiotic Prophylaxis
Routine laboratory testing
2008
2007
2006
2005
2003
2002
2001
Year
Domain 2 Item SM #4 #5 #6 #7
#8
#9
1–3
513
Methodological quality of guidelines in gastroenterology
Authors
Rui Malheiro1, Matilde de Monteiro-Soares1, Cesare Hassan2, Mário Dinis-Ribeiro1
Institutions
1
Bibliography DOI http://dx.doi.org/ 10.1055/s-0034-1365394 Published online: 23.4.2014 Endoscopy 2014; 46: 513–525 © Georg Thieme Verlag KG Stuttgart · New York ISSN 0013-726X Corresponding author Mário Dinis-Ribeiro, MD CINTESIS/CIDES – Health Information and Decision Sciences Department Oporto Faculty of Medicine Rua Dr. Plácido da Costa, s/n 4200-450 Oporto Portugal Fax: +351-22-5513623 [email protected]
CINTESIS/CIDES – Health Information and Decision Sciences Department, Oporto Faculty of Medicine, Oporto, Portugal Gastroenterology and Digestive Endoscopy, Nuovo Regina Margherita Hospital, Rome, Italy
Background and study aims: Clinical guidelines are a common feature in modern endoscopy practice and they are being produced faster than ever. However, their methodological quality is rarely assessed. This study evaluated the methodological quality of current clinical guidelines in the field of gastroenterology, with an emphasis on endoscopy. Materials and methods: Practice guidelines published by the American College of Gastroenterology (ACG), American Gastroenterological Association (AGA), American Society for Gastrointestinal Endoscopy (ASGE), European Society of Gastrointestinal Endoscopy (ESGE), British Society of Gastroenterology (BSG), National Institute for Health and Care Excellence (NICE), and the Scottish Intercollegiate Guidelines Network (SIGN) were searched between September and October 2012 and evaluated using the AGREE II (Appraisal of Guide-
lines for Research and Evaluation) instrument (23 items, scores 1 – 7 for each item; higher scores mean better quality). Results: A total of 100 guidelines were assessed. The mean number of items scoring 6 or 7 per guideline was 9.2 (out of 23 items). Overall, 99 % of guidelines failed to include the target population in the development process, and 96 % did not report facilitators and barriers to guideline application. In addition, 86 % did not include advice or tools, and 94 % did not present monitoring or auditing criteria. Conclusion: The global methodological quality of clinical guidelines in the field of gastroenterology is poor, particularly regarding involvement of the target population in the development of guidelines and in the provision of clear suggestions to practitioners.
Introduction
The use of clinical guidelines is critical to maximize the benefit:risk ratio when implementing endoscopic technologies and new treatments in gastroenterology, and thus, guidelines help to protect patients from an under- or overuse of such innovations. The primary aim of this study was to assess the methodological quality of current clinical guidelines within the gastroenterology area, with particular reference to endoscopy, using the AGREE II instrument. The gaps in developmental methodology that were identified were used to recommend areas for further improvement.
!
Scan this QR-Code to watch the video comment.
Guidelines are a common feature of modern clinical practice [1]. As their number grows internationally, recognized standards to improve the development and reporting of clinical guidelines have been established [2]. Although guideline development is monitored by some of the best experts in the respective fields and they are published by some of the most respected international organizations, their scientific validity, use, and reliability are rarely assessed [1]. This is critical when considering that guidelines are mainly intended to be clinical tools for the decision-making process, and as such are characterized by benefit and risks. In order to face such reality, the Appraisal of Guidelines for Research and Evaluation (AGREE) instrument, which was published in 2003 and updated in 2010, was created as an international instrument to assess the quality of the process and reporting of clinical practice guideline development [1].
Materials and methods !
Selection of guidelines In order to retrieve the most recent guidelines on gastroenterology, and in particular endoscopy, the websites of the following large European and US societies were searched between September
Malheiro Rui et al. Methodological quality of guidelines in gastroenterology … Endoscopy 2014; 46: 513–525
This document was downloaded for personal use only. Unauthorized distribution is strictly prohibited.
2
Review
and October 2012: American College of Gastroenterology (ACG), American Gastroenterological Association (AGA), American Society for Gastrointestinal Endoscopy (ASGE), European Society of Gastrointestinal Endoscopy (ESGE), British Society of Gastroenterology (BSG), National Institute for Health and Care Excellence (NICE), and the Scottish Intercollegiate Guidelines Network (SIGN). All available guidelines were evaluated and those related to gastrointestinal endoscopy were selected. All selected guidelines were in English. Position statements and technical reviews were not included.
Procedures and quality of guidelines The AGREE II instrument [1] comprises a checklist of 23 items grouped into 6 domains for use in clinical practice guideline de" Table 1). In the current study, the AGREE velopment reporting (● II instrument was used to evaluate the methodological quality of the retrieved guidelines. The AGREE II scoring was applied as recommended using a 7-point scale, where a score of 7 applied to an item for which the quality of reporting was exceptional and where the full criteria and considerations articulated in the User’s Manual had been met, and a score of 1 applied to items with no relevant information according to AGREE II or where the concept was very poorly reported [1]. The ratings used in the current study were based on these principles: explicit adherence to items (score 6 or 7); suboptimal adherence (score 4 or 5); deficient adherence (score 1 – 3). All guidelines were evaluated by one reviewer (R. M.), who read the complete printed format (title, abstract, main text, and tables), and used only this information to analyze and score each guideline according to the AGREE II instrument. There was no search for supplementary data in any internet files or appendices.
Item #
The following items were interpreted differently from the suggestions in the AGREE II User Manual [1]. ▶ Item 4: The guideline development group includes individuals from all relevant professional groups. As the definition of all relevant professional groups is rather subjective, the guidelines were classified in the current study by the extent of information given about the authors (i. e. name, area of expertise, institution, geographical location, and the member’s role in the guideline development group). If guidelines presented all this information, they scored 7, even if no professional group was identified. ▶ Item 7: Systematic methods were used to search for evidence. The methods were classified according to whether or not they were explicit, and not whether or not they were systematic. ▶ Item 18: The guideline describes facilitators and barriers to its application. Clarity on the facilitators and barriers to guideline application were scored according to whether or not there was an obvious section describing these issues. The classification “Not clear” was used if the information was omitted or somewhat dispersed throughout the article. ▶ Item 22: The views of the funding body have not influenced the content of the guideline. For the current study, this item was classified according to whether or not the funding body was named and recorded, as it was considered that an evaluation of influence over the final guideline content would result in questionable classifications due to subjectivity and difficulty in interpretation.
Item description
Domain 1 Scope and purpose Item #1
The overall objective(s) of the guideline is (are) specifically described
Item #2
The health question(s) covered by the guideline is (are) specifically described
Item #3
The population (patients, public, etc.) to whom the guideline is meant to apply is specifically described
Domain 2 Stakeholder involvement Item #4
The guideline development group includes individuals from all relevant professional groups
Item #5
The views and preferences of the target population (patients, public, etc.) have been sought
Item #6
The target users of the guideline are clearly defined
Domain 3 Rigor of development Item #7
Systematic methods were used to search for evidence
Item #8
The criteria for selecting the evidence are clearly described
Item #9
The strengths and limitations of the body of evidence are clearly described
Item #10
The methods for formulating the recommendations are clearly described
Item #11
The health benefits, side effects, and risks have been considered in formulating the recommendations
Item #12
There is an explicit link between the recommendations and the supporting evidence
Item #13
The guideline has been externally reviewed by experts prior to its publication
Item #14
A procedure for updating the guideline is provided
Domain 4 Clarity of presentation Item #15
The recommendations are specific and unambiguous
Item #16
The different options for management of the condition or health issue are clearly presented
Item #17
Key recommendations are easily identifiable
Domain 5 Applicability Item #18
The guideline describes facilitators and barriers to its application
Item #19
The guideline provides advice and/or tools on how the recommendations can be put into practice
Item #20
The potential resource implications of applying the recommendations have been considered
Item #21
The guideline presents monitoring and/or auditing criteria
Domain 6 Editorial independence Item #22
The views of the funding body have not influenced the content of the guideline
Item #23
Competing interests of guideline development group members have been recorded and addressed
Malheiro Rui et al. Methodological quality of guidelines in gastroenterology … Endoscopy 2014; 46: 513–525
Table 1 The Appraisal of Guidelines for Research and Evaluation (AGREE II) instrument checklist used for the quality assessment of methodology in retrieved guidelines.
This document was downloaded for personal use only. Unauthorized distribution is strictly prohibited.
514
Review
Year
Endoscopy 2001
Table 2 topic.
Topics [reference] IBD
Oncology
[9, 10]
2003
[11]
2005
[14, 15]
2006
[21, 22]
2007
[28, 29]
2008
[32 – 37]
2009
[38]
[12, 13] [16] [23]
[24 – 26]
[39 – 42] [43 – 45]
[46 – 48]
2012
[49 – 51]
[52]
1996
[53]
1997
[54, 55] [56]
1999
[57]
2004
[60]
[62]
2006
[63] [64]
2008 [69]
[70]
[72]
[73, 74]
[71]
2012 2001
[4]
2007
[75, 76]
2008
[77, 78]
2010
[79 – 81]
2011
[82, 83]
2012
[84 – 86]
2008
[90, 91]
2009
[92]
[87]
2010
[93]
[94] [98]
2012
[99]
[100]
2003
[101]
2011
[95]
[102]
2012 NICE SIGN
[88, 89]
[96, 97]
2011
AGA
[65 – 67] [68]
2009 2010
[58, 59] [61]
2005 2007
BSG
[27]
2011
1998
ESGE
[17, 20] [30, 31]
2010
ACG
Other
[8]
2002
ASGE
Guidelines according to
[103]
2004
[104]
2011
[105]
2008
[3]
ACG, American College of Gastroenterology; AGA, American Gastroenterological Association; ASGE, American Society for Gastrointestinal Endoscopy; ESGE, European Society of Gastrointestinal Endoscopy; BSG, British Society of Gastroenterology; IBD, inflammatory bowel disease; NICE, National Institute for Health and Care Excellence; SIGN, Scottish Intercollegiate Guidelines Network. * Guidelines are ordered, within each society, by year of publication and AGREE II final score.
For each guideline, a final score was calculated using the following formula: obtained score – minimum possible score maximum possible score – minimum possible score The maximum possible score was calculated as: 7 × number of items × the number of evaluators and the minimum possible score was calculated as: 1 × number of items × the number of evaluators The total score was then converted to a percentage ( × 100), and could vary from 0 (the obtained score was equal to the minimum score) to 100. In this study, as only one of the authors conducted the evaluation of the guidelines, the formula was simplified to:
obtained score – number of items (7 × number of items) – number of items [1]. The mean and median scores for items were calculated for each guideline. In addition to this global score, for each guideline the proportion of explicit (score 6 or 7), suboptimal (score 4 or 5), deficient or absent (score 1 – 3) reporting was calculated. A summary measure per domain was also included that represented at least a 50 % completeness of the items (score ≥ 6). Data are presented on a per society basis and ordered by year and AGREE II mean score.
Malheiro Rui et al. Methodological quality of guidelines in gastroenterology … Endoscopy 2014; 46: 513–525
This document was downloaded for personal use only. Unauthorized distribution is strictly prohibited.
Society*
515
Review
Other data In addition to the information required to evaluate the guidelines, the year of publication, the specific society, and the overall topic were recorded for each guideline. Microsoft Excel was used to aggregate all the collected data.
Results !
General results A total of 100 guidelines were retrieved. These were published " Table 2). between 1996 and 2012 (● Of the 23 items in the AGREE II checklist, the mean number of items scoring 6 or 7 for each guideline was 9.2. The overall mean (median) score was 54 % (55 %), ranging from 90 % [3] down to 28 " Fig. 1). Approximately a third (34 %) of the guidelines % [4] (● scored below 50 %. Overall, 99 % of the guidelines had a score of 3 or less for item #5 (views and preferences of the target population), 84 % did not provide a procedure for updating the guideline (item #14), and two-thirds had a “not clear” classification for item #13, as there was no explicit information about external review prior to publi" Fig. 1). The worst classified domain was “Applicability” cation (● (items #18 – 21): 97 % of guidelines did not provide a clear description of facilitators and barriers to guideline application (item #18); 87 % did not provide advice and/or tools in order to apply the recommendations (item #19), scoring 3 or less; and 94 % got the lowest score possible (score 1) on item #21, as they did not present monitoring or auditing criteria. The best scored items were: item #11 (93 % scored 6 or higher), regarding the health benefits, side effects, and risks; item #15, regarding the specificity and ambiguity (76 %); and item #22, considering the influence of the funding body over the content of the guideline (98 %).
Results by society For all societies, ≥ 50 % of the items in the domains of “Editorial independence” and “Clarity of presentation” scored 6 or 7 " Fig. 1). (●
ASGE The mean number of items scoring 6 or 7 in ASGE guidelines was 7.8. The mean score was 50 % and the median value was 48 %. Item #2 (health questions), item #11 (health benefits, side effects, and risks), items #15 – 17 (specificity and ambiguity of recommendations, different options for managing a condition, and key recommendations), and item #22 (influence of the funding body) typi" Fig. 1). The most poorly described cally had a score of 6 or 7 (● items (score ≤ 3) were: item #3 (target population), item #5 (patients’ views and preferences), item #6 (target users), item #8 (criteria for selecting the evidence), item #14 (procedure for updating the guideline), item #19 (advice or tools for application of recommendations), item #21 (monitoring or auditing criteria), and item #23 (conflict of interest).
ACG In ACG guidelines, the mean number of items scoring 6 or 7 was 10.4. The mean and median scores were both 54 %. The items that scored highest were item #2 (health questions), item #11 (health benefits, side effects, and risks), item #16 (different options for managing a condition), item #20 (potential resource implications of applying the recommendations), and item #22 (influence of
" Fig. 1). The items that were poorly reportthe funding body) (● ed were the same as for the ASGE, except for item #3, which scored ≥ 6 in nearly half of the guidelines.
ESGE The mean number of items scoring 6 or 7 in ESGE guidelines was 11.6. The mean score was 62 % and the median score was 67 %. The most adequately reported items were item #1 (objective of the guideline), item #2 (health questions), item #4 (relevant professional groups), item #11 (health benefits, side effects, and risks), item #15 (specificity and ambiguity of recommendations), item #16 (different options for managing a condition), and item " Fig. 1). The weakest re#22 (influence of the funding body) (● ported items were item #5 (patients’ views and preferences) and item #21 (monitoring or auditing criteria).
BSG The mean number of items scoring of 6 or 7 in BSG guidelines was 12.3. The mean score was 64 % and the median score was 63 %. The items with the highest score were item #4 (relevant professional groups), item #10 (methods for formulating recommendations), item #15 (specificity and ambiguity of recommendations), item #16 (different options for managing a condition), item #22 (influence of the funding body), and item #23 (conflict " Fig. 1). The poorest items were item #5 (patients’ of interest) (● views and preferences), item #8 (criteria for selection evidence), item #14 (procedure for updating the guideline), item #19 (advice or tools for application of recommendations), and item #21 (monitoring or auditing criteria).
AGA Three AGA guidelines were retrieved (2003, 2011, and 2012), which scored 50 %, 61 %, and 70 %, respectively. All three guidelines scored over 6 on item #11 (health benefits, adverse effects, and risks), item #16 (different options for managing a condition), and item #22 (influence of the funding body), and below 3 on item #5 (patients’ views and preferences), item #14 (procedure for updating the guideline), item #19 (advice or tools for application of recommendations), and item #21 (monitoring or auditing criteria).
NICE Two guidelines from NICE (from 2004 and 2011) were reviewed, and scored 73 % and 70 %, respectively. Both of them scored over 6 on item #4 (relevant professional groups), items #9 – 11 (strengths and limitations of evidence; methods for formulating recommendations; health benefits, adverse effects, and risks), items #14 – 16 (procedure for updating the guideline; specificity and ambiguity; different options for managing a condition), item #19 (advice or tools for application of recommendations), item #20 (potential resource implications), item #22 (influence of the funding body), and item #23 (conflict of interest).
SIGN Finally, only a single guideline was published by the SIGN group (2008) and this achieved the best overall score (90 %). It is notable that this guideline had only one negative evaluation, which was for item #8 (criteria for selecting evidence).
Malheiro Rui et al. Methodological quality of guidelines in gastroenterology … Endoscopy 2014; 46: 513–525
This document was downloaded for personal use only. Unauthorized distribution is strictly prohibited.
516
[33]
[36]
Antibiotic Prophylaxis
Routine laboratory testing
2008
2007
2006
2005
2003
2002
2001
Year
Domain 2 Item SM #4 #5 #6 #7
#8
#9
1–3
