bs_bs_banner
International Journal of Nursing Practice 2015; 21: 683–686
COCHRANE NURSING CARE NETWORK
Risk of bias reporting in Cochrane systematic reviews Lisa Hopp PhD RN FAAN Director, Indiana Center for Evidence Based Nursing Practice: A Joanna Briggs Collaborating Center, Hammond, Indiana, USA Professor, College of Nursing, Purdue University Calumet, Hammond, Indiana, USA
Accepted for publication June 2013 Hopp L. International Journal of Nursing Practice 2015; 21: 683–686 Risk of bias reporting in Cochrane systematic reviews Risk of bias is an inherent quality of primary research and therefore of systematic reviews. This column addresses the Cochrane Collaboration’s approach to assessing, risks of bias, the meaning of each, indicators of low, high and uncertain, and ways that risk of bias can be represented in a Cochrane systematic review report. The sources of risk of bias that reviewers evaluate include selection, performance, detection, attrition and reporting bias. Each poses threat to the internal validity of the primary studies and requires the reviewer to judge the level of risk as high, low or unclear. Reviewers need to address how studies of higher risk of bias might impact the pooled effect. Key words: Cochrane Collaboration, risk of bias, systematic review.
High-quality systematic reviews represent the strongest evidence in most evidence hierarchies. They sit on top of the evidence heap because their methodologies fit an ideal of ‘best available’ evidence. If the reviewer uses an exhaustive approach to uncovering all evidence, systematic reviews represent what is available. This column will address how, when they transparently provide an assessment of risk of bias, systematic reviews address the ‘best’ portion of the definition. Cochrane systematic reviews aim to reach conclusions about the effect of treatments and interventions. When clinicians use these conclusions to inform their decisions about the comparative effectiveness of treatment options, they also need to know the trustworthiness of the information. Internal validity is the degree that the researcher
Correspondence: Lisa Hopp, College of Nursing, Purdue University Calumet, 2200 169th St, Hammond, IN 46323, USA. Email: [email protected] doi:10.1111/ijn.12252
can make cause-and-effect claims about the independent (cause) and the dependent variable (effect).1 A variety of biases can threaten internal validity and causality, and therefore sully the trustworthiness of the conclusions. Both reviewers and consumers of reviews are concerned about the included studies’ veracity and their related internal validity.
NATURE OF BIAS Bias is ‘a systematic error, or deviation from the truth, in results or inferences’.2 Systematic error leads to either under- or overestimating the true effects. Random error is quite different and relates to precision and the degree of variability attributed to sampling error. Small sample sizes in underpowered studies will lead to greater random error; if researchers replicate an imprecise study, the answers will vary from study to study. Whereas, a set of biased studies conducted in a similar yet flawed manner can yield consistently incorrect answers. In a metaanalysis, imprecise studies contribute less weight to the © 2014 Wiley Publishing Asia Pty Ltd
684
L Hopp
pooled effect whereas biased studies can contribute great weight if they are precise and lead to conclusions that are far from the truth.2 Cochrane reviewers address ‘risk of bias’ rather than make definitive statements about the presence, absence or degree of bias. An individual study’s bias and its impact on the pooled effect cannot be absolutely known because the truth always remains illusive. On the other hand, a study’s result might be unbiased despite methodological flaws. The aim of the risk of bias tools within a Cochrane review is to transparently represent the reviewers’ judgments about individual factors that can threaten the validity of individual studies as well as across studies. Clinicians and others can use the figures, tables and discussion to understand how the risks of bias impact the believability of the review and the strength of recommendations that might emerge from the review. A Cochrane methodology uses a ‘domain-based’ system to assess risk of bias. Rather than using a summed score based on a quality scale or checklist, this approach allows reviewers to assess potential threats based on the particular research design and its conduct. They reach a judgment of high, low or unclear risk of bias in each domain for each included study. Reviewers judge each domain based on flaws in the research design or its conduct, and try to avoid making judgments based on imperfections in reporting. They contact the original study scientists to pose open-ended questions aimed at uncovering missing information to better inform their assessment of risk for bias.2
SOURCES OF RISK OF BIAS The ideal study for a Cochrane effectiveness review is a well-designed and conducted randomized controlled trial that leads to precise estimates of effect. Therefore, the sources of bias relate to threats relevant to this research design and the ability to make cause–effect claims. The domains of bias are selection, performance, detection, attrition, reporting and a general domain of ‘other’. Within each, there are specific elements to consider. Review authors report on eight specific elements. Selection bias refers to systematic differences that exist between or among groups at the baseline of the trial. Random assignment to groups is meant to prevent selection bias. However, the reviewer must judge whether or not the method of generating the allocation sequence is truly random and if the person assigning participants © 2014 Wiley Publishing Asia Pty Ltd
does not know what or who comes next. To minimize systematic differences related to assignment to groups, the person who enrols the participant must not know the next allocation and strictly adheres to the method that generates the random sequence.2 Markers of low risk of bias for random sequence generation include computer-generated random numbers, coin tossing, card/envelope shuffling, etc. In each case, the chance of group assignment is equal. High-risk markers indicate an element of systematic, non-random approaches like using odd or even birthdates for group allocation, a rule for every nth admission to hospital, self or clinician assignment to groups, etc. If reviewers cannot uncover enough information about the sequence of allocation, then they assess the risk of bias as unclear.2 Markers of low risk of selection bias related to allocation methods include a remote and centralized way to avoid knowing what group or person is next in line or opaque, sequentially numbered envelopes. Markers of higher selection risk due to inadequate concealment are an open random table at the point of care, envelopes without safeguards to conceal what is inside, using date of birth or medical record number. Unclear risk often relates to inadequate reporting like no mention of the opacity of envelopes, location of the allocator or a simple claim of blinding without mention of the mechanism to achieve it. Performance bias is systematic differences between groups based on participants or personnel responding or behaving in a particular way because they know group assignments. For example, in a study of respiratory muscle training, participants might take extra care to take their prescribed medications and to engage in regular activity coincident to knowing that they are using a device to exercise their inspiratory muscles. The best way to manage risk of performance bias is to blind both the participants and study personnel to group assignment. In the example of respiratory muscle training, a low risk of bias marker would be that participants used a sham trainer and the study personnel were not able to detect the group assignments based on their interactions at any time during the study. Indications of high risk of performance bias include lack of blinding so the outcome might be influenced by knowledge of group assignment or a reasonable chance that blinding could have been broken. Judging the degree of risk associated with performance bias requires that reviewers understand the relationship between the intervention mechanics and outcomes measurement.
Risk of bias
685
Researchers who transparently explain the relationship help the reviewer understand the threat posed by a lack of blinding.2 Detection bias refers to systematic differences that arise from how researchers gather and measure outcomes.2 Detection bias is especially important when the measurement of the outcome requires a subjective assessment of some sort. For example, detection bias might be particularly important in a study of the effect of a wound intervention on pressure ulcer stage. The risk could be less important in a study of a relaxation intervention on a precise, invasive, digital measurement of blood pressure. Low risk of detection bias would include blinding of outcome assessors without a chance that this blinding would be broken, or that it would be unlikely the outcome measurement would be affected by the knowledge of group assignment. Markers of high risk of detection bias are the opposite of these characteristics.2 Attrition bias relates to systematic differences due to differential loss of outcome data. It might occur due to withdrawal or mortality from one group over another. Higher risk would be assessed when the missing outcome data are related to the group assignment, or when researchers exclude outcome data and these missing outcome data relate to the intervention. Lower risk would include few missing data or too few to make a difference on effect size.2 Reporting bias refers to systematic difference due to selective reporting of outcomes that show a significant difference and lack of reporting of others. Reporting bias is most easily judged if the study protocol is available to compare what the researcher planned to measure and what they reported.2 Other bias refers to any other source of bias that might be specific to the particular study design, related to fraudulent data or other sources of bias.2
REPORTING RISK OF BIAS In Cochrane reviews, risk of bias can be presented in three ways. These include a ‘risk of bias’ summary figure of cross-tabulations of each study across the domains of potential bias (see Table 1). The second figure is a bar chart of the ‘risk of bias graph’ representing the proportion of included studies ratings on each domain (see Fig. 1). Clinicians can use these visual representations to gain an at-a-glance impression of the risk of bias. Red bars or circles indicate high risk, green represents low risk, and yellow reflects unclear risk of bias. Readers can glean a deeper understanding of the risks of bias through a detailed table for each study where each row represents a domain element, a rating of the risk and support for their judgments that might include verbatim quotations from the study or other justification. Risk of bias is a concern for authors of systematic reviews. In this column, the Cochrane approach was
Table 1 Risk of bias summary example
Study 1 Study 2 Study 3
Risk of bias A
Risk of bias B
Risk of bias C
Risk of bias D
Risk of bias E
+ − ?
+ ? +
− ? +
+ − −
− + ?
ROB, theoretical risk of bias (the specific risk of bias would be named in an actual Cochrane systematic review); +, low risk of bias (represented with a green circle in a Cochrane publication); −, high risk of bias (represented with a red circle in a Cochrane publication); ?, unclear risk of bias (represented with a yellow circle in a Cochrane publication).
Risk of bias A
Figure 1. Risk of bias graph.
Risk of bias B
The figure represents the proportion of studies with low, high or unclear risk of biases
Risk of bias C
listed in each row. In a Cochrane publication, low risk of bias bars would be green, high risk of bias bars would be red and unclear risk of bias bars would be yellow. , low risk of bias; , high risk of bias; , unclear risk of bias.
Risk of bias D Risk of bias E
0%
25%
50%
75%
100%
% of studies with risk of bias
© 2014 Wiley Publishing Asia Pty Ltd
686
the focus. Other methodology groups, like the Joanna Briggs Institute, address risk of bias in very similar ways with a clear focus on the same threats to internal validity.3 It is not possible to know the impact of bias on any one trial and therefore a systematic review of trials.2 Reviewers assess the risk of bias and seek to understand how these threats might influence the pooled effect through analytical and logical approaches. Clinicians and other decision makers use the representations of risk of bias to determine the trustworthiness of the conclusions of the review.
© 2014 Wiley Publishing Asia Pty Ltd
L Hopp
REFERENCES 1 Card NA. Applied Meta-Analysis for Social Science Research. New York: The Guilford Press, 2012. 2 Higgins JPT, Green S (eds). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from URL: http://www.cochrane-handbook.org. Accessed 8 April 2013. 3 Joanna Briggs Institute. Reviewer’s Manual, 2011 Edition. Available from URL: http://www.joannabriggs.edu.au/ SUMARI. Accessed 8 April 2013.
International Journal of Nursing Practice 2015; 21: 683–686
COCHRANE NURSING CARE NETWORK
Risk of bias reporting in Cochrane systematic reviews Lisa Hopp PhD RN FAAN Director, Indiana Center for Evidence Based Nursing Practice: A Joanna Briggs Collaborating Center, Hammond, Indiana, USA Professor, College of Nursing, Purdue University Calumet, Hammond, Indiana, USA
Accepted for publication June 2013 Hopp L. International Journal of Nursing Practice 2015; 21: 683–686 Risk of bias reporting in Cochrane systematic reviews Risk of bias is an inherent quality of primary research and therefore of systematic reviews. This column addresses the Cochrane Collaboration’s approach to assessing, risks of bias, the meaning of each, indicators of low, high and uncertain, and ways that risk of bias can be represented in a Cochrane systematic review report. The sources of risk of bias that reviewers evaluate include selection, performance, detection, attrition and reporting bias. Each poses threat to the internal validity of the primary studies and requires the reviewer to judge the level of risk as high, low or unclear. Reviewers need to address how studies of higher risk of bias might impact the pooled effect. Key words: Cochrane Collaboration, risk of bias, systematic review.
High-quality systematic reviews represent the strongest evidence in most evidence hierarchies. They sit on top of the evidence heap because their methodologies fit an ideal of ‘best available’ evidence. If the reviewer uses an exhaustive approach to uncovering all evidence, systematic reviews represent what is available. This column will address how, when they transparently provide an assessment of risk of bias, systematic reviews address the ‘best’ portion of the definition. Cochrane systematic reviews aim to reach conclusions about the effect of treatments and interventions. When clinicians use these conclusions to inform their decisions about the comparative effectiveness of treatment options, they also need to know the trustworthiness of the information. Internal validity is the degree that the researcher
Correspondence: Lisa Hopp, College of Nursing, Purdue University Calumet, 2200 169th St, Hammond, IN 46323, USA. Email: [email protected] doi:10.1111/ijn.12252
can make cause-and-effect claims about the independent (cause) and the dependent variable (effect).1 A variety of biases can threaten internal validity and causality, and therefore sully the trustworthiness of the conclusions. Both reviewers and consumers of reviews are concerned about the included studies’ veracity and their related internal validity.
NATURE OF BIAS Bias is ‘a systematic error, or deviation from the truth, in results or inferences’.2 Systematic error leads to either under- or overestimating the true effects. Random error is quite different and relates to precision and the degree of variability attributed to sampling error. Small sample sizes in underpowered studies will lead to greater random error; if researchers replicate an imprecise study, the answers will vary from study to study. Whereas, a set of biased studies conducted in a similar yet flawed manner can yield consistently incorrect answers. In a metaanalysis, imprecise studies contribute less weight to the © 2014 Wiley Publishing Asia Pty Ltd
684
L Hopp
pooled effect whereas biased studies can contribute great weight if they are precise and lead to conclusions that are far from the truth.2 Cochrane reviewers address ‘risk of bias’ rather than make definitive statements about the presence, absence or degree of bias. An individual study’s bias and its impact on the pooled effect cannot be absolutely known because the truth always remains illusive. On the other hand, a study’s result might be unbiased despite methodological flaws. The aim of the risk of bias tools within a Cochrane review is to transparently represent the reviewers’ judgments about individual factors that can threaten the validity of individual studies as well as across studies. Clinicians and others can use the figures, tables and discussion to understand how the risks of bias impact the believability of the review and the strength of recommendations that might emerge from the review. A Cochrane methodology uses a ‘domain-based’ system to assess risk of bias. Rather than using a summed score based on a quality scale or checklist, this approach allows reviewers to assess potential threats based on the particular research design and its conduct. They reach a judgment of high, low or unclear risk of bias in each domain for each included study. Reviewers judge each domain based on flaws in the research design or its conduct, and try to avoid making judgments based on imperfections in reporting. They contact the original study scientists to pose open-ended questions aimed at uncovering missing information to better inform their assessment of risk for bias.2
SOURCES OF RISK OF BIAS The ideal study for a Cochrane effectiveness review is a well-designed and conducted randomized controlled trial that leads to precise estimates of effect. Therefore, the sources of bias relate to threats relevant to this research design and the ability to make cause–effect claims. The domains of bias are selection, performance, detection, attrition, reporting and a general domain of ‘other’. Within each, there are specific elements to consider. Review authors report on eight specific elements. Selection bias refers to systematic differences that exist between or among groups at the baseline of the trial. Random assignment to groups is meant to prevent selection bias. However, the reviewer must judge whether or not the method of generating the allocation sequence is truly random and if the person assigning participants © 2014 Wiley Publishing Asia Pty Ltd
does not know what or who comes next. To minimize systematic differences related to assignment to groups, the person who enrols the participant must not know the next allocation and strictly adheres to the method that generates the random sequence.2 Markers of low risk of bias for random sequence generation include computer-generated random numbers, coin tossing, card/envelope shuffling, etc. In each case, the chance of group assignment is equal. High-risk markers indicate an element of systematic, non-random approaches like using odd or even birthdates for group allocation, a rule for every nth admission to hospital, self or clinician assignment to groups, etc. If reviewers cannot uncover enough information about the sequence of allocation, then they assess the risk of bias as unclear.2 Markers of low risk of selection bias related to allocation methods include a remote and centralized way to avoid knowing what group or person is next in line or opaque, sequentially numbered envelopes. Markers of higher selection risk due to inadequate concealment are an open random table at the point of care, envelopes without safeguards to conceal what is inside, using date of birth or medical record number. Unclear risk often relates to inadequate reporting like no mention of the opacity of envelopes, location of the allocator or a simple claim of blinding without mention of the mechanism to achieve it. Performance bias is systematic differences between groups based on participants or personnel responding or behaving in a particular way because they know group assignments. For example, in a study of respiratory muscle training, participants might take extra care to take their prescribed medications and to engage in regular activity coincident to knowing that they are using a device to exercise their inspiratory muscles. The best way to manage risk of performance bias is to blind both the participants and study personnel to group assignment. In the example of respiratory muscle training, a low risk of bias marker would be that participants used a sham trainer and the study personnel were not able to detect the group assignments based on their interactions at any time during the study. Indications of high risk of performance bias include lack of blinding so the outcome might be influenced by knowledge of group assignment or a reasonable chance that blinding could have been broken. Judging the degree of risk associated with performance bias requires that reviewers understand the relationship between the intervention mechanics and outcomes measurement.
Risk of bias
685
Researchers who transparently explain the relationship help the reviewer understand the threat posed by a lack of blinding.2 Detection bias refers to systematic differences that arise from how researchers gather and measure outcomes.2 Detection bias is especially important when the measurement of the outcome requires a subjective assessment of some sort. For example, detection bias might be particularly important in a study of the effect of a wound intervention on pressure ulcer stage. The risk could be less important in a study of a relaxation intervention on a precise, invasive, digital measurement of blood pressure. Low risk of detection bias would include blinding of outcome assessors without a chance that this blinding would be broken, or that it would be unlikely the outcome measurement would be affected by the knowledge of group assignment. Markers of high risk of detection bias are the opposite of these characteristics.2 Attrition bias relates to systematic differences due to differential loss of outcome data. It might occur due to withdrawal or mortality from one group over another. Higher risk would be assessed when the missing outcome data are related to the group assignment, or when researchers exclude outcome data and these missing outcome data relate to the intervention. Lower risk would include few missing data or too few to make a difference on effect size.2 Reporting bias refers to systematic difference due to selective reporting of outcomes that show a significant difference and lack of reporting of others. Reporting bias is most easily judged if the study protocol is available to compare what the researcher planned to measure and what they reported.2 Other bias refers to any other source of bias that might be specific to the particular study design, related to fraudulent data or other sources of bias.2
REPORTING RISK OF BIAS In Cochrane reviews, risk of bias can be presented in three ways. These include a ‘risk of bias’ summary figure of cross-tabulations of each study across the domains of potential bias (see Table 1). The second figure is a bar chart of the ‘risk of bias graph’ representing the proportion of included studies ratings on each domain (see Fig. 1). Clinicians can use these visual representations to gain an at-a-glance impression of the risk of bias. Red bars or circles indicate high risk, green represents low risk, and yellow reflects unclear risk of bias. Readers can glean a deeper understanding of the risks of bias through a detailed table for each study where each row represents a domain element, a rating of the risk and support for their judgments that might include verbatim quotations from the study or other justification. Risk of bias is a concern for authors of systematic reviews. In this column, the Cochrane approach was
Table 1 Risk of bias summary example
Study 1 Study 2 Study 3
Risk of bias A
Risk of bias B
Risk of bias C
Risk of bias D
Risk of bias E
+ − ?
+ ? +
− ? +
+ − −
− + ?
ROB, theoretical risk of bias (the specific risk of bias would be named in an actual Cochrane systematic review); +, low risk of bias (represented with a green circle in a Cochrane publication); −, high risk of bias (represented with a red circle in a Cochrane publication); ?, unclear risk of bias (represented with a yellow circle in a Cochrane publication).
Risk of bias A
Figure 1. Risk of bias graph.
Risk of bias B
The figure represents the proportion of studies with low, high or unclear risk of biases
Risk of bias C
listed in each row. In a Cochrane publication, low risk of bias bars would be green, high risk of bias bars would be red and unclear risk of bias bars would be yellow. , low risk of bias; , high risk of bias; , unclear risk of bias.
Risk of bias D Risk of bias E
0%
25%
50%
75%
100%
% of studies with risk of bias
© 2014 Wiley Publishing Asia Pty Ltd
686
the focus. Other methodology groups, like the Joanna Briggs Institute, address risk of bias in very similar ways with a clear focus on the same threats to internal validity.3 It is not possible to know the impact of bias on any one trial and therefore a systematic review of trials.2 Reviewers assess the risk of bias and seek to understand how these threats might influence the pooled effect through analytical and logical approaches. Clinicians and other decision makers use the representations of risk of bias to determine the trustworthiness of the conclusions of the review.
© 2014 Wiley Publishing Asia Pty Ltd
L Hopp
REFERENCES 1 Card NA. Applied Meta-Analysis for Social Science Research. New York: The Guilford Press, 2012. 2 Higgins JPT, Green S (eds). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from URL: http://www.cochrane-handbook.org. Accessed 8 April 2013. 3 Joanna Briggs Institute. Reviewer’s Manual, 2011 Edition. Available from URL: http://www.joannabriggs.edu.au/ SUMARI. Accessed 8 April 2013.
