Letters COMMENT & RESPONSE
Problematic Consequences of Using Standard Errors Rather Than Standard Deviations: Calculation of Effect Sizes To the Editor In the article recently published in JAMA Pediatrics on a cluster randomized trial of the Healthy Buddies program, Santos and colleagues1 reported effect sizes for the impact of intervention on the assessed outcomes, with waist circumference and body mass index z scores serving as primary outcomes. The formula given for calculating each effect size was the difference between treatment and control groups in baseline to follow-up differences on the outcome measure divided by the pooled standard error of the measure. The use of a measure’s standard error for calculating an effect size is problematic for at least 2 reasons. First, there are no known indices of effect size that use the metric of a measure’s standard error (or for that matter, the standard error of an associated test statistic in the event this is what is being referenced in the formula provided). Rather, the widely accepted metric for standardizing effect sizes that involves the difference between treatment and control groups on a continuous measure is the measure’s standard deviation.2 This distinction is consequential3 because as a rule, the standard error of a measure will be much smaller than its standard deviation, the former equal to the standard deviation divided by the square root of the sample size. To illustrate, for a study with a sample size of 400, if the standard deviation of a measure administered to the sample is 4, its standard error will be only 0.20. A standard error–based effect size for this measure would be 20-fold larger than one based on its standard deviation. A second cause for concern stems from the article’s incorrect description of the computed effect sizes as being in the standard deviation–based metric of Cohen d. The authors do so according to the Cohen widely applied rules of thumb for what constitutes an effect of small (0.20), medium (0.50), or large (0.80) magnitude when reported in this metric.4 Yet, as the preceding discussion makes clear, application of these guidelines to standard error–based effect sizes is inappropriate and likely to be quite misleading. Researchers are increasingly being encouraged to move away from null hypothesis testing in favor of greater focus on estimating the magnitude of the effects or associations they are studying. There is much to recommend such a shift.5 Yet, without both accuracy and consistency in the methods used, its intended benefits are unlikely to be realized. David L. DuBois, PhD Author Affiliation: Institute for Health Research and Policy, University of Illinois at Chicago, Chicago. jamapediatrics.com
Corresponding Author: David L. DuBois, PhD, Institute for Health Research and Policy, University of Illinois at Chicago, 1747 W Roosevelt Rd, Chicago, IL 60608 ([email protected]). Published Online: December 15, 2014. doi:10.1001/jamapediatrics.2014.2986. Conflict of Interest Disclosures: None reported. 1. Santos RG, Durksen A, Rabbanni R, et al. Effectiveness of peer-based healthy living lesson plans on anthropometric measures and physical activity in elementary school students: a cluster randomized trial. JAMA Pediatr. 2014;168 (4):330-337. 2. Cooper H. Research Synthesis and Meta-Analysis: A Step-By-Step Approach. 4th ed. Thousand Oaks, CA: Sage Publications; 2010. 3. Higgins JPT, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration; 2011. http://www.cochrane-handbook.org. Accessed July 12, 2014. 4. Cohen J. Statistical Power Analysis for the Social Sciences. 2nd ed. New York, NY: Academic Press; 1988. 5. Cumming G. The new statistics: why and how. Psychol Sci. 2014;25(1):7-29.
In Reply On behalf of my coauthors, I thank Dr Dubois for highlighting the error of reporting treatment effect in the article that described the results of the cluster randomized clinical trial of the Healthy Buddies Curriculum, recently published in JAMA Pediatrics.1 Dr Dubois correctly highlighted the fact that the “the use of a measure’s standard error for calculating an effect size is problematic” because it would inflate the estimate of the effect size. We incorrectly reported the formula used to calculate the treatment effect. In fact, the treatment effect was calculated using the mean gain scored difference: Treatment effect = mean difference between groups/pooled SD Pooled SD = SDdifference/兹[2(1 − r)], where r indicates pre-post correlation and SD indicates standard deviation.2 We apologize for the error in the original publication and have submitted a correction that will accompany the article to address this error. Robert G. Santos, PhD Anita Durksen, MSc Jonathan M. McGavock, PhD Author Affiliations: Healthy Child Manitoba Office, Government of Manitoba, Winnipeg, Canada (Santos); Department of Community Health Sciences, Faculty of Medicine, University of Manitoba, Winnipeg, Canada (Santos); Manitoba Institute of Child Health, Department of Pediatrics and Child Health, University of Manitoba, Winnipeg, Canada (Durksen, McGavock). Corresponding Author: Jonathan M. McGavock, PhD, Faculty of Medicine, Manitoba Institute of Child Health, Department of Pediatrics and Child Health, University of Manitoba, 715 McDermot Ave, 510A JBRC, Winnipeg, Manitoba, Canada, R3E 3P4 ([email protected]). Published Online: December 15, 2014. doi:10.1001/jamapediatrics.2014.2989. JAMA Pediatrics January 2015 Volume 169, Number 1
Copyright 2015 American Medical Association. All rights reserved.
Downloaded From: http://archpedi.jamanetwork.com/ by a University of California - San Diego User on 06/05/2015
95
Letters
Conflict of Interest Disclosures: None reported. 1. Santos RG, Durksen A, Rabbanni R, et al. Effectiveness of peer-based healthy living lesson plans on anthropometric measures and physical activity in elementary school students: a cluster randomized trial. JAMA Pediatr. 2014;168 (4):330-337. 2. Boerenstein M, Hedges LV, Higgins JPT, Rothstein HR. Introduction to Meta Analysis. John Wiley and Sons, Ltd; 2009.
CORRECTIONS Minor Errors in Statistical Methods: In the article titled “Effectiveness of PeerBased Healthy Living Lesson Plans on Anthropometric Measures and Physical Activity in Elementary School Students: A Cluster Randomized Trial,” published online February 10, 2014, and also in the April 2014 print issue of JAMA Pediatrics (2014; 168[4]:330-337. doi:10.1001/jamapediatrics.2013.3688), several minor errors existed in the statistical methods used in the study. Once recalculated, the treatment effects were modified slightly; however, these modifications did not influence the original statistically significant differences between the treatment and
96
control arms of the trial, and the results and conclusions remain unchanged. In addition, the third author’s last name should have appeared as “Rabbani” in the byline, Author Affiliations, and Author Contributions. This article was corrected online.
Incorrect Grant Information: In the article titled “Influence of Age at Virologic Control on Peripheral Blood Human Immunodeficiency Virus Reservoir Size and Serostatus in Perinatally Infected Adolescents,” published online October 6, 2014, and also in the December 2014 print issue of JAMA Pediatrics (2014;168[12]:11381146. doi:10.1001/jamapediatrics.2014.1560), 2 grant numbers in the Funding/ Support section were incorrect and another grant was inadvertently omitted. On page 1145, the first sentence of the Funding/Support section should have read as follows: “The laboratory studies were supported by grants R01HD080474 and R21AI100656 from the National Institutes of Health (NIH) (Dr Persaud); by funding from the American Foundation for AIDS Research (Drs Persaud and Luzuriaga); and by grants UO1-AI-068632 from the International Maternal Pediatric Adolescent AIDS Clinical Trials Network (Dr Persaud) and P30-AI094189 from the Johns Hopkins University Center for AIDS Research (Dr Persaud).” This article was corrected online and in print.
JAMA Pediatrics January 2015 Volume 169, Number 1
Copyright 2015 American Medical Association. All rights reserved.
Downloaded From: http://archpedi.jamanetwork.com/ by a University of California - San Diego User on 06/05/2015
jamapediatrics.com
Problematic Consequences of Using Standard Errors Rather Than Standard Deviations: Calculation of Effect Sizes To the Editor In the article recently published in JAMA Pediatrics on a cluster randomized trial of the Healthy Buddies program, Santos and colleagues1 reported effect sizes for the impact of intervention on the assessed outcomes, with waist circumference and body mass index z scores serving as primary outcomes. The formula given for calculating each effect size was the difference between treatment and control groups in baseline to follow-up differences on the outcome measure divided by the pooled standard error of the measure. The use of a measure’s standard error for calculating an effect size is problematic for at least 2 reasons. First, there are no known indices of effect size that use the metric of a measure’s standard error (or for that matter, the standard error of an associated test statistic in the event this is what is being referenced in the formula provided). Rather, the widely accepted metric for standardizing effect sizes that involves the difference between treatment and control groups on a continuous measure is the measure’s standard deviation.2 This distinction is consequential3 because as a rule, the standard error of a measure will be much smaller than its standard deviation, the former equal to the standard deviation divided by the square root of the sample size. To illustrate, for a study with a sample size of 400, if the standard deviation of a measure administered to the sample is 4, its standard error will be only 0.20. A standard error–based effect size for this measure would be 20-fold larger than one based on its standard deviation. A second cause for concern stems from the article’s incorrect description of the computed effect sizes as being in the standard deviation–based metric of Cohen d. The authors do so according to the Cohen widely applied rules of thumb for what constitutes an effect of small (0.20), medium (0.50), or large (0.80) magnitude when reported in this metric.4 Yet, as the preceding discussion makes clear, application of these guidelines to standard error–based effect sizes is inappropriate and likely to be quite misleading. Researchers are increasingly being encouraged to move away from null hypothesis testing in favor of greater focus on estimating the magnitude of the effects or associations they are studying. There is much to recommend such a shift.5 Yet, without both accuracy and consistency in the methods used, its intended benefits are unlikely to be realized. David L. DuBois, PhD Author Affiliation: Institute for Health Research and Policy, University of Illinois at Chicago, Chicago. jamapediatrics.com
Corresponding Author: David L. DuBois, PhD, Institute for Health Research and Policy, University of Illinois at Chicago, 1747 W Roosevelt Rd, Chicago, IL 60608 ([email protected]). Published Online: December 15, 2014. doi:10.1001/jamapediatrics.2014.2986. Conflict of Interest Disclosures: None reported. 1. Santos RG, Durksen A, Rabbanni R, et al. Effectiveness of peer-based healthy living lesson plans on anthropometric measures and physical activity in elementary school students: a cluster randomized trial. JAMA Pediatr. 2014;168 (4):330-337. 2. Cooper H. Research Synthesis and Meta-Analysis: A Step-By-Step Approach. 4th ed. Thousand Oaks, CA: Sage Publications; 2010. 3. Higgins JPT, Green S, eds. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration; 2011. http://www.cochrane-handbook.org. Accessed July 12, 2014. 4. Cohen J. Statistical Power Analysis for the Social Sciences. 2nd ed. New York, NY: Academic Press; 1988. 5. Cumming G. The new statistics: why and how. Psychol Sci. 2014;25(1):7-29.
In Reply On behalf of my coauthors, I thank Dr Dubois for highlighting the error of reporting treatment effect in the article that described the results of the cluster randomized clinical trial of the Healthy Buddies Curriculum, recently published in JAMA Pediatrics.1 Dr Dubois correctly highlighted the fact that the “the use of a measure’s standard error for calculating an effect size is problematic” because it would inflate the estimate of the effect size. We incorrectly reported the formula used to calculate the treatment effect. In fact, the treatment effect was calculated using the mean gain scored difference: Treatment effect = mean difference between groups/pooled SD Pooled SD = SDdifference/兹[2(1 − r)], where r indicates pre-post correlation and SD indicates standard deviation.2 We apologize for the error in the original publication and have submitted a correction that will accompany the article to address this error. Robert G. Santos, PhD Anita Durksen, MSc Jonathan M. McGavock, PhD Author Affiliations: Healthy Child Manitoba Office, Government of Manitoba, Winnipeg, Canada (Santos); Department of Community Health Sciences, Faculty of Medicine, University of Manitoba, Winnipeg, Canada (Santos); Manitoba Institute of Child Health, Department of Pediatrics and Child Health, University of Manitoba, Winnipeg, Canada (Durksen, McGavock). Corresponding Author: Jonathan M. McGavock, PhD, Faculty of Medicine, Manitoba Institute of Child Health, Department of Pediatrics and Child Health, University of Manitoba, 715 McDermot Ave, 510A JBRC, Winnipeg, Manitoba, Canada, R3E 3P4 ([email protected]). Published Online: December 15, 2014. doi:10.1001/jamapediatrics.2014.2989. JAMA Pediatrics January 2015 Volume 169, Number 1
Copyright 2015 American Medical Association. All rights reserved.
Downloaded From: http://archpedi.jamanetwork.com/ by a University of California - San Diego User on 06/05/2015
95
Letters
Conflict of Interest Disclosures: None reported. 1. Santos RG, Durksen A, Rabbanni R, et al. Effectiveness of peer-based healthy living lesson plans on anthropometric measures and physical activity in elementary school students: a cluster randomized trial. JAMA Pediatr. 2014;168 (4):330-337. 2. Boerenstein M, Hedges LV, Higgins JPT, Rothstein HR. Introduction to Meta Analysis. John Wiley and Sons, Ltd; 2009.
CORRECTIONS Minor Errors in Statistical Methods: In the article titled “Effectiveness of PeerBased Healthy Living Lesson Plans on Anthropometric Measures and Physical Activity in Elementary School Students: A Cluster Randomized Trial,” published online February 10, 2014, and also in the April 2014 print issue of JAMA Pediatrics (2014; 168[4]:330-337. doi:10.1001/jamapediatrics.2013.3688), several minor errors existed in the statistical methods used in the study. Once recalculated, the treatment effects were modified slightly; however, these modifications did not influence the original statistically significant differences between the treatment and
96
control arms of the trial, and the results and conclusions remain unchanged. In addition, the third author’s last name should have appeared as “Rabbani” in the byline, Author Affiliations, and Author Contributions. This article was corrected online.
Incorrect Grant Information: In the article titled “Influence of Age at Virologic Control on Peripheral Blood Human Immunodeficiency Virus Reservoir Size and Serostatus in Perinatally Infected Adolescents,” published online October 6, 2014, and also in the December 2014 print issue of JAMA Pediatrics (2014;168[12]:11381146. doi:10.1001/jamapediatrics.2014.1560), 2 grant numbers in the Funding/ Support section were incorrect and another grant was inadvertently omitted. On page 1145, the first sentence of the Funding/Support section should have read as follows: “The laboratory studies were supported by grants R01HD080474 and R21AI100656 from the National Institutes of Health (NIH) (Dr Persaud); by funding from the American Foundation for AIDS Research (Drs Persaud and Luzuriaga); and by grants UO1-AI-068632 from the International Maternal Pediatric Adolescent AIDS Clinical Trials Network (Dr Persaud) and P30-AI094189 from the Johns Hopkins University Center for AIDS Research (Dr Persaud).” This article was corrected online and in print.
JAMA Pediatrics January 2015 Volume 169, Number 1
Copyright 2015 American Medical Association. All rights reserved.
Downloaded From: http://archpedi.jamanetwork.com/ by a University of California - San Diego User on 06/05/2015
jamapediatrics.com
