International Journal of Cardiology 184 (2015) 510–511
Contents lists available at ScienceDirect
International Journal of Cardiology journal homepage: www.elsevier.com/locate/ijcard
Letter to the Editor
Promotion of “Low T” and citation bias in testosterone studies Jie Zhao a, C. Mary Schooling a,b,⁎ a b
School of Public Health, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Hong Kong Special Administrative Region School of Urban Public Health, Hunter College, CUNY School of Public Health, New York, NY, USA
a r t i c l e
i n f o
Article history: Received 16 January 2015 Accepted 1 March 2015 Available online 3 March 2015 Keywords: Testosterone Cardiovascular disease Citation bias
The “Low T” marketing campaign playing on older men's concerns about flagging masculinity has driven a dramatic increase in testosterone prescription in the last few years, particularly in North America. However, sales of testosterone are now falling again [1], as regulators, particularly Health Canada [2], have drawn attention to the cardiovascular risk of testosterone supplementation. Health Canada, the Food and Drug Administration and the European Medicines Agency have all advised that testosterone prescription should be restricted [2–4], consistent with the lack of established benefit and the possibility of putting millions of older men at risk of venous thromboembolism or heart attacks [2]. We wondered whether this situation had arisen from a lack of evidence or as an example of the triumph of marketing over science. Undercitation of previous research is a particularly egregious problem when the selection of previous studies is biased [5]. Initiatives, such as trial registration and publication based on study quality rather than novelty, are designed to reduce publication bias, which can also be detected by meta-analytic techniques. Systematic review and meta-analysis of the randomized controlled trials (RCTs) giving evidence concerning the cardiovascular effects of testosterone has already identified missing trials favoring the placebo [6]. Citation bias favoring significant results has been reported across the field of medical research [5], however, little is known about its existence in the specific topic of testosterone and cardiovascular disease (CVD). Here, given the population health impact we assessed whether citation bias existed in studies concerning testosterone and CVD. ⁎ Corresponding author at: School of Urban Public Health, Hunter College, CUNY School of Public Health, New York, NY, USA. E-mail address: [email protected] (C.M. Schooling).
http://dx.doi.org/10.1016/j.ijcard.2015.03.010 0167-5273/© 2015 Elsevier Ireland Ltd. All rights reserved.
1. Methods To provide a comprehensive assessment, we compared by direction and significance the citation counts of two systematic reviews and meta-analyses of trials summarizing the effects of testosterone on CVD events [6,7] and of all prospective studies (18 studies) included in a recent meta-analysis of observational studies showing the association of testosterone with CVD [8]. Citations per year up until Dec 9th 2014 and the latest journal impact factors were obtained from the ISI Web of Science database. To account for the effect of publication time, a generalized linear model (GLM) adjusted for publication year was used to assess whether the direction and significance of findings was associated with citation counts. The analysis was conducted using Stata 13.1 (StataCorp LP, College Station, Texas). 2. Results The meta-analysis showing no significantly increased risk of selected cardiovascular outcomes on testosterone therapy [7], had more citations per year than the meta-analysis showing testosterone increased the risk of cardiovascular-related events [6] (Table 1). The two metaanalyses were published in journals with similar impact factors (Table 1). Among the 18 studies included in a meta-analysis of observational studies [8], 6 showed testosterone inversely associated with CVD (references 13 and 15–19 in the meta-analysis) and 12 showed no association (references 2–12 and reference 14 in the meta-analysis). Studies showing an inverse association (i.e., potentially beneficial effects of testosterone) were more often cited (Fig. 1). After adjustment for publication year, studies showing an inverse association of testosterone with cardiovascular disease had 12.2 more citations per year (95% confidence interval 0.4 to 24.0). 3. Discussion Allowing for publication time, we found that a meta-analysis of trials indicating no effect of testosterone on cardiovascular outcomes [7] was more often cited than one that showed increased risk of cardiovascularrelated events on testosterone [6]. We also found that observational studies showing an inverse association of testosterone with CVD events were more often cited than those showing no association. Not only are these findings internally inconsistent, when authors might be expected to place most weight on evidence showing consistency across study designs, but they also suggest citation bias.
J. Zhao, C.M. Schooling / International Journal of Cardiology 184 (2015) 510–511
511
Table 1 Citations, journal impact factor, and effect size for meta-analysis of testosterone and CVD events, 2010–2013. First author, year
Average citations per year
Journal (impact factor)
Effect size (OR or RR)
95% CI
Xu et al., 2013 [6] Fernandez-Balsells et al., 2010 [7]
25.0 37.2
BMC Med (7.3) J Clin Endocrinol Metab (6.3)
1.54 0.91
1.09, 2.18 0.29, 2.82
OR, odds ratio; RR, relative risk; CI, confidence interval.
men at risk, methods to detect and prevent citation bias are clearly urgently needed. Conflict of interest The authors report no relationships that could be construed as a conflict of interest. References
Fig. 1. Comparison of average citations per year for 18 prospective studies with beneficial versus non-beneficial results on testosterone and cardiovascular disease in healthy men, 1987–2009.
Usually, statistically significant findings are more cited than nonsignificant findings [5], but the citations for the two meta-analyses of trials do not corroborate this pattern. On the other hand, the observational studies showing an inverse association of testosterone with CVD were more often cited, even though these studies were at variance with the experimental evidence, and as such a poor guide to the effect of the relevant intervention, i.e., testosterone supplementation. As such, differences in citation counts due to citation for hypothesis refutation seem unlikely. Selective citation appears to be affecting knowledge construction concerning the role of testosterone in CVD. Testosterone supplementation is still being portrayed as a reasonable option for healthy older men [9], although the best evidence available suggests no benefit and potential harm [10]. Although, this is only one example, albeit with millions of
[1] AndroGel giving AbbVie performance anxietyAvailable from http://www. chicagobusiness.com/article/20141122/ISSUE01/311229982/androgel-givingabbvie-performance-anxiety. [2] Health Canada, Summary safety review—testosterone replacement productscardiovascular riskAvailable from http://www.hc-sc.gc.ca/dhp-mps/medeff/reviews-examens/testosterone-eng.php. [3] FDA advisory panel urges restrictions on testosterone useAvailable from http:// www.medscape.com/viewarticle/831897#vp_2. [4] European Medicines Agency, Available from http://www.ema.europa.eu/ema/index. jsp?curl=pages/news_and_events/news/2014/04/news_detail_002069.jsp&mid= WC0b01ac058004d5c1. [5] I. Chalmers, M.B. Bracken, B. Djulbegovic, et al., How to increase value and reduce waste when research priorities are set, Lancet 383 (2014) 156–165. [6] L. Xu, G. Freeman, B.J. Cowling, et al., Testosterone therapy and cardiovascular events among men: a systematic review and meta-analysis of placebo-controlled randomized trials, BMC Med. 11 (2013) 108. [7] M.M. Fernandez-Balsells, M.H. Murad, M. Lane, et al., Clinical review 1: adverse effects of testosterone therapy in adult men: a systematic review and meta-analysis, J. Clin. Endocrinol. Metab. 95 (2010) 2560–2575. [8] J.B. Ruige, A.M. Mahmoud, D. De Bacquer, et al., Endogenous testosterone and cardiovascular disease in healthy men: a meta-analysis, Heart 97 (2011) 870–875. [9] R. Swerdloff, B.D. Anawalt, Clinical decisions. Testosterone-replacement therapy, N. Engl. J. Med. 371 (2014) 2032–2034. [10] Advisory Committee Industry Briefing Document, Testosterone replacement therapyAvailable from http://www.fda.gov/downloads/AdvisoryCommittees/ CommitteesMeetingMaterials/Drugs/ReproductiveHealthDrugsAdvisoryCommittee/ UCM412537.pdf.
Contents lists available at ScienceDirect
International Journal of Cardiology journal homepage: www.elsevier.com/locate/ijcard
Letter to the Editor
Promotion of “Low T” and citation bias in testosterone studies Jie Zhao a, C. Mary Schooling a,b,⁎ a b
School of Public Health, Li Ka Shing Faculty of Medicine, The University of Hong Kong, Hong Kong Special Administrative Region School of Urban Public Health, Hunter College, CUNY School of Public Health, New York, NY, USA
a r t i c l e
i n f o
Article history: Received 16 January 2015 Accepted 1 March 2015 Available online 3 March 2015 Keywords: Testosterone Cardiovascular disease Citation bias
The “Low T” marketing campaign playing on older men's concerns about flagging masculinity has driven a dramatic increase in testosterone prescription in the last few years, particularly in North America. However, sales of testosterone are now falling again [1], as regulators, particularly Health Canada [2], have drawn attention to the cardiovascular risk of testosterone supplementation. Health Canada, the Food and Drug Administration and the European Medicines Agency have all advised that testosterone prescription should be restricted [2–4], consistent with the lack of established benefit and the possibility of putting millions of older men at risk of venous thromboembolism or heart attacks [2]. We wondered whether this situation had arisen from a lack of evidence or as an example of the triumph of marketing over science. Undercitation of previous research is a particularly egregious problem when the selection of previous studies is biased [5]. Initiatives, such as trial registration and publication based on study quality rather than novelty, are designed to reduce publication bias, which can also be detected by meta-analytic techniques. Systematic review and meta-analysis of the randomized controlled trials (RCTs) giving evidence concerning the cardiovascular effects of testosterone has already identified missing trials favoring the placebo [6]. Citation bias favoring significant results has been reported across the field of medical research [5], however, little is known about its existence in the specific topic of testosterone and cardiovascular disease (CVD). Here, given the population health impact we assessed whether citation bias existed in studies concerning testosterone and CVD. ⁎ Corresponding author at: School of Urban Public Health, Hunter College, CUNY School of Public Health, New York, NY, USA. E-mail address: [email protected] (C.M. Schooling).
http://dx.doi.org/10.1016/j.ijcard.2015.03.010 0167-5273/© 2015 Elsevier Ireland Ltd. All rights reserved.
1. Methods To provide a comprehensive assessment, we compared by direction and significance the citation counts of two systematic reviews and meta-analyses of trials summarizing the effects of testosterone on CVD events [6,7] and of all prospective studies (18 studies) included in a recent meta-analysis of observational studies showing the association of testosterone with CVD [8]. Citations per year up until Dec 9th 2014 and the latest journal impact factors were obtained from the ISI Web of Science database. To account for the effect of publication time, a generalized linear model (GLM) adjusted for publication year was used to assess whether the direction and significance of findings was associated with citation counts. The analysis was conducted using Stata 13.1 (StataCorp LP, College Station, Texas). 2. Results The meta-analysis showing no significantly increased risk of selected cardiovascular outcomes on testosterone therapy [7], had more citations per year than the meta-analysis showing testosterone increased the risk of cardiovascular-related events [6] (Table 1). The two metaanalyses were published in journals with similar impact factors (Table 1). Among the 18 studies included in a meta-analysis of observational studies [8], 6 showed testosterone inversely associated with CVD (references 13 and 15–19 in the meta-analysis) and 12 showed no association (references 2–12 and reference 14 in the meta-analysis). Studies showing an inverse association (i.e., potentially beneficial effects of testosterone) were more often cited (Fig. 1). After adjustment for publication year, studies showing an inverse association of testosterone with cardiovascular disease had 12.2 more citations per year (95% confidence interval 0.4 to 24.0). 3. Discussion Allowing for publication time, we found that a meta-analysis of trials indicating no effect of testosterone on cardiovascular outcomes [7] was more often cited than one that showed increased risk of cardiovascularrelated events on testosterone [6]. We also found that observational studies showing an inverse association of testosterone with CVD events were more often cited than those showing no association. Not only are these findings internally inconsistent, when authors might be expected to place most weight on evidence showing consistency across study designs, but they also suggest citation bias.
J. Zhao, C.M. Schooling / International Journal of Cardiology 184 (2015) 510–511
511
Table 1 Citations, journal impact factor, and effect size for meta-analysis of testosterone and CVD events, 2010–2013. First author, year
Average citations per year
Journal (impact factor)
Effect size (OR or RR)
95% CI
Xu et al., 2013 [6] Fernandez-Balsells et al., 2010 [7]
25.0 37.2
BMC Med (7.3) J Clin Endocrinol Metab (6.3)
1.54 0.91
1.09, 2.18 0.29, 2.82
OR, odds ratio; RR, relative risk; CI, confidence interval.
men at risk, methods to detect and prevent citation bias are clearly urgently needed. Conflict of interest The authors report no relationships that could be construed as a conflict of interest. References
Fig. 1. Comparison of average citations per year for 18 prospective studies with beneficial versus non-beneficial results on testosterone and cardiovascular disease in healthy men, 1987–2009.
Usually, statistically significant findings are more cited than nonsignificant findings [5], but the citations for the two meta-analyses of trials do not corroborate this pattern. On the other hand, the observational studies showing an inverse association of testosterone with CVD were more often cited, even though these studies were at variance with the experimental evidence, and as such a poor guide to the effect of the relevant intervention, i.e., testosterone supplementation. As such, differences in citation counts due to citation for hypothesis refutation seem unlikely. Selective citation appears to be affecting knowledge construction concerning the role of testosterone in CVD. Testosterone supplementation is still being portrayed as a reasonable option for healthy older men [9], although the best evidence available suggests no benefit and potential harm [10]. Although, this is only one example, albeit with millions of
[1] AndroGel giving AbbVie performance anxietyAvailable from http://www. chicagobusiness.com/article/20141122/ISSUE01/311229982/androgel-givingabbvie-performance-anxiety. [2] Health Canada, Summary safety review—testosterone replacement productscardiovascular riskAvailable from http://www.hc-sc.gc.ca/dhp-mps/medeff/reviews-examens/testosterone-eng.php. [3] FDA advisory panel urges restrictions on testosterone useAvailable from http:// www.medscape.com/viewarticle/831897#vp_2. [4] European Medicines Agency, Available from http://www.ema.europa.eu/ema/index. jsp?curl=pages/news_and_events/news/2014/04/news_detail_002069.jsp&mid= WC0b01ac058004d5c1. [5] I. Chalmers, M.B. Bracken, B. Djulbegovic, et al., How to increase value and reduce waste when research priorities are set, Lancet 383 (2014) 156–165. [6] L. Xu, G. Freeman, B.J. Cowling, et al., Testosterone therapy and cardiovascular events among men: a systematic review and meta-analysis of placebo-controlled randomized trials, BMC Med. 11 (2013) 108. [7] M.M. Fernandez-Balsells, M.H. Murad, M. Lane, et al., Clinical review 1: adverse effects of testosterone therapy in adult men: a systematic review and meta-analysis, J. Clin. Endocrinol. Metab. 95 (2010) 2560–2575. [8] J.B. Ruige, A.M. Mahmoud, D. De Bacquer, et al., Endogenous testosterone and cardiovascular disease in healthy men: a meta-analysis, Heart 97 (2011) 870–875. [9] R. Swerdloff, B.D. Anawalt, Clinical decisions. Testosterone-replacement therapy, N. Engl. J. Med. 371 (2014) 2032–2034. [10] Advisory Committee Industry Briefing Document, Testosterone replacement therapyAvailable from http://www.fda.gov/downloads/AdvisoryCommittees/ CommitteesMeetingMaterials/Drugs/ReproductiveHealthDrugsAdvisoryCommittee/ UCM412537.pdf.
