American Journal of Epidemiology © The Author 2016. Published by Oxford University Press on behalf of the Johns Hopkins Bloomberg School of Public Health. All rights reserved. For permissions, please e-mail: [email protected]
Vol. 184, No. 7
Letters to the Editor RE: “RECEIVER OPERATING CHARACTERISTIC CURVE INFERENCE FROM A SAMPLE WITH A LIMIT OF DETECTION” where d ⁎ = d /β, h (t ) = t α− 1 exp (−t ) /Γ(α) and Γ(α ; d)=
Biomarkers with a limit of detection (LOD) are commonly encountered in clinical studies (i.e., interleukin-18 or kidney injury molecule-1 for kidney research) (1, 2). To assess the discrimination ability of biomarkers for disease diagnosis, the area under the receiver operating characteristic curve (AUC) is usually explored. In the presence of an LOD for biomarkers, the simplest way to determine the discriminatory ability is to delete the data below the LOD or impute the data with a value less than or equal to the LOD. Results from the literature have shown that these ad hoc methods may lead to biased estimates or may overestimate the precision (3, 4). Thus, investigators need to consider the use of advanced methods to achieve valid inference, and several studies based on the maximum likelihood approach have shown that it performs well in such situations (5, 6). In the article “Receiver Operating Characteristic Curve Inference From a Sample With a Limit of Detection” by Perkins et al. (6), the authors utilized maximum likelihood techniques and investigated the asymptotic properties of the estimators of the AUC for biomarkers with normal and gamma distributions. However, we question the theoretical derivation of the covariance matrices, and the Fisher information matrices of θˆ for both scenarios with normal and gamma distributions are incorrect (7, 8). To ensure the validity of this approach in future applications, it is crucial to correct those errors. Based on our derivation, for biomarkers from a normal distribution with the parameters θ = (μ, σ)T , the Fisher information matrix is
(
(
) )
(
ηφ(η) q
2
φ(η) 1 + η +
(
2
2p + ηφ(η) 1 + η +
)
ηφ(η) q
)
ments of the matrix have errors. In the Web Material (available at http://aje.oxfordjournals. org/), we provide the details. Web Appendix 1 contains a general formula for the AUC estimator under the exponential family distributions. Web Appendix 2 shows the asymptotic variances of the AUC estimator for the cases with both normally and gamma-distributed biomarkers. In Web Appendix 3 and Web Table 1, we show partial simulation results based on our derived formulas using the same set-ups that were used in the article by Perkins et al. (6). We have also included the coding for the R program for the cases with both normally and gamma-distributed biomarkers.
ACKNOWLEDGMENTS The work of M.W. was supported by the start-up funding from the Department of Public Health Sciences at Pennsylvania State University and by the National Center for Advancing Translational Sciences (grant KL2 TR000126). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. Conflict of interest: none declared.
REFERENCES
⎞ ⎟ ⎟, ⎟⎟ ⎠
1. Vaidya VS, Waikar SS, Ferguson MA, et al. Urinary biomarkers for sensitive and specific detection of acute kidney injury in humans. Clin Transl Sci. 2008;1(3):200–208. 2. Siew ED, Ware LB, Ikizler TA. Biological markers of acute kidney injury. J Am Soc Nephrol. 2011;22(5):810–820. 3. Domthong U, Parikh CR, Kimmel PL, et al. Assessing the agreement of biomarker data in the presence of left-censoring. BMC Nephrol. 2014;15:144. 4. Vexler A, Liu A, Eliseeva E, et al. Maximum likelihood ratio tests for comparing the discriminatory ability of biomarkers subject to limit of detection. Biometrics. 2008;64(3):895–903. 5. Hornung RW, Reed LD. Estimation of average concentration in the presence of nondetectable values. Appl Occup Environ Hyg. 1990;5(1):46–51. 6. Perkins NJ, Schisterman EF, Vexler A. Receiver operating characteristic curve inference from a sample with a limit of detection. Am J Epidemiol. 2007;165(3):325–333. 7. Gupta AK. Estimation of the mean and standard deviation of a normal population from a censored sample. Biometrika. 1952; 39(3–4):260–273.
d−μ
where η = σ , q = Φ(η) and p = 1 − Φ(η). In the original paper, the off-diagonal entry is not correct and the term of 12 is missing. On the other hand, for biomarkers from a σ gamma distribution with the parameters θ = (α , β)T , the Fisher information matrix is ⎞ ⎛ d2 1⎡ d ⁎h (d ⁎) ⎟ ⎜ 2 [log (Γ(α))] ⎢p + q β⎣ ⎟ ⎜ dα ⎟ ⎜ Γ(α; d ⁎)Γ′′ (α; d ⁎) − Γ′2 (α; d ⁎) Γ′ (α; d ⁎) ⎤ ⁎ × (q log (d ) − Γ(α) ) ⎦ 2 ⎟ ⎜− q Γ (α) ⎟, I (θ) = ⎜ ⁎ ⁎ ⁎ ⎡ 1 2(Γ(α + 1) − Γ(α + 1;d ) ⎟ ⎜ 1 ⎡ p + d h (d ) −αp + 2 ⎣ Γ(α) ⎟ ⎜ β ⎣⎢ q β ⎟ ⎜ ⁎ ⁎ ⁎ ⎤ Γ′ (α; d ) ⎤ d h (d ) ⁎ ⁎ ⁎ ⁎ ⎜ × (q log (d ) − )⎦ − q ( (α + 1 − d ) q − d h (d ) ) ⎥⎦ ⎟ Γ(α) ⎠ ⎝
552
Am J Epidemiol. 2016;184(7):552–553
Downloaded from http://aje.oxfordjournals.org/ at Serials Dept / Cornell University Medical Library on October 24, 2016
⎛ φ(η) p + φ(η) η + q 1 ⎜ I (θ) = 2 ⎜ σ ⎜⎜ φ(η) 1 + η2 + ηφ(η) q ⎝
d
∫0 t α− 1exp (−t ) dt. In the original paper, both diagonal ele-
Letters to the Editor 553 1
8. Harter HL, Moore AH. Asymptotic variances and covariances of maximum-likelihood estimators, from censored samples, of the parameters of Weibull and gamma populations. Ann Math Stat. 1967;38(2):557–570.
Department of Public Health Sciences, College of Medicine, Pennsylvania State University, Hershey, PA 2 Department of Medicine, University of Texas Health Science Center at San Antonio, San Antonio, TX
Ming Wang1, Zheng Li1, and Brian Reeves2 (e-mail: [email protected])
DOI: 10.1093/aje/kww090; Advance Access publication: September 12, 2016
Downloaded from http://aje.oxfordjournals.org/ at Serials Dept / Cornell University Medical Library on October 24, 2016
Am J Epidemiol. 2016;184(7):552–553
Vol. 184, No. 7
Letters to the Editor RE: “RECEIVER OPERATING CHARACTERISTIC CURVE INFERENCE FROM A SAMPLE WITH A LIMIT OF DETECTION” where d ⁎ = d /β, h (t ) = t α− 1 exp (−t ) /Γ(α) and Γ(α ; d)=
Biomarkers with a limit of detection (LOD) are commonly encountered in clinical studies (i.e., interleukin-18 or kidney injury molecule-1 for kidney research) (1, 2). To assess the discrimination ability of biomarkers for disease diagnosis, the area under the receiver operating characteristic curve (AUC) is usually explored. In the presence of an LOD for biomarkers, the simplest way to determine the discriminatory ability is to delete the data below the LOD or impute the data with a value less than or equal to the LOD. Results from the literature have shown that these ad hoc methods may lead to biased estimates or may overestimate the precision (3, 4). Thus, investigators need to consider the use of advanced methods to achieve valid inference, and several studies based on the maximum likelihood approach have shown that it performs well in such situations (5, 6). In the article “Receiver Operating Characteristic Curve Inference From a Sample With a Limit of Detection” by Perkins et al. (6), the authors utilized maximum likelihood techniques and investigated the asymptotic properties of the estimators of the AUC for biomarkers with normal and gamma distributions. However, we question the theoretical derivation of the covariance matrices, and the Fisher information matrices of θˆ for both scenarios with normal and gamma distributions are incorrect (7, 8). To ensure the validity of this approach in future applications, it is crucial to correct those errors. Based on our derivation, for biomarkers from a normal distribution with the parameters θ = (μ, σ)T , the Fisher information matrix is
(
(
) )
(
ηφ(η) q
2
φ(η) 1 + η +
(
2
2p + ηφ(η) 1 + η +
)
ηφ(η) q
)
ments of the matrix have errors. In the Web Material (available at http://aje.oxfordjournals. org/), we provide the details. Web Appendix 1 contains a general formula for the AUC estimator under the exponential family distributions. Web Appendix 2 shows the asymptotic variances of the AUC estimator for the cases with both normally and gamma-distributed biomarkers. In Web Appendix 3 and Web Table 1, we show partial simulation results based on our derived formulas using the same set-ups that were used in the article by Perkins et al. (6). We have also included the coding for the R program for the cases with both normally and gamma-distributed biomarkers.
ACKNOWLEDGMENTS The work of M.W. was supported by the start-up funding from the Department of Public Health Sciences at Pennsylvania State University and by the National Center for Advancing Translational Sciences (grant KL2 TR000126). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. Conflict of interest: none declared.
REFERENCES
⎞ ⎟ ⎟, ⎟⎟ ⎠
1. Vaidya VS, Waikar SS, Ferguson MA, et al. Urinary biomarkers for sensitive and specific detection of acute kidney injury in humans. Clin Transl Sci. 2008;1(3):200–208. 2. Siew ED, Ware LB, Ikizler TA. Biological markers of acute kidney injury. J Am Soc Nephrol. 2011;22(5):810–820. 3. Domthong U, Parikh CR, Kimmel PL, et al. Assessing the agreement of biomarker data in the presence of left-censoring. BMC Nephrol. 2014;15:144. 4. Vexler A, Liu A, Eliseeva E, et al. Maximum likelihood ratio tests for comparing the discriminatory ability of biomarkers subject to limit of detection. Biometrics. 2008;64(3):895–903. 5. Hornung RW, Reed LD. Estimation of average concentration in the presence of nondetectable values. Appl Occup Environ Hyg. 1990;5(1):46–51. 6. Perkins NJ, Schisterman EF, Vexler A. Receiver operating characteristic curve inference from a sample with a limit of detection. Am J Epidemiol. 2007;165(3):325–333. 7. Gupta AK. Estimation of the mean and standard deviation of a normal population from a censored sample. Biometrika. 1952; 39(3–4):260–273.
d−μ
where η = σ , q = Φ(η) and p = 1 − Φ(η). In the original paper, the off-diagonal entry is not correct and the term of 12 is missing. On the other hand, for biomarkers from a σ gamma distribution with the parameters θ = (α , β)T , the Fisher information matrix is ⎞ ⎛ d2 1⎡ d ⁎h (d ⁎) ⎟ ⎜ 2 [log (Γ(α))] ⎢p + q β⎣ ⎟ ⎜ dα ⎟ ⎜ Γ(α; d ⁎)Γ′′ (α; d ⁎) − Γ′2 (α; d ⁎) Γ′ (α; d ⁎) ⎤ ⁎ × (q log (d ) − Γ(α) ) ⎦ 2 ⎟ ⎜− q Γ (α) ⎟, I (θ) = ⎜ ⁎ ⁎ ⁎ ⎡ 1 2(Γ(α + 1) − Γ(α + 1;d ) ⎟ ⎜ 1 ⎡ p + d h (d ) −αp + 2 ⎣ Γ(α) ⎟ ⎜ β ⎣⎢ q β ⎟ ⎜ ⁎ ⁎ ⁎ ⎤ Γ′ (α; d ) ⎤ d h (d ) ⁎ ⁎ ⁎ ⁎ ⎜ × (q log (d ) − )⎦ − q ( (α + 1 − d ) q − d h (d ) ) ⎥⎦ ⎟ Γ(α) ⎠ ⎝
552
Am J Epidemiol. 2016;184(7):552–553
Downloaded from http://aje.oxfordjournals.org/ at Serials Dept / Cornell University Medical Library on October 24, 2016
⎛ φ(η) p + φ(η) η + q 1 ⎜ I (θ) = 2 ⎜ σ ⎜⎜ φ(η) 1 + η2 + ηφ(η) q ⎝
d
∫0 t α− 1exp (−t ) dt. In the original paper, both diagonal ele-
Letters to the Editor 553 1
8. Harter HL, Moore AH. Asymptotic variances and covariances of maximum-likelihood estimators, from censored samples, of the parameters of Weibull and gamma populations. Ann Math Stat. 1967;38(2):557–570.
Department of Public Health Sciences, College of Medicine, Pennsylvania State University, Hershey, PA 2 Department of Medicine, University of Texas Health Science Center at San Antonio, San Antonio, TX
Ming Wang1, Zheng Li1, and Brian Reeves2 (e-mail: [email protected])
DOI: 10.1093/aje/kww090; Advance Access publication: September 12, 2016
Downloaded from http://aje.oxfordjournals.org/ at Serials Dept / Cornell University Medical Library on October 24, 2016
Am J Epidemiol. 2016;184(7):552–553
