ALCOHOLISM: CLINICAL AND EXPERIMENTAL RESEARCH
Vol. 37, No. 12 December 2013
Commentary: Indefensible Methods of Handling Missing Data in Clinical Trials Stephan Arndt
Background: Literature continues to appear using inappropriate statistical methods when dealing with missing data in treatment trials. I regularly see last observation carried forward, assumed worstcase scenario, or some other static imputation method still being submitted and published in journals. Methods: We briefly cover a few reasons why the use of more modern and defensible methods may not have completely saturated the literature. Results: While some delay in complete permeation of appropriate methods is understandable, we are currently past the point for reasonable delay. Conclusions: Editors and reviewers should demand appropriate statistical methods in published literature. Key Words: Missing Data, Alcohol Use Disorder, Relapse, Treatment, Clinical Trials.
T
HE PAPER BY Hallgren and Witkiewitz (2013) provides a service for research in studies of alcoholism and its treatment. There are previous reports in the literature, several of which are referenced in Hallgren and Witkiewitz attending to the same issue, how to best deal with missing data. Unfortunately, the problem remains an issue. Literature continues to appear using inappropriate statistical methods when dealing with missing data. I regularly see last observation carried forward, assumed worst-case scenario, or some other static imputation method still being submitted and published in journals. So, while the issue addressed by Hallgren and Witkiewitz is not new, the continued problem suggests that more researchers, reviewers, editors, and funders need to be told more often to use appropriate methods. Donald Rubin’s and Roderick J. A. Little’s early work in missing data (Little and Rubin, 1987; Rubin, 1976, 1987) introduced us to several new concepts. Actually, the concepts were not totally new, but these authors defined the problems well and offered a tightly integrated solution. We attribute them with the nomenclature we now use to describe missing data, that is, missing completely at random (MCAR), missing at random, and not missing at random. While this was and is important, a perhaps more important result from these From the Department of Psychiatry (SA), Carver College of Medicine, University of Iowa, Iowa City, Iowa; Department of Biostatistics (SA), College of Public Health, University of Iowa, Iowa City, Iowa; and Iowa Consortium for Substance Abuse Research and Evaluation (SA), University of Iowa, Iowa City, Iowa. Received for publication September 1, 2013; accepted September 7, 2013. Reprint requests: Stephan Arndt, PhD, Iowa Consortium for Substance Abuse Research and Evaluation, University of Iowa, 100 MTP4, Iowa City, IA 52240-5000; Tel.: 319-335-4921; Fax: 319-3354484; E-mail: [email protected] Copyright © 2013 by the Research Society on Alcoholism. DOI: 10.1111/acer.12294 Alcohol Clin Exp Res, Vol 37, No 12, 2013: pp 1997–1998
works has to do with recognizing the uncertainty caused by missing data. Their introduction of the multiple imputation (MI) technique mechanistically estimates the uncertainty of the missing data imputation and, literally, adds that error back into consideration to generate a final test statistic. Maximum likelihood (ML) approaches share a similar timeline, being introduced in the 70s (e.g., Dempster et al., 1977). They have several current methods depending on the type of data and the originator, for example, mixed models, random regression, generalized estimating equations, and full model ML. These methods base their estimates of effect size and error on the information at hand. There is no need for complete data or imputation. These methods enjoy widespread use in a variety of software packages, for example SAS (SAS Institute, Cary, NC), SPSS (IBM, Armonk, NY), STATA (StataCorp, College Station, TX), and R (R Foundation for Statistical Computing, Vienna, Austria), and appear in multiple disciplines and journals. As these approaches were developed so long ago, why are we still seeing inappropriate methods being used, especially when they have repeatedly been shown to be biased? Understandably, new statistical techniques need to be tested, proven robust, and valid before they become widely accepted. Again, however, an extensive literature has shown these newer methods to be less biased (or unbiased), robust, and more valid than the early static imputation methods. Hallgren and Witkiewitz (2013) give a brief literature review in their introduction. Widespread availability of software had something to do with the delay in implementation. However, many statistical packages began implementing MI and ML methods in the 1980s and 1990s. Similarly, these newer methods would have been taught to younger researchers about the time the software came out, that is, over 20 years ago. Clinical trial application sometimes imposes an additional delay for the adoption of methodology. Frequently, researchers specify the statistical analyses before the start of 1997
COMMENTARY
1998
a study that might take 3 to 5 years to complete. Investigators may still feel obliged to use the prespecified analysis even after obtaining the data and discovering that the initial analysis choice is no longer defensible by current standards. All these factors might explain a 10 to 15 year delay in these methods, usurping the simplistic and biased static imputation methods. So, why might these issues still exist today? To be sure, there is a little more work necessary using the more appropriate methods. In the case of the likelihood methods, the analyst must review covariates that may predict whether data are missing, as these must be included in the model for the MCAR assumption to hold. In the case of MI, extra steps are added to the analysis. Again, covariates might be selected to improve the efficiency of the imputations. Yet, this extra work is really on the order of an extra few hours or days. This extra work is really a small price to pay after months or years of data collection. Researchers may hold onto the older static method because of preconceived beliefs about the nature of the missing data (Arndt, 2009). For example, the worst-case scenario imputation assumes that the person in an alcohol treatment study missed a visit or dropped out because of a relapse. Alternatively, researchers may naively assume that the static imputation method is more conservative. This is naive because the static imputation does more than just affect the individual’s or the group mean score. It affects the standard error and creates a misapprehension of larger degrees of freedom. The static imputation also pays no mind to the inherent uncertainty surrounding the missing
value. Finally, there may be some very rare instances where some form of static method might be arguably appropriate (MCAR), but it is difficult to see those situations in realworld settings and even more difficult to provide evidence that the assumptions hold. Whatever the causes of continued use of static imputation methods, such as mean replacement, last observation carried forward, or worst-case scenario, it is time to move on. These methods are no longer defensible and should almost never be used. Authors should detail what method was used for missing data, if there are any. Sensitivity analyses, repeating the analysis with more than one kind of modern method, should be encouraged. Reviewers and editors should expect this and raise issues with inappropriate or indefensible methods, if not simply create an editorial policy. REFERENCES Arndt S (2009) Stereotyping and the treatment of missing data for drug and alcohol clinical trials. Subst Abuse Treat Prev Policy 4: 2. Dempster AP, Laird NM, Rubin DB (1977) Maximum likelihood from incomplete data via the EM algorithm. J R Stat Soc Series B (Methodological) 39:1–38. Hallgren KA, Witkiewitz K (2013) Missing data in alcohol clinical trials: a comparison of methods. Alcohol Clin Exp Res doi: 10.1111/acer.12205. [Epub ahead of print]. Little RJA, Rubin DB (1987) Statistical Analysis with Missing Data. John Wiley & Sons, New York, NY. Rubin DB (1976) Inference and missing data. Biometrika 63:581–592. Rubin DR (1987) Multiple Imputation for Nonresponse in Surveys. John Wiley & Sons, New York, NY.
Copyright of Alcoholism: Clinical & Experimental Research is the property of WileyBlackwell and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
Vol. 37, No. 12 December 2013
Commentary: Indefensible Methods of Handling Missing Data in Clinical Trials Stephan Arndt
Background: Literature continues to appear using inappropriate statistical methods when dealing with missing data in treatment trials. I regularly see last observation carried forward, assumed worstcase scenario, or some other static imputation method still being submitted and published in journals. Methods: We briefly cover a few reasons why the use of more modern and defensible methods may not have completely saturated the literature. Results: While some delay in complete permeation of appropriate methods is understandable, we are currently past the point for reasonable delay. Conclusions: Editors and reviewers should demand appropriate statistical methods in published literature. Key Words: Missing Data, Alcohol Use Disorder, Relapse, Treatment, Clinical Trials.
T
HE PAPER BY Hallgren and Witkiewitz (2013) provides a service for research in studies of alcoholism and its treatment. There are previous reports in the literature, several of which are referenced in Hallgren and Witkiewitz attending to the same issue, how to best deal with missing data. Unfortunately, the problem remains an issue. Literature continues to appear using inappropriate statistical methods when dealing with missing data. I regularly see last observation carried forward, assumed worst-case scenario, or some other static imputation method still being submitted and published in journals. So, while the issue addressed by Hallgren and Witkiewitz is not new, the continued problem suggests that more researchers, reviewers, editors, and funders need to be told more often to use appropriate methods. Donald Rubin’s and Roderick J. A. Little’s early work in missing data (Little and Rubin, 1987; Rubin, 1976, 1987) introduced us to several new concepts. Actually, the concepts were not totally new, but these authors defined the problems well and offered a tightly integrated solution. We attribute them with the nomenclature we now use to describe missing data, that is, missing completely at random (MCAR), missing at random, and not missing at random. While this was and is important, a perhaps more important result from these From the Department of Psychiatry (SA), Carver College of Medicine, University of Iowa, Iowa City, Iowa; Department of Biostatistics (SA), College of Public Health, University of Iowa, Iowa City, Iowa; and Iowa Consortium for Substance Abuse Research and Evaluation (SA), University of Iowa, Iowa City, Iowa. Received for publication September 1, 2013; accepted September 7, 2013. Reprint requests: Stephan Arndt, PhD, Iowa Consortium for Substance Abuse Research and Evaluation, University of Iowa, 100 MTP4, Iowa City, IA 52240-5000; Tel.: 319-335-4921; Fax: 319-3354484; E-mail: [email protected] Copyright © 2013 by the Research Society on Alcoholism. DOI: 10.1111/acer.12294 Alcohol Clin Exp Res, Vol 37, No 12, 2013: pp 1997–1998
works has to do with recognizing the uncertainty caused by missing data. Their introduction of the multiple imputation (MI) technique mechanistically estimates the uncertainty of the missing data imputation and, literally, adds that error back into consideration to generate a final test statistic. Maximum likelihood (ML) approaches share a similar timeline, being introduced in the 70s (e.g., Dempster et al., 1977). They have several current methods depending on the type of data and the originator, for example, mixed models, random regression, generalized estimating equations, and full model ML. These methods base their estimates of effect size and error on the information at hand. There is no need for complete data or imputation. These methods enjoy widespread use in a variety of software packages, for example SAS (SAS Institute, Cary, NC), SPSS (IBM, Armonk, NY), STATA (StataCorp, College Station, TX), and R (R Foundation for Statistical Computing, Vienna, Austria), and appear in multiple disciplines and journals. As these approaches were developed so long ago, why are we still seeing inappropriate methods being used, especially when they have repeatedly been shown to be biased? Understandably, new statistical techniques need to be tested, proven robust, and valid before they become widely accepted. Again, however, an extensive literature has shown these newer methods to be less biased (or unbiased), robust, and more valid than the early static imputation methods. Hallgren and Witkiewitz (2013) give a brief literature review in their introduction. Widespread availability of software had something to do with the delay in implementation. However, many statistical packages began implementing MI and ML methods in the 1980s and 1990s. Similarly, these newer methods would have been taught to younger researchers about the time the software came out, that is, over 20 years ago. Clinical trial application sometimes imposes an additional delay for the adoption of methodology. Frequently, researchers specify the statistical analyses before the start of 1997
COMMENTARY
1998
a study that might take 3 to 5 years to complete. Investigators may still feel obliged to use the prespecified analysis even after obtaining the data and discovering that the initial analysis choice is no longer defensible by current standards. All these factors might explain a 10 to 15 year delay in these methods, usurping the simplistic and biased static imputation methods. So, why might these issues still exist today? To be sure, there is a little more work necessary using the more appropriate methods. In the case of the likelihood methods, the analyst must review covariates that may predict whether data are missing, as these must be included in the model for the MCAR assumption to hold. In the case of MI, extra steps are added to the analysis. Again, covariates might be selected to improve the efficiency of the imputations. Yet, this extra work is really on the order of an extra few hours or days. This extra work is really a small price to pay after months or years of data collection. Researchers may hold onto the older static method because of preconceived beliefs about the nature of the missing data (Arndt, 2009). For example, the worst-case scenario imputation assumes that the person in an alcohol treatment study missed a visit or dropped out because of a relapse. Alternatively, researchers may naively assume that the static imputation method is more conservative. This is naive because the static imputation does more than just affect the individual’s or the group mean score. It affects the standard error and creates a misapprehension of larger degrees of freedom. The static imputation also pays no mind to the inherent uncertainty surrounding the missing
value. Finally, there may be some very rare instances where some form of static method might be arguably appropriate (MCAR), but it is difficult to see those situations in realworld settings and even more difficult to provide evidence that the assumptions hold. Whatever the causes of continued use of static imputation methods, such as mean replacement, last observation carried forward, or worst-case scenario, it is time to move on. These methods are no longer defensible and should almost never be used. Authors should detail what method was used for missing data, if there are any. Sensitivity analyses, repeating the analysis with more than one kind of modern method, should be encouraged. Reviewers and editors should expect this and raise issues with inappropriate or indefensible methods, if not simply create an editorial policy. REFERENCES Arndt S (2009) Stereotyping and the treatment of missing data for drug and alcohol clinical trials. Subst Abuse Treat Prev Policy 4: 2. Dempster AP, Laird NM, Rubin DB (1977) Maximum likelihood from incomplete data via the EM algorithm. J R Stat Soc Series B (Methodological) 39:1–38. Hallgren KA, Witkiewitz K (2013) Missing data in alcohol clinical trials: a comparison of methods. Alcohol Clin Exp Res doi: 10.1111/acer.12205. [Epub ahead of print]. Little RJA, Rubin DB (1987) Statistical Analysis with Missing Data. John Wiley & Sons, New York, NY. Rubin DB (1976) Inference and missing data. Biometrika 63:581–592. Rubin DR (1987) Multiple Imputation for Nonresponse in Surveys. John Wiley & Sons, New York, NY.
Copyright of Alcoholism: Clinical & Experimental Research is the property of WileyBlackwell and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
