Psychological Reports: Measures & Statistics 2013, 112, 3, 835-844. © Psychological Reports 2013
ALPHA VALUES AS A FUNCTION OF SAMPLE SIZE, EFFECT SIZE, AND POWER: ACCURACY OVER INFERENCE1 M. T. BRADLEY
A. BRAND
University of New Brunswick
Kings College: Institute of Psychiatry
Summary.—Tables of alpha values as a function of sample size, effect size, and desired power were presented. The tables indicated expected alphas for small, medium, and large effect sizes given a variety of sample sizes. It was evident that sample sizes for most psychological studies are adequate for large effect sizes defined at .8. The typical alpha level of .05 and desired power of 90% can be achieved with 70 participants in two groups. It was perhaps doubtful if these ideal levels of alpha and power have generally been achieved for medium effect sizes in actual research, since 170 participants would be required. Small effect sizes have rarely been tested with an adequate number of participants or power. Implications were discussed.
Power, the probability to report effects from experimental manipulations when effects should be present, is influenced by sample size, alpha levels, and the power criterion itself. Large true effect sizes, everything else being equal, are tested at higher power than small effect sizes. Large sample sizes and more liberal alpha levels increase the likelihood of accepting any effect size as a significant difference. Willingness to accept a higher probability of Type II errors (lower power), either tacitly or explicitly, may determine whether or not a study will be conducted. Thus, there are several potential considerations related to the decision to conduct a study as well as to understand the results in a meaningful fashion. These considerations are regardless of whether the guiding approach is from Neyman-Pearson, Fisher, or a hybrid of the two perspectives (Gigerenzer, 1993). Traditional alpha or p > .05 is often where the line between statistical significance and non-significance is drawn, and this level, in turn, influences what is published or what is not. Ultimately, effect size estimates that are published are the ones available to researchers. If those effectsize estimates are distorted through possible but improbable Type I errors, problems are created. Type I errors occur when results of statistical tests in a probabilistic sense should not yield significance but do. Type II errors occur when tests should in a probabilistic sense find significance but do not. This paper examines alpha levels associated with different Ns, effect sizes, and Type II error levels to illustrate the alpha values expected when traditional significance and power levels may not be met. Address correspondence to Michael T. Bradley, Department of Psychology, University of New Brunswick, P.O. Box 5050, Tucker Park Road, Saint John, NB, Canada, E2L 4L5 or e-mail ([email protected]). 1
DOI 10.2466/03.49.PR0.112.3.835-844
13_PR_Bradley_FMS_051013_130014.indd 835
ISSN 0033-2941
21/08/13 5:32 PM
836
M. T. BRADLEY & A. BRAND
The recommended approach to increasing power is to increase sample size. Statistical and experimental design texts (Kraemer & Theimann, 1987) devote space to calculating a sample size to give reasonable power to reject the null hypothesis at the .05 level given a certain estimated effect size. This approach has been advocated and widely available since Cohen (1962). The difficulty is that even with that clear recommendation and the availability of methods, sample sizes have not markedly increased over the last 60 years (Marszalek, Barber, Kohlhart, & Holmes, 2011). In fact, Marszalek, et al. (2011) focused on four leading APA journals documenting sample size for the years 1955, 1977, 1995 and 2006. The earliest year, 1955, is before Cohen’s (1962) lucid description of often inadequate sample size for small and medium effects, and his prescription amounting to a request for larger sample sizes. The later years follow Cohen’s message, and the final year is after publication of the APA’s Task Force on Statistical Inference (Wilkinson & The Task Force on Statistical Inference, 1999) that also described problems with inadequate sample size. Further, during the years included in the study, other authors, again perhaps inspired by Cohen, have commented on the issue of inadequate power (Maddock & Rossi, 2001; Maxwell, 2004). In general, there has been ample opportunity to appreciate the importance of the problem, but little has changed during this period. There are possible explanations why sample sizes have not increased. It is possible that psychologists believe power is adequate for their various studies. A common routine, although not common enough, is to calculate power based on effect sizes generated in published studies. Researchers, however, may be misled by these reported values. Brand, Bradley, Best, and Stoica, (2008) showed that literature published under .05 significance criteria would yield estimates exaggerated up to 190% for a small effect size investigated with the typical numbers of measures or participants used in psychological studies. Brand, et al. (2008) looked at the average of effect sizes that were significant in a Monte Carlo study when the effect size between two groups of 38 participants each was set at .2, .5, or .8 and statistical significance was accepted at the .05 level. The exaggeration of the values associated with statistical significance and therefore publication would mislead readers and suggest that a true small effect size (.20) is at least a medium effect size of .58. In addition, these small effect sizes would give a statistically significant result for 14% of the studies conducted. Thus, given a large number of psychologists examining hypotheses, a 14% success rate could create a large literature, but unfortunately it is a literature promoting exaggerated estimates. It is worth noting that Brand, et al. (2008) concentrated on point estimations, but these estimations lie on a continuum. Thus, these estimations of exaggeration of effect sizes vary, but they follow very basic rules.
13_PR_Bradley_FMS_051013_130014.indd 836
21/08/13 5:32 PM
ALTERNATIVE ALPHA LEVELS
837
The distortions or effect size exaggerations go up as true effect size diminishes. The mitigating factor is the probability of obtaining the exaggeration associated with statistical significance goes down, ultimately to 5%, as the true underlying effect size decreases. In a similar way, the reciprocal is true. As the effect size increases, the exaggeration diminishes but the chance of reporting an exaggeration increases. Thus, it is possible that a literature of published effect size exaggerations gives a false impression that significance tests are being conducted at adequate power and, of course, some percentage of the 125,000 North American psychologists, plus their European confreres, and their students, publish further exaggerated estimates through replication of erroneous results. Adding to the general confusion is that usual sample sizes may give the impression that they yield adequate power for the hypotheses that psychologists test, because this is actually the case for large effect sizes (0.80; Cohen, 1977). It, however, is demonstrably not true for Cohen’s medium (.50) and small (0.20) effect sizes. In fact, Maddock and Rossi (2001) reported the mean power to be 36% for the 1997 volumes of three health psychology journals. Published effect sizes exaggerated by including only statistically significant estimates in the calculations may create a variety of difficulties. For example, there could be an effect on obtaining grants. Social science grants are often small, but a researcher may believe from calculations that the small grant provides enough support to do an adequate study. In truth, however, the researcher may fail to include an appropriate number of participants in a study. Reviewers of the grant application may limit the grant amounts because they also calculate power on the published literature and reinforce the belief that what is actually an inadequate sample size is sufficient. The probable result of such an exercise is non significance, failure to publish, and a waste of the funding. Related to the sample size problem is that available participants for any given study might be limited. Maxwell (2004) suggested multisite studies to deal with this, but of course, this requires contacting and inspiring interest in other researchers who might be busy with their own projects. There is some possible balance to this process, but it is slow and cumbersome. The majority of conducted studies will not, of course, support past results and researchers may gradually drop out of a particular area of investigation leaving only the serially lucky to reproduce results. As Maxwell (2004) suggested, the process of correction would be quicker if the thousands of statistically non-significant results could be noted in the published literature, but in current practice such non-significant results are rarely published, and they have little or no effect on any given literature (Iaonnidis, 2005). It is appropriate to note here that there is an exception to the general rule that non-significant results are not reported.
13_PR_Bradley_FMS_051013_130014.indd 837
21/08/13 5:32 PM
838
M. T. BRADLEY & A. BRAND
The Journal of Articles in Support of the Null Hypothesis publishes exclusively statistically non-significant results. The situation described above creates a difficulty for the scientific development of psychology. A hallmark of science is accurate measurement. Chronically inaccurate measurements created by a publication process that presents exaggerated values have no place in science. It is no more justified to include exaggerated measures in reports, than it is to report results more readily explainable by chance than by an actual effect. Again, Maxwell (2004) suggested de-emphasizing significance and publishing more precise estimates. To our way of thinking, it is a question that a researcher in any given area has to approach with reason. If there is a suspicion that a given sample size does not afford enough power, then the researcher has to ask what is a reasonable alpha to suggest that the null hypothesis actually has been rejected. That reasoning has not been applied with the traditional and ever-present .05 level of statistical significance testing. The literature stretching back to Cohen shows that power with small and medium effect sizes and typical sample sizes is inadequate for the .05 level of significance. We believe that one route to achieve an appreciation of obtained results is to endorse Maxwell’s suggestion that traditional significance be de-emphasized. We describe what Type I error rates look like under various conditions a researcher might encounter. We also argue that the .05 level may be appropriate for some areas, but in other areas the consequences of a Type I error may not be serious, and psychologists should be able to read about accurate estimates of an effect. Our comments echo those of Christensen (2005), who summed up the situation by saying the focus on small alpha levels often leads to bad decisions with regards to both Type I and Type II errors. Since the most obvious and heavily advocated solution to the problem, the increase in sample size, has not yet been generally achieved, other ways are explored to allow researchers to consider Type I and Type II errors. Presented are tables for small, medium, and large effect sizes. These tables show the expected alpha level or probability that chance could offer an explanation to rival the experimentally hypothesized explanation for a finding given various sample sizes and power. The power is expressed in the reciprocal form of Type II error rates of .1, .2, and .3. If there were a limited number of participants available for testing a small effect size, it is not possible to obtain the .05 level of statistical significance and be accurate in estimates of effect size. The question then becomes what value of alpha would be reasonable given the estimated effect size, sample size, and power level? With that knowledge, a researcher could then determine if a result that is not statistically significant by traditional levels may actually be accurate in terms of the expected values such that an effect size
13_PR_Bradley_FMS_051013_130014.indd 838
21/08/13 5:32 PM
ALTERNATIVE ALPHA LEVELS
839
calculated from the study is actually a valuable contribution. As such, it would, at the least, be worth communicating to fellow scientists or pursuing as replicable if more participants became available. This approach, at the least, would encourage more pilot studies and places the results in a readily comprehendible context. METHOD A series of criterion power analyses were conducted to determine the alpha level, for ranges of standardized effect sizes (d), desired Type II error rate, and sample size specifications. The criterion power analyses used the statistical package R (R Development Core Team, 2009, http:// www.r-project.org/). Note that statistical power analysis packages such as GPower or nQuery Adviser could also be used to conduct these analyses. The criterion power analyses performed assumed that data would be analyzed using a two-tailed, independent t test where the number of subjects per condition was equal. The values used for the standardized effect sizes were based on Cohen’s (1977) benchmark for small (d = 0.20), medium (d = 0.50) and large (d = 0.80) effect sizes. The values used for the Type II error rates were 0.10, 0.20, and 0.30, which correspond to statistical power of 90%, 80%, and 70%, respectively. A set of nine different sample sizes was used for each of the three effect sizes. For the small effect (d = 0.20), the sample sizes were from 700 to 1,100 incrementing by 50. For the medium effect (d = 0.50), the sample sizes were from 100 to 180 incrementing by 10. For the large effect (d = 0.80), the sample sizes were from 30 to 110 incrementing by 10. The sample size ranges are not exhaustive and were chosen to be illustrative of results that a researcher might find of interest for practical reasons. The computed alphas for the three effect sizes are listed in Table 1. RESULTS Tables 1, 2, and 3 display the alpha levels expected for various sample sizes, over small, medium, and large effect sizes and three power levels. The numbers are straightforward but there are some notable features. The range of sample sizes giving reasonable alpha level estimates change with different effect sizes. Thus, for a small effect size, the tables start at 700 participants and go up by increments of 50 to reach 1,100 participants. For medium effect sizes, a range of 100 to 180 is reasonable with increments of 10. For large effect sizes, depending on the willingness to accept a Type II error, sample sizes might be as low as 30, but if sample size goes up to 70 the traditional ideal is achieved of p < .05 for a Type I error with p < .10 for a Type II error. The increments differ substantially in the tables. For a small effect size an increment of 50 is necessary to make a difference, whereas 10 is adequate for a medium effect size and very effective for a large effect size.
13_PR_Bradley_FMS_051013_130014.indd 839
21/08/13 5:32 PM
840
M. T. BRADLEY & A. BRAND TABLE 1 ALPHA VALUES AS A FUNCTION OF SAMPLE SIZE AND TYPE II ERROR RATE WHEN THE TRUE EFFECT SIZE IS SMALL (d = .2)
Type II Error Rates
700
750
800
850
Sample Size (N) 900
950
1000
1050
1100
0.10
0.173
0.146
0.122
0.103
0.086
0.072
0.060
0.051
0.042
0.20
0.072
0.058
0.047
0.038
0.031
0.025
0.021
0.017
0.014
0.30
0.034
0.027
0.021
0.017
0.013
0.011
0.008
0.007
0.005
Note.—All alpha values are reported to three decimal places.
TABLE 2 ALPHA VALUES AS A FUNCTION OF SAMPLE SIZE AND TYPE II ERROR RATE WHEN THE TRUE EFFECT SIZE IS MEDIUM (d = .5) Sample Size (N)
Type II Error Rates
100
110
120
130
140
150
160
170
180
0.10
0.226
0.184
0.148
0.120
0.097
0.078
0.062
0.050
0.040
0.20
0.101
0.078
0.061
0.047
0.036
0.028
0.022
0.017
0.013
0.30
0.051
0.038
0.029
0.022
0.016
0.012
0.009
0.007
0.005
Note.—All alpha values are reported to three decimal places.
TABLE 3 ALPHA VALUES AS A FUNCTION OF SAMPLE SIZE AND TYPE II ERROR RATE WHEN THE TRUE EFFECT SIZE IS LARGE (d = .8) Sample Size (N)
Type II Error Rates
30
40
50
60
70
80
90
100
110
0.10
0.369
0.221
0.130
0.076
0.044
0.025
0.014
0.008
0.005
0.20
0.188
0.100
0.053
0.028
0.015
0.008
0.004
0.002
0.001
0.30
0.106
0.052
0.026
0.013
0.006
0.003
0.002
0.001
0.000
Note.—All alpha values are reported to three decimal places.
DISCUSSION The present data show the expected alpha levels for various effect sizes, sample sizes, and power levels. The first thing that is evident is that the Ns (number of participants/measurements) differ considerably depending upon the expected effect size in a given area. It is worth pointing out that a study of a small effect size requires 1,050 participants to be both accurate and approximate statistical significance at the .05 level
13_PR_Bradley_FMS_051013_130014.indd 840
21/08/13 5:32 PM
ALTERNATIVE ALPHA LEVELS
841
(.051). Only an N of 70 is required to achieve the same alpha with a large effect size. Also, a medium sized effect can be tested with a fairly modest N of 170. Of course, difficulties abound with obtaining accurate effect size estimates given the current state of the literature, and we suspect that many hypotheses in cognitive and social psychology (Lipsey & Wilson, 1993; Richard, Bond, Jr., & Stokes-Zoota, 2003) have an effect size between .2 and .5. Depending upon which level they approach, the number of participants might have to be substantial. This being said, it must be realized that these tables portray the results for participants in two equal sized groups who each respond to a single measure. More complex designs may both reduce the divisor for the standard error and degrees of freedom for significance, and, as a consequence, make significance more difficult to obtain. On the other hand, multiple measures that are aggregated or averaged on each single individual may increase power (Brand, Bradley, Best, & Stoica, 2011). Overall, the tables serve as a rudimentary benchmark to allow a researcher to estimate beforehand what might be expected from an analysis involving specified N. We use the word “rudimentary” because each researcher has to deal with the peculiarities that pertain to her own research area. If researchers attend to the presented numbers they definitely may improve the estimates and their understanding of these estimates in the case of small effect size, and possibly also with medium effect sizes. Understanding data with these tables can go well beyond the common .05 level of significance. If there was reason to suspect that a research interest involved a small effect size, then for 800 measures an alpha of .122 would be the mean result obtained with a power level of 90%. With such a reporting style, the current practice of forgetting the findings in a “file drawer” could be curtailed. The researcher may decide there is something of interest and pursue collaboration, replication, or tolerate a greater chance for a Type II error to establish the finding. If, through some chance factor, a researcher obtained an alpha of .05 when that value should have been .122, the calculated effect size would be exaggerated and potentially misleading. Researchers would then simply warn others that the current results could be an exaggeration, and others who have close but statistically non-significant results by traditional standards are actually accurately measuring the phenomena and might have their replications recognized. In other words, a mean alpha of .122 for a small effect size would mean that 90% of the time this would be obtained and the effect size estimate would be accurate. It is perhaps forgotten that Fisher (1973) pointed out that insisting on a .05 level of significance showed a lack of statistical sophistication, and his point is well made with the tables presented here.
13_PR_Bradley_FMS_051013_130014.indd 841
21/08/13 5:32 PM
842
M. T. BRADLEY & A. BRAND
The move towards reporting effect sizes particularly highlights the problems with the current practice of a rigid statistical significance level governing whether a paper is published or not. Statistical significance at the .05 level is a state proposition, either an effect is present or it is not. Effect sizes are different. Effect sizes are about accurate measurement. The goal should be about the best estimate of an effect size. It may be a bit of an oxymoron to write of accurate measurement with variable data, but the goal is to get as close as we can to assembling a set of effect sizes that best reflect the phenomena under study. All or none inferences with exclusion of non-significant results do not allow this to be done, and worse, if a study is underpowered the effect size estimate must, by the mechanics of statistical formulae, be exaggerated if the study results are reported as statistically significant. To put it in stark terms, a researcher cannot have statistical significance and effect size accuracy if the study is underpowered. In the ideal situation a researcher would have the resources to have enough participants to allow for inference testing that did not distort reported values. However, as Marszalek, et al. (2011) have reported, sample size has not changed over 60 years. Therefore, a researcher has to be cognizant that it is not possible to obtain an alpha level of .05 and an accurate effect size estimate when the effect size is small and probably even when the effect size is medium. It is, however, very probable when the effect size is large. It is worth emphasizing the, perhaps unintended, role of the 1999 APA Task Force (Wilkinson, & Task Force, 1999) advocating the report of effect sizes along with significant tests. Effect sizes reflect measurement, and accurate measurement is at the core of science. As soon as effect sizes were featured it became obvious that chance factors associated with significance could result in the promulgation of effect size exaggerations–sometimes of a substantial nature (Brand, et al., 2008), especially when statistical significance is coupled with inadequate power. Cohen (1962), as mentioned, has been making this point since 1962, and a logical outcome should have been to increase power by increasing sample size. What the tables show, however, is that less rigorous alpha levels may reduce or eliminate effect size exaggerations. That may be a key or at least a step to resolving the file drawer problem (Begg, 1994; Bradley & Gupta, 1997). There exist in all developed literatures a proportion of known replications at the .05 level against a proportion of unknown failures to replicate. As such, there is no way to know the true effect size without knowing the effect of the file drawer. The tables should inspire, at the least, researchers to keep track of their file drawer studies. Coupled with experience and knowledge in an area, this could allow a scientist to understand more accurately the magnitude of the phenomenon of his interest and perhaps share this information with colleagues.
13_PR_Bradley_FMS_051013_130014.indd 842
21/08/13 5:32 PM
ALTERNATIVE ALPHA LEVELS
843
A comment from an unknown reviewer on this approach was that researchers should plan beforehand. This practice is acceptable and laudatory if it can be done accurately. The point, however, is that the planning cannot be done accurately if correct estimates are excluded from the literature because only improbable results are published due to small sample sizes. This paper in advocating the adjustment of alphas (Type I error rate) is not an endorsement of the Null Hypothesis Statistics Testing (NHST) model. The arguments are presented in the belief that adjusting the alphas may prove to be a small, minimally disruptive step in the right direction away from the rigid form of NHST. Although NHST has been strongly criticized for decades (Sterling, 1959; Cohen, 1994; Nickerson, 2000), experience shows the progress towards adoption of other approaches such as the construction of confidence intervals (CIs) for effect sizes and Bayesian analysis has been minimal at best. (Note: effect sizes are being more frequently reported as a supplement to the p value). Using the NHST approach with flexibility in regards to alpha levels will, it is believed, have three beneficial effects. Firstly, and crucially, it will help minimize the distortion of effect size reported in newly published literature. Secondly, it will not alienate researchers: they will not have to learn a new statistical framework (Bayesian) or encounter problems calculating unfamiliar statistics (CIs), and they will still be able to make dichotomous decisions about whether an effect exists or not, although we would rather they did not. Thirdly, and arguably most importantly, it may help make researchers further aware of the problems with the NHST approach and therefore consider adopting alternatives such as CIs and Bayesian analyses in the future. In sum, the tables present alpha levels expected from small, medium, and large effect sizes for various sample sizes. This is new and important information if there is “a file drawer effect” in the literature that misleads a researcher to believe that an effect size is larger than it actually is. At the very least, the tables describe the alpha values expected as a function of sample size and power if the effect size were smaller than that reported in the published literature. More ambitiously, it allows a researcher with some knowledge of her area to estimate probable outcomes of a research endeavour. With such knowledge, the researcher can be prepared to absorb projected findings into a current data stream in a reasoned way. With already-present outcomes a researcher can see if the results fit with theory and other findings. If the results do fit, all is well and good. If the results don’t fit, perhaps the hypothesis is not worth pursuing, or perhaps the hypothesis must be pursued with a larger sample. This could be on the initiative of a particular researcher on that person’s own, or perhaps the researcher could enlist others in different laboratories to collaborate until collectively they obtain a reasonable number of participants.
13_PR_Bradley_FMS_051013_130014.indd 843
21/08/13 5:32 PM
844
M. T. BRADLEY & A. BRAND REFERENCES
BEGG, C. B. (1994) Publication bias. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis. New York: Russell Sage Foundation. Pp. 400-409. BRADLEY, M. T., & GUPTA, R. D. (1997) Estimating the effect of the file drawer problem in meta-analysis. Perceptual & Motor Skills, 85, 719-722. BRAND, A., BRADLEY, M. T., BEST, L., & STOICA, G. (2008) Accuracy of effect size estimates from published psychological research. Perceptual & Motor Skills, 106, 645-649. BRAND, A., BRADLEY, M. T., BEST, L., & STOICA, G. (2011) Multiple trials may yield exaggerated effect size estimates. Journal of General Psychology, 138(1), 1-11. CHRISTENSEN, R. (2005) Testing Fisher, Neyman, Pearson and Bayes. American Statistical Association, 59, 121-126. COHEN, J. (1962) Statistical power of abnormal-social psychological research: a review. Journal of Abnormal Psychology, 65, 145-153. COHEN, J. (1977) Statistical power analysis for the behavioural sciences. New York: Academic Press. COHEN, J. (1994) The earth is round (p < .05). American Psychologist, 49(12), 997-1003. FISHER, R. A. (1973) Statistical methods and scientific inference. (3rd ed.) New York: Hafner Press. GIGERENZER, G. (1993) The superego, the ego, and the id in statistical reasoning. In G. Keren & C. Lewis (Eds.), A handbook for data analysis in the behavioral sciences: methodological issues. Hillsdale, NJ: Erlbaum. Pp. 311-339. IOANNIDIS, J. P. A. (2005) Why most published research findings are false. PLoS Medicine, 2(8), e 124. DOI: 10.1371/journal.pmed.0020124. KRAEMER, H. C., & THEIMANN, S. (1987) How many subjects? Statistical power analysis in research. Newbury Park, CA: Sage. LIPSEY, M., & WILSON, D. B. (1993) The efficacy of psychological, educational, and behavioral treatment: confirmation from meta-analysis. American Psychologist, 48, 1181-1209. MADDOCK, J. E., & ROSSI, J. S. (2001) Statistical power of articles published in three health psychology-related journals. Health Psychology, 20, 76-78. MARSZALEK, J. M., BARBER, C., KOHLHART, J., & HOLMES, C. B. (2011) Sample size in psychological research over the past 30 years. Perceptual & Motor Skills, 112, 331-348. MAXWELL, S. E. (2004) The persistence of underpowered studies in psychological research: causes, consequences, and remedies. Psychological Methods, 9, 147-163. NICKERSON, R. S. (2000) Null hypothesis significance testing: a review of an old and continuing controversy. Psychological Methods, 5(2), 241-301. RICHARD, F. D., BOND, C. F., JR., & STOKES-ZOOTA, J. J. (2003) One hundred years of social psychology quantitatively described. Review of General Psychology, 7, 331-363. STERLING, T. D. (1959) Publication decisions and their possible effects on inferences drawn from tests on significance or vice versa. Journal of the American Statistical Association, 54, 10-34. WILKINSON, L., & TASK FORCE ON STATISTICAL INFERENCE AMERICAN PSYCHOLOGICAL ASSOCIATION, SCIENCE DIRECTORATE. (1999) Statistical methods in psychology journals: guidelines and explanations. American Psychologist, 54(8), 594-604. Accepted May 10, 2013.
13_PR_Bradley_FMS_051013_130014.indd 844
21/08/13 5:32 PM
Copyright of Psychological Reports is the property of Ammons Scientific, Ltd. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
ALPHA VALUES AS A FUNCTION OF SAMPLE SIZE, EFFECT SIZE, AND POWER: ACCURACY OVER INFERENCE1 M. T. BRADLEY
A. BRAND
University of New Brunswick
Kings College: Institute of Psychiatry
Summary.—Tables of alpha values as a function of sample size, effect size, and desired power were presented. The tables indicated expected alphas for small, medium, and large effect sizes given a variety of sample sizes. It was evident that sample sizes for most psychological studies are adequate for large effect sizes defined at .8. The typical alpha level of .05 and desired power of 90% can be achieved with 70 participants in two groups. It was perhaps doubtful if these ideal levels of alpha and power have generally been achieved for medium effect sizes in actual research, since 170 participants would be required. Small effect sizes have rarely been tested with an adequate number of participants or power. Implications were discussed.
Power, the probability to report effects from experimental manipulations when effects should be present, is influenced by sample size, alpha levels, and the power criterion itself. Large true effect sizes, everything else being equal, are tested at higher power than small effect sizes. Large sample sizes and more liberal alpha levels increase the likelihood of accepting any effect size as a significant difference. Willingness to accept a higher probability of Type II errors (lower power), either tacitly or explicitly, may determine whether or not a study will be conducted. Thus, there are several potential considerations related to the decision to conduct a study as well as to understand the results in a meaningful fashion. These considerations are regardless of whether the guiding approach is from Neyman-Pearson, Fisher, or a hybrid of the two perspectives (Gigerenzer, 1993). Traditional alpha or p > .05 is often where the line between statistical significance and non-significance is drawn, and this level, in turn, influences what is published or what is not. Ultimately, effect size estimates that are published are the ones available to researchers. If those effectsize estimates are distorted through possible but improbable Type I errors, problems are created. Type I errors occur when results of statistical tests in a probabilistic sense should not yield significance but do. Type II errors occur when tests should in a probabilistic sense find significance but do not. This paper examines alpha levels associated with different Ns, effect sizes, and Type II error levels to illustrate the alpha values expected when traditional significance and power levels may not be met. Address correspondence to Michael T. Bradley, Department of Psychology, University of New Brunswick, P.O. Box 5050, Tucker Park Road, Saint John, NB, Canada, E2L 4L5 or e-mail ([email protected]). 1
DOI 10.2466/03.49.PR0.112.3.835-844
13_PR_Bradley_FMS_051013_130014.indd 835
ISSN 0033-2941
21/08/13 5:32 PM
836
M. T. BRADLEY & A. BRAND
The recommended approach to increasing power is to increase sample size. Statistical and experimental design texts (Kraemer & Theimann, 1987) devote space to calculating a sample size to give reasonable power to reject the null hypothesis at the .05 level given a certain estimated effect size. This approach has been advocated and widely available since Cohen (1962). The difficulty is that even with that clear recommendation and the availability of methods, sample sizes have not markedly increased over the last 60 years (Marszalek, Barber, Kohlhart, & Holmes, 2011). In fact, Marszalek, et al. (2011) focused on four leading APA journals documenting sample size for the years 1955, 1977, 1995 and 2006. The earliest year, 1955, is before Cohen’s (1962) lucid description of often inadequate sample size for small and medium effects, and his prescription amounting to a request for larger sample sizes. The later years follow Cohen’s message, and the final year is after publication of the APA’s Task Force on Statistical Inference (Wilkinson & The Task Force on Statistical Inference, 1999) that also described problems with inadequate sample size. Further, during the years included in the study, other authors, again perhaps inspired by Cohen, have commented on the issue of inadequate power (Maddock & Rossi, 2001; Maxwell, 2004). In general, there has been ample opportunity to appreciate the importance of the problem, but little has changed during this period. There are possible explanations why sample sizes have not increased. It is possible that psychologists believe power is adequate for their various studies. A common routine, although not common enough, is to calculate power based on effect sizes generated in published studies. Researchers, however, may be misled by these reported values. Brand, Bradley, Best, and Stoica, (2008) showed that literature published under .05 significance criteria would yield estimates exaggerated up to 190% for a small effect size investigated with the typical numbers of measures or participants used in psychological studies. Brand, et al. (2008) looked at the average of effect sizes that were significant in a Monte Carlo study when the effect size between two groups of 38 participants each was set at .2, .5, or .8 and statistical significance was accepted at the .05 level. The exaggeration of the values associated with statistical significance and therefore publication would mislead readers and suggest that a true small effect size (.20) is at least a medium effect size of .58. In addition, these small effect sizes would give a statistically significant result for 14% of the studies conducted. Thus, given a large number of psychologists examining hypotheses, a 14% success rate could create a large literature, but unfortunately it is a literature promoting exaggerated estimates. It is worth noting that Brand, et al. (2008) concentrated on point estimations, but these estimations lie on a continuum. Thus, these estimations of exaggeration of effect sizes vary, but they follow very basic rules.
13_PR_Bradley_FMS_051013_130014.indd 836
21/08/13 5:32 PM
ALTERNATIVE ALPHA LEVELS
837
The distortions or effect size exaggerations go up as true effect size diminishes. The mitigating factor is the probability of obtaining the exaggeration associated with statistical significance goes down, ultimately to 5%, as the true underlying effect size decreases. In a similar way, the reciprocal is true. As the effect size increases, the exaggeration diminishes but the chance of reporting an exaggeration increases. Thus, it is possible that a literature of published effect size exaggerations gives a false impression that significance tests are being conducted at adequate power and, of course, some percentage of the 125,000 North American psychologists, plus their European confreres, and their students, publish further exaggerated estimates through replication of erroneous results. Adding to the general confusion is that usual sample sizes may give the impression that they yield adequate power for the hypotheses that psychologists test, because this is actually the case for large effect sizes (0.80; Cohen, 1977). It, however, is demonstrably not true for Cohen’s medium (.50) and small (0.20) effect sizes. In fact, Maddock and Rossi (2001) reported the mean power to be 36% for the 1997 volumes of three health psychology journals. Published effect sizes exaggerated by including only statistically significant estimates in the calculations may create a variety of difficulties. For example, there could be an effect on obtaining grants. Social science grants are often small, but a researcher may believe from calculations that the small grant provides enough support to do an adequate study. In truth, however, the researcher may fail to include an appropriate number of participants in a study. Reviewers of the grant application may limit the grant amounts because they also calculate power on the published literature and reinforce the belief that what is actually an inadequate sample size is sufficient. The probable result of such an exercise is non significance, failure to publish, and a waste of the funding. Related to the sample size problem is that available participants for any given study might be limited. Maxwell (2004) suggested multisite studies to deal with this, but of course, this requires contacting and inspiring interest in other researchers who might be busy with their own projects. There is some possible balance to this process, but it is slow and cumbersome. The majority of conducted studies will not, of course, support past results and researchers may gradually drop out of a particular area of investigation leaving only the serially lucky to reproduce results. As Maxwell (2004) suggested, the process of correction would be quicker if the thousands of statistically non-significant results could be noted in the published literature, but in current practice such non-significant results are rarely published, and they have little or no effect on any given literature (Iaonnidis, 2005). It is appropriate to note here that there is an exception to the general rule that non-significant results are not reported.
13_PR_Bradley_FMS_051013_130014.indd 837
21/08/13 5:32 PM
838
M. T. BRADLEY & A. BRAND
The Journal of Articles in Support of the Null Hypothesis publishes exclusively statistically non-significant results. The situation described above creates a difficulty for the scientific development of psychology. A hallmark of science is accurate measurement. Chronically inaccurate measurements created by a publication process that presents exaggerated values have no place in science. It is no more justified to include exaggerated measures in reports, than it is to report results more readily explainable by chance than by an actual effect. Again, Maxwell (2004) suggested de-emphasizing significance and publishing more precise estimates. To our way of thinking, it is a question that a researcher in any given area has to approach with reason. If there is a suspicion that a given sample size does not afford enough power, then the researcher has to ask what is a reasonable alpha to suggest that the null hypothesis actually has been rejected. That reasoning has not been applied with the traditional and ever-present .05 level of statistical significance testing. The literature stretching back to Cohen shows that power with small and medium effect sizes and typical sample sizes is inadequate for the .05 level of significance. We believe that one route to achieve an appreciation of obtained results is to endorse Maxwell’s suggestion that traditional significance be de-emphasized. We describe what Type I error rates look like under various conditions a researcher might encounter. We also argue that the .05 level may be appropriate for some areas, but in other areas the consequences of a Type I error may not be serious, and psychologists should be able to read about accurate estimates of an effect. Our comments echo those of Christensen (2005), who summed up the situation by saying the focus on small alpha levels often leads to bad decisions with regards to both Type I and Type II errors. Since the most obvious and heavily advocated solution to the problem, the increase in sample size, has not yet been generally achieved, other ways are explored to allow researchers to consider Type I and Type II errors. Presented are tables for small, medium, and large effect sizes. These tables show the expected alpha level or probability that chance could offer an explanation to rival the experimentally hypothesized explanation for a finding given various sample sizes and power. The power is expressed in the reciprocal form of Type II error rates of .1, .2, and .3. If there were a limited number of participants available for testing a small effect size, it is not possible to obtain the .05 level of statistical significance and be accurate in estimates of effect size. The question then becomes what value of alpha would be reasonable given the estimated effect size, sample size, and power level? With that knowledge, a researcher could then determine if a result that is not statistically significant by traditional levels may actually be accurate in terms of the expected values such that an effect size
13_PR_Bradley_FMS_051013_130014.indd 838
21/08/13 5:32 PM
ALTERNATIVE ALPHA LEVELS
839
calculated from the study is actually a valuable contribution. As such, it would, at the least, be worth communicating to fellow scientists or pursuing as replicable if more participants became available. This approach, at the least, would encourage more pilot studies and places the results in a readily comprehendible context. METHOD A series of criterion power analyses were conducted to determine the alpha level, for ranges of standardized effect sizes (d), desired Type II error rate, and sample size specifications. The criterion power analyses used the statistical package R (R Development Core Team, 2009, http:// www.r-project.org/). Note that statistical power analysis packages such as GPower or nQuery Adviser could also be used to conduct these analyses. The criterion power analyses performed assumed that data would be analyzed using a two-tailed, independent t test where the number of subjects per condition was equal. The values used for the standardized effect sizes were based on Cohen’s (1977) benchmark for small (d = 0.20), medium (d = 0.50) and large (d = 0.80) effect sizes. The values used for the Type II error rates were 0.10, 0.20, and 0.30, which correspond to statistical power of 90%, 80%, and 70%, respectively. A set of nine different sample sizes was used for each of the three effect sizes. For the small effect (d = 0.20), the sample sizes were from 700 to 1,100 incrementing by 50. For the medium effect (d = 0.50), the sample sizes were from 100 to 180 incrementing by 10. For the large effect (d = 0.80), the sample sizes were from 30 to 110 incrementing by 10. The sample size ranges are not exhaustive and were chosen to be illustrative of results that a researcher might find of interest for practical reasons. The computed alphas for the three effect sizes are listed in Table 1. RESULTS Tables 1, 2, and 3 display the alpha levels expected for various sample sizes, over small, medium, and large effect sizes and three power levels. The numbers are straightforward but there are some notable features. The range of sample sizes giving reasonable alpha level estimates change with different effect sizes. Thus, for a small effect size, the tables start at 700 participants and go up by increments of 50 to reach 1,100 participants. For medium effect sizes, a range of 100 to 180 is reasonable with increments of 10. For large effect sizes, depending on the willingness to accept a Type II error, sample sizes might be as low as 30, but if sample size goes up to 70 the traditional ideal is achieved of p < .05 for a Type I error with p < .10 for a Type II error. The increments differ substantially in the tables. For a small effect size an increment of 50 is necessary to make a difference, whereas 10 is adequate for a medium effect size and very effective for a large effect size.
13_PR_Bradley_FMS_051013_130014.indd 839
21/08/13 5:32 PM
840
M. T. BRADLEY & A. BRAND TABLE 1 ALPHA VALUES AS A FUNCTION OF SAMPLE SIZE AND TYPE II ERROR RATE WHEN THE TRUE EFFECT SIZE IS SMALL (d = .2)
Type II Error Rates
700
750
800
850
Sample Size (N) 900
950
1000
1050
1100
0.10
0.173
0.146
0.122
0.103
0.086
0.072
0.060
0.051
0.042
0.20
0.072
0.058
0.047
0.038
0.031
0.025
0.021
0.017
0.014
0.30
0.034
0.027
0.021
0.017
0.013
0.011
0.008
0.007
0.005
Note.—All alpha values are reported to three decimal places.
TABLE 2 ALPHA VALUES AS A FUNCTION OF SAMPLE SIZE AND TYPE II ERROR RATE WHEN THE TRUE EFFECT SIZE IS MEDIUM (d = .5) Sample Size (N)
Type II Error Rates
100
110
120
130
140
150
160
170
180
0.10
0.226
0.184
0.148
0.120
0.097
0.078
0.062
0.050
0.040
0.20
0.101
0.078
0.061
0.047
0.036
0.028
0.022
0.017
0.013
0.30
0.051
0.038
0.029
0.022
0.016
0.012
0.009
0.007
0.005
Note.—All alpha values are reported to three decimal places.
TABLE 3 ALPHA VALUES AS A FUNCTION OF SAMPLE SIZE AND TYPE II ERROR RATE WHEN THE TRUE EFFECT SIZE IS LARGE (d = .8) Sample Size (N)
Type II Error Rates
30
40
50
60
70
80
90
100
110
0.10
0.369
0.221
0.130
0.076
0.044
0.025
0.014
0.008
0.005
0.20
0.188
0.100
0.053
0.028
0.015
0.008
0.004
0.002
0.001
0.30
0.106
0.052
0.026
0.013
0.006
0.003
0.002
0.001
0.000
Note.—All alpha values are reported to three decimal places.
DISCUSSION The present data show the expected alpha levels for various effect sizes, sample sizes, and power levels. The first thing that is evident is that the Ns (number of participants/measurements) differ considerably depending upon the expected effect size in a given area. It is worth pointing out that a study of a small effect size requires 1,050 participants to be both accurate and approximate statistical significance at the .05 level
13_PR_Bradley_FMS_051013_130014.indd 840
21/08/13 5:32 PM
ALTERNATIVE ALPHA LEVELS
841
(.051). Only an N of 70 is required to achieve the same alpha with a large effect size. Also, a medium sized effect can be tested with a fairly modest N of 170. Of course, difficulties abound with obtaining accurate effect size estimates given the current state of the literature, and we suspect that many hypotheses in cognitive and social psychology (Lipsey & Wilson, 1993; Richard, Bond, Jr., & Stokes-Zoota, 2003) have an effect size between .2 and .5. Depending upon which level they approach, the number of participants might have to be substantial. This being said, it must be realized that these tables portray the results for participants in two equal sized groups who each respond to a single measure. More complex designs may both reduce the divisor for the standard error and degrees of freedom for significance, and, as a consequence, make significance more difficult to obtain. On the other hand, multiple measures that are aggregated or averaged on each single individual may increase power (Brand, Bradley, Best, & Stoica, 2011). Overall, the tables serve as a rudimentary benchmark to allow a researcher to estimate beforehand what might be expected from an analysis involving specified N. We use the word “rudimentary” because each researcher has to deal with the peculiarities that pertain to her own research area. If researchers attend to the presented numbers they definitely may improve the estimates and their understanding of these estimates in the case of small effect size, and possibly also with medium effect sizes. Understanding data with these tables can go well beyond the common .05 level of significance. If there was reason to suspect that a research interest involved a small effect size, then for 800 measures an alpha of .122 would be the mean result obtained with a power level of 90%. With such a reporting style, the current practice of forgetting the findings in a “file drawer” could be curtailed. The researcher may decide there is something of interest and pursue collaboration, replication, or tolerate a greater chance for a Type II error to establish the finding. If, through some chance factor, a researcher obtained an alpha of .05 when that value should have been .122, the calculated effect size would be exaggerated and potentially misleading. Researchers would then simply warn others that the current results could be an exaggeration, and others who have close but statistically non-significant results by traditional standards are actually accurately measuring the phenomena and might have their replications recognized. In other words, a mean alpha of .122 for a small effect size would mean that 90% of the time this would be obtained and the effect size estimate would be accurate. It is perhaps forgotten that Fisher (1973) pointed out that insisting on a .05 level of significance showed a lack of statistical sophistication, and his point is well made with the tables presented here.
13_PR_Bradley_FMS_051013_130014.indd 841
21/08/13 5:32 PM
842
M. T. BRADLEY & A. BRAND
The move towards reporting effect sizes particularly highlights the problems with the current practice of a rigid statistical significance level governing whether a paper is published or not. Statistical significance at the .05 level is a state proposition, either an effect is present or it is not. Effect sizes are different. Effect sizes are about accurate measurement. The goal should be about the best estimate of an effect size. It may be a bit of an oxymoron to write of accurate measurement with variable data, but the goal is to get as close as we can to assembling a set of effect sizes that best reflect the phenomena under study. All or none inferences with exclusion of non-significant results do not allow this to be done, and worse, if a study is underpowered the effect size estimate must, by the mechanics of statistical formulae, be exaggerated if the study results are reported as statistically significant. To put it in stark terms, a researcher cannot have statistical significance and effect size accuracy if the study is underpowered. In the ideal situation a researcher would have the resources to have enough participants to allow for inference testing that did not distort reported values. However, as Marszalek, et al. (2011) have reported, sample size has not changed over 60 years. Therefore, a researcher has to be cognizant that it is not possible to obtain an alpha level of .05 and an accurate effect size estimate when the effect size is small and probably even when the effect size is medium. It is, however, very probable when the effect size is large. It is worth emphasizing the, perhaps unintended, role of the 1999 APA Task Force (Wilkinson, & Task Force, 1999) advocating the report of effect sizes along with significant tests. Effect sizes reflect measurement, and accurate measurement is at the core of science. As soon as effect sizes were featured it became obvious that chance factors associated with significance could result in the promulgation of effect size exaggerations–sometimes of a substantial nature (Brand, et al., 2008), especially when statistical significance is coupled with inadequate power. Cohen (1962), as mentioned, has been making this point since 1962, and a logical outcome should have been to increase power by increasing sample size. What the tables show, however, is that less rigorous alpha levels may reduce or eliminate effect size exaggerations. That may be a key or at least a step to resolving the file drawer problem (Begg, 1994; Bradley & Gupta, 1997). There exist in all developed literatures a proportion of known replications at the .05 level against a proportion of unknown failures to replicate. As such, there is no way to know the true effect size without knowing the effect of the file drawer. The tables should inspire, at the least, researchers to keep track of their file drawer studies. Coupled with experience and knowledge in an area, this could allow a scientist to understand more accurately the magnitude of the phenomenon of his interest and perhaps share this information with colleagues.
13_PR_Bradley_FMS_051013_130014.indd 842
21/08/13 5:32 PM
ALTERNATIVE ALPHA LEVELS
843
A comment from an unknown reviewer on this approach was that researchers should plan beforehand. This practice is acceptable and laudatory if it can be done accurately. The point, however, is that the planning cannot be done accurately if correct estimates are excluded from the literature because only improbable results are published due to small sample sizes. This paper in advocating the adjustment of alphas (Type I error rate) is not an endorsement of the Null Hypothesis Statistics Testing (NHST) model. The arguments are presented in the belief that adjusting the alphas may prove to be a small, minimally disruptive step in the right direction away from the rigid form of NHST. Although NHST has been strongly criticized for decades (Sterling, 1959; Cohen, 1994; Nickerson, 2000), experience shows the progress towards adoption of other approaches such as the construction of confidence intervals (CIs) for effect sizes and Bayesian analysis has been minimal at best. (Note: effect sizes are being more frequently reported as a supplement to the p value). Using the NHST approach with flexibility in regards to alpha levels will, it is believed, have three beneficial effects. Firstly, and crucially, it will help minimize the distortion of effect size reported in newly published literature. Secondly, it will not alienate researchers: they will not have to learn a new statistical framework (Bayesian) or encounter problems calculating unfamiliar statistics (CIs), and they will still be able to make dichotomous decisions about whether an effect exists or not, although we would rather they did not. Thirdly, and arguably most importantly, it may help make researchers further aware of the problems with the NHST approach and therefore consider adopting alternatives such as CIs and Bayesian analyses in the future. In sum, the tables present alpha levels expected from small, medium, and large effect sizes for various sample sizes. This is new and important information if there is “a file drawer effect” in the literature that misleads a researcher to believe that an effect size is larger than it actually is. At the very least, the tables describe the alpha values expected as a function of sample size and power if the effect size were smaller than that reported in the published literature. More ambitiously, it allows a researcher with some knowledge of her area to estimate probable outcomes of a research endeavour. With such knowledge, the researcher can be prepared to absorb projected findings into a current data stream in a reasoned way. With already-present outcomes a researcher can see if the results fit with theory and other findings. If the results do fit, all is well and good. If the results don’t fit, perhaps the hypothesis is not worth pursuing, or perhaps the hypothesis must be pursued with a larger sample. This could be on the initiative of a particular researcher on that person’s own, or perhaps the researcher could enlist others in different laboratories to collaborate until collectively they obtain a reasonable number of participants.
13_PR_Bradley_FMS_051013_130014.indd 843
21/08/13 5:32 PM
844
M. T. BRADLEY & A. BRAND REFERENCES
BEGG, C. B. (1994) Publication bias. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis. New York: Russell Sage Foundation. Pp. 400-409. BRADLEY, M. T., & GUPTA, R. D. (1997) Estimating the effect of the file drawer problem in meta-analysis. Perceptual & Motor Skills, 85, 719-722. BRAND, A., BRADLEY, M. T., BEST, L., & STOICA, G. (2008) Accuracy of effect size estimates from published psychological research. Perceptual & Motor Skills, 106, 645-649. BRAND, A., BRADLEY, M. T., BEST, L., & STOICA, G. (2011) Multiple trials may yield exaggerated effect size estimates. Journal of General Psychology, 138(1), 1-11. CHRISTENSEN, R. (2005) Testing Fisher, Neyman, Pearson and Bayes. American Statistical Association, 59, 121-126. COHEN, J. (1962) Statistical power of abnormal-social psychological research: a review. Journal of Abnormal Psychology, 65, 145-153. COHEN, J. (1977) Statistical power analysis for the behavioural sciences. New York: Academic Press. COHEN, J. (1994) The earth is round (p < .05). American Psychologist, 49(12), 997-1003. FISHER, R. A. (1973) Statistical methods and scientific inference. (3rd ed.) New York: Hafner Press. GIGERENZER, G. (1993) The superego, the ego, and the id in statistical reasoning. In G. Keren & C. Lewis (Eds.), A handbook for data analysis in the behavioral sciences: methodological issues. Hillsdale, NJ: Erlbaum. Pp. 311-339. IOANNIDIS, J. P. A. (2005) Why most published research findings are false. PLoS Medicine, 2(8), e 124. DOI: 10.1371/journal.pmed.0020124. KRAEMER, H. C., & THEIMANN, S. (1987) How many subjects? Statistical power analysis in research. Newbury Park, CA: Sage. LIPSEY, M., & WILSON, D. B. (1993) The efficacy of psychological, educational, and behavioral treatment: confirmation from meta-analysis. American Psychologist, 48, 1181-1209. MADDOCK, J. E., & ROSSI, J. S. (2001) Statistical power of articles published in three health psychology-related journals. Health Psychology, 20, 76-78. MARSZALEK, J. M., BARBER, C., KOHLHART, J., & HOLMES, C. B. (2011) Sample size in psychological research over the past 30 years. Perceptual & Motor Skills, 112, 331-348. MAXWELL, S. E. (2004) The persistence of underpowered studies in psychological research: causes, consequences, and remedies. Psychological Methods, 9, 147-163. NICKERSON, R. S. (2000) Null hypothesis significance testing: a review of an old and continuing controversy. Psychological Methods, 5(2), 241-301. RICHARD, F. D., BOND, C. F., JR., & STOKES-ZOOTA, J. J. (2003) One hundred years of social psychology quantitatively described. Review of General Psychology, 7, 331-363. STERLING, T. D. (1959) Publication decisions and their possible effects on inferences drawn from tests on significance or vice versa. Journal of the American Statistical Association, 54, 10-34. WILKINSON, L., & TASK FORCE ON STATISTICAL INFERENCE AMERICAN PSYCHOLOGICAL ASSOCIATION, SCIENCE DIRECTORATE. (1999) Statistical methods in psychology journals: guidelines and explanations. American Psychologist, 54(8), 594-604. Accepted May 10, 2013.
13_PR_Bradley_FMS_051013_130014.indd 844
21/08/13 5:32 PM
Copyright of Psychological Reports is the property of Ammons Scientific, Ltd. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
