Sample Size Determination Based on Fisher's Exact Test for Use in 2 x 2 Comparative Trials with Low Event Rates Ronald G. Thomas, PhD, and Michael Conlon, PhD VA Medical Center, Palo Alto, California (R.G.T.), and Department of Statistics, University of Florida, Gainesville, Florida (M.C.)
ABSTRACT: A collection of sample size tables are presented for designing comparative trials
when the event rates pl and p2 are low. The tables are based on exact power calculations for Fisher's Exact Test. Both one-sided and two-sided alternative hypotheses are considered. A comparison is made between these sample sizes and those obtained by using popular asymptotic approximations. KEY WORDS: Sample size, Fisher's Exact Test
INTRODUCTION Sample size determination is a fundamental c o m p o n e n t of the clinical trial planning process. In the specific case of a 2 x 2 comparative trial aimed at contrasting event rates between experimental and control therapies, the calculation of the required sample size d e p e n d s on the values of four parameters: (1) pl, the true event rate in the experimental group; (2) P2, the true event rate in the control group; (3) c~, the probability of incorrectly rejecting the null hypothesis H0 : pl = p2, i.e., the type I error; and (4) ~, the probability of incorrectly failing to reject the null hypothesis, i.e., the type II error. Once these parameters have been specified, the trial designers can proceed to consult any of several sets of tables available in textbooks or in the literature to obtain n, the required m i n i m u m n u m b e r of observations per group. The appropriate table to use is determined by the testing procedure planned for use at final analysis. For example, if the hypothesis H0 : pl = p2 will be tested using the standard X2 statistic, then tables contained in Fleiss [1] and elsewhere are appropriate. If, instead, the arc sine test is to be applied, tables in Cochran and Cox [2] or Cohen [3] are available as are isographs in Feigl [4]. A third alternative is Fisher's Exact Test, for which Gail and Gart [5], H a s e m a n [6], and Casagrande et al. [7] have presented tables. Finally, Suissa and Shuster
Address reprint requests to: Ronald G. Thomas, PhD, Cooperative Studies Program Coordinating Center, 151-K, VA Medical Center, 3801 Miranda Avenue, Palo Alto, CA 94204. Received April 25, 1991; revised September 12, 1991.
134 0197-2456/92/$5.00
Controlled Clinical Trials 13:134-147 (1992) ¢~, Elsevier Science Publishing Co., Inc. 1992 655 A v e n u e t~f tht~ Americas, New York, New York 10010
Sample Size Based on Fisher's Exact Test
135
[8] have published sample size tables based on an exact unconditional testing procedure. Due to the ready availability of tabled values, calculating sample sizes is usually a straightforward procedure. However, if the anticipated event rates (in one or both groups) are low, say less than 0.1, then determining n can be problematic. To illustrate, consider an example borrowed (with modification) from Lemeshow et al. [9]. A trial is being designed to evaluate a new cancer screening and treatment program. From previous studies, the mortality rate from this disease for men over 60 can be reliably estimated at 400/100,000. Data from a recently completed pilot study offer evidence that an aggressive early screening program may reduce the death rate to as low as 100/100,000. Based on this evidence; the trial designers set the expected event rates at Pl = 0.001 and P2 = 0.004. A statistician charged with determining a sample size for this trial will soon realize that none of the existing tables contain tabulated values for event rates this low. This is not a serious problem, however, if the anticipated final analysis procedure is one that allows a closed form expression for n. For example, the standard ×2 test yields the formula:
n . (z~_~,V~p~+ . . . .
zl_.X/plql+ . p2q2) 2
(1)
P2 - pl where qi = 1 - pi, i = 1, 2, p is the observed proportion of successes in both groups, and z ~ _ ~ denotes the (1 - 00th deviate of the standard normal distribution. From this expression, n can be calculated for any combination of parameters p~, p2, 0~, and [3. Most testing procedures which appeal to asymptotic normality for inference can be "inverted" to provide a closed form expression for n. Unfortunately, when Pl and P2 are low--as in our example--most statisticians would be hesitant to use a testing procedure which relies on the assumption of approximate normality. For cases where the normality assumption is questionable, the usual textbook recommendation is to use Fisher's Exact Test. See, for example, Armitage [10] and Snedecor and Cochran [11]. However, the literature contains no tables of n based on Fisher's Exact Test for low p~ and P2. Sample size calculation for Fisher's Exact Test is a computationally intensive, iterative procedure. First, the empirical power of the test for a given n, Pl, P2, and oLmust be determined. Then, the sample size is adjusted in iterative fashion until the smallest n for which the empirical power is greater than or equal to 1 - ~ is found. The computational difficulty increases exponentially as n increases. The realization of the need for tabulated values of n based on Fisher's Exact Test for low p~ and p2 prompted the authors to develop an algorithm for the calculation of the power function for Fisher's Exact Test [12]. Through the use of this algorithm, it is possible with modest computing expenditure to determine a value for n even when the underlying event rates are very low. Clearly, in today's medical research environment, therapeutic improvement is usually evolutionary rather than revolutionary. Consequently, more and more clinical trials, particularly in chronic diseases, are being designed
136
R.G. Thomas and M. Conlon to detect a small reduction in an event rate that is already relatively low. Recent examples of such trials are ISIS-2 [13] a n d the Physician's Health S t u d y [14]. As the resources to c o n d u c t large-scale projects of this type b e c o m e scarce, the n e e d to be able to determine sample sizes based on e x a c t - - r a t h e r than approximate---calculations becomes increasingly important. The r e m a i n d e r of the p a p e r contains a description of the p o w e r function for Fisher's Exact Test, followed by an exploration of the validity of the normal approximation to the binomial w h e n event rates are low. The "Results" section presents tables containing calculated values for the m i n i m u m sample size required for a level ct and p o w e r 1 - {3 Fisher's Exact Test of H0 : p1 = p2 against both one- a n d two-sided alternatives.
POWER F U N C T I O N FOR FISHER'S EXACT TEXT Consider a 2 x 2 comparative trial d e s i g n e d to c o m p a r e e v e n t rates b e t w e e n treatment a n d control groups. Define X as the n u m b e r of events occurring in the treatment g r o u p and Y as the n u m b e r of events in the control group. Let both X and Y be binomially distributed r a n d o m variables with index n and parameters Pl and P2, respectively. With this notation we can display the o u t c o m e of the trial in tabular form as
Treatment Group Control Group
Event
No event
x Y
n - x n - Y
I n / n
r
2n - r
2n
I I
w h e r e R = X + Y d e n o t e s the total n u m b e r of events. Fisher's Exact Test is based on the a r g u m e n t that inference regarding the null h y p o t h e s i s can be d r a w n conditionally on r and n. Fisher [15] d e m o n strated that the conditional distribution of X, given R = r, d e p e n d s on the event rates Pl and P2 only t h r o u g h the o d d s ratio 0, w h e r e 0 = pl(1 - p2)/p2(1 - pl). That is, 0x
h(xlr; O) = Pr[X = xlr; O] =
min(r,n)~(~)( lZ ) i ~ max(O,r-
~t)
/"
--
0i
i
U n d e r the null h y p o t h e s i s H0 : Pl = p2 ( 0 = 1), h simplifies to the h y p e r g e o metric distribution a n d the test can be defined i n d e p e n d e n t l y of the event rates. Thus, Fisher's one-sided level ot test rejects the null h y p o t h e s i s if the conditional probability of observing a value of X at least as extreme as x is less than or equal to or. The conditional rejection region is defined as
C~r~ = {(x,y) : x + y = r
and
~. i = max(O,r
h(ilr; 0 = 1) ~< et} n)
137
Sample Size Based on Fisher's Exact Test The p o w e r of the test is defined as 2n
~,(n) = Pr[(x,y) ~ C~r)IH1] = ~ ~ f(x,ylp~, P2, n),
(2)
r = OC(r)
where
f(x,ylp,,p2,n) =
x
Y
pI(1 - pl)"-Xp~(1 - p2)n-y
the joint binomial mass function. If we let 2n
= U cF) r=O
d e n o t e the unconditional rejection region, then we can alternatively write the p o w e r function as
~l(n) = ~_~ f(x,ylp,,p2,n)
(3)
CF
Several authors, e.g., Berkson [16], have p o i n t e d out that Fisher's Exact Test is conservative. That is, the empirical type I error of the test is usually substantially less than the nominal significance level o~. It is straightforward to calculate the empirical type I error as
o~* = ~ f(x,ylp2,P2,n) CF
For each combination of p~ and p2 in Tables 2-4, we have calculated ~* along with the sample size. It is left to the reader to make his or her o w n j u d g m e n t as to w h e t h e r the d e g r e e of conservativeness precludes the usefulness of Fisher's Exact Test for a given combination of parameters. Since the sample size is calculated b y iterating n until the smallest n such that ~/(n)/> 1 - .B is d e t e r m i n e d , the empirical type II error will be, at most, only slightly less than J3 and n e v e r greater. EXACT ERROR EVALUATION W h e n trying to decide b e t w e e n sample size tables based on exact versus asymptotic calculations, it is reasonable to ask, U n d e r w h a t circumstances is the normal approximation to the binomial inappropriate? In other words, for w h a t combinations of pl and p2 should large sample m e t h o d s be a b a n d o n e d in favor of Fisher's Exact Test or some other exact m e t h o d ? Intuitively, it w o u l d seem that as one or both of the e v e n t rates becomes low, the validity of the a s s u m p t i o n of approximate normality w o u l d deteriorate. But h o w low is low? To address this question, we have evaluated the empirical type I and type II errors associated with sample sizes obtained using equation (1) for a range of Pl and p2 values and for oL = 0.05 and J3 = 0.1. The results are given in Table 1 (to facilitate interpretation Table 1 presents 1 - 13" rather than ~*). Equation (1) is based on the standard ×2 statistic, Z 2, or equivalently Z, the normalized difference b e t w e e n observed proportions.
138
R.G. Thomas and M. Conlon
Table 1
F r o m left to r i g h t : M i n i m u m S a m p l e Size, n (c, = 0.05 a n d 1 - f~ = 0.9), B a s e d o n S t a n d a r d (×2); E m p i r i c a l T y p e I E r r o r a* f r o m S t a n d a r d ×2; E m p i r i c a l P o w e r 1 - [3* f r o m S t a n d a r d ×2; n B a s e d o n F i s h e r ' s Exact T e s t (FET); n B a s e d o n C a s a g r a n d e , P i k e , a n d S m i t h (1978) (CPS); n B a s e d o n D o b s o n a n d G e b s k i (1986) (DG) 1
n
n
n
p,
P2
n (×2)
~,(×2)
~,(×2)
-
(FET)
(CPS)
(DG)
0.00001
0.00010 0.00100 0.01000 0.02000 0.04000 0.06000 0.08000 0.10000
115600 8649 841 400 196 121 100 64
0.04882 0.04927 0.04955 0.04866 0.05053 0.05470 0.05286 0.05141
0.94975 0.98058 0.98906 0.98618 0.98563 0.97839 0.98865 0.96098
116576 8300 801 399 199 132 98 78
137597 10743 1044 518 256 169 125 99
120071 10506 1728 1071 677 522 435 377
0.00010
0.00100 0.01000 0.02000 0.04000 0.06000 0.08000 0.10000
11449 841 400 196 121 100 64
0.04874 0.04955 0.04866 0.05053 0.05470 0.05286 0.05141
0.94804 0.97816 0.97991 0.98402 0.97706 0.98793 0.95993
11656 829 406 200 133 99 79
13753 1069 525 258 169 126 99
12003 1049 582 336 247 200 170
0.00100
0.01000 0.02000 0.04000 0.06000 0.08000 0.10000
1089 484 196 121 100 81
0.04799 0.04943 0.05053 0.05470 0.05286 0.05107
0.93874 0.95860 0.96567 0.96289 0.98017 0.98384
1164 484 218 140 103 81
1368 592 274 176 129 102
1197 513 245 165 126 103
0.01000
0.02000 0.04000 0.06000 0.08000 0.10000
2500 441 225 144 100
0.05039 0.05188 0.04960 0.05259 0.05099
0.90356 0.91112 0.92654 0.93802 0.94007
2654 486 239 156 114
2725 527 268 176 130
2661 491 244 158 116
0.02000
0.04000 0.06000 0.08000 0.10000
1225 400 196 144
0.05023 0.05059 0.04923 0.05226
0.90285 0.91236 0.88986 0.91736
1311 437 236 159
1343 458 257 174
1312 437 240 161
0.04000
0.06000 0.08000 0.10000
2025 576 289
0.04998 0.05008 0.04971
0.90116 0.89380 0.89263
2113 634 327
2131 651 341
2114 638 330
0.06000
0.08000 0.10000
2704 784
0.04998 0.05018
0.89337 0.90314
2872 826
2885 836
2874 826
0.08000
0.10000
3481
0.05007
0.89903
3597
3604
3596
If t h e n o r m a l a p p r o x i m a t i o n to t h e b i n o m i a l is a d e q u a t e , w e w o u l d e x p e c t t h e e m p i r i c a l t y p e I a n d t y p e II e r r o r s (x* a n d [3* for s a m p l e s i z e s b a s e d o n Z to b e c l o s e to t h e d e s i g n p a r a m e t e r s oL a n d fk T h e e m p i r i c a l t y p e I e r r o r for Z is d e f i n e d a s
eL* = Pr[Z > Zl-~IHo] = ~ f(x,yIp2,p2,n) Cz
139
Sample Size Based on Fisher's Exact Test w h e r e C~ = fix,y) : Z > zl - ~}. Similarly, the empirical type II error can be calculated as [3* = Pr[Z < zl-~lH1] = ~_, f(x,y[p,,p2,n) = 1 - ~ f(x,ylpl,p2,n) C~
CZ
c
w h e r e Cz = ~ - Cz, and ~ = {0. . . . . n } ~) {0. . . . . n}--the sample space. Table 1 shows that u n d e r the null h y p o t h e s i s the normal approximation to the binomial is good, e v e n w h e n p2 is as low as 10 -4. H o w e v e r , u n d e r the alternative, p~ < pa, the empirical type II error can be as m u c h as 90% less than [3. In fact, for some combinations in this table the "conservative" Fisher's Exact Test yields a smaller n than does the standard ×a. Clearly, as Pl a n d P2 b e c o m e more disparate, the less accurate the approximation becomes. H o w ever, Table 1 also shows that for tabled values with Pl greater than or equal to 0.04 both the empirical type I and type II errors are v e r y close to the optimal values e~ -- 0.05 and 13 = 0.1. If one adopts the ad hoc rule that both empirical errors should be within 10% of the design values (i.e., 0.045 ~ or* ~ 0.055 and 0.89 ~ 1 - 13" ~ 0.91), t h e n a crude interpretation of this table suggests that both p~ and p2 need to be at least as large as 0.04 for the normal approximation to the binomial to be applied. (Unless P2 - Pl is small, say 0.01, t h e n p~ could be as low as 0.01.) This table also presents values based on Fisher's Exact Test and on the sample size formulas in Casagrande et al. (CPS) [17] and Dobson and Gebski (DG) [18]. The CPS formula was derived as a c o m p r o m i s e b e t w e e n calculations based on the uncorrected s t a n d a r d X2, and the ×2 with Yates's correction. The authors s h o w e d that for 0.05 < p~,P2 < 0.75 their formula gave a close approximation to the sample size based on Fisher's Exact Test. H o w e v e r , w h e n p~ and p2 are low, as in Table 1, the CPS formula gives n u m b e r s as m u c h as 29% larger than Fisher's Exact Test. The DG equation is derived from the continuity corrected arc sine m e t h o d of Walters [19] and gives an excellent approximation to the exact results for "larger" values of pl and Pa, but it too gives values considerably larger than Fisher's Exact Test for m a n y of the p a r a m e t e r combinations in Table 1. RESULTS Tables 2 - 4 present sample sizes for use in designing 2 x 2 comparative trials with low expected e v e n t probabilities. Table 2 covers pl and P2 in the range Pl -- 0.01(0.01)0.9 a n d p2 = pl + 0.01(0.01)0.1. (The notation Pl = 0.01(0.01)0.9, for example, indicates that pl takes on all values in the range 0.01 to 0.9 in increments of 0.01.) Table 3 is for Pl and p2 in the range Pl = 0.005(0.005)0.045 and P2 = Pl + 0.005(0.005)0.05. Table 4 gives sample sizes for Pl and p2 in the range Pl = 0.001(0.001)0.009 and P2 = pl + 0.001(0.001)0.01. Each r o w of each table is organized into four sets of three n u m b e r s . The sets c o r r e s p o n d to different design objectives for type I and type II error r a t e s - one- and two-sided cx = 0.05 and 80% and 90% power. Within each set of objectives, three n u m b e r s are presented---the m i n i m u m sample size required in each g r o u p to achieve the design objectives using Fisher's Exact Test, the empirical type I error rate, and the empirical p o w e r of the study. To describe the use of the tables, consider the example of the cancer screen-
140 Table 2
R.G. Thomas and M. Conlon M i n i m u m S a m p l e S i z e s B a s e d o n F i s h e r ' s Exact Test. E v e n t R a t e R a n g e s : pl = 0.01(0.01)0.9; P2 = pl + 0.01(0.01)0.1 o~ = 0.05 (one-sided) 1 - [8 = 0.9
1 - [8 = 0.08
pl 0.01
p2 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.10
n 2654 881 486 323 239 189 156 132 114
~* 0.04052 0.03827 0.03474 0.03338 0.03468 0.03311 0.03165 0.03051 0.03334
1 - [8* 0.90002 0.90009 0.90051 0.90043 0.90017 0.90032 0.90122 0.90094 0.90057
n 1982 666 369 250 188 150 124 105 87
a* 0.03939 0.03572 0.03284 0.03376 0.03162 0.02991 0.02849 0.03061 0.02888
1 - [8* 0.80015 0.80005 0.80104 0.80103 0.80177 0.80322 0.80350 0.80227 0.80121
0.02
0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.10
4333 1311 677 437 314 236 191 159
0.04392 0.04037 0.03843 0.03717 0.03636 0.03521 0.03472 0.03306
0.90003 0.90017 0.90018 0.90021 0.90064 0.90027 0.90083 0.90068
3183 966 511 331 235 182 148 124
0.04295 0.03946 0.03668 0.03582 0.03506 0.03354 0.03251 0.03052
0.80008 0.80036 0.80011 0.80038 0.80143 0.80036 0.80172 0.80194
0.03
0.04 0.05 0.06 0.07 0.08 0.09 0.10
5955 1719 862 538 375 287 223
0.04533 0.04250 0.04027 0.03893 0.03812 0.03700 0.03566
0.90003 0.90010 0.90016 0.90010 0.90053 0.90043 0.90090
4355 1268 638 400 278 213 171
0.04458 0.04142 0.03963 0.03761 0.03611 0.03611 0.03391
0.80006 0.80028 0.80034 0.80003 0.80006 0.80090 0.80049
0.04
0.05 0.06 0.07 0.08 0.09 0.10
7536 2113 1037 634 440 327
0.04632 0.04361 0.04168 0.04061 0.03906 0.03833
0.90000 0.90003 0.90019 0.90022 0.90037 0.90027
5498 1554 767 476 328 244
0.04559 0.04261 0.04067 0.03928 0.03757 0.03720
0.80000 0.80018 0.80035 0.80020 0.80062 0.80053
0.05
0.06 0.07 0.08 0.09 0.10
9084 2495 1207 732 503
0.04686 0.04452 0.04277 0.04159 0.04004
0.90001 0.90001 0.90017 0.90034 0.90001
6612 1830 889 543 371
0.04641 0.04391 0.04177 0.03991 0,03891
0.80001 0.80009 0.80037 0.80051 {).80004
0.06
0.07 0.08 0.09 0.10
10593 2872 1373 826
0.04730 0.04526 0.04352 0.04211
0.90001 0.90003 0.90011 0.90033
7702 2102 1013 607
0.04682 0.04458 0.04251 0.04108
0.80004 0.80016 0.80029 0.80017
0.07
0.08 0.09 0.10
12068 3237 1537
0.04760 0.04578 0.04419
0.90002 0.90005 0.90009
8767 2366 1127
0.04723 0.04515 0.04330
0.80003 0.80007 0.80024
0.08
0.09 0.10
13507 3597
0.04785 0.04615
0.90001 0.90007
9807 2623
0.04750 0.04552
0.80000 0.80010
0.09
0.10
14916
0.04804
0.90002
10824
0.04770
0.80003
141
Sample Size Based on Fisher's Exact Test Table 2
Continued ~x = 0.05 (two-sided) 1 - B -- 0.9
1 - B = 0.8
pl 0.01
p2 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.10
n 3224 1067 579 381 283 224 184 155 134
~* 0.02041 0.01875 0.01814 0.01684 0.01595 0.01638 0.01548 0.01624 0.01535
1 - B* 0.90009 0.90009 0.90027 0.90063 0.90044 0.90120 0.90132 0.90059 0.90170
n 2456 826 444 302 226 179 147 124 106
~* 0.01973 0.01776 0.01697 0.01605 0.01586 0.01498 0.01490 0.01397 0.01476
1 - ~* 0.80003 0.80047 0.80031 0.80101 0.80121 0.80189 0.80156 0.80042 0.80314
0.02
0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.10
5270 1585 817 519 371 283 226 188
0.02186 0.02016 0.01912 0.01849 0.01790 0.01730 0.01721 0.01636
0.90005 0.90010 0.90010 0.90052 0.90044 0.90080 0.90010 0.90024
3989 1202 622 400 286 221 179 150
0.02141 0.01981 0.01896 0.01778 0.01655 0.01680 0.01566 0.01652
0.80001 0.80001 0.80051 0.80001 0.80057 0.80188 0.80168 0.80226
0.03
0.04 0.05 0.06 0.07 0.08 0.09 0.10
7257 2084 1043 651 454 343 267
0.02269 0.02116 0.02035 0.01971 0.01847 0.01832 0.01814
0.90001 0.90004 0.90012 0.90004 0.90038 0.90040 0.90049
5478 1586 797 496 351 262 209
0.02229 0.02071 0.01960 0.01867 0.01802 0.01793 0.01721
0.80003 0.80025 0.80016 0.80026 0.80069 0.80168 0.80107
0.04
0.05 0.06 0.07 0.08 0.09 0.10
9203 2571 1256 771 531 394
0.02310 0.02180 0.02072 0.01991 0.01930 0.01910
0.90001 0.90011 0.90010 0.90017 0.90002 0.90004
6928 1942 960 586 407 306
0.02288 0.02127 0.02024 0.01941 0.01885 0.01839
0.80005 0.80001 0.80006 0.80073 0.80085 0.80157
0.05
0.06 0.07 0.08 0.09 0.10
11096 3043 1468 889 606
0.02343 0.02226 0.02146 0.02063 0.02013
0.90001 0.90002 0.90017 0.90004 0.90010
8338 2302 1116 676 466
0.02320 0.02180 0.02088 0.02008 0.01938
0.80001 0.80016 0.80026 0.80032 0.80092
O.06
0.07 O.08 0.09 0.10
12950 3504 1672 1001
O.02362 0.02262 0.02168 0.02112
O.90001 O.90008 0.90014 0.90001
9726 2643 1267 763
O.02340 0.02220 0.02127 0.02046
O.80004 0.80007 0.80032 0.80045
0.07
0.08 O.09 0.10
14764 3952 1870
0.02377 O.02283 0.02207
0.90001 O.90005 0.90016
11080 2974 1414
0.02360 O.02250 0.02153
0.80002 O.80001 0.80015
0.08
0.09 O. 10
16528 4391
0.02390 0.02303
0.90000 0.90006
12398 3304
0.02375 O.02274
0.80000 O.80002
0.09
O. 10
18255
O.02400
O.90001
13688
O.02385
O.80003
142 Table 3
R.G. Thomas a n d M. Conlon M i n i m u m S a m p l e Sizes B a s e d o n F i s h e r ' s Exact Test. E v e n t Rate R a n g e s : pl = 0.005(0.005)0.045; p2 = Pl + 0.005(0.005)0.05 ¢x = 0.05 (one-sided) 1 - f~ = 0.9
1 - f~ = 0.08
pl 0.005
p2 0.010 0.015 0.020 0.025 0.030 0.035 0.040 0.045 0.050
n 5325 1771 976 650 482 381 314 267 231
a* 0.04130 0.03696 0.03622 0.03428 0.03295 0.03207 0.03121 0.03356 0.03265
1 - 6" 0.90004 0.90003 0.90016 0.90035 0.90026 0.90005 0.90008 0.90109 0.90085
n 3972 1337 740 502 377 301 249 211 183
c~* 0.04004 0.03580 0.03383 0.03215 0.03116 0.03042 0.03118 0.03019 0.02937
1 - 6" 0.80001 0.80028 0.80043 0.80047 0.80026 0.80159 0.80173 0.80033 0.80023
0.010
0.015 0.020 0.025 0.030 0.035 0.040 0.045 0.050
8757 2654 1384 881 634 486 389 323
0.04416 0.04052 0.03894 0.03827 0.03643 0.03474 0.03483 0.03338
0.90002 0.90002 0.90004 0.90009 0.90045 0.90051 0.90020 0.90043
6442 1982 1028 666 479 369 298 250
0.04308 0.03939 0.03737 0.03572 0.03383 0.03284 0.031 44 0.03376
0.80005 0.80015 0.80021 0.80005 0.80067 0.80104 0.80018 0.80103
0.015
0.020 0.025 0.030 0.035 0.040 0.045 0.050
12119 3496 1763 1106 771 585 465
0.04551 0.04267 0.04099 0.03982 0.03823 0.03733 0.03580
0.90001 0.90005 0.90003 0.90004 0.90009 0.90015 0.90008
8861 2590 1309 819 582 443 352
0.04485 0.04128 0.03938 0.03831 0.03655 0.03466 0.03470
0.80000 0.80001 0.80004 0.80010 0.80025 0.80043 0.80036
0.020
0.025 0.030 0.035 0.040 0.045 0.050
15423 4333 2131 1311 903 677
0.04637 0.04392 0.04240 0.04037 0.03937 0.03843
0.90000 0.90003 0.90012 0.90017 0.90014 0.90018
11236 3183 1579 966 677 511
0.04582 0.04295 0.04084 0.03946 0.03792 0.03668
0.80002 0.80008 0.80015 0.80036 0.80022 0.80011
0.025
0.030 0.035 0.040 0.045 0.050
18684 5151 2497 1516 1038
0.04698 0.04478 0.04327 0.04210 0.04069
0.90000 0.90001 0.90010 0.90006 0.90015
13596 3768 1835 1117 771
0.04647 0.04417 0.04191 0.04086 0.03907
0.80001 0.80001 0.80016 0.80026 0.80042
0.030
0.035 0.040 0.045 0.050
21912 5955 2851 1719
0.04736 0.04533 0.04374 0.04250
0.90000 0.90003 0.90003 0.90010
15922 4355 2092 1268
0.04699 0.04458 0.04298 0.04142
0.80002 0.80006 0.80015 0.80028
0.035
0.040 0.045 0.050
25097 6751 3205
0.04771 0.04594 0.04430
0.90001 0.90001 0.90000
18231 4931 2351
0.04732 0.04524 0.04349
0.80001 0.80007 0.80001
0.040
0.045 0.050
28258 7536
0.04795 0.04632
0.90001 0.90000
20504 5498
0.04762 0.04559
0.80001 0.80000
0.045
0.050
31376
0.04814
0.90000
22757
0.04784
0.80001
143
Sample Size Based o n Fisher's Exact Test Table 3
Continued ~x = 0.05 (two-sided) 1 - ~ = 0.9
1 -
~--
0.8
pl 0.005
p2 0.010 0.015 0.020 0.025 0.030 0.035 0.040 0.045 0.050
n 6465 2142 1184 788 577 450 370 313 270
ct* 0.02040 0.01836 0.01739 0.01662 0.01723 0.01681 0.01644 0.01613 0.01625
1 - 6" 0.90001 0.90003 0.90014 0.90023 0.90053 0.90001 0.90009 0.90048 0.90037
n 4923 1656 922 612 454 359 296 250 217
a* 0.01978 0.01783 0.01657 0.01681 0.01671 0.01627 0.01582 0.01544 0.01500
1 - 6" 0.80001 0.80026 0.80032 0.80017 0.80085 0.80024 0.80149 0.80050 0.80241
0.010
0.015 0.020 0.025 0.030 0.035 0.040 0.045 0.050
10680 3224 1661 1067 764 579 458 381
0.02200 0.02041 0.01935 0.01875 0.01802 0.01814 0.01742 0.01684
0.90000 0.90009 0.90009 0.90009 0.90015 0.90027 0.90053 0.90063
8059 2456 1282 826 588 444 360 302
0.02154 0.01973 0.01854 0.01776 0.01788 0.01697 0.01643 0.01605
0.80000 0.80003 0.80013 0.80047 0.80065 0.80031 0.80036 0.80101
0.015
0.020 0.025 0.030 0.035 0.040 0.045 0.050
14777 4247 2128 1330 938 698 552
0.02266 0.02130 0.02031 0.01978 0.01901 0.01826 0.01833
0.90000 0.90004 0.90001 0.90001 0.90019 0.90002 0.90040
11149 3225 1627 1025 723 538 424
0.02240 0.02072 0.01969 0.01910 0.01816 0.01806 0.01731
0.80004 0.80013 0.80007 0.80033 0.80028 0.80018 0.80078
0.020
0.025 0.030 0.035 0.040 0.045 0.050
18822 5270 2578 1585 1093 817
0.02314 0.02186 0.02088 0.02016 0.01967 0.01912
0.90000 0.90005 0.90006 0.90010 0.90007 0.90010
14173 3989 1961 1202 839 622
0.02286 0.02141 0.02027 0.01981 0.01894 0.01896
0.80000 0.80001 0.80001 0.80001 0.80012 0.80051
0.025
0.030 0.035 0.040 0.045 0.050
22833 6275 3029 1836 1256
0.02345 0.02234 0.02133 0.02086 0.02032
0.90001 0.90000 0.90003 0.90013 0.90022
17163 4741 2295 1402 960
0.02323 0.02195 0.02100 0.02017 0.01930
0.80001 0.80008 0.80008 0.80015 0.80024
0.030
0.035 0.040 0.045 0.050
26791 7257 3467 2084
0.02367 0.02269 0.02187 0.02116
0.90001 0.90001 0.90007 0.90004
20110 5478 2629 1586
0.02346 0.02229 0.02138 0.02071
0.80002 0.80003 0.80009 0.80025
0.035
0.040 0.045 0.050
30704 8237 3906
0.02383 0.02291 0.02215
0.90000 0.90002 0.90007
23036 6202 2950
0.02363 0.02262 0.02173
0.80000 0.80005 0.80002
0.040
0.045 0.050
34573 9203
0.02395 0.02310
0.90001 0.90001
25931 6928
0.02379 0.02288
0.80001 0.80005
0.045
0.050
38403
0.02405
0.90000
28786
0.02391
0.80001
144 Table 4
R.G. Thomas and M. Conlon M i n i m u m S a m p l e S i z e s B a s e d o n F i s h e r ' s Ex act Test. E v e n t R a t e R a n g e s : Pl = 0.001(0.001)0.009; P2 = pl + 0.001(0.001)0.01 c~ = 0.05 (one-sided) 1 - [3 = 0.9
1 - 13 = 0.08
pl 0.001
p2 0.002 0.003 0.004 0.005 0.006 0.007 0.008 0.009 0.010
n 26977 8884 4898 3263 2423 1918 1582 1343 1164
~* 0.04100 0.03785 0.03651 0.03558 0.03462 0.03389 0.03336 0.03297 0.03265
1 - ~* 0.90000 0.90001 0.90001 0.90001 0.90009 0.90007 0.90008 0.90022 0.90028
n 19929 6698 3709 2518 1893 1508 1249 1062 922
~* 0.03922 0.03665 0.03493 0.03353 0.03227 0.03226 0.03186 0.03150 0.03118
1 - ~* 0.80001 0.80005 0.80008 0.80014 0.80012 0.80000 0.80037 0.80027 0.80029
0.002
0.003 0.004 0.005 0.006 0.007 0.008 0.009 0.010
44265 13481 6963 4439 3189 2447 1963 1630
0.04406 0.04084 0.03928 0.03763 0.03668 0.03622 0.03576 0.03523
0.90000 0.90002 0.90004 0.90005 0.90001 0.90008 0.90014 0.90012
32557 9960 5173 3347 2406 1853 1501 1258
0.04329 0.03907 0.03753 0.03644 0.03564 0.03465 0.03381 0.03318
0.80001 0.80003 0.80002 0.80006 0.80014 0.80008 0.80025 0.80022
0.003
0.004 0.005 0.006 0.007 0.008 0.009 0.010
61352 17770 8930 5592 3925 2957 2350
0.04565 0.04293 0.04068 0.04019 0.03848 0.03741 0.03667
0.90000 0.90000 0.90001 0.90001 0.90007 0.90007 0.90003
44830 13113 6627 4159 2930 2230 1774
0,04483 0.04169 0.03962 0.03807 0.03681 0.03622 0.03571
0.80000 0.80002 0.80004 0.80006 0.80001 0.80008 0.80013
0.004
0.005 0.006 0.007 0.008 0.009 0.010
78437 22070 10891 6694 4647 3476
0.04645 0.04404 0.04258 0.04077 0.04037 0.03910
0.90000 0.90001 0.90001 0.90004 0.90001 0.90000
57191 16201 8037 4968 3461 2584
0.04581 0.04307 0.04074 0.04020 0.03835 0.03717
0.80000 0.80000 0.80000 0.80006 0.80003 0.80009
0.005
0.006 0.007 0.008 0.009 0.010
95471 26336 12782 7798 5325
0.04703 0.04486 0.04305 0.04202 0.04130
0.90000 0.90000 0.90000 0.90002 0.90004
69490 19293 9433 5721 3972
0,04654 0.04401 0.04247 0.04121 0.04004
0.80000 0.80002 0.80002 3.80002 0.80001
0.006
0.007 0.008 0.009 0.010
112434 30613 14698 8847
0.04746 0.04564 0.04405 0.04307
0.90000 0.90000 0.90001 0.90000
81709 22369 10764 6540
0.04708 0.04492 0.04294 0.04184
0.80000 0.80000 0.80002 0.80004
0.007
0.008 0.009 0.010
129359 34843 16592
0.04778 0.04598 0.04464
0.90000 0.90000 0.90001
93946 25400 12145
0.04739 0.04542 0.04365
0.80000 0.80001 0.80002
0.008
0.009 0.010
146209 39079
0.04802 0.04644
0.90000 0.90000
106089 28463
0.04771 0.04594
0.80000 0,80001
0.009
0.010
163034
0.04823
0.90000
118272
0.04790
0.80000
145
Sample Size Based on Fisher's Exact Test Table 4
Continued c~ = 0.05 (two-sided) 1
-
13 =
0.9
1
-
0.8
13 =
pl 0.001
p2 0.002 0.003 0.004 0.005 0.006 0.007 0.008 0.009 0.010
n 32628 10742 5940 3956 2902 2278 1868 1577 1362
~* 0.02028 0.01876 0.01800 0.01743 0.01698 0.01651 0.01607 0.01569 0.01539
1 - 13" 0.90000 0.90002 0.90004 0.90005 0.90009 0.90013 0.90017 0.90002 0.90023
n 24886 8295 4621 3074 2284 1805 1485 1257 1088
~* 0.01962 0.01816 0.01728 0.01638 0.01559 0.01510 0.01480 0.01461 0.01448
1 - 13" 0.80002 0.80005 0.80009 0.80009 0.80019 0.80014 0.80024 0.80011 0.80014
0.002
0.003 0.004 0.005 0.006 0.007 0.008 0.009 0.010
53848 16255 8405 5367 3845 2967 2388 1976
0.02197 0.02022 0.01975 0.01880 0.01816 0.01781 0.01749 0.01721
0.90000 0.90000 0.90002 0.90002 0.90006 0.90001 0.90011 0.90009
40716 12405 6437 4145 3007 2309 1849 1536
0.02153 0.01999 0.01884 0.01802 0.01752 0.01710 0.01665 0.01616
0.80001 0.80000 0.80000 0.80003 0.80014 0.80012 0.80002 0.80021
0.003
0.004 0.005 0.006 0.007 0.008 0.009 0.010
74925 21604 10801 6746 4741 3576 2833
0.02270 0.02141 0.02044 0.01983 0.01938 0.01865 0.01815
0.90000 0.90001 0.90003 0.90002 0.90006 0.90008 0.90000
56503 16421 8212 5170 3639 2762 2212
0.02237 0.02083 0.01999 0.01926 0.01832 0.01788 0.01752
0.80000 0.80002 0.80001 0.80003 0.80005 0.80007 0.80021
0.004
0.005 0.006 0.007 0.008 0.009 0.010
95825 26894 13169 8086 5601 4197
0.02319 0.02197 0.02106 0.02050 0.01990 0.01951
0.90000 0.90001 0.90001 0.90002 0.90001 0.90003
72089 20308 10017 6157 4300 3216
0.02292 0.02157 0.02060 0.01988 0.01939 0.01860
0.80000 0.80001 0.80001 0.80006 0.80003 0.80010
0.005
0.006 0.007 0.008 0.009 0.010
116666 32117 15541 9434 6465
0.02350 0.02243 0.02157 0.02097 0.02040
0.90000 0.90001 0.90001 0.90001 0.90001
87695 24251 11753 7159 4923
0.02325 0.02198 0.02102 0.02045 0.01978
0.80000 0.80001 0.80001 0.80001 0.80001
0.006
0.007 0.008 0.009 0.010
137514 37328 17844 10747
0.02371 0.02279 0.02203 0.02120
0.90000 0.90000 0.90002 0.90002
103239 28157 13528 8145
0.02351 0.02238 0.02146 0.02083
0.80000 0.80001 0.80002 0.80001
0.007
0.008 0.009 0.010
158255 42529 20198
0.02386 0.02297 0.02232
0.90000 0.90000 0.90001
118721 32020 15241
0.02370 0.02269 0.02187
0.80000 0.80001 0.80002
0.008
0.009 0.010
178924 47724
0.02398 0.02315
0.90000 0.90000
134191 35874
0.02384 0.02294
0.80000 0.80001
0.009
0.010
199618
0.02409
0.90000
149581
0.02395
0.80000
146
R.G. Thomas and M. Cordon ing trial given in the "Introduction." In this trial the expected e v e n t rates are pl = 0.001 a n d P2 0.004. Let the specified values of o~ and 1 - [3 be 0.05 (two-sided) and 0.8, respectively. Table 4 provides calculations of n based on Fisher's Exact Test for p l , p 2 = 0.001, 0.001(0,01) and pl < P2. The third r o w of the table c o r r e s p o n d s to the (Pl,p2) = (0.001, 0.004) e v e n t rate combination. The third c o l u m n from the right contains 4621, the required m i n i m u m sample size in b o t h the experimental and control group. The associated empirical type I error for this p a r a m e t e r combination is 0.0173. This oL*should be comp a r e d to the design value 0.025. =
SUMMARY
This p a p e r provides sample size calculations based o n Fisher's Exact Test for 2 × 2 comparative trials in w h i c h the expected e v e n t rates in b o t h the experimental and control g r o u p are low. The empirical type I errors are calculated to provide a quantification of the conservativeness of the test for a given combination of s t u d y parameters. In m a n y cases, the sample sizes p r e s e n t e d in these tables are smaller t h a n sample sizes calculated using formulae based on asymptotic p r o c e d u r e s since these expressions d o not p r o d u c e accurate results w h e n the e v e n t rates are low.
REFERENCES
1. Fleiss J: Statistical Methods for Rates and Proportions. New York, Wiley, 1973 2. Cochran WG, Cox GM: Experimental Designs, 2nd ed. New York, Wiley, 1957 3. Cohen J: Statistical Power Analysis for the Behavioral Sciences. New York, Academic Press, 1969 4. Feigl P: A graphical aid for determining sample size when comparing two independent proportions. Biometrics 34:111-122, 1978 5. Gail M, Gart JJ: The determination of sample sizes for use with the conditional test in 2 x 2 comparative trials. Biometrics 29:441-448, 1973 6. Haseman JK: Exact sample sizes for use with the Fisher-Irwin Test for 2 x 2 tables. Biometrics 34:106-109, 1978 7. Casagrande JT, Pike MC, Smith PG: Algorithm AS 129. The power function of the "exact" test for comparing two binomial distributions. Appl Stat 27:212-219, 1978 8. Suissa S, Shuster JJ: Exact unconditional sample sizes for the 2 × 2 comparative trial. J Roy Stat Soc, Series A 148:317-327, 1985 9. Lemeshow S, Hosmer DW, Stewart JP: A comparison of sample size determination methods in the two group trial where the underlying disease is rare. Commun Stat--Simulation and Computation B10(5):437-449, 1981 10. Armitage P: Statistical Methods in Medical Research. Oxford, Blackwell Scientific Publications, 1971 11. Snedecor GW, Cochran WG: Statistical Methods. Ames, Iowa, Iowa State University Press, 1980 12. Thomas RG, Cordon M: An algorithm for the rapid evaluation of the power function for Fisher's Exact Test. University of Florida Department of Statistics Technical Report, No. 382, 1991 13. ISIS-2 Collaborative Group: Randomized trial of intravenous streptokinase, oral
Sample Size Based on Fisher's Exact Test
14. 15. 16. 17. 18. 19.
147
aspirin, both or neither among 17,187 cases of suspected acute myocardial infarction: ISIS-2. Lancet 2:349, 1988 Steering Committee of the Physician's Health Study Research Group: Final report on the aspirin component of the ongoing physicians' health study. N Engl J Med 321:129, 1989 Fisher RA: The logic of inductive inference. J Roy Stat Soc 98:39-54, 1935 Berkson J: In dispraise of the exact test: Do the marginal totals of the 2 x 2 table contain relevant information respecting the table proportions? J Star Plan Inf 2:2742, 1978 Casagrande JT, Pike MC, Smith PG: An improved formula for calculating sample sizes for comparing two binomial distributions. Biometrics 34:483-486, 1978 Dobson VJ, Gebski V: Sample sizes for comparing two independent proportions using the continuity-corrected arc sine transformation. Statistidan 35:51-53, 1986 Waiters DE: In defense of the arc sine approximation. The Statistician 28:219-222, 1979
ABSTRACT: A collection of sample size tables are presented for designing comparative trials
when the event rates pl and p2 are low. The tables are based on exact power calculations for Fisher's Exact Test. Both one-sided and two-sided alternative hypotheses are considered. A comparison is made between these sample sizes and those obtained by using popular asymptotic approximations. KEY WORDS: Sample size, Fisher's Exact Test
INTRODUCTION Sample size determination is a fundamental c o m p o n e n t of the clinical trial planning process. In the specific case of a 2 x 2 comparative trial aimed at contrasting event rates between experimental and control therapies, the calculation of the required sample size d e p e n d s on the values of four parameters: (1) pl, the true event rate in the experimental group; (2) P2, the true event rate in the control group; (3) c~, the probability of incorrectly rejecting the null hypothesis H0 : pl = p2, i.e., the type I error; and (4) ~, the probability of incorrectly failing to reject the null hypothesis, i.e., the type II error. Once these parameters have been specified, the trial designers can proceed to consult any of several sets of tables available in textbooks or in the literature to obtain n, the required m i n i m u m n u m b e r of observations per group. The appropriate table to use is determined by the testing procedure planned for use at final analysis. For example, if the hypothesis H0 : pl = p2 will be tested using the standard X2 statistic, then tables contained in Fleiss [1] and elsewhere are appropriate. If, instead, the arc sine test is to be applied, tables in Cochran and Cox [2] or Cohen [3] are available as are isographs in Feigl [4]. A third alternative is Fisher's Exact Test, for which Gail and Gart [5], H a s e m a n [6], and Casagrande et al. [7] have presented tables. Finally, Suissa and Shuster
Address reprint requests to: Ronald G. Thomas, PhD, Cooperative Studies Program Coordinating Center, 151-K, VA Medical Center, 3801 Miranda Avenue, Palo Alto, CA 94204. Received April 25, 1991; revised September 12, 1991.
134 0197-2456/92/$5.00
Controlled Clinical Trials 13:134-147 (1992) ¢~, Elsevier Science Publishing Co., Inc. 1992 655 A v e n u e t~f tht~ Americas, New York, New York 10010
Sample Size Based on Fisher's Exact Test
135
[8] have published sample size tables based on an exact unconditional testing procedure. Due to the ready availability of tabled values, calculating sample sizes is usually a straightforward procedure. However, if the anticipated event rates (in one or both groups) are low, say less than 0.1, then determining n can be problematic. To illustrate, consider an example borrowed (with modification) from Lemeshow et al. [9]. A trial is being designed to evaluate a new cancer screening and treatment program. From previous studies, the mortality rate from this disease for men over 60 can be reliably estimated at 400/100,000. Data from a recently completed pilot study offer evidence that an aggressive early screening program may reduce the death rate to as low as 100/100,000. Based on this evidence; the trial designers set the expected event rates at Pl = 0.001 and P2 = 0.004. A statistician charged with determining a sample size for this trial will soon realize that none of the existing tables contain tabulated values for event rates this low. This is not a serious problem, however, if the anticipated final analysis procedure is one that allows a closed form expression for n. For example, the standard ×2 test yields the formula:
n . (z~_~,V~p~+ . . . .
zl_.X/plql+ . p2q2) 2
(1)
P2 - pl where qi = 1 - pi, i = 1, 2, p is the observed proportion of successes in both groups, and z ~ _ ~ denotes the (1 - 00th deviate of the standard normal distribution. From this expression, n can be calculated for any combination of parameters p~, p2, 0~, and [3. Most testing procedures which appeal to asymptotic normality for inference can be "inverted" to provide a closed form expression for n. Unfortunately, when Pl and P2 are low--as in our example--most statisticians would be hesitant to use a testing procedure which relies on the assumption of approximate normality. For cases where the normality assumption is questionable, the usual textbook recommendation is to use Fisher's Exact Test. See, for example, Armitage [10] and Snedecor and Cochran [11]. However, the literature contains no tables of n based on Fisher's Exact Test for low p~ and P2. Sample size calculation for Fisher's Exact Test is a computationally intensive, iterative procedure. First, the empirical power of the test for a given n, Pl, P2, and oLmust be determined. Then, the sample size is adjusted in iterative fashion until the smallest n for which the empirical power is greater than or equal to 1 - ~ is found. The computational difficulty increases exponentially as n increases. The realization of the need for tabulated values of n based on Fisher's Exact Test for low p~ and p2 prompted the authors to develop an algorithm for the calculation of the power function for Fisher's Exact Test [12]. Through the use of this algorithm, it is possible with modest computing expenditure to determine a value for n even when the underlying event rates are very low. Clearly, in today's medical research environment, therapeutic improvement is usually evolutionary rather than revolutionary. Consequently, more and more clinical trials, particularly in chronic diseases, are being designed
136
R.G. Thomas and M. Conlon to detect a small reduction in an event rate that is already relatively low. Recent examples of such trials are ISIS-2 [13] a n d the Physician's Health S t u d y [14]. As the resources to c o n d u c t large-scale projects of this type b e c o m e scarce, the n e e d to be able to determine sample sizes based on e x a c t - - r a t h e r than approximate---calculations becomes increasingly important. The r e m a i n d e r of the p a p e r contains a description of the p o w e r function for Fisher's Exact Test, followed by an exploration of the validity of the normal approximation to the binomial w h e n event rates are low. The "Results" section presents tables containing calculated values for the m i n i m u m sample size required for a level ct and p o w e r 1 - {3 Fisher's Exact Test of H0 : p1 = p2 against both one- a n d two-sided alternatives.
POWER F U N C T I O N FOR FISHER'S EXACT TEXT Consider a 2 x 2 comparative trial d e s i g n e d to c o m p a r e e v e n t rates b e t w e e n treatment a n d control groups. Define X as the n u m b e r of events occurring in the treatment g r o u p and Y as the n u m b e r of events in the control group. Let both X and Y be binomially distributed r a n d o m variables with index n and parameters Pl and P2, respectively. With this notation we can display the o u t c o m e of the trial in tabular form as
Treatment Group Control Group
Event
No event
x Y
n - x n - Y
I n / n
r
2n - r
2n
I I
w h e r e R = X + Y d e n o t e s the total n u m b e r of events. Fisher's Exact Test is based on the a r g u m e n t that inference regarding the null h y p o t h e s i s can be d r a w n conditionally on r and n. Fisher [15] d e m o n strated that the conditional distribution of X, given R = r, d e p e n d s on the event rates Pl and P2 only t h r o u g h the o d d s ratio 0, w h e r e 0 = pl(1 - p2)/p2(1 - pl). That is, 0x
h(xlr; O) = Pr[X = xlr; O] =
min(r,n)~(~)( lZ ) i ~ max(O,r-
~t)
/"
--
0i
i
U n d e r the null h y p o t h e s i s H0 : Pl = p2 ( 0 = 1), h simplifies to the h y p e r g e o metric distribution a n d the test can be defined i n d e p e n d e n t l y of the event rates. Thus, Fisher's one-sided level ot test rejects the null h y p o t h e s i s if the conditional probability of observing a value of X at least as extreme as x is less than or equal to or. The conditional rejection region is defined as
C~r~ = {(x,y) : x + y = r
and
~. i = max(O,r
h(ilr; 0 = 1) ~< et} n)
137
Sample Size Based on Fisher's Exact Test The p o w e r of the test is defined as 2n
~,(n) = Pr[(x,y) ~ C~r)IH1] = ~ ~ f(x,ylp~, P2, n),
(2)
r = OC(r)
where
f(x,ylp,,p2,n) =
x
Y
pI(1 - pl)"-Xp~(1 - p2)n-y
the joint binomial mass function. If we let 2n
= U cF) r=O
d e n o t e the unconditional rejection region, then we can alternatively write the p o w e r function as
~l(n) = ~_~ f(x,ylp,,p2,n)
(3)
CF
Several authors, e.g., Berkson [16], have p o i n t e d out that Fisher's Exact Test is conservative. That is, the empirical type I error of the test is usually substantially less than the nominal significance level o~. It is straightforward to calculate the empirical type I error as
o~* = ~ f(x,ylp2,P2,n) CF
For each combination of p~ and p2 in Tables 2-4, we have calculated ~* along with the sample size. It is left to the reader to make his or her o w n j u d g m e n t as to w h e t h e r the d e g r e e of conservativeness precludes the usefulness of Fisher's Exact Test for a given combination of parameters. Since the sample size is calculated b y iterating n until the smallest n such that ~/(n)/> 1 - .B is d e t e r m i n e d , the empirical type II error will be, at most, only slightly less than J3 and n e v e r greater. EXACT ERROR EVALUATION W h e n trying to decide b e t w e e n sample size tables based on exact versus asymptotic calculations, it is reasonable to ask, U n d e r w h a t circumstances is the normal approximation to the binomial inappropriate? In other words, for w h a t combinations of pl and p2 should large sample m e t h o d s be a b a n d o n e d in favor of Fisher's Exact Test or some other exact m e t h o d ? Intuitively, it w o u l d seem that as one or both of the e v e n t rates becomes low, the validity of the a s s u m p t i o n of approximate normality w o u l d deteriorate. But h o w low is low? To address this question, we have evaluated the empirical type I and type II errors associated with sample sizes obtained using equation (1) for a range of Pl and p2 values and for oL = 0.05 and J3 = 0.1. The results are given in Table 1 (to facilitate interpretation Table 1 presents 1 - 13" rather than ~*). Equation (1) is based on the standard ×2 statistic, Z 2, or equivalently Z, the normalized difference b e t w e e n observed proportions.
138
R.G. Thomas and M. Conlon
Table 1
F r o m left to r i g h t : M i n i m u m S a m p l e Size, n (c, = 0.05 a n d 1 - f~ = 0.9), B a s e d o n S t a n d a r d (×2); E m p i r i c a l T y p e I E r r o r a* f r o m S t a n d a r d ×2; E m p i r i c a l P o w e r 1 - [3* f r o m S t a n d a r d ×2; n B a s e d o n F i s h e r ' s Exact T e s t (FET); n B a s e d o n C a s a g r a n d e , P i k e , a n d S m i t h (1978) (CPS); n B a s e d o n D o b s o n a n d G e b s k i (1986) (DG) 1
n
n
n
p,
P2
n (×2)
~,(×2)
~,(×2)
-
(FET)
(CPS)
(DG)
0.00001
0.00010 0.00100 0.01000 0.02000 0.04000 0.06000 0.08000 0.10000
115600 8649 841 400 196 121 100 64
0.04882 0.04927 0.04955 0.04866 0.05053 0.05470 0.05286 0.05141
0.94975 0.98058 0.98906 0.98618 0.98563 0.97839 0.98865 0.96098
116576 8300 801 399 199 132 98 78
137597 10743 1044 518 256 169 125 99
120071 10506 1728 1071 677 522 435 377
0.00010
0.00100 0.01000 0.02000 0.04000 0.06000 0.08000 0.10000
11449 841 400 196 121 100 64
0.04874 0.04955 0.04866 0.05053 0.05470 0.05286 0.05141
0.94804 0.97816 0.97991 0.98402 0.97706 0.98793 0.95993
11656 829 406 200 133 99 79
13753 1069 525 258 169 126 99
12003 1049 582 336 247 200 170
0.00100
0.01000 0.02000 0.04000 0.06000 0.08000 0.10000
1089 484 196 121 100 81
0.04799 0.04943 0.05053 0.05470 0.05286 0.05107
0.93874 0.95860 0.96567 0.96289 0.98017 0.98384
1164 484 218 140 103 81
1368 592 274 176 129 102
1197 513 245 165 126 103
0.01000
0.02000 0.04000 0.06000 0.08000 0.10000
2500 441 225 144 100
0.05039 0.05188 0.04960 0.05259 0.05099
0.90356 0.91112 0.92654 0.93802 0.94007
2654 486 239 156 114
2725 527 268 176 130
2661 491 244 158 116
0.02000
0.04000 0.06000 0.08000 0.10000
1225 400 196 144
0.05023 0.05059 0.04923 0.05226
0.90285 0.91236 0.88986 0.91736
1311 437 236 159
1343 458 257 174
1312 437 240 161
0.04000
0.06000 0.08000 0.10000
2025 576 289
0.04998 0.05008 0.04971
0.90116 0.89380 0.89263
2113 634 327
2131 651 341
2114 638 330
0.06000
0.08000 0.10000
2704 784
0.04998 0.05018
0.89337 0.90314
2872 826
2885 836
2874 826
0.08000
0.10000
3481
0.05007
0.89903
3597
3604
3596
If t h e n o r m a l a p p r o x i m a t i o n to t h e b i n o m i a l is a d e q u a t e , w e w o u l d e x p e c t t h e e m p i r i c a l t y p e I a n d t y p e II e r r o r s (x* a n d [3* for s a m p l e s i z e s b a s e d o n Z to b e c l o s e to t h e d e s i g n p a r a m e t e r s oL a n d fk T h e e m p i r i c a l t y p e I e r r o r for Z is d e f i n e d a s
eL* = Pr[Z > Zl-~IHo] = ~ f(x,yIp2,p2,n) Cz
139
Sample Size Based on Fisher's Exact Test w h e r e C~ = fix,y) : Z > zl - ~}. Similarly, the empirical type II error can be calculated as [3* = Pr[Z < zl-~lH1] = ~_, f(x,y[p,,p2,n) = 1 - ~ f(x,ylpl,p2,n) C~
CZ
c
w h e r e Cz = ~ - Cz, and ~ = {0. . . . . n } ~) {0. . . . . n}--the sample space. Table 1 shows that u n d e r the null h y p o t h e s i s the normal approximation to the binomial is good, e v e n w h e n p2 is as low as 10 -4. H o w e v e r , u n d e r the alternative, p~ < pa, the empirical type II error can be as m u c h as 90% less than [3. In fact, for some combinations in this table the "conservative" Fisher's Exact Test yields a smaller n than does the standard ×a. Clearly, as Pl a n d P2 b e c o m e more disparate, the less accurate the approximation becomes. H o w ever, Table 1 also shows that for tabled values with Pl greater than or equal to 0.04 both the empirical type I and type II errors are v e r y close to the optimal values e~ -- 0.05 and 13 = 0.1. If one adopts the ad hoc rule that both empirical errors should be within 10% of the design values (i.e., 0.045 ~ or* ~ 0.055 and 0.89 ~ 1 - 13" ~ 0.91), t h e n a crude interpretation of this table suggests that both p~ and p2 need to be at least as large as 0.04 for the normal approximation to the binomial to be applied. (Unless P2 - Pl is small, say 0.01, t h e n p~ could be as low as 0.01.) This table also presents values based on Fisher's Exact Test and on the sample size formulas in Casagrande et al. (CPS) [17] and Dobson and Gebski (DG) [18]. The CPS formula was derived as a c o m p r o m i s e b e t w e e n calculations based on the uncorrected s t a n d a r d X2, and the ×2 with Yates's correction. The authors s h o w e d that for 0.05 < p~,P2 < 0.75 their formula gave a close approximation to the sample size based on Fisher's Exact Test. H o w e v e r , w h e n p~ and p2 are low, as in Table 1, the CPS formula gives n u m b e r s as m u c h as 29% larger than Fisher's Exact Test. The DG equation is derived from the continuity corrected arc sine m e t h o d of Walters [19] and gives an excellent approximation to the exact results for "larger" values of pl and Pa, but it too gives values considerably larger than Fisher's Exact Test for m a n y of the p a r a m e t e r combinations in Table 1. RESULTS Tables 2 - 4 present sample sizes for use in designing 2 x 2 comparative trials with low expected e v e n t probabilities. Table 2 covers pl and P2 in the range Pl -- 0.01(0.01)0.9 a n d p2 = pl + 0.01(0.01)0.1. (The notation Pl = 0.01(0.01)0.9, for example, indicates that pl takes on all values in the range 0.01 to 0.9 in increments of 0.01.) Table 3 is for Pl and p2 in the range Pl = 0.005(0.005)0.045 and P2 = Pl + 0.005(0.005)0.05. Table 4 gives sample sizes for Pl and p2 in the range Pl = 0.001(0.001)0.009 and P2 = pl + 0.001(0.001)0.01. Each r o w of each table is organized into four sets of three n u m b e r s . The sets c o r r e s p o n d to different design objectives for type I and type II error r a t e s - one- and two-sided cx = 0.05 and 80% and 90% power. Within each set of objectives, three n u m b e r s are presented---the m i n i m u m sample size required in each g r o u p to achieve the design objectives using Fisher's Exact Test, the empirical type I error rate, and the empirical p o w e r of the study. To describe the use of the tables, consider the example of the cancer screen-
140 Table 2
R.G. Thomas and M. Conlon M i n i m u m S a m p l e S i z e s B a s e d o n F i s h e r ' s Exact Test. E v e n t R a t e R a n g e s : pl = 0.01(0.01)0.9; P2 = pl + 0.01(0.01)0.1 o~ = 0.05 (one-sided) 1 - [8 = 0.9
1 - [8 = 0.08
pl 0.01
p2 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.10
n 2654 881 486 323 239 189 156 132 114
~* 0.04052 0.03827 0.03474 0.03338 0.03468 0.03311 0.03165 0.03051 0.03334
1 - [8* 0.90002 0.90009 0.90051 0.90043 0.90017 0.90032 0.90122 0.90094 0.90057
n 1982 666 369 250 188 150 124 105 87
a* 0.03939 0.03572 0.03284 0.03376 0.03162 0.02991 0.02849 0.03061 0.02888
1 - [8* 0.80015 0.80005 0.80104 0.80103 0.80177 0.80322 0.80350 0.80227 0.80121
0.02
0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.10
4333 1311 677 437 314 236 191 159
0.04392 0.04037 0.03843 0.03717 0.03636 0.03521 0.03472 0.03306
0.90003 0.90017 0.90018 0.90021 0.90064 0.90027 0.90083 0.90068
3183 966 511 331 235 182 148 124
0.04295 0.03946 0.03668 0.03582 0.03506 0.03354 0.03251 0.03052
0.80008 0.80036 0.80011 0.80038 0.80143 0.80036 0.80172 0.80194
0.03
0.04 0.05 0.06 0.07 0.08 0.09 0.10
5955 1719 862 538 375 287 223
0.04533 0.04250 0.04027 0.03893 0.03812 0.03700 0.03566
0.90003 0.90010 0.90016 0.90010 0.90053 0.90043 0.90090
4355 1268 638 400 278 213 171
0.04458 0.04142 0.03963 0.03761 0.03611 0.03611 0.03391
0.80006 0.80028 0.80034 0.80003 0.80006 0.80090 0.80049
0.04
0.05 0.06 0.07 0.08 0.09 0.10
7536 2113 1037 634 440 327
0.04632 0.04361 0.04168 0.04061 0.03906 0.03833
0.90000 0.90003 0.90019 0.90022 0.90037 0.90027
5498 1554 767 476 328 244
0.04559 0.04261 0.04067 0.03928 0.03757 0.03720
0.80000 0.80018 0.80035 0.80020 0.80062 0.80053
0.05
0.06 0.07 0.08 0.09 0.10
9084 2495 1207 732 503
0.04686 0.04452 0.04277 0.04159 0.04004
0.90001 0.90001 0.90017 0.90034 0.90001
6612 1830 889 543 371
0.04641 0.04391 0.04177 0.03991 0,03891
0.80001 0.80009 0.80037 0.80051 {).80004
0.06
0.07 0.08 0.09 0.10
10593 2872 1373 826
0.04730 0.04526 0.04352 0.04211
0.90001 0.90003 0.90011 0.90033
7702 2102 1013 607
0.04682 0.04458 0.04251 0.04108
0.80004 0.80016 0.80029 0.80017
0.07
0.08 0.09 0.10
12068 3237 1537
0.04760 0.04578 0.04419
0.90002 0.90005 0.90009
8767 2366 1127
0.04723 0.04515 0.04330
0.80003 0.80007 0.80024
0.08
0.09 0.10
13507 3597
0.04785 0.04615
0.90001 0.90007
9807 2623
0.04750 0.04552
0.80000 0.80010
0.09
0.10
14916
0.04804
0.90002
10824
0.04770
0.80003
141
Sample Size Based on Fisher's Exact Test Table 2
Continued ~x = 0.05 (two-sided) 1 - B -- 0.9
1 - B = 0.8
pl 0.01
p2 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.10
n 3224 1067 579 381 283 224 184 155 134
~* 0.02041 0.01875 0.01814 0.01684 0.01595 0.01638 0.01548 0.01624 0.01535
1 - B* 0.90009 0.90009 0.90027 0.90063 0.90044 0.90120 0.90132 0.90059 0.90170
n 2456 826 444 302 226 179 147 124 106
~* 0.01973 0.01776 0.01697 0.01605 0.01586 0.01498 0.01490 0.01397 0.01476
1 - ~* 0.80003 0.80047 0.80031 0.80101 0.80121 0.80189 0.80156 0.80042 0.80314
0.02
0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.10
5270 1585 817 519 371 283 226 188
0.02186 0.02016 0.01912 0.01849 0.01790 0.01730 0.01721 0.01636
0.90005 0.90010 0.90010 0.90052 0.90044 0.90080 0.90010 0.90024
3989 1202 622 400 286 221 179 150
0.02141 0.01981 0.01896 0.01778 0.01655 0.01680 0.01566 0.01652
0.80001 0.80001 0.80051 0.80001 0.80057 0.80188 0.80168 0.80226
0.03
0.04 0.05 0.06 0.07 0.08 0.09 0.10
7257 2084 1043 651 454 343 267
0.02269 0.02116 0.02035 0.01971 0.01847 0.01832 0.01814
0.90001 0.90004 0.90012 0.90004 0.90038 0.90040 0.90049
5478 1586 797 496 351 262 209
0.02229 0.02071 0.01960 0.01867 0.01802 0.01793 0.01721
0.80003 0.80025 0.80016 0.80026 0.80069 0.80168 0.80107
0.04
0.05 0.06 0.07 0.08 0.09 0.10
9203 2571 1256 771 531 394
0.02310 0.02180 0.02072 0.01991 0.01930 0.01910
0.90001 0.90011 0.90010 0.90017 0.90002 0.90004
6928 1942 960 586 407 306
0.02288 0.02127 0.02024 0.01941 0.01885 0.01839
0.80005 0.80001 0.80006 0.80073 0.80085 0.80157
0.05
0.06 0.07 0.08 0.09 0.10
11096 3043 1468 889 606
0.02343 0.02226 0.02146 0.02063 0.02013
0.90001 0.90002 0.90017 0.90004 0.90010
8338 2302 1116 676 466
0.02320 0.02180 0.02088 0.02008 0.01938
0.80001 0.80016 0.80026 0.80032 0.80092
O.06
0.07 O.08 0.09 0.10
12950 3504 1672 1001
O.02362 0.02262 0.02168 0.02112
O.90001 O.90008 0.90014 0.90001
9726 2643 1267 763
O.02340 0.02220 0.02127 0.02046
O.80004 0.80007 0.80032 0.80045
0.07
0.08 O.09 0.10
14764 3952 1870
0.02377 O.02283 0.02207
0.90001 O.90005 0.90016
11080 2974 1414
0.02360 O.02250 0.02153
0.80002 O.80001 0.80015
0.08
0.09 O. 10
16528 4391
0.02390 0.02303
0.90000 0.90006
12398 3304
0.02375 O.02274
0.80000 O.80002
0.09
O. 10
18255
O.02400
O.90001
13688
O.02385
O.80003
142 Table 3
R.G. Thomas a n d M. Conlon M i n i m u m S a m p l e Sizes B a s e d o n F i s h e r ' s Exact Test. E v e n t Rate R a n g e s : pl = 0.005(0.005)0.045; p2 = Pl + 0.005(0.005)0.05 ¢x = 0.05 (one-sided) 1 - f~ = 0.9
1 - f~ = 0.08
pl 0.005
p2 0.010 0.015 0.020 0.025 0.030 0.035 0.040 0.045 0.050
n 5325 1771 976 650 482 381 314 267 231
a* 0.04130 0.03696 0.03622 0.03428 0.03295 0.03207 0.03121 0.03356 0.03265
1 - 6" 0.90004 0.90003 0.90016 0.90035 0.90026 0.90005 0.90008 0.90109 0.90085
n 3972 1337 740 502 377 301 249 211 183
c~* 0.04004 0.03580 0.03383 0.03215 0.03116 0.03042 0.03118 0.03019 0.02937
1 - 6" 0.80001 0.80028 0.80043 0.80047 0.80026 0.80159 0.80173 0.80033 0.80023
0.010
0.015 0.020 0.025 0.030 0.035 0.040 0.045 0.050
8757 2654 1384 881 634 486 389 323
0.04416 0.04052 0.03894 0.03827 0.03643 0.03474 0.03483 0.03338
0.90002 0.90002 0.90004 0.90009 0.90045 0.90051 0.90020 0.90043
6442 1982 1028 666 479 369 298 250
0.04308 0.03939 0.03737 0.03572 0.03383 0.03284 0.031 44 0.03376
0.80005 0.80015 0.80021 0.80005 0.80067 0.80104 0.80018 0.80103
0.015
0.020 0.025 0.030 0.035 0.040 0.045 0.050
12119 3496 1763 1106 771 585 465
0.04551 0.04267 0.04099 0.03982 0.03823 0.03733 0.03580
0.90001 0.90005 0.90003 0.90004 0.90009 0.90015 0.90008
8861 2590 1309 819 582 443 352
0.04485 0.04128 0.03938 0.03831 0.03655 0.03466 0.03470
0.80000 0.80001 0.80004 0.80010 0.80025 0.80043 0.80036
0.020
0.025 0.030 0.035 0.040 0.045 0.050
15423 4333 2131 1311 903 677
0.04637 0.04392 0.04240 0.04037 0.03937 0.03843
0.90000 0.90003 0.90012 0.90017 0.90014 0.90018
11236 3183 1579 966 677 511
0.04582 0.04295 0.04084 0.03946 0.03792 0.03668
0.80002 0.80008 0.80015 0.80036 0.80022 0.80011
0.025
0.030 0.035 0.040 0.045 0.050
18684 5151 2497 1516 1038
0.04698 0.04478 0.04327 0.04210 0.04069
0.90000 0.90001 0.90010 0.90006 0.90015
13596 3768 1835 1117 771
0.04647 0.04417 0.04191 0.04086 0.03907
0.80001 0.80001 0.80016 0.80026 0.80042
0.030
0.035 0.040 0.045 0.050
21912 5955 2851 1719
0.04736 0.04533 0.04374 0.04250
0.90000 0.90003 0.90003 0.90010
15922 4355 2092 1268
0.04699 0.04458 0.04298 0.04142
0.80002 0.80006 0.80015 0.80028
0.035
0.040 0.045 0.050
25097 6751 3205
0.04771 0.04594 0.04430
0.90001 0.90001 0.90000
18231 4931 2351
0.04732 0.04524 0.04349
0.80001 0.80007 0.80001
0.040
0.045 0.050
28258 7536
0.04795 0.04632
0.90001 0.90000
20504 5498
0.04762 0.04559
0.80001 0.80000
0.045
0.050
31376
0.04814
0.90000
22757
0.04784
0.80001
143
Sample Size Based o n Fisher's Exact Test Table 3
Continued ~x = 0.05 (two-sided) 1 - ~ = 0.9
1 -
~--
0.8
pl 0.005
p2 0.010 0.015 0.020 0.025 0.030 0.035 0.040 0.045 0.050
n 6465 2142 1184 788 577 450 370 313 270
ct* 0.02040 0.01836 0.01739 0.01662 0.01723 0.01681 0.01644 0.01613 0.01625
1 - 6" 0.90001 0.90003 0.90014 0.90023 0.90053 0.90001 0.90009 0.90048 0.90037
n 4923 1656 922 612 454 359 296 250 217
a* 0.01978 0.01783 0.01657 0.01681 0.01671 0.01627 0.01582 0.01544 0.01500
1 - 6" 0.80001 0.80026 0.80032 0.80017 0.80085 0.80024 0.80149 0.80050 0.80241
0.010
0.015 0.020 0.025 0.030 0.035 0.040 0.045 0.050
10680 3224 1661 1067 764 579 458 381
0.02200 0.02041 0.01935 0.01875 0.01802 0.01814 0.01742 0.01684
0.90000 0.90009 0.90009 0.90009 0.90015 0.90027 0.90053 0.90063
8059 2456 1282 826 588 444 360 302
0.02154 0.01973 0.01854 0.01776 0.01788 0.01697 0.01643 0.01605
0.80000 0.80003 0.80013 0.80047 0.80065 0.80031 0.80036 0.80101
0.015
0.020 0.025 0.030 0.035 0.040 0.045 0.050
14777 4247 2128 1330 938 698 552
0.02266 0.02130 0.02031 0.01978 0.01901 0.01826 0.01833
0.90000 0.90004 0.90001 0.90001 0.90019 0.90002 0.90040
11149 3225 1627 1025 723 538 424
0.02240 0.02072 0.01969 0.01910 0.01816 0.01806 0.01731
0.80004 0.80013 0.80007 0.80033 0.80028 0.80018 0.80078
0.020
0.025 0.030 0.035 0.040 0.045 0.050
18822 5270 2578 1585 1093 817
0.02314 0.02186 0.02088 0.02016 0.01967 0.01912
0.90000 0.90005 0.90006 0.90010 0.90007 0.90010
14173 3989 1961 1202 839 622
0.02286 0.02141 0.02027 0.01981 0.01894 0.01896
0.80000 0.80001 0.80001 0.80001 0.80012 0.80051
0.025
0.030 0.035 0.040 0.045 0.050
22833 6275 3029 1836 1256
0.02345 0.02234 0.02133 0.02086 0.02032
0.90001 0.90000 0.90003 0.90013 0.90022
17163 4741 2295 1402 960
0.02323 0.02195 0.02100 0.02017 0.01930
0.80001 0.80008 0.80008 0.80015 0.80024
0.030
0.035 0.040 0.045 0.050
26791 7257 3467 2084
0.02367 0.02269 0.02187 0.02116
0.90001 0.90001 0.90007 0.90004
20110 5478 2629 1586
0.02346 0.02229 0.02138 0.02071
0.80002 0.80003 0.80009 0.80025
0.035
0.040 0.045 0.050
30704 8237 3906
0.02383 0.02291 0.02215
0.90000 0.90002 0.90007
23036 6202 2950
0.02363 0.02262 0.02173
0.80000 0.80005 0.80002
0.040
0.045 0.050
34573 9203
0.02395 0.02310
0.90001 0.90001
25931 6928
0.02379 0.02288
0.80001 0.80005
0.045
0.050
38403
0.02405
0.90000
28786
0.02391
0.80001
144 Table 4
R.G. Thomas and M. Conlon M i n i m u m S a m p l e S i z e s B a s e d o n F i s h e r ' s Ex act Test. E v e n t R a t e R a n g e s : Pl = 0.001(0.001)0.009; P2 = pl + 0.001(0.001)0.01 c~ = 0.05 (one-sided) 1 - [3 = 0.9
1 - 13 = 0.08
pl 0.001
p2 0.002 0.003 0.004 0.005 0.006 0.007 0.008 0.009 0.010
n 26977 8884 4898 3263 2423 1918 1582 1343 1164
~* 0.04100 0.03785 0.03651 0.03558 0.03462 0.03389 0.03336 0.03297 0.03265
1 - ~* 0.90000 0.90001 0.90001 0.90001 0.90009 0.90007 0.90008 0.90022 0.90028
n 19929 6698 3709 2518 1893 1508 1249 1062 922
~* 0.03922 0.03665 0.03493 0.03353 0.03227 0.03226 0.03186 0.03150 0.03118
1 - ~* 0.80001 0.80005 0.80008 0.80014 0.80012 0.80000 0.80037 0.80027 0.80029
0.002
0.003 0.004 0.005 0.006 0.007 0.008 0.009 0.010
44265 13481 6963 4439 3189 2447 1963 1630
0.04406 0.04084 0.03928 0.03763 0.03668 0.03622 0.03576 0.03523
0.90000 0.90002 0.90004 0.90005 0.90001 0.90008 0.90014 0.90012
32557 9960 5173 3347 2406 1853 1501 1258
0.04329 0.03907 0.03753 0.03644 0.03564 0.03465 0.03381 0.03318
0.80001 0.80003 0.80002 0.80006 0.80014 0.80008 0.80025 0.80022
0.003
0.004 0.005 0.006 0.007 0.008 0.009 0.010
61352 17770 8930 5592 3925 2957 2350
0.04565 0.04293 0.04068 0.04019 0.03848 0.03741 0.03667
0.90000 0.90000 0.90001 0.90001 0.90007 0.90007 0.90003
44830 13113 6627 4159 2930 2230 1774
0,04483 0.04169 0.03962 0.03807 0.03681 0.03622 0.03571
0.80000 0.80002 0.80004 0.80006 0.80001 0.80008 0.80013
0.004
0.005 0.006 0.007 0.008 0.009 0.010
78437 22070 10891 6694 4647 3476
0.04645 0.04404 0.04258 0.04077 0.04037 0.03910
0.90000 0.90001 0.90001 0.90004 0.90001 0.90000
57191 16201 8037 4968 3461 2584
0.04581 0.04307 0.04074 0.04020 0.03835 0.03717
0.80000 0.80000 0.80000 0.80006 0.80003 0.80009
0.005
0.006 0.007 0.008 0.009 0.010
95471 26336 12782 7798 5325
0.04703 0.04486 0.04305 0.04202 0.04130
0.90000 0.90000 0.90000 0.90002 0.90004
69490 19293 9433 5721 3972
0,04654 0.04401 0.04247 0.04121 0.04004
0.80000 0.80002 0.80002 3.80002 0.80001
0.006
0.007 0.008 0.009 0.010
112434 30613 14698 8847
0.04746 0.04564 0.04405 0.04307
0.90000 0.90000 0.90001 0.90000
81709 22369 10764 6540
0.04708 0.04492 0.04294 0.04184
0.80000 0.80000 0.80002 0.80004
0.007
0.008 0.009 0.010
129359 34843 16592
0.04778 0.04598 0.04464
0.90000 0.90000 0.90001
93946 25400 12145
0.04739 0.04542 0.04365
0.80000 0.80001 0.80002
0.008
0.009 0.010
146209 39079
0.04802 0.04644
0.90000 0.90000
106089 28463
0.04771 0.04594
0.80000 0,80001
0.009
0.010
163034
0.04823
0.90000
118272
0.04790
0.80000
145
Sample Size Based on Fisher's Exact Test Table 4
Continued c~ = 0.05 (two-sided) 1
-
13 =
0.9
1
-
0.8
13 =
pl 0.001
p2 0.002 0.003 0.004 0.005 0.006 0.007 0.008 0.009 0.010
n 32628 10742 5940 3956 2902 2278 1868 1577 1362
~* 0.02028 0.01876 0.01800 0.01743 0.01698 0.01651 0.01607 0.01569 0.01539
1 - 13" 0.90000 0.90002 0.90004 0.90005 0.90009 0.90013 0.90017 0.90002 0.90023
n 24886 8295 4621 3074 2284 1805 1485 1257 1088
~* 0.01962 0.01816 0.01728 0.01638 0.01559 0.01510 0.01480 0.01461 0.01448
1 - 13" 0.80002 0.80005 0.80009 0.80009 0.80019 0.80014 0.80024 0.80011 0.80014
0.002
0.003 0.004 0.005 0.006 0.007 0.008 0.009 0.010
53848 16255 8405 5367 3845 2967 2388 1976
0.02197 0.02022 0.01975 0.01880 0.01816 0.01781 0.01749 0.01721
0.90000 0.90000 0.90002 0.90002 0.90006 0.90001 0.90011 0.90009
40716 12405 6437 4145 3007 2309 1849 1536
0.02153 0.01999 0.01884 0.01802 0.01752 0.01710 0.01665 0.01616
0.80001 0.80000 0.80000 0.80003 0.80014 0.80012 0.80002 0.80021
0.003
0.004 0.005 0.006 0.007 0.008 0.009 0.010
74925 21604 10801 6746 4741 3576 2833
0.02270 0.02141 0.02044 0.01983 0.01938 0.01865 0.01815
0.90000 0.90001 0.90003 0.90002 0.90006 0.90008 0.90000
56503 16421 8212 5170 3639 2762 2212
0.02237 0.02083 0.01999 0.01926 0.01832 0.01788 0.01752
0.80000 0.80002 0.80001 0.80003 0.80005 0.80007 0.80021
0.004
0.005 0.006 0.007 0.008 0.009 0.010
95825 26894 13169 8086 5601 4197
0.02319 0.02197 0.02106 0.02050 0.01990 0.01951
0.90000 0.90001 0.90001 0.90002 0.90001 0.90003
72089 20308 10017 6157 4300 3216
0.02292 0.02157 0.02060 0.01988 0.01939 0.01860
0.80000 0.80001 0.80001 0.80006 0.80003 0.80010
0.005
0.006 0.007 0.008 0.009 0.010
116666 32117 15541 9434 6465
0.02350 0.02243 0.02157 0.02097 0.02040
0.90000 0.90001 0.90001 0.90001 0.90001
87695 24251 11753 7159 4923
0.02325 0.02198 0.02102 0.02045 0.01978
0.80000 0.80001 0.80001 0.80001 0.80001
0.006
0.007 0.008 0.009 0.010
137514 37328 17844 10747
0.02371 0.02279 0.02203 0.02120
0.90000 0.90000 0.90002 0.90002
103239 28157 13528 8145
0.02351 0.02238 0.02146 0.02083
0.80000 0.80001 0.80002 0.80001
0.007
0.008 0.009 0.010
158255 42529 20198
0.02386 0.02297 0.02232
0.90000 0.90000 0.90001
118721 32020 15241
0.02370 0.02269 0.02187
0.80000 0.80001 0.80002
0.008
0.009 0.010
178924 47724
0.02398 0.02315
0.90000 0.90000
134191 35874
0.02384 0.02294
0.80000 0.80001
0.009
0.010
199618
0.02409
0.90000
149581
0.02395
0.80000
146
R.G. Thomas and M. Cordon ing trial given in the "Introduction." In this trial the expected e v e n t rates are pl = 0.001 a n d P2 0.004. Let the specified values of o~ and 1 - [3 be 0.05 (two-sided) and 0.8, respectively. Table 4 provides calculations of n based on Fisher's Exact Test for p l , p 2 = 0.001, 0.001(0,01) and pl < P2. The third r o w of the table c o r r e s p o n d s to the (Pl,p2) = (0.001, 0.004) e v e n t rate combination. The third c o l u m n from the right contains 4621, the required m i n i m u m sample size in b o t h the experimental and control group. The associated empirical type I error for this p a r a m e t e r combination is 0.0173. This oL*should be comp a r e d to the design value 0.025. =
SUMMARY
This p a p e r provides sample size calculations based o n Fisher's Exact Test for 2 × 2 comparative trials in w h i c h the expected e v e n t rates in b o t h the experimental and control g r o u p are low. The empirical type I errors are calculated to provide a quantification of the conservativeness of the test for a given combination of s t u d y parameters. In m a n y cases, the sample sizes p r e s e n t e d in these tables are smaller t h a n sample sizes calculated using formulae based on asymptotic p r o c e d u r e s since these expressions d o not p r o d u c e accurate results w h e n the e v e n t rates are low.
REFERENCES
1. Fleiss J: Statistical Methods for Rates and Proportions. New York, Wiley, 1973 2. Cochran WG, Cox GM: Experimental Designs, 2nd ed. New York, Wiley, 1957 3. Cohen J: Statistical Power Analysis for the Behavioral Sciences. New York, Academic Press, 1969 4. Feigl P: A graphical aid for determining sample size when comparing two independent proportions. Biometrics 34:111-122, 1978 5. Gail M, Gart JJ: The determination of sample sizes for use with the conditional test in 2 x 2 comparative trials. Biometrics 29:441-448, 1973 6. Haseman JK: Exact sample sizes for use with the Fisher-Irwin Test for 2 x 2 tables. Biometrics 34:106-109, 1978 7. Casagrande JT, Pike MC, Smith PG: Algorithm AS 129. The power function of the "exact" test for comparing two binomial distributions. Appl Stat 27:212-219, 1978 8. Suissa S, Shuster JJ: Exact unconditional sample sizes for the 2 × 2 comparative trial. J Roy Stat Soc, Series A 148:317-327, 1985 9. Lemeshow S, Hosmer DW, Stewart JP: A comparison of sample size determination methods in the two group trial where the underlying disease is rare. Commun Stat--Simulation and Computation B10(5):437-449, 1981 10. Armitage P: Statistical Methods in Medical Research. Oxford, Blackwell Scientific Publications, 1971 11. Snedecor GW, Cochran WG: Statistical Methods. Ames, Iowa, Iowa State University Press, 1980 12. Thomas RG, Cordon M: An algorithm for the rapid evaluation of the power function for Fisher's Exact Test. University of Florida Department of Statistics Technical Report, No. 382, 1991 13. ISIS-2 Collaborative Group: Randomized trial of intravenous streptokinase, oral
Sample Size Based on Fisher's Exact Test
14. 15. 16. 17. 18. 19.
147
aspirin, both or neither among 17,187 cases of suspected acute myocardial infarction: ISIS-2. Lancet 2:349, 1988 Steering Committee of the Physician's Health Study Research Group: Final report on the aspirin component of the ongoing physicians' health study. N Engl J Med 321:129, 1989 Fisher RA: The logic of inductive inference. J Roy Stat Soc 98:39-54, 1935 Berkson J: In dispraise of the exact test: Do the marginal totals of the 2 x 2 table contain relevant information respecting the table proportions? J Star Plan Inf 2:2742, 1978 Casagrande JT, Pike MC, Smith PG: An improved formula for calculating sample sizes for comparing two binomial distributions. Biometrics 34:483-486, 1978 Dobson VJ, Gebski V: Sample sizes for comparing two independent proportions using the continuity-corrected arc sine transformation. Statistidan 35:51-53, 1986 Waiters DE: In defense of the arc sine approximation. The Statistician 28:219-222, 1979
