0895-4356/91 $3.00+ 0.00 Pergamonpressplc
J clin Epidmiol Vol. 44, No. 8, pp. 763-770,1991 Printedin Great Britain
LIKELIHOOD RATIOS WITH CONFIDENCE: SAMPLE SIZE ESTIMATION FOR DIAGNOSTIC TEST STUDIES DAVID L. SIMEL,* GREGORY P. SAMSA and DAVID B. MATCHAR
Medical Center and Divisionof General Internal Medicine,Duke UniversityMedicalCenter, Durham, North Carolina.U.S.A.
Center for Health Services Research in Primary Care, Durham Veterans Administration
(Received in revised form 19 December 1990)
Abstract-Confidence intervals are important summary measures that provide useful information from clinical investigations, especially when comparing data from different populations or sites. Studies of a diagnostic test should include both point estimates and confidence intervals for the tests’ sensitivity and specificity. Equally important measures of a test’s efficiency are likelihood ratios at each test outcome level. We present a method for calculating likelihood ratio confidence intervals for tests that have positive or negative results, tests with non-positive/non-negative results, and tests reported on an ordinal outcome scale. In addition, we demonstrate a sample size estimation procedure for diagnostic test studies based on the desired likelihood ratio confidence interval. The renewed interest in confidence intervals in the medica literature is important, and should be extended to studies analyzing diagnostic tests. Likelihood ratio
Confidence interval
INTRODUCTION
Confidence intervals are important summary measures that provide useful information from clinical investigations , especially when comparing data from different populations or sites [l, 21. In particular, they allow more critical evaluation of point estimates describing the sensitivity and specificity of diagnostic testsvalues that inform us about a test’s diagnostic efficiency. Sample size estimates for studies that investigate tests generally depend on the length of the desired confidence interval around the operating characteristics. In addition to sensitivity and specificity, likelihood ratios are also used to characterize the behavior of diagnostic tests[3]. Sensitivity (the proportion of diseased patients with posi*Address all correspondence to: David L. Simel, Ambulatory Care Service (1 lC), Durham Veterans Administration Medical Center, Durham, NC 27705, U.S.A.
Sample size
Diagnostic test
tive results) and specificity (the proportion of non-diseased patients with negative results) describe the behavior of tests given patients’ disease status. For 2 x 2 tables, the positive likelihood ratio (LR+) is defined as sensitivity/(1 - specificity), while the negative likelihood ratio (LR-) is (1 - sensitivity)/specificity (Fig. 1). Likelihood ratios algebraically combine sensitivity and specificity to describe more than the independent values themselves-they describe the change in odds favoring disease given a particular test result. When a likelihood ratio exceeds 1, the odds favoring disease increase; when the likelihood ratio becomes less than 1, the odds favoring disease decrease; when a likelihood ratio approaches 1, the odds favoring disease do not change and the test is indeterminate [4]. Since likelihood ratios refer to actual test results before disease status is known, they are more immediately useful to clinicians than sensitivity and specificity. 763
764
DAWDL. SIMEL ef al.
Disease Present
Positive
A
Negative
C
Test Result
f
nl
Absent
3 B
D
n2
Fig. 1. Conventional 2 x 2 matrix in which A = true positive, B = false positive, C = false negative, and D = true negative results. The total patients with disease = PI, and total patients without disease = nz .
Likelihood ratios have three important properties, outlined by others, that together make them extremely useful [3]. First, likelihood ratios do not change with the pretest probability of disease. Second, likelihood ratios can be calculated for multiple levels of test outcomes rather than just two nominal levels. Finally, likelihood ratios allow us to assess the impact of a test result on the odds that a patient will have disease. The intent of this paper is not to convince investigators to report only likelihood ratios, but to emphasize that likelihood ratios, like other diagnostic test indices, also have confidence intervals. The likelihood ratio confidence intervals can be determined from the dual functions of sensitivity and specificity, and they can be used to guide sample size estimation for clinical investigations. The point estimate for sensitivity expresses the percentage of patients with disease who have a positive test result, whereas the point estimate for specificity expresses the percentage of patients without disease who have a negative test result. Since the precision of these point estimates varies with the sample size, confidence intervals allow expression of a plausible range for the sensitivity and specificity. Thus, a 95% confidence interval contains the point estimate about 95 times in 100 replications of the study; in about 5 replications the point estimate would fall outside the range of the confidence interval. Because the width of the confidence interval decreases as the sample size increases, we have more confidence in the value of point estimates from larger populations. Calculating the confidence interval for sensitivity and specificity is straightforward. How to calculate the confidence interval or estimate the sample size requirements for likelihood ratios is
less obvious since, not only do likelihood ratios combine two values (sensitivity and specificity), they also may apply to tests having multiple possible outcome levels (e.g. positive, non-positive/non-negative, or negative). We will show how to calculate the confidence intervals around likelihood ratios for tests with positive or negative results (2 x 2 table); conditional likelihood ratios for tests with positive, non-positive/nonnegative, or negative results (3 x 2 table); and conventional likelihood ratios for tests with results transformed to a multilevel ordinal scale (r x 2 table, where r represents the number of rows). In this last example, neither conventional sensitivity and specificity values or conditional likelihood ratios completely describe multilevel test outcomes, so that conventional likelihood ratios at each test result level are preferred. Finally, we will demonstrate the method for estimating the sample size required to show that a likelihood ratio falls within an expected range. The equations and working examples should serve as reference guides for reporting likelihood ratio confidence intervals and estimating sample sizes based on the desired confidence interval length. CONFIDENCE INTERVALS FOR LIKELIHOOD RATIOS FROM 2 x 2 TABLES
The simple 2 x 2 table displays the results of a diagnostic test vs the reference standard for a population size, n, where (Fig. 1): sensitivity = A/(A + C) and specificity = D/(B + D). Each of these sensitivity and specificity values represents a simple proportion. The approximate 95% confidence interval for any simple proportion, p, drawn from a normal distribution is p f
1.96.&m,
(1)
where n represents the size of the population. As an examole, 95% confidence __ _ . the aunroximate interval for the sensitivity from Fig. 1 is
01 f 1.96,/[A /(A + Cl1*[C/U + Cllh
V/V
+
(2)
or (sensitivity) f 1.96 Jsensitivity
‘(1 - sensitivity)/n,,
(3)
Likelihood Ratios and Sample Si
where n, represents the number of patients with disease. The approximate 95% confidence interval for the specificity from Fig. 1 is P/P
+
011
k 1.96,/P l@ + 011.PI@ + D)I/n;! (4) or (specificity) If: 1.96 Jspecificity
. (1 - specificity)/n,,
(5)
where n2 represents the number of patients without disease. The confidence interval for sensitivity varies in proportion to the number of patients with disease, just as the confidence interval for specificity varies in proportion to the number of patients without disease. The positive likelihood ratio (LR + ) expresses the change in odds favoring disease given a positive test result, whereas the negative likelihood ratio (LR -) expresses the change in odds favoring disease given a negative test result. These likelihood ratios for the 2 x 2 table in Fig. 1 are calculated from the sensitivity and specificity: LR + = sensitivity/( 1 - specificity)
(6)
LR - = (1 - sensitivity)/specificity.
(7)
and
In
sensitivity + 1.96 1 - specificity
where A and B are values of the respective cells from the 2 x 2 table and exp is the base of the natural logarithm. For the negative likelihood ratio: LR-
=exp
In
1 - sensitivity + 1 g6 specificity - .
765
where C and D are the values of the respective cells from the 2 x 2 table and exp is the base of the natural logarithm. Equations (8) and (9) are examples of the general formula, derived in the Appendix, for the approximate 95% confidence interval for risk ratios. The general formula represents the likelihood ratio 95% confidence interval that can be calculated for any level of test result, x, by the equation LR,=exp(lnEk where
for
1.96*J%+z),(lO) LR, = LR+,
p1 = sensitivity, and pzn, = B. When specifying the negative likelihood ratio (LR, = LR-), p, = 1 - sensitivity, p2 = specificity, p,n, = C, and p2n2 = D. For example, when the values from Table 1 are A = 30, B = 5, C = 10, and D = 45, the following operating characteristics and 95% confidence intervals are obtained:
pz = 1 - specificity,
p,n, = A,
sensitivity = 0.75 (95% confidence interval 0.62-0.88), specificity = 0.90 (0.82-0.98), positive likelihood
An approximation to the confidence interval for the likelihood ratio could be obtained by substituting the upper and lower approximate 95% confidence limits for sensitivity and specificity into equations (6) and (7). Unfortunately, this approximation reveals confidence intervals that are too wide, suggesting inappropriately less precision. For the positive likelihood ratio, the approximate 95% confidence intervals should be calculated by the equation LR+ =exp
Determinations
ratio = 7.5 (3.20-17.56),
and negative likelihood ratio = 0.28 (0.16-0.48). Equation (10) may not appear intuitive, but the equation is algebraically identical with the formula for calculating the confidence interval for relative risk ratios. The formula can be explained by examining the exponentiated component parts. The value p,/p2 represents the likelihood ratio. The upper and lower bounds of the 95% confidence interval around the log likelihood ratio (lnp, /p2, where In is the natural logarithm) is determined after multiplying 1.96 times the standard error of the log likelihood ratio. The value (1 -pI )/p, n, refers to subjects with disease, while (1 -p2)/p2n2 is a separate and analogous calculation referring to subjects without disease. The key to understanding the equation is recognizing the separability between diseased and non-diseased subjects so that sensitivity and specificity have independent effects on the likelihood ratio confidence interval. The magnitude of the total effect cannot be interpreted from the confidence interval around the independent proportions of sensitivity and specificity.
766
DAVIDL. &MELer al.
CONDITIONAL CONFIDENCE INTERVALS FOR LIKELIHOOD RATIOS FROM 3 x 2 TABLES
LR’+ = sensitivity//( 1 - specificity’)
(11)
LR’ - = (1 - sensitivity’)/specificity’,
(12)
and The clinical examination is one type of diagnostic test that yields frequent non-positive/ non-negative results. An example is the physical examination for ascites: the physician may be confident that ascites is present (positive), confident that it is absent (negative), or not sure (non-positive/non-negative) [5]. When nonpositive/non-negative or intermediate results are obtained, the likelihood ratios and test operating characteristics may be reported as conditional on obtaining a positive or negative result [4]. This may be especially useful for tests that have excellent predictive value when positive or negative, but poor predictive value when the test result is less obvious. Conditional operating characteristics allow the investigator to disentangle the impact of an extremely useful positive or negative result from a less useful non-positive/non-negative outcome. This information is especially important when the diagnostic test under consideration is risky or expensive [4]. Tests that are neither positive or,negative may have an important impact on clinical decision making. Sometimes, a test that is neither positive or negative may increase the odds favoring disease to a level between the positive and negative likelihood ratio, an intermediate result. In other cases, a non-positive/non-negative result may not add useful information to the clinical examination. The lack of diagnostic utility at any test level, detected by a likelihood ratio approaching 1, defines an indeterminate result [4]. The conditional operating characterstics from the 3 x 2 table of Fig. 2 are sensitivity’ = A /(A + C) and specificity’ = D/(B + D). The conditional likelihood ratios are:
where LR’+ is the positive and LR’- is the negative conditional likelihood ratio. Confidence intervals for the positive and negative conditional likelihood ratios are identical with the conventional likelihood ratios for the 2 x 2 case in equations (8) and (9), except the conditional values for sensitivity and specificity are substituted. The likelihood ratio for non-positive/nonnegative results (LR -I-) is calculated in terms of the test yield. The test yield provides useful information by describing the probability of obtaining a more definite positive or negative result, as opposed to a non-positive/non-negative or intermediate result. The positive yield (YD+) is the probability of a positive or a negative result when disease is present:
YD+ =(A +C)/(A +C+E);
(13)
whereas the negative yield (YD-) is the probability of a positive or a negative result when disease is absent, YD-
=(B+D)/(B+D+F).
(14)
The likelihood ratio for a non-positive/nonnegative result (LR & ) is LRf
= (1 - YD+)/(l
- YD-).
(15)
The confidence interval for the likelihood ratio for non-positive/non-negative results can be derived from equation (10):
+ 1.96$~),
(16)
1 -YD+,p,= 1 -YD-,p,q=E and p2n, = Fare the values from the 3 x 2 table
wherep,=
Disease
Present
Test Result
Absent
(Fig. 2).
Positive
A
B
CONVENTIONAL CONFIDENCE INTERVALS FOR LIKELIHOOD RATIOS FOR TESTS WITH MULTILEVEL RESULTS
Non-positive Non-negative
E
F
c
D
Ill
n2
The conventional or conditional sensitivity and specificity values do not completely describe multilevel ordinal outcomes. An example is the exercise tolerance test, transformed to an ordinal scale outcome based on the millimeters of ST-segment depression. For such tests, the usual descriptors of its performance are the
Negative
EH
Fig. 2. The 3 x 2 matrix when obtaining results that are non-positive/non-negative.
Likelihood Ratios and Sample Size Determinations
Disease PWSWI
A
B
E
F
Level 3
G
H
Level 4
C
D
fll
n2
Level 1
Test
Absent
Level 2
Result
Fig. 3. The I x 2 matrix when obtaining multilevel test results.
conventional multilevel likelihood ratios [3]. These likelihood ratios show the increase in odds favoring disease at each test outcome level. The multilevel likelihood ratio is calculated by describing the proportion of diseased patients with a given test result (p,) divided by the proportion of non-diseased patients with the same test result (p2). For a subject with the second level outcome (LR,=,), using the notation from Fig. 3, the point estimate for LR, is
_p,_W+E+G+C)
LR
2
~2
E/n,
F/(B+F+H+D)=Fln,’
(17)
where n, describes the subjects with disease and n, describes the subjects without disease. Based on equation (lo), the approximate 95% confidence interval can be calculated for any level of test result, x, by the general equation for risk ratio confidence intervals. In this case, the values for equation (10) (pl, p2, n,, and n2) represent the values described above. Equation (17) uses a standard notation for multilevel ordinal outcomes. In other instances, a clinician may be most insterested in choosing a single cutpoint that defines normality. For example, if a clinician considers Level 1 or Level 2 results abnormal and Level 3 or Level 4 results normal, then equation (17) becomes
=&=(A +W(A +E+G+C)
LR
”
~2
(B+F)/(B+F+H+D)
Fortunately, just as in equation (17), the appropriate values for p, , n, , p2, and n2 can be reinserted into equation (10) to determine the confidence interval for LR c *. The effect of different cutpoints for normality are evaluated
161
by plotting the true positive rate vs the false positive rate in a receiver operating characteristic (ROC) curve. It is important to recognize that the slope at each point on the ROC curve represents a positive likelihood ratio. One method for choosing the cutpoint involves identifying the point on the curve balancing the pretest probability of disease with the relative importance to the patient of avoiding false positive and negative results [6]. The confidence interval for any point on the ROC curve can be determined with the above method. The utility of the general formula [equation (lo)] should now be apparent. However one chooses to define the cutpoint describing normality, the equation can be used to calculate the relative confidence interval for the corresponding likelihood ratio. By understanding the interpretation of p, and p2, the equation works for the positive likelihood ratio, negative likelihood ratio, or likelihood ratio at multiple levels. SAMPLE SIZE ESTIMATES FOR EVALUATING LIKELIHOOD RATIOS
Sample size estimates for studies describing the operating characteristics of diagnostic tests may reflect the degree of confidence the investigator wishes to establish for the sensitivity point estimate (or, alternatively the specificity point estimate). However, when a clinician uses a diagnostic test, the question becomes not one of confidence in the operating characteristics, but confidence in the impact of the result on the likelihood of disease. The impact suggests that sample size estimates should reflect concern with the confidence interval around the likelihood ratio rather than just the confidence interval around sensitivity or specificity. Although methods for calculating sample sizes for constructing confidence intervals around proportions are well described [I, the appropriate method applicable to likelihood ratios is not generally appreciated by clinical investigators. In many situations, the desired threshold limits for the likelihood ratio confidence interval may be determined by expert opinion or through the tools of decision analysis [4,6]. These threshold values may also serve as the upper or lower bounds of our desired confidence interval. We will show the steps required for estimating the sample size of an investigation designed to have an appropriately narrow confidence interval around a clinically meaningful likelihood ratio.
768
DAVIDL. &MELet al.
WORKING EXAMPLES
Problem I
Based on pilot data, we believe the sensitivity of a new test is at least 80%, specificity = 73%, and positive likelihood ratio = 2.96 [derived from sensitivity/( 1 - specificity)]. Suppose expert opinion, or decision analysis, suggests the test would be clinically useful when the positive likelihood ratio was 2 2.0 (LR + min).Our study goal is to demonstrate that the positive likelihood ratio 95% confidence interval around 2.96 does not contain 2.0. How many patients are required with and without disease? Solution. We use equation (10) to solve for n, = n2 = number of required patients with and without disease. In this case, p, = sensitivity = 0.80, pz = 1 - specificity = 0.27, and LR+ = 2.96. The values LR+ti, = 2.0, p, , and pz are inserted into equation (10) and solved for n,=n,=n: 2.0 = exp{ln(0.8/0.27)
- 1.96*,/(l/n)*[(0.2/0.8)
+ (0.73/0.27)]}.
After taking the logarithm of both sides of the equation, we find n = 73.4. Thus, 74 patients with disease and 74 patients without disease are required to show that a test with a sensitivity of at least 80%, specificity of at least 73%, and likelihood ratio of 2.96 does not contain the threshold likelihood ratio of 2.0 in its 95% confidence interval. It should be apparent from equation (10) that the standard error of the log likelihood ratio decreases, resulting in a narrower LR+ confidence interval, when (1) sensitivity improves, (2) specificity improves, or (3) the number of patients with disease increases. Problem 2
We hypothesize that a new screening test would be clinically useful if the negative likelihood ratio was ~0.4 (LR-,,). We believe the sensitivity of this new test is at least 90%, the negative likelihood ratio is 0.2, and want to show that the negative likelihood ratio 95% confidence interval around 0.2 does not contain 0.4. How many patients are required with and without disease? Solution. We use equation (10) to solve for n, = n, = number of required patients with and without disease. In this case, p, = 1 sensitivity = 0.1 and LR- = 0.2; therefore, from equation (7), p2 = specificity = 0.5. The
values LR -man = 0.4, p,, and p2 are inserted into equation (10) and solved for n, = n, = n: 0.4 = exp{ln(0.1/0.5)
+ 1.96*J(l/n)*[(O.9/0.1)
+ (O.S/O.S)]}.
After taking the logarithm of both sides of the equation, we find n = 79.9. Thus, 80 patients with disease and 80 patients without disease are required to show that a test with a sensitivity of at least 90% and negative likelihood ratio of 0.2 does not contain a likelihood ratio of 0.4 in its 95% confidence interval. If we have more than 80 patients in either group, or a sensitivity >0.90, our confidence that the negative likelihood ratio does not contain 0.4 becomes greater. Problem 3
Often, clinical investigators have difficulty finding patients with the disease of interest. Suppose, after estimating the sample size in Problem 1, the investigators realize they will be able to enroll only about 1 patient with disease for every 5 without disease. How many patients are required with and without disease to assure that the 95% confidence interval excludes LR+ = 2? To answer the question, we must use the general form for the likelihood ratio confidence interval [equation (lo)]. From Problem 1 we have p, = sensitivity = 0.80 and p2 = (1 specificity) = 0.27. However, now n, # n2 and instead n, = 0.2 (n2): 2 = exp ln(0.8/0.27)
After taking the logarithm of both sides of the equation we find n2 = 98.3. Thus, the investigator requires 99 patients without disease and 20 diseased patients to exclude a positive likelihood ratio of 2 from the predicted positive likelihood ratio of 2.96. The paradoxical finding of Problem 3 is that the imbalance between diseased vs non-diseased patients actually decreases the overall sample size required to exclude the minimally acceptable positive likelihood ratio. However, enrolling fewer subjects with disease has an confidence intervals adverse impact-the around the negative likelihood and the sensitivity become progressively wider. The paradox
Likelihood Ratios and Sample Size Determinations
highlights the strength of using likelihood ratios to estimate sample sizes for diagnostic test studies: equation (10) can be used to select the minimum number of diseased and non-diseased patients to achieve the desired likelihood ratio confidence interval. Close inspection of equation (10) reveals that many n,, n, pairs satisfy the desired positive likelihood ratio confidence interval solution excluding the minimum positive likelihood ratio. Although the simplest solution to sample size estimation is to set n, = n2, this may not lead to a total sample size minimizing the total number of patients for a specified LR+ confidence interval. Table 1 demonstrates the total sample size required and the predicted confidence interval around the negative likelihood ratio using the values from Problem 3: sensitivity = 0.8, specificity = 0.73, and LR+,i” = 2. In Table 1, r is the ratio between patients without disease (n2) and with disease (n,). When patients are selected from a population with known disease status, the investigator has control over the ratio between non-diseased and diseased patients. In that case, the smallest sample size for the desired positive likelihood ratio confidence interval can be selected. For the example, the minimum sample size occurs with r = 3.0 or, 86 without disease and 29 patients with disease. In other cases, the investigator may enroll patients consecutively before their disease status is defined. This strategy is often used to evaluate screening tests in large populations. Large screening programs for diseases with relatively low prevalence generally seek to minimize false positive results; here the prime concern is Table 1. Total sample size required for the desired positive likelihood ratio 95% confidence interval with LR+,, = 2 when the investigator can control the ratio, r, of nondiseased to diseased subjects
r = n&r, 0.05 0.1 0.5 1.0 2.0 3.0 5.0 10.0
Subjects with disease (n,) 1351 619 141 14 40 29 20 13
Subjects without disease (n2)
Total sample size
95% Negative likelihood ratio confidence interval
68 68 71 14 80 86 99 130
1419 141 212 148 120 115 119 143
0.240.31 0.23-0.32 0.194.39 0.17-0.44 0.15-0.51 0.13-0.57 0.11-0.67 0.09-0.82
A ratio of 3 : 1 provides the minimum sample size to exclude the minimum positive likelihood ratio with the data from Problem 3.
769
Table 2. Total sample size required for the desired negative likelihood ratio 95% confidence interval with LR-, = 0.4 when the investigator cannot control the ratio of nondiseased to diseased subjects
Prevalence 0.50 0.33 0.20 0.10 0.075 0.05
Subjects with disease (n,)
Subjects without disease (n2)
Total sample size
95% Positive likelihood ratio confidence interval
80 76 74 73 73 73
80 152 296 656 896 1376
160 228 370 729 969 1449
1.43-2.21 1.51-2.15 1.57-2.06 1.62-2.01 1.63-1.99 1641.98
The table uses data from Problem 2 and varies the prevalence of disease among consecutively enrolled subjects.
obtaining a high specificity and minimizing the negative likelihood ratio [3]. For sample size estimation we focus on the desired upper limit to the 95% confidence interval for the negative likelihood ratio. Since the ratio between nondiseased and diseased subjects now depends on prevalence, the investigator cannot control r. The total sample size predicted to exclude a maximum negative likelihood ratio from a 95% confidence interval must be selected given the anticipated disease prevalence. Table 2 demonstrates the total sample size required and the predicted confidence interval around the positive likelihood ratio using the values from Problem 2: sensitivity = 0.9, specificity = 0.5, and LR - ,,,= = 0.4. Table 2 demonstrates clearly that sample size depends strongly on the number of non-diseased patients when we are seeking to exclude a maximum negative likelihood ratio (LR-,,,) and cannot control the ratio of non-diseased to diseased patients. In fact, the required number of diseased patients changes little below a prevalence of 50%. SUMMARY
Recognizing that likelihood ratios are algebraically identical to relative risk ratios allows calculation of their confidence intervals. This algebraic identity suggests analogies between diagnostic test results and epidemiologic cohort studies designed to assess the impact of risk factors for disease. For diagnostic tests, the risk conferred by having a certain test result is analogous to “exposure” to a risk factor during epidemiologic cohort studies. The confidence interval around a likelihood ratio describes the precision of the relative risk of disease at that test outcome level and can be calculated with
DAVIDL.
170
a hand-held calculator containing logarithmic functions. * Sample size calculations for investigations of diagnostic tests may be best guided by estimating the meaningful confidence interval around the desired likelihood ratio. We agree with the trend to place more emphasis on confidence intervals and urge that likelihood ratio confidence intervals be used to estimate sample sizes for diagnostic test studies. Acknowledgements-We
thank Eugene Oddone and John Williams for their critical review of earlier versions of the manuscript. REFERENCES 1. Gardner MJ, Altman DG, Eds. Statistics with Coniidence: Conildence Intervals and Statistical Guidelines. London: British Medical Journal; 1989. 2. Braitman LE. Confidence intervals extract clinically useful information from data. Ann Intern Med 1988: 108: 296-298. 3. Sackett DL, Haynes RB, Tugwell P. Clinical Epidemiology: a Basic Science for Clinical Medicine. Boston, Mass.: Little, Brown; 1985. 4. Simel DL, Feussner JR, Delong ER, Matchar DB. Intermediate, indeterminate, and uninterpretable diagnostic test results. Med Decis Making 1987; 7: 107-l 14. 5. Simel DL, Halvorsen RA, Feussner JR. Quantitating bedside diagnosis. J Gen Intern Med 1988; 3: 423428. 6. Sox HC, Blatt MA, Higgins MC, Marton KI. Medical Decision Making. New York: Butterworths; 1988.
*The authors will provide a program, on request, that conveniently calculates likelihood ratio confidence intervals using the SAS language. Please send a DOS formatted floppy disk.
SIMEL et al.
7.
Bristol DR. Sample sixes for constructing confidence intervals and testing hypotheses. Stat Med 1989, 8: RI-i?4 11 APPENDIX
Consider the problem of obtaining a 95% confidence interval for the risk ratio p, /p2, where p, denotes some proportion of patients with disease and pz denotes some proportion of patients without disease. For example, in the definition of LR+ the values p, = sensitivity and p2 = 1 - specificity. Taking the logarithm (to the base e) of the risk ratio p,/p2 yields: 10, /P& = ln@, I- Mph
(A.11
and var(ln~,/~J
= var(ln~J
+ vm(ln~~)
(A.21
because the assessments on the patients with disease vs those without disease are statistically independent. We will use a Taylor series expansion to estimate the two variances on the right hand side of equation (A.2). For any function&) the Taylor series approimation’to iis variance is varl..,fhI \ Iax I.
J clin Epidmiol Vol. 44, No. 8, pp. 763-770,1991 Printedin Great Britain
LIKELIHOOD RATIOS WITH CONFIDENCE: SAMPLE SIZE ESTIMATION FOR DIAGNOSTIC TEST STUDIES DAVID L. SIMEL,* GREGORY P. SAMSA and DAVID B. MATCHAR
Medical Center and Divisionof General Internal Medicine,Duke UniversityMedicalCenter, Durham, North Carolina.U.S.A.
Center for Health Services Research in Primary Care, Durham Veterans Administration
(Received in revised form 19 December 1990)
Abstract-Confidence intervals are important summary measures that provide useful information from clinical investigations, especially when comparing data from different populations or sites. Studies of a diagnostic test should include both point estimates and confidence intervals for the tests’ sensitivity and specificity. Equally important measures of a test’s efficiency are likelihood ratios at each test outcome level. We present a method for calculating likelihood ratio confidence intervals for tests that have positive or negative results, tests with non-positive/non-negative results, and tests reported on an ordinal outcome scale. In addition, we demonstrate a sample size estimation procedure for diagnostic test studies based on the desired likelihood ratio confidence interval. The renewed interest in confidence intervals in the medica literature is important, and should be extended to studies analyzing diagnostic tests. Likelihood ratio
Confidence interval
INTRODUCTION
Confidence intervals are important summary measures that provide useful information from clinical investigations , especially when comparing data from different populations or sites [l, 21. In particular, they allow more critical evaluation of point estimates describing the sensitivity and specificity of diagnostic testsvalues that inform us about a test’s diagnostic efficiency. Sample size estimates for studies that investigate tests generally depend on the length of the desired confidence interval around the operating characteristics. In addition to sensitivity and specificity, likelihood ratios are also used to characterize the behavior of diagnostic tests[3]. Sensitivity (the proportion of diseased patients with posi*Address all correspondence to: David L. Simel, Ambulatory Care Service (1 lC), Durham Veterans Administration Medical Center, Durham, NC 27705, U.S.A.
Sample size
Diagnostic test
tive results) and specificity (the proportion of non-diseased patients with negative results) describe the behavior of tests given patients’ disease status. For 2 x 2 tables, the positive likelihood ratio (LR+) is defined as sensitivity/(1 - specificity), while the negative likelihood ratio (LR-) is (1 - sensitivity)/specificity (Fig. 1). Likelihood ratios algebraically combine sensitivity and specificity to describe more than the independent values themselves-they describe the change in odds favoring disease given a particular test result. When a likelihood ratio exceeds 1, the odds favoring disease increase; when the likelihood ratio becomes less than 1, the odds favoring disease decrease; when a likelihood ratio approaches 1, the odds favoring disease do not change and the test is indeterminate [4]. Since likelihood ratios refer to actual test results before disease status is known, they are more immediately useful to clinicians than sensitivity and specificity. 763
764
DAWDL. SIMEL ef al.
Disease Present
Positive
A
Negative
C
Test Result
f
nl
Absent
3 B
D
n2
Fig. 1. Conventional 2 x 2 matrix in which A = true positive, B = false positive, C = false negative, and D = true negative results. The total patients with disease = PI, and total patients without disease = nz .
Likelihood ratios have three important properties, outlined by others, that together make them extremely useful [3]. First, likelihood ratios do not change with the pretest probability of disease. Second, likelihood ratios can be calculated for multiple levels of test outcomes rather than just two nominal levels. Finally, likelihood ratios allow us to assess the impact of a test result on the odds that a patient will have disease. The intent of this paper is not to convince investigators to report only likelihood ratios, but to emphasize that likelihood ratios, like other diagnostic test indices, also have confidence intervals. The likelihood ratio confidence intervals can be determined from the dual functions of sensitivity and specificity, and they can be used to guide sample size estimation for clinical investigations. The point estimate for sensitivity expresses the percentage of patients with disease who have a positive test result, whereas the point estimate for specificity expresses the percentage of patients without disease who have a negative test result. Since the precision of these point estimates varies with the sample size, confidence intervals allow expression of a plausible range for the sensitivity and specificity. Thus, a 95% confidence interval contains the point estimate about 95 times in 100 replications of the study; in about 5 replications the point estimate would fall outside the range of the confidence interval. Because the width of the confidence interval decreases as the sample size increases, we have more confidence in the value of point estimates from larger populations. Calculating the confidence interval for sensitivity and specificity is straightforward. How to calculate the confidence interval or estimate the sample size requirements for likelihood ratios is
less obvious since, not only do likelihood ratios combine two values (sensitivity and specificity), they also may apply to tests having multiple possible outcome levels (e.g. positive, non-positive/non-negative, or negative). We will show how to calculate the confidence intervals around likelihood ratios for tests with positive or negative results (2 x 2 table); conditional likelihood ratios for tests with positive, non-positive/nonnegative, or negative results (3 x 2 table); and conventional likelihood ratios for tests with results transformed to a multilevel ordinal scale (r x 2 table, where r represents the number of rows). In this last example, neither conventional sensitivity and specificity values or conditional likelihood ratios completely describe multilevel test outcomes, so that conventional likelihood ratios at each test result level are preferred. Finally, we will demonstrate the method for estimating the sample size required to show that a likelihood ratio falls within an expected range. The equations and working examples should serve as reference guides for reporting likelihood ratio confidence intervals and estimating sample sizes based on the desired confidence interval length. CONFIDENCE INTERVALS FOR LIKELIHOOD RATIOS FROM 2 x 2 TABLES
The simple 2 x 2 table displays the results of a diagnostic test vs the reference standard for a population size, n, where (Fig. 1): sensitivity = A/(A + C) and specificity = D/(B + D). Each of these sensitivity and specificity values represents a simple proportion. The approximate 95% confidence interval for any simple proportion, p, drawn from a normal distribution is p f
1.96.&m,
(1)
where n represents the size of the population. As an examole, 95% confidence __ _ . the aunroximate interval for the sensitivity from Fig. 1 is
01 f 1.96,/[A /(A + Cl1*[C/U + Cllh
V/V
+
(2)
or (sensitivity) f 1.96 Jsensitivity
‘(1 - sensitivity)/n,,
(3)
Likelihood Ratios and Sample Si
where n, represents the number of patients with disease. The approximate 95% confidence interval for the specificity from Fig. 1 is P/P
+
011
k 1.96,/P l@ + 011.PI@ + D)I/n;! (4) or (specificity) If: 1.96 Jspecificity
. (1 - specificity)/n,,
(5)
where n2 represents the number of patients without disease. The confidence interval for sensitivity varies in proportion to the number of patients with disease, just as the confidence interval for specificity varies in proportion to the number of patients without disease. The positive likelihood ratio (LR + ) expresses the change in odds favoring disease given a positive test result, whereas the negative likelihood ratio (LR -) expresses the change in odds favoring disease given a negative test result. These likelihood ratios for the 2 x 2 table in Fig. 1 are calculated from the sensitivity and specificity: LR + = sensitivity/( 1 - specificity)
(6)
LR - = (1 - sensitivity)/specificity.
(7)
and
In
sensitivity + 1.96 1 - specificity
where A and B are values of the respective cells from the 2 x 2 table and exp is the base of the natural logarithm. For the negative likelihood ratio: LR-
=exp
In
1 - sensitivity + 1 g6 specificity - .
765
where C and D are the values of the respective cells from the 2 x 2 table and exp is the base of the natural logarithm. Equations (8) and (9) are examples of the general formula, derived in the Appendix, for the approximate 95% confidence interval for risk ratios. The general formula represents the likelihood ratio 95% confidence interval that can be calculated for any level of test result, x, by the equation LR,=exp(lnEk where
for
1.96*J%+z),(lO) LR, = LR+,
p1 = sensitivity, and pzn, = B. When specifying the negative likelihood ratio (LR, = LR-), p, = 1 - sensitivity, p2 = specificity, p,n, = C, and p2n2 = D. For example, when the values from Table 1 are A = 30, B = 5, C = 10, and D = 45, the following operating characteristics and 95% confidence intervals are obtained:
pz = 1 - specificity,
p,n, = A,
sensitivity = 0.75 (95% confidence interval 0.62-0.88), specificity = 0.90 (0.82-0.98), positive likelihood
An approximation to the confidence interval for the likelihood ratio could be obtained by substituting the upper and lower approximate 95% confidence limits for sensitivity and specificity into equations (6) and (7). Unfortunately, this approximation reveals confidence intervals that are too wide, suggesting inappropriately less precision. For the positive likelihood ratio, the approximate 95% confidence intervals should be calculated by the equation LR+ =exp
Determinations
ratio = 7.5 (3.20-17.56),
and negative likelihood ratio = 0.28 (0.16-0.48). Equation (10) may not appear intuitive, but the equation is algebraically identical with the formula for calculating the confidence interval for relative risk ratios. The formula can be explained by examining the exponentiated component parts. The value p,/p2 represents the likelihood ratio. The upper and lower bounds of the 95% confidence interval around the log likelihood ratio (lnp, /p2, where In is the natural logarithm) is determined after multiplying 1.96 times the standard error of the log likelihood ratio. The value (1 -pI )/p, n, refers to subjects with disease, while (1 -p2)/p2n2 is a separate and analogous calculation referring to subjects without disease. The key to understanding the equation is recognizing the separability between diseased and non-diseased subjects so that sensitivity and specificity have independent effects on the likelihood ratio confidence interval. The magnitude of the total effect cannot be interpreted from the confidence interval around the independent proportions of sensitivity and specificity.
766
DAVIDL. &MELer al.
CONDITIONAL CONFIDENCE INTERVALS FOR LIKELIHOOD RATIOS FROM 3 x 2 TABLES
LR’+ = sensitivity//( 1 - specificity’)
(11)
LR’ - = (1 - sensitivity’)/specificity’,
(12)
and The clinical examination is one type of diagnostic test that yields frequent non-positive/ non-negative results. An example is the physical examination for ascites: the physician may be confident that ascites is present (positive), confident that it is absent (negative), or not sure (non-positive/non-negative) [5]. When nonpositive/non-negative or intermediate results are obtained, the likelihood ratios and test operating characteristics may be reported as conditional on obtaining a positive or negative result [4]. This may be especially useful for tests that have excellent predictive value when positive or negative, but poor predictive value when the test result is less obvious. Conditional operating characteristics allow the investigator to disentangle the impact of an extremely useful positive or negative result from a less useful non-positive/non-negative outcome. This information is especially important when the diagnostic test under consideration is risky or expensive [4]. Tests that are neither positive or,negative may have an important impact on clinical decision making. Sometimes, a test that is neither positive or negative may increase the odds favoring disease to a level between the positive and negative likelihood ratio, an intermediate result. In other cases, a non-positive/non-negative result may not add useful information to the clinical examination. The lack of diagnostic utility at any test level, detected by a likelihood ratio approaching 1, defines an indeterminate result [4]. The conditional operating characterstics from the 3 x 2 table of Fig. 2 are sensitivity’ = A /(A + C) and specificity’ = D/(B + D). The conditional likelihood ratios are:
where LR’+ is the positive and LR’- is the negative conditional likelihood ratio. Confidence intervals for the positive and negative conditional likelihood ratios are identical with the conventional likelihood ratios for the 2 x 2 case in equations (8) and (9), except the conditional values for sensitivity and specificity are substituted. The likelihood ratio for non-positive/nonnegative results (LR -I-) is calculated in terms of the test yield. The test yield provides useful information by describing the probability of obtaining a more definite positive or negative result, as opposed to a non-positive/non-negative or intermediate result. The positive yield (YD+) is the probability of a positive or a negative result when disease is present:
YD+ =(A +C)/(A +C+E);
(13)
whereas the negative yield (YD-) is the probability of a positive or a negative result when disease is absent, YD-
=(B+D)/(B+D+F).
(14)
The likelihood ratio for a non-positive/nonnegative result (LR & ) is LRf
= (1 - YD+)/(l
- YD-).
(15)
The confidence interval for the likelihood ratio for non-positive/non-negative results can be derived from equation (10):
+ 1.96$~),
(16)
1 -YD+,p,= 1 -YD-,p,q=E and p2n, = Fare the values from the 3 x 2 table
wherep,=
Disease
Present
Test Result
Absent
(Fig. 2).
Positive
A
B
CONVENTIONAL CONFIDENCE INTERVALS FOR LIKELIHOOD RATIOS FOR TESTS WITH MULTILEVEL RESULTS
Non-positive Non-negative
E
F
c
D
Ill
n2
The conventional or conditional sensitivity and specificity values do not completely describe multilevel ordinal outcomes. An example is the exercise tolerance test, transformed to an ordinal scale outcome based on the millimeters of ST-segment depression. For such tests, the usual descriptors of its performance are the
Negative
EH
Fig. 2. The 3 x 2 matrix when obtaining results that are non-positive/non-negative.
Likelihood Ratios and Sample Size Determinations
Disease PWSWI
A
B
E
F
Level 3
G
H
Level 4
C
D
fll
n2
Level 1
Test
Absent
Level 2
Result
Fig. 3. The I x 2 matrix when obtaining multilevel test results.
conventional multilevel likelihood ratios [3]. These likelihood ratios show the increase in odds favoring disease at each test outcome level. The multilevel likelihood ratio is calculated by describing the proportion of diseased patients with a given test result (p,) divided by the proportion of non-diseased patients with the same test result (p2). For a subject with the second level outcome (LR,=,), using the notation from Fig. 3, the point estimate for LR, is
_p,_W+E+G+C)
LR
2
~2
E/n,
F/(B+F+H+D)=Fln,’
(17)
where n, describes the subjects with disease and n, describes the subjects without disease. Based on equation (lo), the approximate 95% confidence interval can be calculated for any level of test result, x, by the general equation for risk ratio confidence intervals. In this case, the values for equation (10) (pl, p2, n,, and n2) represent the values described above. Equation (17) uses a standard notation for multilevel ordinal outcomes. In other instances, a clinician may be most insterested in choosing a single cutpoint that defines normality. For example, if a clinician considers Level 1 or Level 2 results abnormal and Level 3 or Level 4 results normal, then equation (17) becomes
=&=(A +W(A +E+G+C)
LR
”
~2
(B+F)/(B+F+H+D)
Fortunately, just as in equation (17), the appropriate values for p, , n, , p2, and n2 can be reinserted into equation (10) to determine the confidence interval for LR c *. The effect of different cutpoints for normality are evaluated
161
by plotting the true positive rate vs the false positive rate in a receiver operating characteristic (ROC) curve. It is important to recognize that the slope at each point on the ROC curve represents a positive likelihood ratio. One method for choosing the cutpoint involves identifying the point on the curve balancing the pretest probability of disease with the relative importance to the patient of avoiding false positive and negative results [6]. The confidence interval for any point on the ROC curve can be determined with the above method. The utility of the general formula [equation (lo)] should now be apparent. However one chooses to define the cutpoint describing normality, the equation can be used to calculate the relative confidence interval for the corresponding likelihood ratio. By understanding the interpretation of p, and p2, the equation works for the positive likelihood ratio, negative likelihood ratio, or likelihood ratio at multiple levels. SAMPLE SIZE ESTIMATES FOR EVALUATING LIKELIHOOD RATIOS
Sample size estimates for studies describing the operating characteristics of diagnostic tests may reflect the degree of confidence the investigator wishes to establish for the sensitivity point estimate (or, alternatively the specificity point estimate). However, when a clinician uses a diagnostic test, the question becomes not one of confidence in the operating characteristics, but confidence in the impact of the result on the likelihood of disease. The impact suggests that sample size estimates should reflect concern with the confidence interval around the likelihood ratio rather than just the confidence interval around sensitivity or specificity. Although methods for calculating sample sizes for constructing confidence intervals around proportions are well described [I, the appropriate method applicable to likelihood ratios is not generally appreciated by clinical investigators. In many situations, the desired threshold limits for the likelihood ratio confidence interval may be determined by expert opinion or through the tools of decision analysis [4,6]. These threshold values may also serve as the upper or lower bounds of our desired confidence interval. We will show the steps required for estimating the sample size of an investigation designed to have an appropriately narrow confidence interval around a clinically meaningful likelihood ratio.
768
DAVIDL. &MELet al.
WORKING EXAMPLES
Problem I
Based on pilot data, we believe the sensitivity of a new test is at least 80%, specificity = 73%, and positive likelihood ratio = 2.96 [derived from sensitivity/( 1 - specificity)]. Suppose expert opinion, or decision analysis, suggests the test would be clinically useful when the positive likelihood ratio was 2 2.0 (LR + min).Our study goal is to demonstrate that the positive likelihood ratio 95% confidence interval around 2.96 does not contain 2.0. How many patients are required with and without disease? Solution. We use equation (10) to solve for n, = n2 = number of required patients with and without disease. In this case, p, = sensitivity = 0.80, pz = 1 - specificity = 0.27, and LR+ = 2.96. The values LR+ti, = 2.0, p, , and pz are inserted into equation (10) and solved for n,=n,=n: 2.0 = exp{ln(0.8/0.27)
- 1.96*,/(l/n)*[(0.2/0.8)
+ (0.73/0.27)]}.
After taking the logarithm of both sides of the equation, we find n = 73.4. Thus, 74 patients with disease and 74 patients without disease are required to show that a test with a sensitivity of at least 80%, specificity of at least 73%, and likelihood ratio of 2.96 does not contain the threshold likelihood ratio of 2.0 in its 95% confidence interval. It should be apparent from equation (10) that the standard error of the log likelihood ratio decreases, resulting in a narrower LR+ confidence interval, when (1) sensitivity improves, (2) specificity improves, or (3) the number of patients with disease increases. Problem 2
We hypothesize that a new screening test would be clinically useful if the negative likelihood ratio was ~0.4 (LR-,,). We believe the sensitivity of this new test is at least 90%, the negative likelihood ratio is 0.2, and want to show that the negative likelihood ratio 95% confidence interval around 0.2 does not contain 0.4. How many patients are required with and without disease? Solution. We use equation (10) to solve for n, = n, = number of required patients with and without disease. In this case, p, = 1 sensitivity = 0.1 and LR- = 0.2; therefore, from equation (7), p2 = specificity = 0.5. The
values LR -man = 0.4, p,, and p2 are inserted into equation (10) and solved for n, = n, = n: 0.4 = exp{ln(0.1/0.5)
+ 1.96*J(l/n)*[(O.9/0.1)
+ (O.S/O.S)]}.
After taking the logarithm of both sides of the equation, we find n = 79.9. Thus, 80 patients with disease and 80 patients without disease are required to show that a test with a sensitivity of at least 90% and negative likelihood ratio of 0.2 does not contain a likelihood ratio of 0.4 in its 95% confidence interval. If we have more than 80 patients in either group, or a sensitivity >0.90, our confidence that the negative likelihood ratio does not contain 0.4 becomes greater. Problem 3
Often, clinical investigators have difficulty finding patients with the disease of interest. Suppose, after estimating the sample size in Problem 1, the investigators realize they will be able to enroll only about 1 patient with disease for every 5 without disease. How many patients are required with and without disease to assure that the 95% confidence interval excludes LR+ = 2? To answer the question, we must use the general form for the likelihood ratio confidence interval [equation (lo)]. From Problem 1 we have p, = sensitivity = 0.80 and p2 = (1 specificity) = 0.27. However, now n, # n2 and instead n, = 0.2 (n2): 2 = exp ln(0.8/0.27)
After taking the logarithm of both sides of the equation we find n2 = 98.3. Thus, the investigator requires 99 patients without disease and 20 diseased patients to exclude a positive likelihood ratio of 2 from the predicted positive likelihood ratio of 2.96. The paradoxical finding of Problem 3 is that the imbalance between diseased vs non-diseased patients actually decreases the overall sample size required to exclude the minimally acceptable positive likelihood ratio. However, enrolling fewer subjects with disease has an confidence intervals adverse impact-the around the negative likelihood and the sensitivity become progressively wider. The paradox
Likelihood Ratios and Sample Size Determinations
highlights the strength of using likelihood ratios to estimate sample sizes for diagnostic test studies: equation (10) can be used to select the minimum number of diseased and non-diseased patients to achieve the desired likelihood ratio confidence interval. Close inspection of equation (10) reveals that many n,, n, pairs satisfy the desired positive likelihood ratio confidence interval solution excluding the minimum positive likelihood ratio. Although the simplest solution to sample size estimation is to set n, = n2, this may not lead to a total sample size minimizing the total number of patients for a specified LR+ confidence interval. Table 1 demonstrates the total sample size required and the predicted confidence interval around the negative likelihood ratio using the values from Problem 3: sensitivity = 0.8, specificity = 0.73, and LR+,i” = 2. In Table 1, r is the ratio between patients without disease (n2) and with disease (n,). When patients are selected from a population with known disease status, the investigator has control over the ratio between non-diseased and diseased patients. In that case, the smallest sample size for the desired positive likelihood ratio confidence interval can be selected. For the example, the minimum sample size occurs with r = 3.0 or, 86 without disease and 29 patients with disease. In other cases, the investigator may enroll patients consecutively before their disease status is defined. This strategy is often used to evaluate screening tests in large populations. Large screening programs for diseases with relatively low prevalence generally seek to minimize false positive results; here the prime concern is Table 1. Total sample size required for the desired positive likelihood ratio 95% confidence interval with LR+,, = 2 when the investigator can control the ratio, r, of nondiseased to diseased subjects
r = n&r, 0.05 0.1 0.5 1.0 2.0 3.0 5.0 10.0
Subjects with disease (n,) 1351 619 141 14 40 29 20 13
Subjects without disease (n2)
Total sample size
95% Negative likelihood ratio confidence interval
68 68 71 14 80 86 99 130
1419 141 212 148 120 115 119 143
0.240.31 0.23-0.32 0.194.39 0.17-0.44 0.15-0.51 0.13-0.57 0.11-0.67 0.09-0.82
A ratio of 3 : 1 provides the minimum sample size to exclude the minimum positive likelihood ratio with the data from Problem 3.
769
Table 2. Total sample size required for the desired negative likelihood ratio 95% confidence interval with LR-, = 0.4 when the investigator cannot control the ratio of nondiseased to diseased subjects
Prevalence 0.50 0.33 0.20 0.10 0.075 0.05
Subjects with disease (n,)
Subjects without disease (n2)
Total sample size
95% Positive likelihood ratio confidence interval
80 76 74 73 73 73
80 152 296 656 896 1376
160 228 370 729 969 1449
1.43-2.21 1.51-2.15 1.57-2.06 1.62-2.01 1.63-1.99 1641.98
The table uses data from Problem 2 and varies the prevalence of disease among consecutively enrolled subjects.
obtaining a high specificity and minimizing the negative likelihood ratio [3]. For sample size estimation we focus on the desired upper limit to the 95% confidence interval for the negative likelihood ratio. Since the ratio between nondiseased and diseased subjects now depends on prevalence, the investigator cannot control r. The total sample size predicted to exclude a maximum negative likelihood ratio from a 95% confidence interval must be selected given the anticipated disease prevalence. Table 2 demonstrates the total sample size required and the predicted confidence interval around the positive likelihood ratio using the values from Problem 2: sensitivity = 0.9, specificity = 0.5, and LR - ,,,= = 0.4. Table 2 demonstrates clearly that sample size depends strongly on the number of non-diseased patients when we are seeking to exclude a maximum negative likelihood ratio (LR-,,,) and cannot control the ratio of non-diseased to diseased patients. In fact, the required number of diseased patients changes little below a prevalence of 50%. SUMMARY
Recognizing that likelihood ratios are algebraically identical to relative risk ratios allows calculation of their confidence intervals. This algebraic identity suggests analogies between diagnostic test results and epidemiologic cohort studies designed to assess the impact of risk factors for disease. For diagnostic tests, the risk conferred by having a certain test result is analogous to “exposure” to a risk factor during epidemiologic cohort studies. The confidence interval around a likelihood ratio describes the precision of the relative risk of disease at that test outcome level and can be calculated with
DAVIDL.
170
a hand-held calculator containing logarithmic functions. * Sample size calculations for investigations of diagnostic tests may be best guided by estimating the meaningful confidence interval around the desired likelihood ratio. We agree with the trend to place more emphasis on confidence intervals and urge that likelihood ratio confidence intervals be used to estimate sample sizes for diagnostic test studies. Acknowledgements-We
thank Eugene Oddone and John Williams for their critical review of earlier versions of the manuscript. REFERENCES 1. Gardner MJ, Altman DG, Eds. Statistics with Coniidence: Conildence Intervals and Statistical Guidelines. London: British Medical Journal; 1989. 2. Braitman LE. Confidence intervals extract clinically useful information from data. Ann Intern Med 1988: 108: 296-298. 3. Sackett DL, Haynes RB, Tugwell P. Clinical Epidemiology: a Basic Science for Clinical Medicine. Boston, Mass.: Little, Brown; 1985. 4. Simel DL, Feussner JR, Delong ER, Matchar DB. Intermediate, indeterminate, and uninterpretable diagnostic test results. Med Decis Making 1987; 7: 107-l 14. 5. Simel DL, Halvorsen RA, Feussner JR. Quantitating bedside diagnosis. J Gen Intern Med 1988; 3: 423428. 6. Sox HC, Blatt MA, Higgins MC, Marton KI. Medical Decision Making. New York: Butterworths; 1988.
*The authors will provide a program, on request, that conveniently calculates likelihood ratio confidence intervals using the SAS language. Please send a DOS formatted floppy disk.
SIMEL et al.
7.
Bristol DR. Sample sixes for constructing confidence intervals and testing hypotheses. Stat Med 1989, 8: RI-i?4 11 APPENDIX
Consider the problem of obtaining a 95% confidence interval for the risk ratio p, /p2, where p, denotes some proportion of patients with disease and pz denotes some proportion of patients without disease. For example, in the definition of LR+ the values p, = sensitivity and p2 = 1 - specificity. Taking the logarithm (to the base e) of the risk ratio p,/p2 yields: 10, /P& = ln@, I- Mph
(A.11
and var(ln~,/~J
= var(ln~J
+ vm(ln~~)
(A.21
because the assessments on the patients with disease vs those without disease are statistically independent. We will use a Taylor series expansion to estimate the two variances on the right hand side of equation (A.2). For any function&) the Taylor series approimation’to iis variance is varl..,fhI \ Iax I.
