A new risk-adjusted Bernoulli cumulative sum chart for monitoring binary health data Giuseppe Rossi, Simone Del Sarto and Marco Marchi Stat Methods Med Res published online 22 April 2014 DOI: 10.1177/0962280214530883 The online version of this article can be found at: http://smm.sagepub.com/content/early/2014/04/21/0962280214530883

Published by: http://www.sagepublications.com

Additional services and information for Statistical Methods in Medical Research can be found at: Email Alerts: http://smm.sagepub.com/cgi/alerts Subscriptions: http://smm.sagepub.com/subscriptions Reprints: http://www.sagepub.com/journalsReprints.nav Permissions: http://www.sagepub.com/journalsPermissions.nav

>> OnlineFirst Version of Record - Apr 22, 2014 What is This?

Downloaded from smm.sagepub.com at UNIVERSITY OF WINDSOR on October 25, 2014

XML Template (2014) [15.4.2014–1:03pm] [1–10] //blrnas3/cenpro/ApplicationFiles/Journals/SAGE/3B2/SMMJ/Vol00000/140041/APPFile/SG-SMMJ140041.3d (SMM) [PREPRINTER stage]

Article

A new risk-adjusted Bernoulli cumulative sum chart for monitoring binary health data

Statistical Methods in Medical Research 0(0) 1–10 ! The Author(s) 2014 Reprints and permissions: sagepub.co.uk/journalsPermissions.nav DOI: 10.1177/0962280214530883 smm.sagepub.com

Giuseppe Rossi,1 Simone Del Sarto1 and Marco Marchi2

Abstract To monitor a health event in patients with a specific risk of developing the event, a risk-adjusted cumulative sum chart is needed. The risk-adjusted cumulative sum chart proposed in the literature has some limitations. Setting appropriate control limits is not straightforward, there is no simple formula for constructing them, and they remain sensitive to changes in the underlying risk distribution and the baseline incidence rate. To overcome these limits, we propose a new risk-adjusted Bernoulli cumulative sum chart as a simple and efficient solution. Analyses of simulated and real data sets illustrate the performance and usefulness of the proposed procedure. Keywords Bernoulli variate, cumulative sum chart, health data, risk-adjustment, surveillance

1 Introduction Cumulative sum (CUSUM) charts were initially developed by Page1 for industrial quality control, where the main interest was monitoring the production process. This method was designed to detect a shift occurring in the mean of a process (in-control mean) toward a particular mean (out-ofcontrol mean) for which the system was declared ‘‘out of control.’’ The use of CUSUM method is growing in popularity for health-care and public-health surveillance.2 CUSUM chart allows monitoring a Poisson process, where observations consist of counts collected over time. However, the standard Poisson CUSUM3 is not able to treat a variability in the in-control mean, due to timevarying population size, seasonality of the process, or adjustment for risk factors. To take into account varying in-control expected values, adjustments to the standard procedure have been proposed by Rossi et al.,4 Rogerson and Yamada,5 and Ho¨hle and Paul.6 1

Unit of Epidemiology and Biostatistics, Institute of Clinical Physiology, National Research Council and G. Monasterio Foundation, Pisa, Italy 2 Department of Statistics, University of Florence, Florence, Italy Corresponding author: Giuseppe Rossi, Unit of Epidemiology and Biostatistics, Institute of Clinical Physiology, National Research Council and G. Monasterio Foundation, via Moruzzi 1, 56124 Pisa, Italy. Email: [email protected]

Downloaded from smm.sagepub.com at UNIVERSITY OF WINDSOR on October 25, 2014

XML Template (2014) [15.4.2014–1:03pm] [1–10] //blrnas3/cenpro/ApplicationFiles/Journals/SAGE/3B2/SMMJ/Vol00000/140041/APPFile/SG-SMMJ140041.3d (SMM) [PREPRINTER stage]

2

Statistical Methods in Medical Research 0(0)

Measuring quality of medical practice is a key component in improving eﬃciency in health care. Health data often consist of a continuous stream of binary individual observations (patients) and we may be interested in monitoring the incidence rate p of a particular outcome (mortality, infection, etc.). For this kind of data Reynolds and Stoumbos7 developed the Bernoulli CUSUM chart designed to detect a shift in the overall in-control incidence rate p0. However, patients could present very heterogeneous characteristics in terms of risk factors and then of risk of event. Thus, an adjustment for varying in-control risks of event is required because the system could signal an alarm even though a shift in the incidence rate has not occurred but the baseline risk of event has been modiﬁed due to changes in the patient mix and so in the underlying predisposed risk distribution. The Reynolds and Stoumbos Bernoulli CUSUM chart is a risk-unadjusted CUSUM chart and does not allow an adjustment for varying in-control risks of event, thus limiting its use. For this reason Steiner et al.8 proposed a risk-adjusted CUSUM (RA-CUSUM) chart based on log-likelihood score for monitoring surgical performance. This charting scheme plots the CUSUM of log-likelihood ratio scores of patients that are based on the predicted preoperative risks of patients. The preoperative risks are estimated from a logistic regression based on Parsonnet scores, where the Parsonnet score is a method of uniform stratiﬁcation of risk for evaluating the results of surgery in acquired adult heart disease.9 A Markov chain (MC) method was considered by Steiner et al.8 to approximate the average run lengths. This method has been used over time, particularly for monitoring surgical outcomes,10–14 but setting appropriate control limits is not straightforward and there is no simple formula for computing them.15,16 Furthermore, even when the RA-CUSUM chart has been adjusted for the patients’ risks, it remains sensitive to changes in the underlying predisposed risk distribution and the baseline incidence rate.17,18 We propose a risk-adjusted Bernoulli CUSUM (RA-B-CUSUM) chart, very easy to implement and whose setting does not require great computational eﬀort; in addition, it is not sensitive to changes in the underlying predisposed risk distribution and in the baseline incidence rate. This article is organized as follows: in Section 2 the Bernoulli CUSUM chart, the Steiner’s RACUSUM chart, and the proposed RA-B-CUSUM chart are presented. In Section 3 the performance of the proposed RA-B-CUSUM chart is compared with the Steiner’s RA-CUSUM chart by a simulation study. In Section 4 the results of the application of our proposal to a real data set is shown, and ﬁnally we conclude with a brief discussion in Section 5.

2 CUSUM for binary data 2.1 Risk-unadjusted Bernoulli CUSUM When we are concerned about departure from a target a Bernoulli CUSUM was proposed by Reynolds and Stoumbos7 and is based on a decision interval scheme. Let observation yt in the tth patient be distributed as a Bernoulli variate with parameter p, which is the probability that an event will occur. We assume p ¼ p0 when the process is in control and p ¼ p1 when the process is out of control. We are interested in detecting a shift from p0 to p1 as quickly as possible; if this happens the system is declared out of control and the method will signal an alarm. As in the Steiner CUSUM, we expressed p1 in function of the odds ratio (OR). Being OR ¼ [p1 / (1– p1)]/[ p0 / (1– p0)], p1 can be deﬁned as p1 ¼ ORp0/(1 p0 þ ORp0). In most situations in which p is being monitored, the main interest is detecting increases in p. However, detecting decreases will also be of interest in some applications.

Downloaded from smm.sagepub.com at UNIVERSITY OF WINDSOR on October 25, 2014

XML Template (2014) [15.4.2014–1:03pm] [1–10] //blrnas3/cenpro/ApplicationFiles/Journals/SAGE/3B2/SMMJ/Vol00000/140041/APPFile/SG-SMMJ140041.3d (SMM) [PREPRINTER stage]

Rossi et al.

3

When we are interested in an increase or decrease in p, the Bernoulli CUSUM control statistic St is calculated as follows St ¼ max 0, St1 þ ðyt kÞ for an increase in p ð1Þ ð2Þ St ¼ min 0, St1 þ ðyt kÞ for a decrease in p where k > 0 is the suitably chosen reference value and S0 ¼ 0. The chart will signal if St hu in an upward chart (1) and if St hd in a downward chart (2), where hu and hd are the suitably chosen control limits. The reference value k is calculated using p0 and p1, and the appropriate choice of k is7 r1 k¼ ð3Þ r2 where 1 p1 r1 ¼ log 1 p0 p1 ð1 p0 Þ r2 ¼ log p0 ð1 p1 Þ

ð4Þ

The performance of a CUSUM chart is evaluated by the average number of observations to signal (ANOS). When the system is in control the ANOS0 should be as long as possible, since in this context a signal would consist of a false alarm. In the other hand, if the process shifts in the out-ofcontrol state the ANOS1 should be shorter, with the aim of detecting the shift as quickly as possible. The selection of the control limit h is critical because the real performance of the procedure in terms of false alarm and time to signal substantially depends on the choice of h. Reynolds and Stoumbos7 suggested several approaches in order to select a suitable control limit. One approach is based on the MCs19 that give the exact ANOS, but it can be computing intensive due to the large size of the transition probabilities matrix.7 A second approach uses the basic MC formulation but explicitly solves the resulting linear equations involving ANOS; however, it can also require high computational eﬀort as pointed out by Reynolds and Stoumbos.7 A third and easier approach to estimating the ANOS is based on the corrected diﬀusion (CD) approximation.20,21 This latter approach is based on the following approximation of the in-control ANOS0 ANOS0

expðh r2 Þ h r2 1 r2 p0 r1

ð5Þ

where r1 and r2 are determined from equation (4) and h* is an adjusted value of h used by CD approximation pﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ h ¼ h þ "ðp0 Þ p0 ð1 p0 Þ ð6Þ The value of "(p0) can be approximated by the formulas reported in the appendix of the paper of Reynolds and Stoumbos.7 Given p0, p1 and the desired value of ANOS0, one can use equations (5) and (6) to ﬁnd the control limit h that ensures a calculated ANOS0 as close as possible to the desired value.

Downloaded from smm.sagepub.com at UNIVERSITY OF WINDSOR on October 25, 2014

XML Template (2014) [15.4.2014–1:03pm] [1–10] //blrnas3/cenpro/ApplicationFiles/Journals/SAGE/3B2/SMMJ/Vol00000/140041/APPFile/SG-SMMJ140041.3d (SMM) [PREPRINTER stage]

4

Statistical Methods in Medical Research 0(0)

Once k and h are obtained it can be desirable to determine the ANOS1, which measures how fast a shift from p0 to p1 will be detected by the chart. The CD approximation of the out-of-control ANOS1 is given by ANOS1

expðh r2 Þ þ h r2 1 r2 p1 r1

ð7Þ

The above is valid for an upward CUSUM chart (1). If we are interested in using a downward chart (2) the CD approximation of ANOS0 and ANOS1 are the same as earlier, but now the control limit h assumes negative values and the corresponding adjusted value becomes pﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ h ¼ h "ðp0 Þ p0 ð1 p0 Þ ð8Þ

2.2

RA-CUSUM

The RA-CUSUM chart was developed by Steiner et al.8 Because each patient can present diﬀerent risk factors and then diﬀerent baseline risk of event, this chart is designed to detect a shift in the OR of the event adjusting for the baseline risk. When the system is in control the OR0 is equal to 1 and the chart is set to identify a shift to OR1, where OR1 is the OR when the system is considered out of control. For each patient t, having a baseline risk of event p0,t and a Bernoulli outcome yt, the loglikelihood ratio score Wt is determined as 8 > 1 < log 1p0,tOR if yt ¼ 1 þOR1 p0,t ð9Þ Wt ¼ > 1 : log 1p þOR ¼ 0 if y t 0,t 1 p0,t These scores are then used in the following CUSUM scheme St ¼ maxð0, St1 þ Wt Þ when OR1 4 1

ð10Þ

St ¼ minð0, St1 Wt Þ

ð11Þ

when OR1 5 1

where S0 ¼ 0. When OR1 > 1 the system is set to detect an upward shift, when OR1 < 1 the system is set to detect a downward shift. The system is shifted in the out-of-control state when St hu in an upward chart (10) and if St hd in a downward chart (11), where h is the control limit. In order to select a suitable h value, the method requires a training dataset, generally a historical data set with a predisposed risk distribution. Steiner et al.8 suggested using an iterative procedure based on the MC approach to ﬁnd the in-control ANOS0 for any control limit, baseline reference level, and patient mix. In this way one can choose the control limit h which allows obtaining an ANOS0 value as close as possible to the desired one.

2.3

New RA-B-CUSUM

As outlined earlier, the expected in-control value associated with a Bernoulli variable might vary in relation to the t patient characteristics (p0,t; t ¼ 1, 2, . . .). Simply implementing a CUSUM scheme

Downloaded from smm.sagepub.com at UNIVERSITY OF WINDSOR on October 25, 2014

XML Template (2014) [15.4.2014–1:03pm] [1–10] //blrnas3/cenpro/ApplicationFiles/Journals/SAGE/3B2/SMMJ/Vol00000/140041/APPFile/SG-SMMJ140041.3d (SMM) [PREPRINTER stage]

Rossi et al.

5

with constant parameters would have misleading results if the actual values of p0 ﬂuctuate from patient to patient about the constant assumed parameter. As in the approach of Rogerson and Yamada for a Poisson CUSUM,5 we propose a modiﬁed Bernoulli CUSUM chart to take into account varying in-control risks p0,t. Patient-speciﬁc values of the parameters k and h are used. The observed values, yt, are then used in the CUSUM using the following correction to equations (1) and (2) ð12Þ St ¼ max 0, St1 þ ct ðyt kt Þ for OR1 4 1 ð13Þ St ¼ min 0, St1 þ ct ðyt kt Þ for OR1 5 1 where ct is the correction factor; kt and ht are, respectively, the speciﬁc reference value and the speciﬁc control limit for patient t; and h is the global control limit. First, h is obtained on the basis of an overall desired baseline performance level p0, an associated value of k, as deﬁned in the equation (3), and the desired ANOS0. Once h is chosen, next choose kt on the basis of p0,t and p1,t, where 1p1,t log 1p0,t r1,t kt ¼ ¼ r2,t p1,t ð1p0,t Þ log p 1p 0,t ð 1,t Þ and p1,t ¼ OR1p0,t/(1 þ OR1p0,t p0,t). Then, ct is chosen as the ratio h to ht, where the latter is the value of the control limit associated with the desired ANOS0, kt, and values of p0,t and p1,t. Thus, ct ¼ h/ht. The quantity ct is chosen so that the observed value yt will make the proper relative contribution toward the signaling parameter h that is used in the actual CUSUM. If h > ht, then the contribution yt – kt is scaled up by the factor h/ht, while if h < ht, then the contribution yt – kt is scaled down by the factor h/ht. The parameters h and ht can be obtained either by the MC procedure or by the formulas based on the CD approximation,7 using the speciﬁc p0,t, the desired OR or kt, and the desired ANOS0. The chart will issue an alarm if St hu in an upward chart (12) and if St hd in a downward chart (13), where hu and hd are the global control limit h in an upward chart and in a downward chart, respectively.

3 Performance of the RA-CUSUM and RA-B-CUSUM charts To illustrate the performance of the Steiner’s risk adjusted CUSUM (RA-CUSUM) and the proposed RA-B-CUSUM chart, a simulation study was performed using data described in Steiner et al.,8 concerning 30-day mortality after cardiac surgery in patients of a UK center for cardiac surgery over the period 1992–1998. The ﬁrst 2 years’ data (training data set with an overall mortality rate equal to 0.065) were used to determine the control limit h of the Steiner’s RACUSUM and to simulate four diﬀerent scenarios by Monte Carlo sampling. For risk prediction we used the predictive logistic model estimated on the training data set by Steiner et al8

logit p0,t ¼ 3:68 þ 0:077Xt ð14Þ where Xt denotes the Parsonnet score for patient t. The distribution of the Parsonnet score in the training data set is presented in Table 1. With regard to the simulated scenarios, in the ﬁrst (scenario A) the predisposed risk distribution and the baseline reference level were the same as in the training data set. In scenario B the

Downloaded from smm.sagepub.com at UNIVERSITY OF WINDSOR on October 25, 2014

XML Template (2014) [15.4.2014–1:03pm] [1–10] //blrnas3/cenpro/ApplicationFiles/Journals/SAGE/3B2/SMMJ/Vol00000/140041/APPFile/SG-SMMJ140041.3d (SMM) [PREPRINTER stage]

6

Statistical Methods in Medical Research 0(0) Table 1. Parsonnet score distribution in patients from a UK center for cardiac surgery (1992– 1993) with the predicted risk of death (p0,t): logit(p0,t) ¼ –3.68 þ 0.077Parsonnet_scoret. Patients Parsonnet score

Median of class

N

%

p0,t

0 1–5 6–10 11–20 21–30 31–40 41–50 51 Total

0 3 8 16 26 36 46 56

396 710 440 440 140 47 22 23 2218

17.9 32.0 19.8 19.8 6.3 2.1 1.0 1.0 100

0.0246 0.0308 0.0446 0.0796 0.1574 0.2874 0.4656 0.6529

distribution of the predisposed risk was reversed in order to obtain more patients at high risk and less patients at low risk: for example, patients with Parsonnet score equal to 0 became 23 and those with Parsonnet score 51 became 396. Finally, in the last two scenarios (C and D) the predisposed risk distribution was the same as in the training data set (scenario A), but the baseline reference level was decreased by changing the intercept value of the predictive equation (14). The baseline reference level was 0.02 (intercept ¼ 5.05) in scenario C and 0.005 (intercept ¼ 6.54) in the scenario D. For each of the four scenarios, ﬁve sequences of ﬁve million of Bernoulli random data were obtained using the following approach: (1) a speciﬁc distribution of the risks p0,t was deﬁned for each scenario, (2) from each distribution a random sample of p0,t was extracted, and (3) for each sampled p0,t a Bernoulli variable yt was generated. Each scenario was then analyzed using the RA-CUSUM chart and the RA-B-CUSUM chart designed to detect a doubling in the odds of death (OR1 ¼ 2) with a desired ANOS0 ¼ 9600. For the RA-CUSUM chart, two diﬀerent control limit h values were used: 4.50 and 4.76. The ﬁrst value was proposed by Steiner et al.,8 while the second was the h value we obtained following the Steiner’s approach. Results are reported in Table 2. In scenario A, where the predisposed risk distribution and the baseline reference level are the same as in the training data set, RA-CUSUM using the h value of 4.50 shows a low actual ANOS0, since it signals more false alarms (1 every 7433 observations) than desired ones (1 every 9600 observations). Furthermore, RA-CUSUM appears to be very sensitive to changes occurring in the risk distribution or in the baseline reference level. When many patients are high-risk patients (scenario B) RA-CUSUM signals on average one false alarm every 2466 observations, so it issues more false alarms than desired. If the case mix remains unchanged but the baseline reference level decreases (scenarios C and D) RA-CUSUM becomes very conservative, with an actual ANOS0 close to 20,000 and 72,000, respectively. When the h value of 4.76 was used, as expected the RA-CUSUM obtained an actual ANOS0 value very close to the desired one for scenario A and failed for the scenarios B, C, and D. ANOS1 values were consistent with ANOS0 values, with a longer ANOS1 in correspondence of a higher ANOS0. Also the results of the proposed RA-B-CUSUM chart applied to the four scenarios are reported in Table 2. As previously seen, in scenario A, where the predisposed risk distribution and the baseline reference level remain unchanged with respect to the training data set, the RA-CUSUM signals

Downloaded from smm.sagepub.com at UNIVERSITY OF WINDSOR on October 25, 2014

XML Template (2014) [15.4.2014–1:03pm] [1–10] //blrnas3/cenpro/ApplicationFiles/Journals/SAGE/3B2/SMMJ/Vol00000/140041/APPFile/SG-SMMJ140041.3d (SMM) [PREPRINTER stage]

Rossi et al.

7

Table 2. Performance of the risk-adjusted CUSUM (RA-CUSUM) using two different control limit (h) values and of the risk-adjusted Bernoulli CUSUM (RA-B-CUSUM) designed with the corrected diffusion (CD) approximation. Both charts are designed to detect an odds ratio (OR) of 2 with a desired ANOS0 ¼ 9600. The actual in-control ANOS0 and the out-of-control ANOS1 are obtained for four simulated scenarios created using real data from a UK center for cardiac surgery (1992–1993).

Baseline Scenario p0 A

0.065

B

0.36

C

0.02

D

0.005

RA-CUSUM

RA-B-CUSUM

h

Method ANOS0 ANOS1

ANOS0 ANOS1 OR ¼ 1 OR ¼ 1.5 OR ¼ 2 OR ¼ 2.5

4.50 4.76 4.50 4.76 4.50 4.76 4.50 4.76

7433 9612 2466 3204 20,188 26,931 72,215 96,772

549.9 604.8 186.3 204.4 1560.5 1721.3 5816.3 6423.8

207.9 222.2 78.7 84.1 580.7 621.7 2088.0 2232.8

127.8 135.7 52.7 56.0 350.3 372.2 1243.0 1320.2

OR ¼ 1 OR ¼ 1.5 OR ¼ 2 OR ¼ 2.5 CD

10,878

640.8

230.7

140.1

CD

11,465

298.5

109.5

71.4

CD

10,419

1256.2

506.2

308.7

CD

9780

2520.4

1205.9

770.7

more alarms than desired when the h value is 4.50 and a number of alarms equal to the desired when the h value is 4.76. Conversely, the RA-B-CUSUM based on the CD approximation approach works quite well, with an actual ANOS0 suﬃciently close to the desired one of 9600, although it results slightly more conservative. In the other three scenarios (B, C, and D), the RA-B-CUSUM performs in the usual manner in terms of false alarm rate, with a slight overestimation of the ANOS0, while the RA-CUSUM fails substantially in terms of false alarms, irrespective of the h value. When comparing diﬀerent surveillance methods, it is common to Erst maintain the same level of false alarm probability and then compare the detection delay, which is analogous to controlling the type I error in hypothesis testing. Since the two methods diﬀer substantially in ANOS0, the comparison between the out-ofcontrol ANOS1 of the RA-CUSUM and the out-of-control ANOS1 of the RA-B-CUSUM is not warranted. However, when the ANOS0 values of both methods were very similar, as in scenario A, also the ANOS1 values were very close.

4 Real example To better illustrate the characteristics of the proposed RA-B-CUSUM, we applied the riskunadjusted Bernoulli CUSUM and the RA-B-CUSUM chart to real data from a UK center for cardiac surgery over the period 1994–19988 considering the trainee surgeons and an acceptable performance level of 0.065. As acceptable performance level it was used the global 30-day mortality rate of all surgeons, both trainee and experienced surgeons, over the period 1992–1993. Both procedures were designed to detect a halving (OR ¼ 0.5) in the odds of mortality with a desired in-control ANOS0 equal to 9600, corresponding to a false alarm every 9 years, given the frequency of surgeries of this center. Over the period 1994–1998 the trainee surgeons deal with relatively low-risk patients (mean Parsonnet score ¼ 5.1) while all surgeons (experienced ones plus trainee surgeons) receive more complicated surgeries over the reference period 1992–1993 (mean Parsonnet score ¼ 9.6); thus the former will have a lower mortality than the latter.

Downloaded from smm.sagepub.com at UNIVERSITY OF WINDSOR on October 25, 2014

XML Template (2014) [15.4.2014–1:03pm] [1–10] //blrnas3/cenpro/ApplicationFiles/Journals/SAGE/3B2/SMMJ/Vol00000/140041/APPFile/SG-SMMJ140041.3d (SMM) [PREPRINTER stage]

8

Statistical Methods in Medical Research 0(0)

The application of the two CUSUM procedures to the trainee surgeons is plotted in Figure 1. The Bernoulli CUSUM chart (at the top of Figure 1), which makes no adjustment for patient risk, signals a decrease in mortality. Thus, we might conclude that the mortality associated with these surgeons signiﬁcantly decreases below the overall mortality rate of 0.065. The critical point is that the trainee surgeons mainly deal with low-risk patients and so the associated mortality will result lower than the acceptable performance level. The RA-B-CUSUM chart (at the bottom of Figure 1), which takes account of the speciﬁc patient risk, does not signal any decrease in mortality. Hence, we can conclude that the surgical performance of this particular group of surgeons remains constant during the monitored period.

Figure 1. Performance of the Bernoulli CUSUM and the risk-adjusted Bernoulli CUSUM (RA-B-CUSUM), using real data from a UK center for cardiac surgery over the period 1994–1998 and considering an acceptable performance level of 0.065 (global 30-day mortality rate during the period 1992–1993). Both procedures are designed to detect a halving (OR ¼ 0.5) in the odds of mortality with a desired in-control ANOS0 equal to 9600.

Downloaded from smm.sagepub.com at UNIVERSITY OF WINDSOR on October 25, 2014

XML Template (2014) [15.4.2014–1:03pm] [1–10] //blrnas3/cenpro/ApplicationFiles/Journals/SAGE/3B2/SMMJ/Vol00000/140041/APPFile/SG-SMMJ140041.3d (SMM) [PREPRINTER stage]

Rossi et al.

9

5 Discussion This paper has presented a new CUSUM procedure for monitoring binary health data for dealing with the problem of varying in-control risks, has provided a thorough comparison of the two CUSUM charts recently developed for health surveillance, and has discussed design issues of these control charts in practice. Simulation studies have been used to show how the proposed RA-B-CUSUM works in practice, by using simulated data sets over a reasonably broad range of combinations of the baseline incidence rate (p0), the target step shift (OR), and the target in-control average number of events until signal (ANOS0; please refer to the Supplementary Materials at http://smm.sagepub.com/). The procedure has also been applied to a classic data set on mortality in a UK center for cardiac surgery. We found that there was clear evidence that the proposed RA-B-CUSUM works better than the RA-CUSUM previously reported in the literature. The proposed RA-B-CUSUM chart is insensitive to changes occurring in the underlying distribution of case mix and/or in the baseline reference level. The comparison with the alternative RA-CUSUM approach of Steiner et al.8 has shown that the proposed CUSUM approach addresses some of the issues that arise from such existing methodology. Our approach, in particular that based on the CD approximation, is easier to set up, reduces the sensitivity to changes in the underlying distribution of the case mix occurring during the monitored period, and its performance in terms of false alarm rate is close enough to the desired value. Furthermore, the proposed approach may allow you to update the predictive logistic regression model at regular intervals without updating the control chart. We hope that this paper will shed light on practical implications for designing and evaluation of control chart methods in a nonstationary setting, because nonstationarity is a common feature of many data sets in the health care domain. From the practical point of view, it is important to develop robust surveillance methods that are not sensitive to varying in-control risks but still sensitive in detecting changes in incidence rate. This paper has considered only CUSUM-type control chart methods for health care surveillance and it would be interesting to investigate the performance of other risk-adjusted methods and make a comparison between the CUSUM approach and other methods. In our opinion, the RA-B-CUSUM chart proposed in this paper could be a simple and eﬃcient solution, and we hope it will prove useful for a more powerful examination of data sets in the ﬁeld of public health surveillance. The R-Program RA-B-CUSUM and the software manual are available on request. Acknowledgments The authors thank Professor Luigi Donato for advice and encouragement, the Regional Health Agency of Tuscany Region, the Institute of Clinical Physiology of the Italian National Research Council, and the ‘‘G. Monasterio’’ Foundation CNR-Tuscany Region for their support to this work. We are very grateful to the editor and the referees for comments and suggestions which have improved the presentation and content of the paper. A preliminary version was presented at the 46th Scientiﬁc Meeting of the Italian Statics Society, June 2012, Rome.

Funding This research received no speciﬁc grant from any funding agency in the public, commercial, or not-for-proﬁt sectors.

Downloaded from smm.sagepub.com at UNIVERSITY OF WINDSOR on October 25, 2014

XML Template (2014) [15.4.2014–1:03pm] [1–10] //blrnas3/cenpro/ApplicationFiles/Journals/SAGE/3B2/SMMJ/Vol00000/140041/APPFile/SG-SMMJ140041.3d (SMM) [PREPRINTER stage]

10

Statistical Methods in Medical Research 0(0)

Conflict of interest None declared.

References 1. Page ES. Continuous inspection schemes. Biometrika 1954; 41: 100–115. 2. Woodall WH. Use of control charts in health-care and public-health surveillance (with Discussion). J Qual Technol 2006; 38: 89–104. 3. Lucas JM. Counted data CUSUM’s. Technometrics 1985; 27: 129–144. 4. Rossi G, Lampugnani L and Marchi M. An approximate cusum procedure for surveillance of health events. Stat Med 1999; 18: 2111–2122. 5. Rogerson PA and Yamada I. Approaches to syndromic surveillance when data consist of small regional counts. Morb Mortal Wkly Rep 2004; 53: 79–85. 6. Ho¨hle M and Paul M. Count data regression chart for the monitoring of surveillance time series. Comput Stat Data Anal 2008; 52: 4357–4368. 7. Reynolds MR and Stoumbos ZG. A CUSUM chart for monitoring a proportion when inspecting continuously. J Qual Technol 1999; 31: 87–108. 8. Steiner SH, Cook RJ, Farewell VT, et al. Monitoring surgical performance using risk-adjusted cumulative sum charts. Biostatistics 2000; 1: 441–452. 9. Parsonnet V, Dean D and Bernstein AD. A method of uniform stratification of risk for evaluating the results of surgery in acquired adult heart disease. Circulation 1989; 79: 12–13. 10. Forbes TL, Steiner SH, Lawlor DK, et al. Risk-adjusted analysis of outcomes following elective open abdominal aortic aneurysm repair. Ann Vasc Surg 2005; 19: 142–148. 11. Harris JR, Forbes TL, Steiner SH, et al. Risk-adjusted analysis of early mortality after ruptured abdominal aortic aneurysm repair. J Vasc Surg 2005; 42: 387–391. 12. Axelroda DA, Guidingerb MK, Metzgerc RA, et al. Transplant center quality assessment using a continuously

13.

14.

15.

16.

17.

18.

19.

20.

21.

updatable, risk-adjusted technique (CUSUM). Am J Transpl 2006; 6: 313–323. Novick RJ, Fox SA, Stitt LW, et al. Direct comparison of risk-adjusted and non–risk-adjusted CUSUM analyses of coronary artery bypass surgery outcomes. J Thorac Cardiovasc Surg 2006; 132: 386–391. Sibanda T and Sibanda N. The CUSUM chart method as a tool for continuous monitoring of clinical outcomes using routinely collected data. BMC Med Res Methodol 2007; 7: 46. Spiegelhalter DJ, Grigg OA, Kinsman R, et al. Sequential probability ratio tests (SPRTs) for monitoring riskadjusted outcomes. Int J Qual Health Care 2003; 15: 1–7. Chen TT, Chung KP, Hu FC, et al. The use of statistical process control (risk-adjusted CUSUM, risk-adjusted RSPRT and CRAM with prediction limits) for monitoring the outcomes of out-of-hospital cardiac arrest patients rescued by the EMS system. J Eval Clin Pract 2011; 17: 71–77. Loke CK and Gan FF. Joint monitoring scheme for clinical failures and predisposed risks. Qual Technol Quant Manag 2012; 9: 3–21. Jones MA and Steiner SH. Assessing the effect of estimation error on risk-adjusted CUSUM chart performance. Int J Qual Health Care 2012; 24: 176–181. Brook D and Evans DA. An approach to the probability distribution of cusum run length. Biometrika 1972; 59: 539–548. Siegmund D. Corrected diffusion approximations in certain random walk problems. Adv Appl Prob 1979; 11: 701–719. Siegmund D. Sequential analysis. New York: SpringerVerlag, 1985.

Downloaded from smm.sagepub.com at UNIVERSITY OF WINDSOR on October 25, 2014