Sleep Breath DOI 10.1007/s11325-014-0967-z
ORIGINAL ARTICLE
A comparison of radio-frequency biomotion sensors and actigraphy versus polysomnography for the assessment of sleep in normal subjects Emer O’Hare & David Flanagan & Thomas Penzel & Carmen Garcia & Daniela Frohberg & Conor Heneghan
Received: 19 November 2013 / Revised: 6 February 2014 / Accepted: 24 February 2014 # Springer-Verlag Berlin Heidelberg 2014
Abstract Purpose This paper aims to compare the absolute performance of three noncontact sleep measurement devices for measuring sleep parameters in normal subjects against polysomnography and to assess their relative performance. Methods The devices investigated were two noncontact radiofrequency biomotion sensors (SleepMinder (SM) and SleepDesign (HSL-101)) and an actigraphy-based system (Actiwatch). Overnight polysomnography measurements were carried out in 20 normal subjects, with simultaneous assessment of sleep parameters using the three devices. The parameters measured included total sleep time (TST), sleep efficiency (SE), sleep-onset latency (SOL), and wake-aftersleep onset (WASO). The per-epoch agreement level for sleep/ wake distinction was evaluated. Results The TSTs reported by the three devices were 426±34, 434 ± 22, and 441 ± 16 min, for the SM, HSL-101, and Actiwatch, respectively, against polysomnogram (PSG)-reported TST of 391±49 min. The SOLs were 10±10, 5±6, and 3±2 min for the SM, HSL-101 and Actiwatch, respectively against PSG SOL of 19±13 min. The WASO times were 46 ± 33, 43 ± 22, and 38 ± 17 min, as against PSGreported 69±46 min. All three devices had a statistically E. O’Hare : D. Flanagan : C. Heneghan (*) ResMed Sensor Technologies, Belfield Office Park, Blocks 9&10, Clonskeagh, Dublin 4, Ireland e-mail: [email protected] T. Penzel : D. Frohberg Advanced Sleep Research, Luisenstraße 55, 10117 Berlin, Germany D. Flanagan : C. Heneghan School of Electrical, Electronic and Communication Engineering, University College Dublin, Belfield, Dublin 4, Ireland T. Penzel : C. Garcia Sleep Medicine Center, Charité Universitaetsmedizin Berlin, 10117 Berlin, Germany
significant bias to overestimate sleep time and underestimate WASO and SOL compared with PSG. The performance of the three devices was basically equivalent, with only minor interdevice differences. The overall per-epoch agreement levels were 86 % for the SM, 86 % for the HSL-101, and 85 % for the Actiwatch. Conclusions Noncontact biomotion approaches to sleep measurement provided reasonable estimates of TST, but with a bias to over-estimation of sleep. The radio-frequency biomotion sensors provided similar accuracies for sleep/ wake determination in normal subjects as the actigraph used in this study and slightly improved estimates of TST, SOL, and WASO. Keywords Biomotion sensor . Actigraphy . Total sleep time . Wake-after-sleep onset
Introduction Polysomnography is a well-established and validated standard for the measurement of sleep parameters in a laboratory setting. However, its use in observing “normal” sleep patterns in the home environment is limited, and as a result, it is still technically challenging to obtain reliable objective measurements of sleep in the home setting. To overcome this, one commonly used technique for objective home sleep measurement is actigraphy, in which a sensitive accelerometer is used to determine wrist movement [1–4]. Algorithms have then been developed to infer sleep patterns from the movement signals thus measured. Reported accuracies of actigraphy systems vary widely and are typically dependent on the observed population and the presence of sleep disorders such as sleep-disordered breathing, insomnia, and periodic limb movements. For example, in sleep-disordered breathing patients, per-epoch accuracies of 80–86 % have been reported
Sleep Breath
[5]; in insomnia patients, accuracies of 83 % have been achieved [6]. In “normal” subjects, assumed to be free of sleep disorders, accuracies up to 90 % have been recorded [7]. However, the performance of actigraphy is quite dependent on the chosen population and their sleep characteristics [8]. In particular, actigraphy has been shown to have relatively limited wake specificity [7, 9]. Nevertheless, actigraphy has been used with reasonable success in a number of different populations and studies [10, 11]. In the latest American Academy of Sleep Medicine (AASM) practice guidelines on use of actigraphy, its use has now been indicated for a range of clinical uses such as evaluating circadian rhythm disorders and as an estimate of sleep time in respiratory polygraphy when polysomnography is not available [3]. In particular, the ability of actigraphy-based systems to monitor sleep in a person’s normal environment over long periods of time provides a new capability to sleep medicine research and practice. We have been involved in the development and assessment of two products for the measurement of sleep in the home environment—the SM product and the Omron SleepDesign (HSL-101). Both of these devices use short-range radiofrequency sensing to determine a person’s movement and respiration without any direct contact with the subject, and based on this, an algorithm to determine sleep/wake state has been developed. The SM’s ability to measure respiratory effort can provide additional information such as respiration rate [12] and the presence of sleep-disordered breathing [13], and these have been discussed in related publications. The focus in this paper is on sleep/wake estimation. Previously, the absolute performance of the SM system in sleep/wake estimation has been tested against polysomnography in subjects under evaluation for sleep apnea, with a reported per-epoch accuracy of approximately 80 % across the population [13]. However, its performance on “normal” subjects, free of sleep-disordered breathing has not been reported. The performance of the HSL101 device has not been previously published. In this paper, we present the performance of the two radiofrequency-based systems and an actigraphy-based system for the measurement of sleep parameters in a normal adult population, against gold-standard laboratory-based polysomnogram (PSG). (Note—we refer to all three systems as “noncontact” as they do not require adhesive electrodes or tethered sensors such as inductance plethysmography). We also investigate the relative performance of the three noncontact approaches, using a null hypothesis that their performance is statistically equivalent.
Methods Trial protocol and demographics The trial was conducted at the polysomnography laboratory of advanced sleep research in Berlin. Ethics approval for the trial
was granted by the local Ethics Committee, and informed consent was obtained from trial participants. Twenty subjects were recruited from a list of enrolled volunteers who were believed to be free of sleep-disordered breathing and primary insomnia, due to previous evaluation. Polysomnography was carried out using the Embla N7000 system running the Somnologica software. The subject’s sleep parameters were scored by a single scorer using AASM scoring rules [14]. The subject was simultaneously monitored using three different sensor technologies which primarily use biomotion to determine sleep/wake. The three technologies evaluated were (a) a radio-frequency based biomotion sensor operating at 5.8 GHz (SM, ResMed Sensor Technologies, Dublin, Ireland), (b) a radio-frequency-based biomotion sensor operating at 10.525 GHz (HSL-101, Omron, Kyoto, Japan), and (c) a wrist-based actigraphy sensor (Actiwatch (AW)-2, Philips Respironics, USA). The principle of operation of the two radio-frequency-based biomotion sensors is that they detect bodily movement and respiration movements by transmitting a low-power pulse of radio-frequency energy. By monitoring the echo received from the subject, the movement of the subject can be determined. An algorithm has then been developed to classify 30-sec epochs of the received signal as either being wake, sleep, or absent (the subject is deemed absent if they are out of the sensor range which is approximately 1.5 m). This algorithm has been previously described and tested for the SM system [13] on a mixed population of normal and sleep-disordered breathing subjects. In internal testing, we have verified that clothing and blankets have minimal impact on the detected signal (as these materials are relatively transparent to radio waves at these frequencies). There is an effect of body position on the signal amplitude recorded (with the weakest signal generally found when the person’s back is to the sensor), but the algorithm is designed to “auto-calibrate” to the overall signal level. The principle of operation of the AW is that it records wrist movement using a sensitive 3D accelerometer and then infers sleep state by analysis of the recorded movement signal. The AW has been previously validated as a sleep measurement device across a wide variety of populations [1–8]. The subjects were instructed to follow their normal daily routine and to come to the sleep laboratory at approximately 21:00. A technician setup the PSG, AW, and HSL-101 devices, and lights out was at approximately 22:30. The AW was worn on the subject’s nondominant wrist. The SM and HSL101 were placed side-by-side approximately 1 m from the subject, and slightly elevated above the torso (see Fig. 1). Subjects were awoken at approximately 06:30 the following morning. Each of the recording devices has an internal clock with an accuracy of 5 s or less over the 10-h recording period. The synchronization of the devices was ensured by following a protocol to set the time of each device with respect to a
Sleep Breath Table 2 Recorded sleep parameters for the 20 subjects tested in this study Measured sleep parameters Total sleep time (min) Sleep-onset latency (min) Wake-after-sleep onset (min) Sleep efficiency (%) AHI (events/h)
391±49 20±13 69±46 82±11 % 0.9±1
The performance metrics of interest are
Fig. 1 Illustration of the setup at the sleep laboratory
controlled clock prior to each sleep study. Table 1 summarizes the demographic characteristics of the study participants. Analysis The four recording systems (PSG, SM, HSL-101, and AW) were all switched on and off at slightly different times due to practical constraints of equipment setup. Therefore, for the analysis, we only include the period from time in bed to time out of bed. Periods during the night when the person exited the bed (e.g., for bathroom breaks) were omitted from any comparative analysis, as they represented approximately 0.4 % of all epochs in the data set. Each system reports a label for each 30-s epoch. For the PSG system, the valid labels are Wake, Stage N1 sleep, Stage N2 sleep, Stage N3 sleep, REM, and artifact. For analysis purposes, Stages N1–N3 and REM are collectively labeled as sleep. For the HSL-101, the epochs are labeled as sleep, wake, or absent. For the AW, the labels are sleep or wake. The overall goal of the analysis was (a) to determine the absolute performance of the three noncontact system in assessment of sleep metrics, using the PSG results as a gold standard, and (b) to investigate potential differences in the relative performance of the three noncontact devices, under a null hypothesis that their performance is equivalent.
Table 1 Demographics of the study participants Subject Demographics Subjects (M/F) Age BMI AHI
20 (11/9) 30±6 years 23.2±3.6 kg/m2 0.9±1 events/h
(a) Per-epoch accuracy. We report agreement levels between 30-s epoch labels produced by the PSG as sleep or wake (omitting any absence periods) and the corresponding epoch labels produced by the three noncontact methods. (b) Sensitivity to sleep. This is the percentage of actual PSG sleep epochs which are labeled sleep by the noncontact device. (c) Sensitivity to wake. This is the percentage of actual PSG wake epochs which are labeled wake by the noncontact device. (d) Cohen’s kappa. Given the high imbalance of the sleep/ wake classes, per-epoch accuracy figures can give an overly optimistic impression of performance; Cohen’s kappa is a useful tool for showing the level of agreement, which has been achieved above chance alone and provides an alternative and more meaningful metric in some cases. (e) Total sleep time (TST). This is the duration (in minutes) of all epochs labeled as sleep. (f) TST error. This is the difference between the TST reported by the noncontact device and the PSG TST. (g) Sleep efficiency (SE). This is the TST divided by the time in bed expressed as a percentage. (h) Sleep-onset latency (SOL). This is defined as the time from lights out to the first continuous 2-min period of sleep. Table 3 Comparative performance on a pooled-epoch basis
Per-epoch accuracy Sensitivity to sleep Sensitivity to wake Cohen’s kappa
SleepMinder
HSL-101
Actiwatch
0.856 0.954 0.424 0.52
0.855 0.963 0.376 0.52
0.854 0.972 0.332 0.51
The per-epoch accuracy is the number of epochs correctly labeled relative to the gold-standard polysomnogram labels. The sensitivity to sleep is the percentage of true sleep epochs correctly labeled by the system. The sensitivity to wake is the percentage of true wake epochs correctly labeled by the system. Cohen’s Kappa reflects the agreement level observed which is greater than chance alone (Cohen’s Kappa can assume values between −1 and 1, with values of >0.6 typically considered as high)
Sleep Breath Table 4 Comparative performance of the three systems on a per-subject basis
Per-epoch accuracy Sensitivity to sleep Sensitivity to wake Cohen’s kappa Estimated total sleep time (min) Total sleep time error versus PSG (min) Estimated sleep efficiency Estimated SOL (min) Estimated WASO (min)
SleepMinder
HSL-101
Actiwatch
0.856±0.062 0.953±0.038 0.389±0.190 0.533±0.201 426±34b 33±38 0.884±0.075a, b 10±10a, b 46±33b
0.856±0.064 0.964±0.024 0.358±0.1420 0.532±0.207 434±22b 41±37 0.901±0.048b 5±5b 43±22b
0.855±0.063 0.973±0.020 0.339±0.127 0.528±0.201 441±16b 48±41 0.916±0.035a, b 3±2a, b 38±17b
Statistically significant inter-instrument results are marked in the table Significant difference between Actiwatch and SleepMinder estimates
b
Significant difference from PSG-reported value
(i) Wake-after-sleep onset (WASO). This is defined as the total wake time observed between sleep onset and time out of bed To visualize data, we use scatter plots and (modified) Bland-and-Altman analysis and also report Pearson product– moment correlation coefficients against the PSG goldstandard reference values. One-way ANOVA tests were used to establish if the estimates by the three noncontact methods varied from the PSG measurement. Pairwise t tests with adjusted p values (using Holm–Bonferroni correction) were used to see if there were statistically significant variations between the three noncontact methods. Statistical analysis was carried out using R v3.0.2. All statistical significant conclusions are made at an α=0.05 level. When product moment correlation coefficients are quoted, we also include the 95 % confidence interval, using a Fisher transformation, under an assumption that the variables have a bivariate normal distribution.
Results Sleep parameters reported by the PSG analysis are shown in Table 2. The TST and sleep efficiencies reported are consistent with previous lab-based studies of normal volunteers in a sleep lab. The pooled per-epoch, sensitivity to sleep, sensitivity to wake, and Cohen’s kappa values are shown in Table 3. This was obtained by combining all scored epochs from all subjects. There was no statistically significant difference between the three noncontact systems on any of these metrics. The per-subject performance of the three biomotion systems is detailed in Table 4 (in a per-subject analysis, we calculated the per-epoch accuracy, sensitivity to sleep, etc.
individually for each subject, and then averaged results across subjects). Paired t tests with Holm–Bonferroni correction were used to determine the significance of any interinstrument difference between the systems. The null hypothesis (that all three noncontact methods provide equivalence performance) was satisfied for most of these performance metrics, with the exception of SOL and SE estimates, where there was a statistically significant difference between SM and AW performance. To assess the absolute performance of the three noncontact methods, we show the scatter plot of TSTs reported by the SM, HSL-101 and the AW versus the PSG reported TST (Fig. 2). The Pearson’s product moment correlation 500
450
TST SM/HSL/AW (mins)
a
400
350
SM HSL AW 1:1 SM
300
fit
HSL
fit
250
200 200
AWfit
250
300
350
400
450
500
TST PSG (mins)
Fig. 2 A comparison of the TST for the twenty subjects as measured by the PSG versus TST measured by the SM, HSL-101, and AW. The lines show a linear best fit for the three devices and the line of identity
Sleep Breath 150
Fig. 3 A modified Bland-andAltman plot of the TST error for the 20 subjects as measured by the SM, HSL-101, and AW TST SM/HSL/AW −TST PSG(mins)
SM HSL AW SMfit HSL
fit
100
AW
fit
50
0
−50 200
250
300
350
400
450
500
TST PSG (mins)
coefficients between PSG and SM, HSL-101, and AW were 0.646 (p
ORIGINAL ARTICLE
A comparison of radio-frequency biomotion sensors and actigraphy versus polysomnography for the assessment of sleep in normal subjects Emer O’Hare & David Flanagan & Thomas Penzel & Carmen Garcia & Daniela Frohberg & Conor Heneghan
Received: 19 November 2013 / Revised: 6 February 2014 / Accepted: 24 February 2014 # Springer-Verlag Berlin Heidelberg 2014
Abstract Purpose This paper aims to compare the absolute performance of three noncontact sleep measurement devices for measuring sleep parameters in normal subjects against polysomnography and to assess their relative performance. Methods The devices investigated were two noncontact radiofrequency biomotion sensors (SleepMinder (SM) and SleepDesign (HSL-101)) and an actigraphy-based system (Actiwatch). Overnight polysomnography measurements were carried out in 20 normal subjects, with simultaneous assessment of sleep parameters using the three devices. The parameters measured included total sleep time (TST), sleep efficiency (SE), sleep-onset latency (SOL), and wake-aftersleep onset (WASO). The per-epoch agreement level for sleep/ wake distinction was evaluated. Results The TSTs reported by the three devices were 426±34, 434 ± 22, and 441 ± 16 min, for the SM, HSL-101, and Actiwatch, respectively, against polysomnogram (PSG)-reported TST of 391±49 min. The SOLs were 10±10, 5±6, and 3±2 min for the SM, HSL-101 and Actiwatch, respectively against PSG SOL of 19±13 min. The WASO times were 46 ± 33, 43 ± 22, and 38 ± 17 min, as against PSGreported 69±46 min. All three devices had a statistically E. O’Hare : D. Flanagan : C. Heneghan (*) ResMed Sensor Technologies, Belfield Office Park, Blocks 9&10, Clonskeagh, Dublin 4, Ireland e-mail: [email protected] T. Penzel : D. Frohberg Advanced Sleep Research, Luisenstraße 55, 10117 Berlin, Germany D. Flanagan : C. Heneghan School of Electrical, Electronic and Communication Engineering, University College Dublin, Belfield, Dublin 4, Ireland T. Penzel : C. Garcia Sleep Medicine Center, Charité Universitaetsmedizin Berlin, 10117 Berlin, Germany
significant bias to overestimate sleep time and underestimate WASO and SOL compared with PSG. The performance of the three devices was basically equivalent, with only minor interdevice differences. The overall per-epoch agreement levels were 86 % for the SM, 86 % for the HSL-101, and 85 % for the Actiwatch. Conclusions Noncontact biomotion approaches to sleep measurement provided reasonable estimates of TST, but with a bias to over-estimation of sleep. The radio-frequency biomotion sensors provided similar accuracies for sleep/ wake determination in normal subjects as the actigraph used in this study and slightly improved estimates of TST, SOL, and WASO. Keywords Biomotion sensor . Actigraphy . Total sleep time . Wake-after-sleep onset
Introduction Polysomnography is a well-established and validated standard for the measurement of sleep parameters in a laboratory setting. However, its use in observing “normal” sleep patterns in the home environment is limited, and as a result, it is still technically challenging to obtain reliable objective measurements of sleep in the home setting. To overcome this, one commonly used technique for objective home sleep measurement is actigraphy, in which a sensitive accelerometer is used to determine wrist movement [1–4]. Algorithms have then been developed to infer sleep patterns from the movement signals thus measured. Reported accuracies of actigraphy systems vary widely and are typically dependent on the observed population and the presence of sleep disorders such as sleep-disordered breathing, insomnia, and periodic limb movements. For example, in sleep-disordered breathing patients, per-epoch accuracies of 80–86 % have been reported
Sleep Breath
[5]; in insomnia patients, accuracies of 83 % have been achieved [6]. In “normal” subjects, assumed to be free of sleep disorders, accuracies up to 90 % have been recorded [7]. However, the performance of actigraphy is quite dependent on the chosen population and their sleep characteristics [8]. In particular, actigraphy has been shown to have relatively limited wake specificity [7, 9]. Nevertheless, actigraphy has been used with reasonable success in a number of different populations and studies [10, 11]. In the latest American Academy of Sleep Medicine (AASM) practice guidelines on use of actigraphy, its use has now been indicated for a range of clinical uses such as evaluating circadian rhythm disorders and as an estimate of sleep time in respiratory polygraphy when polysomnography is not available [3]. In particular, the ability of actigraphy-based systems to monitor sleep in a person’s normal environment over long periods of time provides a new capability to sleep medicine research and practice. We have been involved in the development and assessment of two products for the measurement of sleep in the home environment—the SM product and the Omron SleepDesign (HSL-101). Both of these devices use short-range radiofrequency sensing to determine a person’s movement and respiration without any direct contact with the subject, and based on this, an algorithm to determine sleep/wake state has been developed. The SM’s ability to measure respiratory effort can provide additional information such as respiration rate [12] and the presence of sleep-disordered breathing [13], and these have been discussed in related publications. The focus in this paper is on sleep/wake estimation. Previously, the absolute performance of the SM system in sleep/wake estimation has been tested against polysomnography in subjects under evaluation for sleep apnea, with a reported per-epoch accuracy of approximately 80 % across the population [13]. However, its performance on “normal” subjects, free of sleep-disordered breathing has not been reported. The performance of the HSL101 device has not been previously published. In this paper, we present the performance of the two radiofrequency-based systems and an actigraphy-based system for the measurement of sleep parameters in a normal adult population, against gold-standard laboratory-based polysomnogram (PSG). (Note—we refer to all three systems as “noncontact” as they do not require adhesive electrodes or tethered sensors such as inductance plethysmography). We also investigate the relative performance of the three noncontact approaches, using a null hypothesis that their performance is statistically equivalent.
Methods Trial protocol and demographics The trial was conducted at the polysomnography laboratory of advanced sleep research in Berlin. Ethics approval for the trial
was granted by the local Ethics Committee, and informed consent was obtained from trial participants. Twenty subjects were recruited from a list of enrolled volunteers who were believed to be free of sleep-disordered breathing and primary insomnia, due to previous evaluation. Polysomnography was carried out using the Embla N7000 system running the Somnologica software. The subject’s sleep parameters were scored by a single scorer using AASM scoring rules [14]. The subject was simultaneously monitored using three different sensor technologies which primarily use biomotion to determine sleep/wake. The three technologies evaluated were (a) a radio-frequency based biomotion sensor operating at 5.8 GHz (SM, ResMed Sensor Technologies, Dublin, Ireland), (b) a radio-frequency-based biomotion sensor operating at 10.525 GHz (HSL-101, Omron, Kyoto, Japan), and (c) a wrist-based actigraphy sensor (Actiwatch (AW)-2, Philips Respironics, USA). The principle of operation of the two radio-frequency-based biomotion sensors is that they detect bodily movement and respiration movements by transmitting a low-power pulse of radio-frequency energy. By monitoring the echo received from the subject, the movement of the subject can be determined. An algorithm has then been developed to classify 30-sec epochs of the received signal as either being wake, sleep, or absent (the subject is deemed absent if they are out of the sensor range which is approximately 1.5 m). This algorithm has been previously described and tested for the SM system [13] on a mixed population of normal and sleep-disordered breathing subjects. In internal testing, we have verified that clothing and blankets have minimal impact on the detected signal (as these materials are relatively transparent to radio waves at these frequencies). There is an effect of body position on the signal amplitude recorded (with the weakest signal generally found when the person’s back is to the sensor), but the algorithm is designed to “auto-calibrate” to the overall signal level. The principle of operation of the AW is that it records wrist movement using a sensitive 3D accelerometer and then infers sleep state by analysis of the recorded movement signal. The AW has been previously validated as a sleep measurement device across a wide variety of populations [1–8]. The subjects were instructed to follow their normal daily routine and to come to the sleep laboratory at approximately 21:00. A technician setup the PSG, AW, and HSL-101 devices, and lights out was at approximately 22:30. The AW was worn on the subject’s nondominant wrist. The SM and HSL101 were placed side-by-side approximately 1 m from the subject, and slightly elevated above the torso (see Fig. 1). Subjects were awoken at approximately 06:30 the following morning. Each of the recording devices has an internal clock with an accuracy of 5 s or less over the 10-h recording period. The synchronization of the devices was ensured by following a protocol to set the time of each device with respect to a
Sleep Breath Table 2 Recorded sleep parameters for the 20 subjects tested in this study Measured sleep parameters Total sleep time (min) Sleep-onset latency (min) Wake-after-sleep onset (min) Sleep efficiency (%) AHI (events/h)
391±49 20±13 69±46 82±11 % 0.9±1
The performance metrics of interest are
Fig. 1 Illustration of the setup at the sleep laboratory
controlled clock prior to each sleep study. Table 1 summarizes the demographic characteristics of the study participants. Analysis The four recording systems (PSG, SM, HSL-101, and AW) were all switched on and off at slightly different times due to practical constraints of equipment setup. Therefore, for the analysis, we only include the period from time in bed to time out of bed. Periods during the night when the person exited the bed (e.g., for bathroom breaks) were omitted from any comparative analysis, as they represented approximately 0.4 % of all epochs in the data set. Each system reports a label for each 30-s epoch. For the PSG system, the valid labels are Wake, Stage N1 sleep, Stage N2 sleep, Stage N3 sleep, REM, and artifact. For analysis purposes, Stages N1–N3 and REM are collectively labeled as sleep. For the HSL-101, the epochs are labeled as sleep, wake, or absent. For the AW, the labels are sleep or wake. The overall goal of the analysis was (a) to determine the absolute performance of the three noncontact system in assessment of sleep metrics, using the PSG results as a gold standard, and (b) to investigate potential differences in the relative performance of the three noncontact devices, under a null hypothesis that their performance is equivalent.
Table 1 Demographics of the study participants Subject Demographics Subjects (M/F) Age BMI AHI
20 (11/9) 30±6 years 23.2±3.6 kg/m2 0.9±1 events/h
(a) Per-epoch accuracy. We report agreement levels between 30-s epoch labels produced by the PSG as sleep or wake (omitting any absence periods) and the corresponding epoch labels produced by the three noncontact methods. (b) Sensitivity to sleep. This is the percentage of actual PSG sleep epochs which are labeled sleep by the noncontact device. (c) Sensitivity to wake. This is the percentage of actual PSG wake epochs which are labeled wake by the noncontact device. (d) Cohen’s kappa. Given the high imbalance of the sleep/ wake classes, per-epoch accuracy figures can give an overly optimistic impression of performance; Cohen’s kappa is a useful tool for showing the level of agreement, which has been achieved above chance alone and provides an alternative and more meaningful metric in some cases. (e) Total sleep time (TST). This is the duration (in minutes) of all epochs labeled as sleep. (f) TST error. This is the difference between the TST reported by the noncontact device and the PSG TST. (g) Sleep efficiency (SE). This is the TST divided by the time in bed expressed as a percentage. (h) Sleep-onset latency (SOL). This is defined as the time from lights out to the first continuous 2-min period of sleep. Table 3 Comparative performance on a pooled-epoch basis
Per-epoch accuracy Sensitivity to sleep Sensitivity to wake Cohen’s kappa
SleepMinder
HSL-101
Actiwatch
0.856 0.954 0.424 0.52
0.855 0.963 0.376 0.52
0.854 0.972 0.332 0.51
The per-epoch accuracy is the number of epochs correctly labeled relative to the gold-standard polysomnogram labels. The sensitivity to sleep is the percentage of true sleep epochs correctly labeled by the system. The sensitivity to wake is the percentage of true wake epochs correctly labeled by the system. Cohen’s Kappa reflects the agreement level observed which is greater than chance alone (Cohen’s Kappa can assume values between −1 and 1, with values of >0.6 typically considered as high)
Sleep Breath Table 4 Comparative performance of the three systems on a per-subject basis
Per-epoch accuracy Sensitivity to sleep Sensitivity to wake Cohen’s kappa Estimated total sleep time (min) Total sleep time error versus PSG (min) Estimated sleep efficiency Estimated SOL (min) Estimated WASO (min)
SleepMinder
HSL-101
Actiwatch
0.856±0.062 0.953±0.038 0.389±0.190 0.533±0.201 426±34b 33±38 0.884±0.075a, b 10±10a, b 46±33b
0.856±0.064 0.964±0.024 0.358±0.1420 0.532±0.207 434±22b 41±37 0.901±0.048b 5±5b 43±22b
0.855±0.063 0.973±0.020 0.339±0.127 0.528±0.201 441±16b 48±41 0.916±0.035a, b 3±2a, b 38±17b
Statistically significant inter-instrument results are marked in the table Significant difference between Actiwatch and SleepMinder estimates
b
Significant difference from PSG-reported value
(i) Wake-after-sleep onset (WASO). This is defined as the total wake time observed between sleep onset and time out of bed To visualize data, we use scatter plots and (modified) Bland-and-Altman analysis and also report Pearson product– moment correlation coefficients against the PSG goldstandard reference values. One-way ANOVA tests were used to establish if the estimates by the three noncontact methods varied from the PSG measurement. Pairwise t tests with adjusted p values (using Holm–Bonferroni correction) were used to see if there were statistically significant variations between the three noncontact methods. Statistical analysis was carried out using R v3.0.2. All statistical significant conclusions are made at an α=0.05 level. When product moment correlation coefficients are quoted, we also include the 95 % confidence interval, using a Fisher transformation, under an assumption that the variables have a bivariate normal distribution.
Results Sleep parameters reported by the PSG analysis are shown in Table 2. The TST and sleep efficiencies reported are consistent with previous lab-based studies of normal volunteers in a sleep lab. The pooled per-epoch, sensitivity to sleep, sensitivity to wake, and Cohen’s kappa values are shown in Table 3. This was obtained by combining all scored epochs from all subjects. There was no statistically significant difference between the three noncontact systems on any of these metrics. The per-subject performance of the three biomotion systems is detailed in Table 4 (in a per-subject analysis, we calculated the per-epoch accuracy, sensitivity to sleep, etc.
individually for each subject, and then averaged results across subjects). Paired t tests with Holm–Bonferroni correction were used to determine the significance of any interinstrument difference between the systems. The null hypothesis (that all three noncontact methods provide equivalence performance) was satisfied for most of these performance metrics, with the exception of SOL and SE estimates, where there was a statistically significant difference between SM and AW performance. To assess the absolute performance of the three noncontact methods, we show the scatter plot of TSTs reported by the SM, HSL-101 and the AW versus the PSG reported TST (Fig. 2). The Pearson’s product moment correlation 500
450
TST SM/HSL/AW (mins)
a
400
350
SM HSL AW 1:1 SM
300
fit
HSL
fit
250
200 200
AWfit
250
300
350
400
450
500
TST PSG (mins)
Fig. 2 A comparison of the TST for the twenty subjects as measured by the PSG versus TST measured by the SM, HSL-101, and AW. The lines show a linear best fit for the three devices and the line of identity
Sleep Breath 150
Fig. 3 A modified Bland-andAltman plot of the TST error for the 20 subjects as measured by the SM, HSL-101, and AW TST SM/HSL/AW −TST PSG(mins)
SM HSL AW SMfit HSL
fit
100
AW
fit
50
0
−50 200
250
300
350
400
450
500
TST PSG (mins)
coefficients between PSG and SM, HSL-101, and AW were 0.646 (p
