pii: jc- 00179-15
http://dx.doi.org/10.5664/jcsm.5398
S CI E NT IF IC IN VES TIGATIONS
Agreement in the Scoring of Respiratory Events Among International Sleep Centers for Home Sleep Testing Ulysses J. Magalang, MD1; Erna S. Arnardottir, PhD2,3; Ning-Hung Chen, MD4; Peter A. Cistulli, MD, PhD5; Thorarinn Gíslason, MD, PhD2,3; Diane Lim, MD7; Thomas Penzel, PhD6; Richard Schwab, MD7; Sergio Tufik, MD, PhD8; Allan I. Pack, MBChB, PhD7, for the SAGIC Investigators Division of Pulmonary, Allergy, Critical Care, and Sleep Medicine, The Ohio State University Wexner Medical Center, Columbus, Ohio; 2Department of Respiratory Medicine and Sleep, Landspitali – The National University Hospital of Iceland, Reykjavik, Iceland; 3Faculty of Medicine, University of Iceland, 4Division of Pulmonary, Critical Care, and Sleep Medicine, Chang Gung Memorial Hospital, Taoyuan, Taiwan; 5Centre for Sleep Health and Research, Department of Respiratory Medicine, Royal North Shore Hospital, and University of Sydney, Sydney, Australia; 6Center of Sleep Medicine, Charité University Hospital, Berlin, Germany; 7Center for Sleep and Circadian Neurobiology, Division of Sleep Medicine, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania; 8Disciplina de Medicina e Biologia do Sono, Departamento de Psicobiologia, Universidade Federal de São Paulo, São Paulo, Brazil. 1
Study Objectives: Home sleep testing (HST) is used worldwide to confirm the presence of obstructive sleep apnea (OSA). We sought to determine the agreement of HST scoring among international sleep centers. Methods: Fifteen HSTs, previously recorded using a type 3 monitor, were deidentified and saved in European Data Format. The studies were scored by nine technologists from the sleep centers of the Sleep Apnea Global Interdisciplinary Consortium (SAGIC) using the locally available software. Each study was scored separately using one of three different airflow signals: nasal pressure (NP), transformed (square root) nasal pressure signal (transformed NP), and uncalibrated respiratory inductive plethysmography (RIP) flow. Only one of the three airflow signals was visible to the scorer at each scoring session. The scoring procedure was repeated to determine the intrarater reliability. Results: The intraclass correlation coefficients (ICCs) using the NP were: apnea-hypopnea index (AHI) = 0.96 (95% confidence interval [CI]: 0.93–0.99); apnea index = 0.91 (0.83–0.96); and hypopnea index = 0.75 (0.59–0.89). The ICCs using the transformed NP were: AHI = 0.98 (0.96–0.99); apnea index = 0.95 (0.90–0.98); and hypopnea index = 0.90 (0.82–0.96). The ICCs using the RIP flow were: AH I = 0.98 (0.96–0.99); apnea index = 0.66 (0.48–0.84); and hypopnea index = 0.78 (0.63–0.90). The mean difference of first and second scoring sessions of the same respiratory variables ranged from −1.02 to 0.75/h. Conclusion: There is a strong agreement in the scoring of the respiratory events for HST among international sleep centers. Our results suggest that centralized scoring of HSTs may not be necessary in future research collaboration among international sites. Commentary: A commentary on this article appears in this issue on page 7. Keywords: home sleep testing, sleep study scoring Citation: Magalang UJ, Arnardottir ES, Chen NH, Cistulli PA, Gíslason T, Lim D, Penzel T, Schwab R, Tufik S, Pack AI, SAGIC Investigators. Agreement in the scoring of respiratory events among international sleep centers for home sleep testing. J Clin Sleep Med 2016;12(1):71–77.
I N T RO D U C T I O N
BRIEF SUMMARY
Current Knowledge/Study Rationale: Home sleep testing (HST) is now commonly used worldwide to confirm the presence of obstructive sleep apnea because it is less labor intensive and less expensive compared to in-laboratory polysomnography (PSG). We previously showed that there was a substantial agreement in the scoring of respiratory events for PSG among international sleep centers, but this has not been studied for HST. Study Impact: We found a strong agreement in the scoring of the respiratory events for HST among international sleep centers. In association with our previous study, the results suggest that centralized scoring of sleep studies may not be necessary in research collaboration among international sites.
Home sleep testing (HST) is commonly used worldwide to confirm the presence of obstructive sleep apnea (OSA) because it is less labor intensive and less expensive compared to in-laboratory polysomnography (PSG).1–4 HST, performed most often using a type 3 portable device, involves monitoring of respiration during the normal sleep hours. The home sleep study is unattended by a technologist and does not record sleep stages.5 Agreement in the scoring of sleep studies has an effect both on clinical management as well as research collaboration. Our group previously showed that there was a substantial agreement in the scoring of respiratory events for in-laboratory polysomnography studies among international sleep centers in the Sleep Apnea Global Interdisciplinary Consortium (SAGIC) www.med.upenn.edu/sleepctr/SAGIC.shtml.6 This previous study on PSG scoring suggested an alternative approach to centralized scoring of PSGs in research collaboration among international centers. However, the agreement in scoring of
respiratory events for HST among international sleep centers is unknown. This would be important to determine given the increasing clinical use of HST and the developing collaboration among sleep researchers worldwide in multicenter international studies.7,8 71
Journal of Clinical Sleep Medicine, Vol. 12, No. 1, 2016
UJ Magalang, ES Arnardottir, NH Chen et al. Agreement in HST Scoring
In addition, patients were asked to demonstrate to the technologist how they will use the HST equipment on themselves. The patient was also provided with written instructions to take home at the end of the session. The sampling rates of the channels were: nasal pressure: 50 Hz; chest and abdominal movement: 50 Hz; and pulse oximetry: 3 Hz. The pulse oximeter (Nonin Medical, Inc, Plymouth, MN, USA) used in acquiring the HSTs used an averaging algorithm of 3 sec or faster for pulse rates of 60 beats per min or greater. The HSTs were deidentified and then converted into European Data Format (EDF), a standard file format that allows the exchange and storage of polygraphic data in a form that is independent of hard- or software environments.13 In our study, it was necessary to use EDF because of the different scoring software used locally at each sleep center. EDF conversion removed all previous scoring of respiratory events.
The signals that are typically included in HSTs are nasal pressure, thorax and abdominal movements, oxyhemoglobin saturation, and pulse rate.9 The airflow derived by the square root transformation of the nasal pressure signal (transformed NP) has been suggested to more closely approximate flow for hypopnea scoring and is frequently used for the scoring of respiratory events.10,11 In addition, the flow derived from respiratory inductive plethysmography (RIP flow) from the thorax and abdominal belt signals has been suggested as an alternative sensor for scoring apneas and hypopneas, if the primary signal fails or is unreliable.12 The agreement in scoring of respiratory events when the transformed NP or RIP flow signals are used in scoring of HSTs is also not known. We sought to determine the agreement of scoring of respiratory events for HSTs in the international member centers of SAGIC. Our primary hypothesis was that there would be strong agreement in HST scoring of the apnea-hypopnea index (AHI) among experienced scorers.
Scoring
At each participating SAGIC site, the studies in EDF format were imported into the local software used for scoring. The scoring software included Remlogic (Natus Neurology, Tonawanda, NY, USA [Berlin, São Paulo, Philadelphia, and Columbus sites]); Compumedics (Compumedics Limited, Victoria, Australia [Sydney and Taipei sites]); and Noxturnal (Nox Medical, Reykjavik, Iceland [Reykjavik site]). The scoring protocol that included a review of the guidelines on scoring of respiratory events (in Microsoft Power Point and Word) was provided to nine experienced scorers. This was supplemented by an educational online conference (WebEx, Cisco Systems, Inc., San Jose, CA, USA) headed by an investigator (UJM) involving the scorers. All nine scorers had at least 5 y of experience in scoring clinical sleep studies and were designated by the investigator at each participating SAGIC site (two scorers in Berlin, two scorers in São Paulo, and one scorer each in Philadelphia, Columbus, Sydney, Taipei, and Reykjavik). The analysis start time (“lights out”) and analysis end time (“lights on”) were provided for each HST. Each 30-sec epoch was manually scored for apneas and hypopneas in accordance with the 2007 American Academy of Sleep Medicine (AASM) manual.14 Briefly, an apnea was scored when all of the following criteria were met: a drop in airflow sensor excursion by ≥ 90% of baseline, the duration of the event lasts ≥ 10 sec, and ≥ 90% of the event duration meets the amplitude reduction criteria. An apnea was classified as obstructive if it was associated with continued or increased respiratory effort throughout the entire period of absent airflow, central if it was associated with absent respiratory effort throughout the entire period of absent airflow, and mixed if it was associated with absent respiratory effort in the initial portion of the event, followed by resumption of respiratory effort in the second portion of the event.14 A hypopnea was scored if the following criteria were met: the airflow signal excursion dropped by ≥ 30% of baseline, there was a ≥ 4% desaturation from pre-event baseline, and at least 90% of the event’s duration met the amplitude reduction criteria.14–16 The computer-derived oxygen desaturation index (ODI), defined as the number of oxygen desaturations ≥ 4% per hour of sleep, was also obtained. All scorers recorded the following variables for each HST: AHI; number of apneas;
METHODS The SAGIC centers who participated in this study are located in the following cities: Sydney, Australia; São Paulo, Brazil; Berlin, Germany; Reykjavik, Iceland; Taipei, Taiwan; and Columbus, Ohio and Philadelphia, Pennsylvania in the United States of America. The study was approved by the Institutional Review Board of The Ohio State University Wexner Medical Center. Informed consent was waived for this study given that the HSTs were previously recorded and deidentified.
Home Sleep Studies
Fifteen previously recorded HSTs in the Columbus, Ohio site were chosen by a SAGIC investigator (UJM). Previously scored HSTs are routinely stored at the Ohio State University Sleep Disorders Center on a quarterly basis. The 15 studies were randomly selected from one folder containing the list of patients who had HSTs obtained in one quarter (74 studies). The written report for each randomly selected study was opened to make sure that the study does not meet any of the exclusion criteria. Studies were included provided the clinical report did not state that the study was not interpretable because of excessive artifacts or absence of adequate data. Exclusion criteria included studies performed while on positive airway pressure therapy or while wearing a mandibular advancement device. During this process, three studies were excluded because they were performed while on continuous positive airway pressure (CPAP) therapy and two were excluded because they were performed while wearing dental device. Fifteen studies were chosen according to a power analysis as explained below. After the 15th sleep study was selected, no further HSTs were evaluated. During this process, no HSTs were excluded on the basis of poor signal quality. All HSTs were originally recorded using an Embletta Gold Type 3 portable device (Natus Neurology, Tonawanda, NY, USA). All patients were seen by a technologist, during a scheduled session, at which time detailed instructions and demonstration of how the sensors should be applied were performed. Journal of Clinical Sleep Medicine, Vol. 12, No. 1, 2016
72
UJ Magalang, ES Arnardottir, NH Chen et al. Agreement in HST Scoring
number of obstructive, central, and mixed apneas; number of hypopneas; and ODI. Each study was scored separately using one of three different airflow signals: nasal pressure (NP), transformed (square root) nasal pressure signal (transformed NP), and uncalibrated respiratory inductive plethysmography (RIP) flow.17–19 Only one of the three airflow signals was visible to the scorer at each scoring session. For example, when the NP was used as the signal for airflow, both the transformed NP and RIP flow were not visible to the scorer. The procedure was repeated for the second time, with at least a 1-w interval between scoring a study, to determine the intrarater reliability of scoring.
Figure 1—The absolute values of the apnea-hypopnea index (AHI) by the nine scorers of each of the 15 home sleep tests are shown.
Sample Size
The primary outcome for the interrater agreement analysis was the intraclass correlation coefficient (ICC) of the AHI. Given nine sleep scorers, the 15 PSGs had a power of 83% to detect an ICC of at least 0.90, assuming a null hypothesis of ICC = 0.70.
Statistical Analysis
The interrater reliability measures used to examine the agreement among the nine different scorers were the ICCs for the different respiratory indices. The levels of agreement using the ICCs of respiratory indices were classified as follows: 0.00– 0.25 = little, 0.26–0.49 = low, 0.50–0.69 = moderate, 0.70– 0.89 = strong, 0.90–1.00 = very strong.20–22 Intrarater scoring agreement assessed how close the scores were on the first and second scoring sessions of the same respiratory variables. For this purpose, the Bland-Altman method was used for assessing agreement between the two measurements by calculating the mean difference (MEANdiff) between the two measurements and the standard deviation (SD) of this difference. The limits of agreement (LoA) were calculated as: mean difference + 1.96*SD of the differences.23 In addition, Bland-Altman plots of the agreement between the AHI values using the three different airflow signals (NP, transformed NP, and RIP flow) were also obtained. Data analyses were performed using SPSS software version 22 (IBM Corp., Armonk, NY, USA). The BlandAltman plots were generated using SigmaPlot software version 13 (Systat Software, Inc, San Jose, CA, USA).
NP, nasal pressure; transformed NP, square root transformation of NP signal; RIP flow, respiratory flow using respiratory inductive plethysmography.
index = 0.91 (0.83–0.96); and hypopnea index = 0.75 (0.59– 0.89). The ICCs using the transformed NP were: AHI = 0.98 (0.96–0.99); apnea index = 0.95 (0.90–0.98); and hypopnea index = 0.90 (0.82–0.96). The ICCs using the RIP flow were: AHI = 0.98 (0.96–0.99); apnea index = 0.66 (0.48–0.84); and hypopnea index = 0.78 (0.63–0.90). Using the transformed NP signal increased the agreement in the scoring of hypopnea index, whereas there was only moderate scoring agreement of the apnea index when the RIP flow signal was used. In addition, use of the transformed NP signal also increased the agreement in the scoring of the central and mixed apnea indices. The agreement of automated ODI scoring between softwares was also very strong. Using the Bland-Altman method, the mean differences between AHI and ODI scores were 2.6 ± 4.8 events/h (NP signal), 0.3 ± 2.7 events/h (transformed NP), and 1.1 ± 2.3 events/h (RIP flow). Tables S1–S3 in the supplemental material display the means and the standard deviations (SD) of the scoring of
R ES U LT S Figure 1 displays the absolute values of the scoring of the AHI of all 15 HSTs by the nine scorers using the three different airflow signals. The mean ± SD of the AHI on the 15 HSTs was 28.0 ± 18.3 events/h (range: 6.3–105.5) using NP, 25.1 ± 16.7 events/h (range: 5.3–73.4) using transformed NP, and 24.0 ± 15.3 events/h (range: 1.7–67.5) using RIP flow.
Interrater Agreement
The interscorer agreement of AHI scoring among the SAGIC scorers was very strong using any of the following as the primary signal for airflow: NP, transformed NP, or RIP flow. As shown in Table 1, the ICCs of the respiratory event scoring using the NP were: AHI = 0.96 (95% CI: 0.93–0.99); apnea 73
Journal of Clinical Sleep Medicine, Vol. 12, No. 1, 2016
UJ Magalang, ES Arnardottir, NH Chen et al. Agreement in HST Scoring
Table 1—Interrater agreement of scoring respiratory events using different airflow signals Variable AHI (events/h) Apnea index OA index CA index MA index Hypopnea index ODI (events/h)
NP 0.96 (0.93–0.99) 0.91 (0.83–0.96) 0.90 (0.82–0.96) 0.62 (0.43–0.81) 0.56 (0.37–0.77) 0.75 (0.59–0.89) 0.98 (0.97–0.99)
Intraclass Correlation Coefficients Transformed NP 0.98 (0.96–0.99) 0.95 (0.90–0.98) 0.94 (0.89–0.98) 0.86 (0.75–0.94) 0.70 (0.52–0.86) 0.90 (0.82–0.96) 0.99 (0.98–0.99)
RIP Flow 0.98 (0.96–0.99) 0.66 (0.48–0.84) 0.60 (0.42–0.80) 0.68 (0.50–0.85) 0.48 (0.29–0.72) 0.78 (0.63–0.90) 0.98 (0.97–0.99)
Values in parenthesis represent the 95% confidence limits. AHI, apnea-hypopnea index; CA, central apnea; MA, mixed apnea; NP, nasal pressure; OA, obstructive apnea; ODI, oxygen desaturation index; RIP flow, flow using respiratory inductive plethysmography; transformed NP, square root transformation of NP signal.
Table 2—Intrarater agreement of scoring respiratory events using different airflow signals Variable AHI (events/h) Apnea index OA index CA index MA index Hypopnea index
MEANdiff 0.02 −0.14 0.17 −0.05 0.06 −0.14
NP SDdiff 1.53 2.22 2.55 0.33 0.34 2.22
LoA −2.98, 3.02 −4.49, 4.21 −4.82, 5.16 −0.69, 0.59 −0.60, 0.73 −4.49, 4.21
MEANdiff −0.26 −0.13 −0.12 −0.01 0.004 −0.13
Transformed NP SDdiff LoA 0.85 −1.91, 1.40 1.19 −2.46, 2.21 1.13 −2.33, 2.09 0.31 −0.61, 0.58 0.23 −0.44, 0.45 1.22 −2.53, 2.27
MEANdiff −0.40 0.75 0.68 −0.06 −0.04 −1.02
RIP Flow SDdiff 1.28 2.66 2.71 0.41 0.42 2.68
LoA −2.91, 2.10 −4.48, 5.96 −4.64, 6.01 −0.87, 0.75 −0.87, 0.78 −6.27, 4.24
AHI, apnea hypopnea index; CA, central apnea; LoA, limits of agreement; MA, mixed apnea; MEANdiff, mean difference between the first and second scoring sessions of the same respiratory variables; NP, nasal pressure; OA, obstructive apnea; ODI, oxygen desaturation index; RIP flow, flow using respiratory inductive plethysmography; transformed NP, square root transformation of NP signal.
Figure 2—Bland-Altman plot of the agreement between the apnea-hypopnea index (AHI) values using nasal pressure (NP) and the square root transformation of the NP signal (transformed NP).
intrarater agreement between the first and second scoring sessions in the scoring of all respiratory indices using any of the three airflow signals.
Comparison of AHI Using Different Airflow Signals
Overall, the mean differences (bias) in AHI scoring using the three airflow signals were small. Figure 2 shows the BlandAltman plot of the AHI values obtained using the NP and transformed NP signals (bias = 2.9 ± 3.6 events/h). The AHI values between the NP signal and RIP flow (Figure 3) had the greatest bias (4.0 ± 6.1 events/h), whereas the AHI values between the transformed NP and the RIP flow (Figure 4) had the least bias (1.1 ± 3.6 events/h). D I SCUS S I O N This study investigated HST scoring agreement of respiratory events across seven international centers within SAGIC. The major findings of this study are: (1) there is a very strong interrater agreement in the scoring of the AHI among experienced sleep technologists regardless of the type of airflow signal used (NP, transformed NP, or RIP flow); (2) there is strong agreement in the scoring of the apnea and hypopnea indices particularly when the transformed NP is used as the airflow signal; (3) there is excellent intrarater agreement between the first and second scoring sessions in the scoring of all respiratory indices
respiratory events in all 15 HSTs by the nine scorers using the NP, transformed NP, and RIP flow, respectively.
Intrarater Agreement
Table 2 shows the intrarater agreement of scoring respiratory events using the different airflow signals. The mean difference of first and second scoring sessions of the same respiratory variables ranged from −1.02 to 0.75/h. There was good Journal of Clinical Sleep Medicine, Vol. 12, No. 1, 2016
74
UJ Magalang, ES Arnardottir, NH Chen et al. Agreement in HST Scoring
Figure 3—Bland-Altman plot of the agreement between the apnea-hypopnea index (AHI) values using nasal pressure (NP) and respiratory flow using respiratory inductive plethysmography (RIP flow).
Figure 4—Bland-Altman plot of the agreement between the apnea-hypopnea index (AHI) values using the square root transformation of the NP signal (transformed NP) and respiratory flow using respiratory inductive plethysmography (RIP flow).
using any of the three airflow signals; and (4) the automated scoring of the ODI also shows very strong agreement despite using various scoring software to derive this index. To our knowledge, this is the first study examining the agreement of HST scoring of respiratory events among international sleep centers. Because HST is now increasingly used as the initial modality to diagnose OSA, the issue of agreement is not only important to researchers who are planning multicenter collaborative efforts but also to clinicians because such information is essential for evaluation of the findings and interpretation. As in our prior study of PSG scoring, the use of different scoring software across the SAGIC centers is a strength of this study because it provides a greater challenge to good agreement and is a real-world application.6 The current study extends our previous report that showed substantial agreement in the scoring of the respiratory events among international sleep centers working in the SAGIC consortium for in-laboratory PSGs. The results of the current study on HST scoring and our previous report on PSG scoring suggest an alternative approach to centralized scoring of sleep studies in research collaboration among international centers. Similar to our in-laboratory PSG scoring study, the agreement in the scoring of central and mixed apneas for HSTs was lower than for obstructive apneas. The reasons for these findings are unclear, but suggest that it is an area that needs to be emphasized in ongoing quality improvement of sleep study scoring among the SAGIC centers. However, it should also be noted that the number of mixed and central apneas in the HSTs was low. For HSTs, it appears that using the transformed NP signal improves the HST scoring of central and mixed apneas, although not to the level of agreement obtained in scoring obstructive apneas. The airflow derived by the square root transformation of the nasal pressure signal (transformed NP) is used by some centers for the scoring of respiratory events.10–12 Our results suggest that using the transformed NP may improve the agreement in the scoring of the hypopneas. The use of the RIP flow from the
thorax and abdominal belt signals, which is suggested as an alternative sensor for scoring respiratory events if the NP signal fails, also showed strong agreement in AHI scoring. To our knowledge, ours is the first study that has examined the AHI scoring agreement using RIP flow. Two prior studies examined the agreement between automatic and manual scoring of respiratory events in HSTs.9,24 The two studies suggest that the agreement between automatic and manual agreement is modest and that automatic agreement consistently underestimated the AHI derived from manual scoring of HSTs. None of these prior publications reported the agreement when different scoring software is used on the same studies, which is a strength of our current study given that this would be the real-world situation in collaborative research involving international sleep centers. Our study has some limitations. First, we used a ≥ 4% oxygen desaturation to define a hypopnea. An update of the AASM PSG scoring rules stipulate the following change in hypopnea definition: a drop of peak signal excursion by ≥ 30% of preevent baseline using the nasal pressure signal for duration of ≥ 10 sec with a ≥ 3% oxygen desaturation from preevent baseline or an associated arousal.12 However, we are aware of only two studies that have systematically examined the association of varying definitions of hypopnea and clinical outcomes.25,26 In a large epidemiologic study examining an outcome-based definition of hypopneas, only desaturations of ≥ 4% were independently associated with cardiovascular disease.25 In contrast, no association was observed between cardiovascular disease and hypopneas associated with milder desaturations (including 3%) or arousals in a population of 6,106 adults.25 A more recent study showed that using a ≥ 4% desaturation to define a hypopnea was most consistently associated with systolic blood pressure in 2,040 participants in the Multi-Ethnic Study of Atherosclerosis (MESA).26 The most robust studies looking at the association of OSA and hypertension27–30 or the association of OSA and cardiovascular events31 75
Journal of Clinical Sleep Medicine, Vol. 12, No. 1, 2016
UJ Magalang, ES Arnardottir, NH Chen et al. Agreement in HST Scoring
all utilized a definition of hypopnea that incorporates ≥ 4% oxyhemoglobin desaturation.27–31 Nonetheless, the effect of the AASM change in hypopnea definition on the interrater reliability of scoring respiratory events for HSTs among international centers is yet to be examined (excluding arousal assessment as it cannot be performed in typical type 3 portable devices). A second limitation is using 15 HSTs acquired from one SAGIC center. However, data acquisition in these HSTs used standard techniques that would not be expected to be different in other international centers given that the equipment used is a type 3 portable device. A third limitation of our study is that it was not specifically powered to examine scoring agreement in those with mild OSA (AHI ≥ 5 to < 15/h). Finally, we did not randomize the order of the scoring of respiratory events when using one of the three airflow signals: NP, transformed NP, and RIP flow. We could not entirely rule out a learning effect during the multiple scoring sessions that could have affected the interrater agreement. However, HSTs were scored with only one airflow signal visible at each scoring session. In addition, the intrarater agreement of respiratory indices scoring was all similarly excellent using the three different airflow signals, suggesting that any order effect on scoring was not significant. In summary, our findings show that there is a strong interrater and intrarater agreement in the scoring of the respiratory events among international sleep centers for HSTs in the SAGIC consortium. Use of the transformed NP as the signal for airflow appears to increase the agreement in the scoring of hypopneas. There is also strong agreement in AHI scoring when alternative signals such as RIP flow is used in scoring of HSTs. Taken together with our previous study on in-laboratory PSG scoring, our results suggest that centralized scoring of HSTs and PSGs may not be necessary in research collaboration among international sites provided adequate recordings are acquired and experienced scorers are involved.
R E FE R E N CES 1. Masa JF, Corral J, Pereira R, et al. Effectiveness of home respiratory polygraphy for the diagnosis of sleep apnoea and hypopnoea syndrome. Thorax 2011;66:567–73. 2. Whittle AT, Finch SP, Mortimore IL, MacKay TW, Douglas NJ. Use of home sleep studies for diagnosis of the sleep apnoea/hypopnoea syndrome. Thorax 1997;52:1068–73. 3. Kuna ST, Badr MS, Kimoff RJ, et al. An official ATS/AASM/ACCP/ERS workshop report: research priorities in ambulatory management of adults with obstructive sleep apnea. Proc Am Thorac Soc 2011;8:1–16. 4. Masa JF, Corral J, Pereira R, et al. Therapeutic decision-making for sleep apnea and hypopnea syndrome using home respiratory polygraphy: a large multicentric study. Am J Respir Crit Care Med 2011;184:964–71. 5. Flemons WW, Littner MR, Rowley JA, et al. Home diagnosis of sleep apnea: a systematic review of the literature. An evidence review cosponsored by the American Academy of Sleep Medicine, the American College of Chest Physicians, and the American Thoracic Society. Chest 2003;124:1543–79. 6. Magalang UJ, Chen NH, Cistulli PA, et al. Agreement in the scoring of respiratory events and sleep among international sleep centers. Sleep 2013;36:591–6. 7. McEvoy RD, Anderson CS, Antic NA, et al. The sleep apnea cardiovascular endpoints (SAVE) trial: rationale and start-up phase. J Thorac Dis 2010;2:138–43. 8. Hedner J, Grote L, Bonsignore M, et al. The European Sleep Apnoea Database (ESADA): report from 22 European sleep laboratories. Eur Respir J 2011;38:635–42. 9. Masa JF, Corral J, Pereira R, et al. Effectiveness of sequential automaticmanual home respiratory polygraphy scoring. Eur Respir J 2013;41:879–87. 10. Thurnheer R, Xie X, Bloch KE. Accuracy of nasal cannula pressure recordings for assessment of ventilation during sleep. Am J Respir Crit Care Med 2001;164:1914–9. 11. Farre R, Rigau J, Montserrat JM, Ballester E, Navajas D. Relevance of linearizing nasal prongs for assessing hypopneas and flow limitation during sleep. Am J Respir Crit Care Med 2001;163:494–7. 12. Berry RB, Budhiraja R, Gottlieb DJ, et al. Rules for scoring respiratory events in sleep: update of the 2007 AASM Manual for the Scoring of Sleep and Associated Events. Deliberations of the Sleep Apnea Definitions Task Force of the American Academy of Sleep Medicine. J Clin Sleep Med 2012;8:597–619. 13. Kemp B, Varri A, Rosa AC, Nielsen KD, Gade J. A simple format for exchange of digitized polygraphic recordings. Electroencephalogr Clin Neurophysiol 1992;82:391–3. 14. Iber C, Ancoli-Israel S, Chesson A, Quan SF. The AASM Manual for the Scoring of Sleep and Associated Events: Rules, Terminology and Technical Specifications, 1st ed. Westchester, IL: American Academy of Sleep Medicine, 2007. 15. Kuna ST, Gurubhagavatula I, Maislin G, et al. Noninferiority of functional outcome in ambulatory management of obstructive sleep apnea. Am J Respir Crit Care Mede 2011;183:1238–44. 16. Rosen CL, Auckley D, Benca R, et al. A multisite randomized trial of portable sleep studies and positive airway pressure autotitration versus laboratorybased polysomnography for the diagnosis and treatment of obstructive sleep apnea: the HomePAP study. Sleep 2012;35:757–67. 17. Dingli K, Coleman EL, Vennelle M, et al. Evaluation of a portable device for diagnosing the sleep apnoea/hypopnoea syndrome. Eur Respir J 2003;21:253–9. 18. Smith LA, Chong DW, Vennelle M, Denvir MA, Newby DE, Douglas NJ. Diagnosis of sleep-disordered breathing in patients with chronic heart failure: evaluation of a portable limited sleep study system. J Sleep Res 2007;16:428–35. 19. Purdy S, Berry R. Portable Monitoring. In: Lee-Chiong T, ed. Fundamentals of sleep technology. Philadelphia, PA: Lippincott Williams & Wilkins, 2012:482–500. 20. Munro BH. Statistical methods for health care research. 5th ed. Philadelphia, PA: Lippincott Williams & Wilkins, 2005:248–9.
A B B R E V I AT I O N S AASM, American Academy of Sleep Medicine AHI, apnea-hypopnea index CPAP, continuous positive airway pressure EDF, european data format HST, home sleep testing ICC, intraclass correlation coefficient LoA, limits of agreement MEANdiff, mean difference ODI, oxygen desaturation index OSA, obstructive sleep apnea PSG, polysomnography RIP, respiratory inductive plethysmography SAGIC, Sleep Apnea Global Interdisciplinary Consortium SD, standard deviation Transformed NP, square root transformation of the nasal pressure signal
Journal of Clinical Sleep Medicine, Vol. 12, No. 1, 2016
76
UJ Magalang, ES Arnardottir, NH Chen et al. Agreement in HST Scoring 21. Cheng JW, Tsai WC, Yu TY, Huang KY. Reproducibility of sonographic measurement of thickness and echogenicity of the plantar fascia. J Clin Ultrasound 2012;40:14–9. 22. Portney LG, Watkins MP. Foundations of clinical research. Applications to practice. New Jersey: Prentice Hall Inc, 2000:560–7. 23. Bland JM, Altman DG. Measuring agreement in method comparison studies. Stat Methods Med Res 1999;8:135–60. 24. Aurora RN, Swartz R, Punjabi NM. Misclassification of OSA severity with automated scoring of home sleep recordings. Chest 2015;147:719–27. 25. Punjabi NM, Newman AB, Young TB, Resnick HE, Sanders MH. Sleepdisordered breathing and cardiovascular disease: an outcome-based definition of hypopneas. Am J Respir Crit Care Med 2008;177:1150–5. 26. Dean DA, Wang R, Jacobs DR, et al. A systematic assessment of the association of polysomnographic indices with blood pressure: the Multi-Ethnic Study of Atherosclerosis (MESA). Sleep 2015;38:587–96. 27. Marin JM, Agusti A, Villar I, et al. Association between treated and untreated obstructive sleep apnea and risk of hypertension. JAMA 2012;307:2169–76. 28. Barbe F, Duran-Cantolla J, Sanchez-de-la-Torre M, et al. Effect of continuous positive airway pressure on the incidence of hypertension and cardiovascular events in nonsleepy patients with obstructive sleep apnea: a randomized controlled trial. JAMA 2012;307:2161–8. 29. Nieto FJ, Young TB, Lind BK, et al. Association of sleep-disordered breathing, sleep apnea, and hypertension in a large community-based study. Sleep Heart Health Study. JAMA 2000;283:1829–36. 30. Peppard PE, Young T, Palta M, Skatrud J. Prospective study of the association between sleep-disordered breathing and hypertension. N Engl J Med 2000;342:1378–84. 31. Marin JM, Carrizo SJ, Vicente E, Agusti AG. Long-term cardiovascular outcomes in men with obstructive sleep apnoea-hypopnoea with or without treatment with continuous positive airway pressure: an observational study. Lancet 2005;365:1046–53.
ACK N O W L E D G M E N T S The authors thank the following individuals who helped in this project: Mohammad Ahmadi, Alexander Blau, Petra Cornell, Silverio Garbuio, Su-Lan Liu, João Reinfelderon, Beth Staley, Magdalena Ósk Sigurgunnarsdóttir, and Sandra Zimmermann.
SUBM I SSI O N & CO R R ESPO NDENCE I NFO R M ATI O N Submitted for publication April, 2015 Submitted in final revised form July, 2015 Accepted for publication July, 2015 Address correspondence to: Ulysses J. Magalang MD, Division of Pulmonary, Allergy, Critical Care, and Sleep Medicine, The Ohio State University Wexner Medical Center, 201 Davis Heart and Lung Research Institute, 473 West 12th Avenue, Columbus, OH, 43210; Tel: (614) 292-6563; Fax: (614) 293-4799; Email [email protected].
D I SCLO S U R E S TAT E M E N T This was not an industry supported study. Support was provided by NHLBI award P01 HL094307 (AIP), HL093463 (UJM) and Award Number Grant UL1TR001070 from the National Center For Advancing Translational Sciences. Dr. Cistulli has received research support from MonoMed Ltd. and ResMed; has consulted for Exploramed Inc. and Zephyr Sleep Technologies; and has received royalties from Quintessence Publishing. Dr. Arnardottir has consulted for Nox Medical and Weinmann. Dr. Schwab has consulted for ApniCure, CryOSA, Foramis Medical Group, and ResMed. Dr. Magalang has received research support from Hill-Rom. Dr. Gíslason owns stock in and receives salary from Nox Medical. The other authors have indicated no financial conflicts of interest.
77
Journal of Clinical Sleep Medicine, Vol. 12, No. 1, 2016
http://dx.doi.org/10.5664/jcsm.5398
S CI E NT IF IC IN VES TIGATIONS
Agreement in the Scoring of Respiratory Events Among International Sleep Centers for Home Sleep Testing Ulysses J. Magalang, MD1; Erna S. Arnardottir, PhD2,3; Ning-Hung Chen, MD4; Peter A. Cistulli, MD, PhD5; Thorarinn Gíslason, MD, PhD2,3; Diane Lim, MD7; Thomas Penzel, PhD6; Richard Schwab, MD7; Sergio Tufik, MD, PhD8; Allan I. Pack, MBChB, PhD7, for the SAGIC Investigators Division of Pulmonary, Allergy, Critical Care, and Sleep Medicine, The Ohio State University Wexner Medical Center, Columbus, Ohio; 2Department of Respiratory Medicine and Sleep, Landspitali – The National University Hospital of Iceland, Reykjavik, Iceland; 3Faculty of Medicine, University of Iceland, 4Division of Pulmonary, Critical Care, and Sleep Medicine, Chang Gung Memorial Hospital, Taoyuan, Taiwan; 5Centre for Sleep Health and Research, Department of Respiratory Medicine, Royal North Shore Hospital, and University of Sydney, Sydney, Australia; 6Center of Sleep Medicine, Charité University Hospital, Berlin, Germany; 7Center for Sleep and Circadian Neurobiology, Division of Sleep Medicine, University of Pennsylvania Perelman School of Medicine, Philadelphia, Pennsylvania; 8Disciplina de Medicina e Biologia do Sono, Departamento de Psicobiologia, Universidade Federal de São Paulo, São Paulo, Brazil. 1
Study Objectives: Home sleep testing (HST) is used worldwide to confirm the presence of obstructive sleep apnea (OSA). We sought to determine the agreement of HST scoring among international sleep centers. Methods: Fifteen HSTs, previously recorded using a type 3 monitor, were deidentified and saved in European Data Format. The studies were scored by nine technologists from the sleep centers of the Sleep Apnea Global Interdisciplinary Consortium (SAGIC) using the locally available software. Each study was scored separately using one of three different airflow signals: nasal pressure (NP), transformed (square root) nasal pressure signal (transformed NP), and uncalibrated respiratory inductive plethysmography (RIP) flow. Only one of the three airflow signals was visible to the scorer at each scoring session. The scoring procedure was repeated to determine the intrarater reliability. Results: The intraclass correlation coefficients (ICCs) using the NP were: apnea-hypopnea index (AHI) = 0.96 (95% confidence interval [CI]: 0.93–0.99); apnea index = 0.91 (0.83–0.96); and hypopnea index = 0.75 (0.59–0.89). The ICCs using the transformed NP were: AHI = 0.98 (0.96–0.99); apnea index = 0.95 (0.90–0.98); and hypopnea index = 0.90 (0.82–0.96). The ICCs using the RIP flow were: AH I = 0.98 (0.96–0.99); apnea index = 0.66 (0.48–0.84); and hypopnea index = 0.78 (0.63–0.90). The mean difference of first and second scoring sessions of the same respiratory variables ranged from −1.02 to 0.75/h. Conclusion: There is a strong agreement in the scoring of the respiratory events for HST among international sleep centers. Our results suggest that centralized scoring of HSTs may not be necessary in future research collaboration among international sites. Commentary: A commentary on this article appears in this issue on page 7. Keywords: home sleep testing, sleep study scoring Citation: Magalang UJ, Arnardottir ES, Chen NH, Cistulli PA, Gíslason T, Lim D, Penzel T, Schwab R, Tufik S, Pack AI, SAGIC Investigators. Agreement in the scoring of respiratory events among international sleep centers for home sleep testing. J Clin Sleep Med 2016;12(1):71–77.
I N T RO D U C T I O N
BRIEF SUMMARY
Current Knowledge/Study Rationale: Home sleep testing (HST) is now commonly used worldwide to confirm the presence of obstructive sleep apnea because it is less labor intensive and less expensive compared to in-laboratory polysomnography (PSG). We previously showed that there was a substantial agreement in the scoring of respiratory events for PSG among international sleep centers, but this has not been studied for HST. Study Impact: We found a strong agreement in the scoring of the respiratory events for HST among international sleep centers. In association with our previous study, the results suggest that centralized scoring of sleep studies may not be necessary in research collaboration among international sites.
Home sleep testing (HST) is commonly used worldwide to confirm the presence of obstructive sleep apnea (OSA) because it is less labor intensive and less expensive compared to in-laboratory polysomnography (PSG).1–4 HST, performed most often using a type 3 portable device, involves monitoring of respiration during the normal sleep hours. The home sleep study is unattended by a technologist and does not record sleep stages.5 Agreement in the scoring of sleep studies has an effect both on clinical management as well as research collaboration. Our group previously showed that there was a substantial agreement in the scoring of respiratory events for in-laboratory polysomnography studies among international sleep centers in the Sleep Apnea Global Interdisciplinary Consortium (SAGIC) www.med.upenn.edu/sleepctr/SAGIC.shtml.6 This previous study on PSG scoring suggested an alternative approach to centralized scoring of PSGs in research collaboration among international centers. However, the agreement in scoring of
respiratory events for HST among international sleep centers is unknown. This would be important to determine given the increasing clinical use of HST and the developing collaboration among sleep researchers worldwide in multicenter international studies.7,8 71
Journal of Clinical Sleep Medicine, Vol. 12, No. 1, 2016
UJ Magalang, ES Arnardottir, NH Chen et al. Agreement in HST Scoring
In addition, patients were asked to demonstrate to the technologist how they will use the HST equipment on themselves. The patient was also provided with written instructions to take home at the end of the session. The sampling rates of the channels were: nasal pressure: 50 Hz; chest and abdominal movement: 50 Hz; and pulse oximetry: 3 Hz. The pulse oximeter (Nonin Medical, Inc, Plymouth, MN, USA) used in acquiring the HSTs used an averaging algorithm of 3 sec or faster for pulse rates of 60 beats per min or greater. The HSTs were deidentified and then converted into European Data Format (EDF), a standard file format that allows the exchange and storage of polygraphic data in a form that is independent of hard- or software environments.13 In our study, it was necessary to use EDF because of the different scoring software used locally at each sleep center. EDF conversion removed all previous scoring of respiratory events.
The signals that are typically included in HSTs are nasal pressure, thorax and abdominal movements, oxyhemoglobin saturation, and pulse rate.9 The airflow derived by the square root transformation of the nasal pressure signal (transformed NP) has been suggested to more closely approximate flow for hypopnea scoring and is frequently used for the scoring of respiratory events.10,11 In addition, the flow derived from respiratory inductive plethysmography (RIP flow) from the thorax and abdominal belt signals has been suggested as an alternative sensor for scoring apneas and hypopneas, if the primary signal fails or is unreliable.12 The agreement in scoring of respiratory events when the transformed NP or RIP flow signals are used in scoring of HSTs is also not known. We sought to determine the agreement of scoring of respiratory events for HSTs in the international member centers of SAGIC. Our primary hypothesis was that there would be strong agreement in HST scoring of the apnea-hypopnea index (AHI) among experienced scorers.
Scoring
At each participating SAGIC site, the studies in EDF format were imported into the local software used for scoring. The scoring software included Remlogic (Natus Neurology, Tonawanda, NY, USA [Berlin, São Paulo, Philadelphia, and Columbus sites]); Compumedics (Compumedics Limited, Victoria, Australia [Sydney and Taipei sites]); and Noxturnal (Nox Medical, Reykjavik, Iceland [Reykjavik site]). The scoring protocol that included a review of the guidelines on scoring of respiratory events (in Microsoft Power Point and Word) was provided to nine experienced scorers. This was supplemented by an educational online conference (WebEx, Cisco Systems, Inc., San Jose, CA, USA) headed by an investigator (UJM) involving the scorers. All nine scorers had at least 5 y of experience in scoring clinical sleep studies and were designated by the investigator at each participating SAGIC site (two scorers in Berlin, two scorers in São Paulo, and one scorer each in Philadelphia, Columbus, Sydney, Taipei, and Reykjavik). The analysis start time (“lights out”) and analysis end time (“lights on”) were provided for each HST. Each 30-sec epoch was manually scored for apneas and hypopneas in accordance with the 2007 American Academy of Sleep Medicine (AASM) manual.14 Briefly, an apnea was scored when all of the following criteria were met: a drop in airflow sensor excursion by ≥ 90% of baseline, the duration of the event lasts ≥ 10 sec, and ≥ 90% of the event duration meets the amplitude reduction criteria. An apnea was classified as obstructive if it was associated with continued or increased respiratory effort throughout the entire period of absent airflow, central if it was associated with absent respiratory effort throughout the entire period of absent airflow, and mixed if it was associated with absent respiratory effort in the initial portion of the event, followed by resumption of respiratory effort in the second portion of the event.14 A hypopnea was scored if the following criteria were met: the airflow signal excursion dropped by ≥ 30% of baseline, there was a ≥ 4% desaturation from pre-event baseline, and at least 90% of the event’s duration met the amplitude reduction criteria.14–16 The computer-derived oxygen desaturation index (ODI), defined as the number of oxygen desaturations ≥ 4% per hour of sleep, was also obtained. All scorers recorded the following variables for each HST: AHI; number of apneas;
METHODS The SAGIC centers who participated in this study are located in the following cities: Sydney, Australia; São Paulo, Brazil; Berlin, Germany; Reykjavik, Iceland; Taipei, Taiwan; and Columbus, Ohio and Philadelphia, Pennsylvania in the United States of America. The study was approved by the Institutional Review Board of The Ohio State University Wexner Medical Center. Informed consent was waived for this study given that the HSTs were previously recorded and deidentified.
Home Sleep Studies
Fifteen previously recorded HSTs in the Columbus, Ohio site were chosen by a SAGIC investigator (UJM). Previously scored HSTs are routinely stored at the Ohio State University Sleep Disorders Center on a quarterly basis. The 15 studies were randomly selected from one folder containing the list of patients who had HSTs obtained in one quarter (74 studies). The written report for each randomly selected study was opened to make sure that the study does not meet any of the exclusion criteria. Studies were included provided the clinical report did not state that the study was not interpretable because of excessive artifacts or absence of adequate data. Exclusion criteria included studies performed while on positive airway pressure therapy or while wearing a mandibular advancement device. During this process, three studies were excluded because they were performed while on continuous positive airway pressure (CPAP) therapy and two were excluded because they were performed while wearing dental device. Fifteen studies were chosen according to a power analysis as explained below. After the 15th sleep study was selected, no further HSTs were evaluated. During this process, no HSTs were excluded on the basis of poor signal quality. All HSTs were originally recorded using an Embletta Gold Type 3 portable device (Natus Neurology, Tonawanda, NY, USA). All patients were seen by a technologist, during a scheduled session, at which time detailed instructions and demonstration of how the sensors should be applied were performed. Journal of Clinical Sleep Medicine, Vol. 12, No. 1, 2016
72
UJ Magalang, ES Arnardottir, NH Chen et al. Agreement in HST Scoring
number of obstructive, central, and mixed apneas; number of hypopneas; and ODI. Each study was scored separately using one of three different airflow signals: nasal pressure (NP), transformed (square root) nasal pressure signal (transformed NP), and uncalibrated respiratory inductive plethysmography (RIP) flow.17–19 Only one of the three airflow signals was visible to the scorer at each scoring session. For example, when the NP was used as the signal for airflow, both the transformed NP and RIP flow were not visible to the scorer. The procedure was repeated for the second time, with at least a 1-w interval between scoring a study, to determine the intrarater reliability of scoring.
Figure 1—The absolute values of the apnea-hypopnea index (AHI) by the nine scorers of each of the 15 home sleep tests are shown.
Sample Size
The primary outcome for the interrater agreement analysis was the intraclass correlation coefficient (ICC) of the AHI. Given nine sleep scorers, the 15 PSGs had a power of 83% to detect an ICC of at least 0.90, assuming a null hypothesis of ICC = 0.70.
Statistical Analysis
The interrater reliability measures used to examine the agreement among the nine different scorers were the ICCs for the different respiratory indices. The levels of agreement using the ICCs of respiratory indices were classified as follows: 0.00– 0.25 = little, 0.26–0.49 = low, 0.50–0.69 = moderate, 0.70– 0.89 = strong, 0.90–1.00 = very strong.20–22 Intrarater scoring agreement assessed how close the scores were on the first and second scoring sessions of the same respiratory variables. For this purpose, the Bland-Altman method was used for assessing agreement between the two measurements by calculating the mean difference (MEANdiff) between the two measurements and the standard deviation (SD) of this difference. The limits of agreement (LoA) were calculated as: mean difference + 1.96*SD of the differences.23 In addition, Bland-Altman plots of the agreement between the AHI values using the three different airflow signals (NP, transformed NP, and RIP flow) were also obtained. Data analyses were performed using SPSS software version 22 (IBM Corp., Armonk, NY, USA). The BlandAltman plots were generated using SigmaPlot software version 13 (Systat Software, Inc, San Jose, CA, USA).
NP, nasal pressure; transformed NP, square root transformation of NP signal; RIP flow, respiratory flow using respiratory inductive plethysmography.
index = 0.91 (0.83–0.96); and hypopnea index = 0.75 (0.59– 0.89). The ICCs using the transformed NP were: AHI = 0.98 (0.96–0.99); apnea index = 0.95 (0.90–0.98); and hypopnea index = 0.90 (0.82–0.96). The ICCs using the RIP flow were: AHI = 0.98 (0.96–0.99); apnea index = 0.66 (0.48–0.84); and hypopnea index = 0.78 (0.63–0.90). Using the transformed NP signal increased the agreement in the scoring of hypopnea index, whereas there was only moderate scoring agreement of the apnea index when the RIP flow signal was used. In addition, use of the transformed NP signal also increased the agreement in the scoring of the central and mixed apnea indices. The agreement of automated ODI scoring between softwares was also very strong. Using the Bland-Altman method, the mean differences between AHI and ODI scores were 2.6 ± 4.8 events/h (NP signal), 0.3 ± 2.7 events/h (transformed NP), and 1.1 ± 2.3 events/h (RIP flow). Tables S1–S3 in the supplemental material display the means and the standard deviations (SD) of the scoring of
R ES U LT S Figure 1 displays the absolute values of the scoring of the AHI of all 15 HSTs by the nine scorers using the three different airflow signals. The mean ± SD of the AHI on the 15 HSTs was 28.0 ± 18.3 events/h (range: 6.3–105.5) using NP, 25.1 ± 16.7 events/h (range: 5.3–73.4) using transformed NP, and 24.0 ± 15.3 events/h (range: 1.7–67.5) using RIP flow.
Interrater Agreement
The interscorer agreement of AHI scoring among the SAGIC scorers was very strong using any of the following as the primary signal for airflow: NP, transformed NP, or RIP flow. As shown in Table 1, the ICCs of the respiratory event scoring using the NP were: AHI = 0.96 (95% CI: 0.93–0.99); apnea 73
Journal of Clinical Sleep Medicine, Vol. 12, No. 1, 2016
UJ Magalang, ES Arnardottir, NH Chen et al. Agreement in HST Scoring
Table 1—Interrater agreement of scoring respiratory events using different airflow signals Variable AHI (events/h) Apnea index OA index CA index MA index Hypopnea index ODI (events/h)
NP 0.96 (0.93–0.99) 0.91 (0.83–0.96) 0.90 (0.82–0.96) 0.62 (0.43–0.81) 0.56 (0.37–0.77) 0.75 (0.59–0.89) 0.98 (0.97–0.99)
Intraclass Correlation Coefficients Transformed NP 0.98 (0.96–0.99) 0.95 (0.90–0.98) 0.94 (0.89–0.98) 0.86 (0.75–0.94) 0.70 (0.52–0.86) 0.90 (0.82–0.96) 0.99 (0.98–0.99)
RIP Flow 0.98 (0.96–0.99) 0.66 (0.48–0.84) 0.60 (0.42–0.80) 0.68 (0.50–0.85) 0.48 (0.29–0.72) 0.78 (0.63–0.90) 0.98 (0.97–0.99)
Values in parenthesis represent the 95% confidence limits. AHI, apnea-hypopnea index; CA, central apnea; MA, mixed apnea; NP, nasal pressure; OA, obstructive apnea; ODI, oxygen desaturation index; RIP flow, flow using respiratory inductive plethysmography; transformed NP, square root transformation of NP signal.
Table 2—Intrarater agreement of scoring respiratory events using different airflow signals Variable AHI (events/h) Apnea index OA index CA index MA index Hypopnea index
MEANdiff 0.02 −0.14 0.17 −0.05 0.06 −0.14
NP SDdiff 1.53 2.22 2.55 0.33 0.34 2.22
LoA −2.98, 3.02 −4.49, 4.21 −4.82, 5.16 −0.69, 0.59 −0.60, 0.73 −4.49, 4.21
MEANdiff −0.26 −0.13 −0.12 −0.01 0.004 −0.13
Transformed NP SDdiff LoA 0.85 −1.91, 1.40 1.19 −2.46, 2.21 1.13 −2.33, 2.09 0.31 −0.61, 0.58 0.23 −0.44, 0.45 1.22 −2.53, 2.27
MEANdiff −0.40 0.75 0.68 −0.06 −0.04 −1.02
RIP Flow SDdiff 1.28 2.66 2.71 0.41 0.42 2.68
LoA −2.91, 2.10 −4.48, 5.96 −4.64, 6.01 −0.87, 0.75 −0.87, 0.78 −6.27, 4.24
AHI, apnea hypopnea index; CA, central apnea; LoA, limits of agreement; MA, mixed apnea; MEANdiff, mean difference between the first and second scoring sessions of the same respiratory variables; NP, nasal pressure; OA, obstructive apnea; ODI, oxygen desaturation index; RIP flow, flow using respiratory inductive plethysmography; transformed NP, square root transformation of NP signal.
Figure 2—Bland-Altman plot of the agreement between the apnea-hypopnea index (AHI) values using nasal pressure (NP) and the square root transformation of the NP signal (transformed NP).
intrarater agreement between the first and second scoring sessions in the scoring of all respiratory indices using any of the three airflow signals.
Comparison of AHI Using Different Airflow Signals
Overall, the mean differences (bias) in AHI scoring using the three airflow signals were small. Figure 2 shows the BlandAltman plot of the AHI values obtained using the NP and transformed NP signals (bias = 2.9 ± 3.6 events/h). The AHI values between the NP signal and RIP flow (Figure 3) had the greatest bias (4.0 ± 6.1 events/h), whereas the AHI values between the transformed NP and the RIP flow (Figure 4) had the least bias (1.1 ± 3.6 events/h). D I SCUS S I O N This study investigated HST scoring agreement of respiratory events across seven international centers within SAGIC. The major findings of this study are: (1) there is a very strong interrater agreement in the scoring of the AHI among experienced sleep technologists regardless of the type of airflow signal used (NP, transformed NP, or RIP flow); (2) there is strong agreement in the scoring of the apnea and hypopnea indices particularly when the transformed NP is used as the airflow signal; (3) there is excellent intrarater agreement between the first and second scoring sessions in the scoring of all respiratory indices
respiratory events in all 15 HSTs by the nine scorers using the NP, transformed NP, and RIP flow, respectively.
Intrarater Agreement
Table 2 shows the intrarater agreement of scoring respiratory events using the different airflow signals. The mean difference of first and second scoring sessions of the same respiratory variables ranged from −1.02 to 0.75/h. There was good Journal of Clinical Sleep Medicine, Vol. 12, No. 1, 2016
74
UJ Magalang, ES Arnardottir, NH Chen et al. Agreement in HST Scoring
Figure 3—Bland-Altman plot of the agreement between the apnea-hypopnea index (AHI) values using nasal pressure (NP) and respiratory flow using respiratory inductive plethysmography (RIP flow).
Figure 4—Bland-Altman plot of the agreement between the apnea-hypopnea index (AHI) values using the square root transformation of the NP signal (transformed NP) and respiratory flow using respiratory inductive plethysmography (RIP flow).
using any of the three airflow signals; and (4) the automated scoring of the ODI also shows very strong agreement despite using various scoring software to derive this index. To our knowledge, this is the first study examining the agreement of HST scoring of respiratory events among international sleep centers. Because HST is now increasingly used as the initial modality to diagnose OSA, the issue of agreement is not only important to researchers who are planning multicenter collaborative efforts but also to clinicians because such information is essential for evaluation of the findings and interpretation. As in our prior study of PSG scoring, the use of different scoring software across the SAGIC centers is a strength of this study because it provides a greater challenge to good agreement and is a real-world application.6 The current study extends our previous report that showed substantial agreement in the scoring of the respiratory events among international sleep centers working in the SAGIC consortium for in-laboratory PSGs. The results of the current study on HST scoring and our previous report on PSG scoring suggest an alternative approach to centralized scoring of sleep studies in research collaboration among international centers. Similar to our in-laboratory PSG scoring study, the agreement in the scoring of central and mixed apneas for HSTs was lower than for obstructive apneas. The reasons for these findings are unclear, but suggest that it is an area that needs to be emphasized in ongoing quality improvement of sleep study scoring among the SAGIC centers. However, it should also be noted that the number of mixed and central apneas in the HSTs was low. For HSTs, it appears that using the transformed NP signal improves the HST scoring of central and mixed apneas, although not to the level of agreement obtained in scoring obstructive apneas. The airflow derived by the square root transformation of the nasal pressure signal (transformed NP) is used by some centers for the scoring of respiratory events.10–12 Our results suggest that using the transformed NP may improve the agreement in the scoring of the hypopneas. The use of the RIP flow from the
thorax and abdominal belt signals, which is suggested as an alternative sensor for scoring respiratory events if the NP signal fails, also showed strong agreement in AHI scoring. To our knowledge, ours is the first study that has examined the AHI scoring agreement using RIP flow. Two prior studies examined the agreement between automatic and manual scoring of respiratory events in HSTs.9,24 The two studies suggest that the agreement between automatic and manual agreement is modest and that automatic agreement consistently underestimated the AHI derived from manual scoring of HSTs. None of these prior publications reported the agreement when different scoring software is used on the same studies, which is a strength of our current study given that this would be the real-world situation in collaborative research involving international sleep centers. Our study has some limitations. First, we used a ≥ 4% oxygen desaturation to define a hypopnea. An update of the AASM PSG scoring rules stipulate the following change in hypopnea definition: a drop of peak signal excursion by ≥ 30% of preevent baseline using the nasal pressure signal for duration of ≥ 10 sec with a ≥ 3% oxygen desaturation from preevent baseline or an associated arousal.12 However, we are aware of only two studies that have systematically examined the association of varying definitions of hypopnea and clinical outcomes.25,26 In a large epidemiologic study examining an outcome-based definition of hypopneas, only desaturations of ≥ 4% were independently associated with cardiovascular disease.25 In contrast, no association was observed between cardiovascular disease and hypopneas associated with milder desaturations (including 3%) or arousals in a population of 6,106 adults.25 A more recent study showed that using a ≥ 4% desaturation to define a hypopnea was most consistently associated with systolic blood pressure in 2,040 participants in the Multi-Ethnic Study of Atherosclerosis (MESA).26 The most robust studies looking at the association of OSA and hypertension27–30 or the association of OSA and cardiovascular events31 75
Journal of Clinical Sleep Medicine, Vol. 12, No. 1, 2016
UJ Magalang, ES Arnardottir, NH Chen et al. Agreement in HST Scoring
all utilized a definition of hypopnea that incorporates ≥ 4% oxyhemoglobin desaturation.27–31 Nonetheless, the effect of the AASM change in hypopnea definition on the interrater reliability of scoring respiratory events for HSTs among international centers is yet to be examined (excluding arousal assessment as it cannot be performed in typical type 3 portable devices). A second limitation is using 15 HSTs acquired from one SAGIC center. However, data acquisition in these HSTs used standard techniques that would not be expected to be different in other international centers given that the equipment used is a type 3 portable device. A third limitation of our study is that it was not specifically powered to examine scoring agreement in those with mild OSA (AHI ≥ 5 to < 15/h). Finally, we did not randomize the order of the scoring of respiratory events when using one of the three airflow signals: NP, transformed NP, and RIP flow. We could not entirely rule out a learning effect during the multiple scoring sessions that could have affected the interrater agreement. However, HSTs were scored with only one airflow signal visible at each scoring session. In addition, the intrarater agreement of respiratory indices scoring was all similarly excellent using the three different airflow signals, suggesting that any order effect on scoring was not significant. In summary, our findings show that there is a strong interrater and intrarater agreement in the scoring of the respiratory events among international sleep centers for HSTs in the SAGIC consortium. Use of the transformed NP as the signal for airflow appears to increase the agreement in the scoring of hypopneas. There is also strong agreement in AHI scoring when alternative signals such as RIP flow is used in scoring of HSTs. Taken together with our previous study on in-laboratory PSG scoring, our results suggest that centralized scoring of HSTs and PSGs may not be necessary in research collaboration among international sites provided adequate recordings are acquired and experienced scorers are involved.
R E FE R E N CES 1. Masa JF, Corral J, Pereira R, et al. Effectiveness of home respiratory polygraphy for the diagnosis of sleep apnoea and hypopnoea syndrome. Thorax 2011;66:567–73. 2. Whittle AT, Finch SP, Mortimore IL, MacKay TW, Douglas NJ. Use of home sleep studies for diagnosis of the sleep apnoea/hypopnoea syndrome. Thorax 1997;52:1068–73. 3. Kuna ST, Badr MS, Kimoff RJ, et al. An official ATS/AASM/ACCP/ERS workshop report: research priorities in ambulatory management of adults with obstructive sleep apnea. Proc Am Thorac Soc 2011;8:1–16. 4. Masa JF, Corral J, Pereira R, et al. Therapeutic decision-making for sleep apnea and hypopnea syndrome using home respiratory polygraphy: a large multicentric study. Am J Respir Crit Care Med 2011;184:964–71. 5. Flemons WW, Littner MR, Rowley JA, et al. Home diagnosis of sleep apnea: a systematic review of the literature. An evidence review cosponsored by the American Academy of Sleep Medicine, the American College of Chest Physicians, and the American Thoracic Society. Chest 2003;124:1543–79. 6. Magalang UJ, Chen NH, Cistulli PA, et al. Agreement in the scoring of respiratory events and sleep among international sleep centers. Sleep 2013;36:591–6. 7. McEvoy RD, Anderson CS, Antic NA, et al. The sleep apnea cardiovascular endpoints (SAVE) trial: rationale and start-up phase. J Thorac Dis 2010;2:138–43. 8. Hedner J, Grote L, Bonsignore M, et al. The European Sleep Apnoea Database (ESADA): report from 22 European sleep laboratories. Eur Respir J 2011;38:635–42. 9. Masa JF, Corral J, Pereira R, et al. Effectiveness of sequential automaticmanual home respiratory polygraphy scoring. Eur Respir J 2013;41:879–87. 10. Thurnheer R, Xie X, Bloch KE. Accuracy of nasal cannula pressure recordings for assessment of ventilation during sleep. Am J Respir Crit Care Med 2001;164:1914–9. 11. Farre R, Rigau J, Montserrat JM, Ballester E, Navajas D. Relevance of linearizing nasal prongs for assessing hypopneas and flow limitation during sleep. Am J Respir Crit Care Med 2001;163:494–7. 12. Berry RB, Budhiraja R, Gottlieb DJ, et al. Rules for scoring respiratory events in sleep: update of the 2007 AASM Manual for the Scoring of Sleep and Associated Events. Deliberations of the Sleep Apnea Definitions Task Force of the American Academy of Sleep Medicine. J Clin Sleep Med 2012;8:597–619. 13. Kemp B, Varri A, Rosa AC, Nielsen KD, Gade J. A simple format for exchange of digitized polygraphic recordings. Electroencephalogr Clin Neurophysiol 1992;82:391–3. 14. Iber C, Ancoli-Israel S, Chesson A, Quan SF. The AASM Manual for the Scoring of Sleep and Associated Events: Rules, Terminology and Technical Specifications, 1st ed. Westchester, IL: American Academy of Sleep Medicine, 2007. 15. Kuna ST, Gurubhagavatula I, Maislin G, et al. Noninferiority of functional outcome in ambulatory management of obstructive sleep apnea. Am J Respir Crit Care Mede 2011;183:1238–44. 16. Rosen CL, Auckley D, Benca R, et al. A multisite randomized trial of portable sleep studies and positive airway pressure autotitration versus laboratorybased polysomnography for the diagnosis and treatment of obstructive sleep apnea: the HomePAP study. Sleep 2012;35:757–67. 17. Dingli K, Coleman EL, Vennelle M, et al. Evaluation of a portable device for diagnosing the sleep apnoea/hypopnoea syndrome. Eur Respir J 2003;21:253–9. 18. Smith LA, Chong DW, Vennelle M, Denvir MA, Newby DE, Douglas NJ. Diagnosis of sleep-disordered breathing in patients with chronic heart failure: evaluation of a portable limited sleep study system. J Sleep Res 2007;16:428–35. 19. Purdy S, Berry R. Portable Monitoring. In: Lee-Chiong T, ed. Fundamentals of sleep technology. Philadelphia, PA: Lippincott Williams & Wilkins, 2012:482–500. 20. Munro BH. Statistical methods for health care research. 5th ed. Philadelphia, PA: Lippincott Williams & Wilkins, 2005:248–9.
A B B R E V I AT I O N S AASM, American Academy of Sleep Medicine AHI, apnea-hypopnea index CPAP, continuous positive airway pressure EDF, european data format HST, home sleep testing ICC, intraclass correlation coefficient LoA, limits of agreement MEANdiff, mean difference ODI, oxygen desaturation index OSA, obstructive sleep apnea PSG, polysomnography RIP, respiratory inductive plethysmography SAGIC, Sleep Apnea Global Interdisciplinary Consortium SD, standard deviation Transformed NP, square root transformation of the nasal pressure signal
Journal of Clinical Sleep Medicine, Vol. 12, No. 1, 2016
76
UJ Magalang, ES Arnardottir, NH Chen et al. Agreement in HST Scoring 21. Cheng JW, Tsai WC, Yu TY, Huang KY. Reproducibility of sonographic measurement of thickness and echogenicity of the plantar fascia. J Clin Ultrasound 2012;40:14–9. 22. Portney LG, Watkins MP. Foundations of clinical research. Applications to practice. New Jersey: Prentice Hall Inc, 2000:560–7. 23. Bland JM, Altman DG. Measuring agreement in method comparison studies. Stat Methods Med Res 1999;8:135–60. 24. Aurora RN, Swartz R, Punjabi NM. Misclassification of OSA severity with automated scoring of home sleep recordings. Chest 2015;147:719–27. 25. Punjabi NM, Newman AB, Young TB, Resnick HE, Sanders MH. Sleepdisordered breathing and cardiovascular disease: an outcome-based definition of hypopneas. Am J Respir Crit Care Med 2008;177:1150–5. 26. Dean DA, Wang R, Jacobs DR, et al. A systematic assessment of the association of polysomnographic indices with blood pressure: the Multi-Ethnic Study of Atherosclerosis (MESA). Sleep 2015;38:587–96. 27. Marin JM, Agusti A, Villar I, et al. Association between treated and untreated obstructive sleep apnea and risk of hypertension. JAMA 2012;307:2169–76. 28. Barbe F, Duran-Cantolla J, Sanchez-de-la-Torre M, et al. Effect of continuous positive airway pressure on the incidence of hypertension and cardiovascular events in nonsleepy patients with obstructive sleep apnea: a randomized controlled trial. JAMA 2012;307:2161–8. 29. Nieto FJ, Young TB, Lind BK, et al. Association of sleep-disordered breathing, sleep apnea, and hypertension in a large community-based study. Sleep Heart Health Study. JAMA 2000;283:1829–36. 30. Peppard PE, Young T, Palta M, Skatrud J. Prospective study of the association between sleep-disordered breathing and hypertension. N Engl J Med 2000;342:1378–84. 31. Marin JM, Carrizo SJ, Vicente E, Agusti AG. Long-term cardiovascular outcomes in men with obstructive sleep apnoea-hypopnoea with or without treatment with continuous positive airway pressure: an observational study. Lancet 2005;365:1046–53.
ACK N O W L E D G M E N T S The authors thank the following individuals who helped in this project: Mohammad Ahmadi, Alexander Blau, Petra Cornell, Silverio Garbuio, Su-Lan Liu, João Reinfelderon, Beth Staley, Magdalena Ósk Sigurgunnarsdóttir, and Sandra Zimmermann.
SUBM I SSI O N & CO R R ESPO NDENCE I NFO R M ATI O N Submitted for publication April, 2015 Submitted in final revised form July, 2015 Accepted for publication July, 2015 Address correspondence to: Ulysses J. Magalang MD, Division of Pulmonary, Allergy, Critical Care, and Sleep Medicine, The Ohio State University Wexner Medical Center, 201 Davis Heart and Lung Research Institute, 473 West 12th Avenue, Columbus, OH, 43210; Tel: (614) 292-6563; Fax: (614) 293-4799; Email [email protected].
D I SCLO S U R E S TAT E M E N T This was not an industry supported study. Support was provided by NHLBI award P01 HL094307 (AIP), HL093463 (UJM) and Award Number Grant UL1TR001070 from the National Center For Advancing Translational Sciences. Dr. Cistulli has received research support from MonoMed Ltd. and ResMed; has consulted for Exploramed Inc. and Zephyr Sleep Technologies; and has received royalties from Quintessence Publishing. Dr. Arnardottir has consulted for Nox Medical and Weinmann. Dr. Schwab has consulted for ApniCure, CryOSA, Foramis Medical Group, and ResMed. Dr. Magalang has received research support from Hill-Rom. Dr. Gíslason owns stock in and receives salary from Nox Medical. The other authors have indicated no financial conflicts of interest.
77
Journal of Clinical Sleep Medicine, Vol. 12, No. 1, 2016
