Downloaded from http://jnis.bmj.com/ on December 17, 2015 - Published by group.bmj.com
Hemorrhagic stroke
ORIGINAL RESEARCH
Interobserver variability of aneurysm morphology: discrimination of the daughter sac Sang Hyun Suh,1 Harry J Cloft,2 John Huston III,2 Kyung Hwa Han,3 David F Kallmes2 1
Department of Radiology, Gangnam Severance Hospital, Yonsei University, Seoul, Korea 2 Department of Radiology, Mayo Clinic, Rochester, Minnesota, USA 3 Biostatistics Collaboration Unit, Gangnam Medical Research Center, Yonsei University College of Medicine, Seoul, Korea Correspondence to Dr S H Suh, Department of Radiology, Gangnam Severance Hospital, Yonsei University, 146-92 Dogok-dong Gangnam-gu, Seoul 135-720, Korea; [email protected] Received 18 September 2014 Revised 21 October 2014 Accepted 23 October 2014 Published Online First 10 November 2014
ABSTRACT Objective Several definitions have been proposed to distinguish the daughter sac when treating unruptured intracranial aneurysms. The aim of this study was to evaluate interobserver variability of aneurysm morphology, including the daughter sac, using criteria from the International Study of Unruptured Intracranial Aneurysms (ISUIA) and the Unruptured Cerebral Aneurysm Study of Japan (UCAS). Materials and methods After approval by the institutional review board, we analyzed three morphological features (daughter sac, lobulation, and irregular margin) from the ISUIA and UCAS using angiographic images from 102 saccular aneurysms. Four independent readers interpreted each morphological criterion using dichotomized scales (existence or not). The κ statistic was used to measure interobserver agreement, and κ>0.6 was considered substantial agreement. Results For discrimination of the daughter sac, interobserver agreement among the four readers was substantial using the UCAS criteria (k=0.626 for twodimensional (2D) and 0.659 for three-dimensional (3D) images) but not for the ISUIA criteria (k=0.487 for 2D and 0.473 for 3D images; significant difference). Irrespective of the images used, pairwise pooled κ values for the UCAS were >0.6, except for one case (score of 0.54 between readers A and B). Regarding the proportion of positive reads, there was a significant difference between reads for the daughter sac using the UCAS and ISUIA criteria. Conclusions For discrimination of the daughter sac, the UCAS definition showed a higher reliability than the ISUIA. However, a further prospective study is necessary to validate this definition as the treatment standard for unruptured intracranial aneurysms.
INTRODUCTION
To cite: Suh SH, Cloft HJ, Huston J, et al. J NeuroIntervent Surg 2016;8:38–41. 38
With the advance of imaging technology, unruptured intracranial aneurysms (UIAs) can be detected more frequently, and treatment of these previously undetected aneurysms is rapidly increasing. However, the natural course of UIAs is still poorly understood and their management remains controversial. The International Study of Unruptured Intracranial Aneurysms (ISUIA)1 has proposed the size and location of the aneurysm as independent risk factors predicting rupture, and recently the Unruptured Cerebral Aneurysm Study of Japan (UCAS)2 has also reported that the natural course of UIAs varies with aneurysm morphology, especially the presence of a daughter sac (DS). However, the definition of aneurysm morphology
is still not clearly defined and is subject to interobserver variation. Although some quantitative or mixed descriptions have been used in previous studies3–7 to overcome this ambiguity, these descriptions have yet to be standardized. Therefore, the purpose of this study was to assess interobserver agreement of aneurysm morphology, including the presence of a DS, using the different morphological criteria from the UCAS and ISUIA.
MATERIALS AND METHODS Patients After approval by the institutional review board for this retrospective study, 100 patients with 102 saccular aneurysms (men:women 20:80; mean age 55.6 years, range 31–89 years), who underwent aneurysm coiling from January 2001 through March 2010, were randomly selected from the institutional database of the Mayo Clinic (Rochester, Minnesota, USA). All selected patients had undergone two-dimensional (2D) DSA and three-dimensional (3D) rotational angiography (3DRA) during their management. Of 102 aneurysms, the location was the anterior cerebral artery in 18 (18%), the internal carotid artery–posterior communicating artery in 21 (21%), the cavernous internal carotid artery in 40 (39%), the middle cerebral artery in 7 (7%), and the vertebrobasilar artery in 16 (15%). All aneurysms were saccular type.
Image acquisition Acquisition of 2D images was accomplished with angiograms of the internal carotid artery or vertebral artery using a 5 or 6 F catheter via femoral artery access. Three-dimensional as well as 2D images, including conventional anteroposterior and lateral projection and working projection, were obtained before the procedure. The working projection was decided based on the image that provided the best discrimination between the aneurysm and parent artery after 3DRA. Three-dimensional images were acquired using a biplane C arm digital angiography suite (Integris; Philips Medical Systems, Best, The Netherlands) with an FOV of 17.78 cm and a frame rate of 30 f/s. Images were obtained with a head end propeller C arm orientation at a rotational speed of 55°/s, covering +120° to -120°. Image acquisition was started 1–3 s after contrast injection (16 mL of non-ionic contrast medium via an injector with a velocity of 4 mL/s); image acquisition time was 4.4 s. Volume rendered reconstruction was performed with 100%
Suh SH, et al. J NeuroIntervent Surg 2016;8:38–41. doi:10.1136/neurintsurg-2014-011471
Downloaded from http://jnis.bmj.com/ on December 17, 2015 - Published by group.bmj.com
Hemorrhagic stroke magnification (a field of 37.56 cm2) and a matrix of 256 pixels3 using the 3DRA volumetric measurement of the system software (Philips Medical Systems). The threshold for the volume rendered image was fixed at the default value provided by the manufacturer. In all cases, the 2D and 3D images were obtained at the initial diagnosis. A total of 618 images (314 3D and 304 2D images) in JPEG format were selected from a Mayo Clinic image database by one interventional neuroradiologist (SHS). A file for online review and evaluation was made in Adobe PDF format, where 2D and 3D images of the appropriate projections were arrayed side by side for comparison. Two different files, one of 2D images and the other of 3D images, were made for interpretation.
Table 2 Pairwise difference in κ values between the morphological features
DS-UCAS vs margin DS-UCAS vs lobulation DS-UCAS vs DS-ISUIA Margin vs lobulation Margin vs DS-ISUIA Lobulation vs DS-ISUIA
Readers and image interpretation All images of 102 aneurysms were evaluated by four experienced readers who worked in two different centers in different countries (USA and Korea). There were three interventional neuroradiologists (HJC and DFK, with more than 10 years of experience; SHS, with 8 years of experience) and one neuroradiologist ( JH, with more than 10 years of experience). For aneurysm morphology, three morphological features were identified from the UCAS2 and ISUIA8: discrimination of a ‘DS’, ‘lobulation’, and ‘margin’ of the aneurysm. ‘DS’ was defined as “an irregular protrusion of the aneurysm wall on 2D or 3D images from MR angiography (MRA), 3D CT angiography (CTA), or DSA” in the UCAS (DS-UCAS), and as “a separate protuberance arising from the margin sac that is less than 25% of the total volume of the sac” in the ISUIA (DS-ISUIA). ‘Lobulation’ was defined as “a protuberance arising directly from the primary neck of the aneurysm or from the main body and representing 25% or more of the apparent volume of the main sac”. The ‘margin’ of the aneurysm was defined as a ‘smooth’ or ‘irregular’ shape of the aneurysm. Each reader made his own morphological assessment independently using a dichotomized scale ( present or not) for the three features, and repeated the second assessment using the same criteria and materials 2 months after the first reading for measurement of intraobserver variability. No additional training or knowledge was provided to the readers.
Statistical analysis Statistical analyses were performed using SAS (V.9.2; SAS Institute Inc, Cary, North Carolina, USA). To determine the degree of intraobserver and interobserver agreement, the κ statistic was used for each morphological feature, using 2D and 3D images. The κ value was interpreted as follows9: slight agreement,
Hemorrhagic stroke
ORIGINAL RESEARCH
Interobserver variability of aneurysm morphology: discrimination of the daughter sac Sang Hyun Suh,1 Harry J Cloft,2 John Huston III,2 Kyung Hwa Han,3 David F Kallmes2 1
Department of Radiology, Gangnam Severance Hospital, Yonsei University, Seoul, Korea 2 Department of Radiology, Mayo Clinic, Rochester, Minnesota, USA 3 Biostatistics Collaboration Unit, Gangnam Medical Research Center, Yonsei University College of Medicine, Seoul, Korea Correspondence to Dr S H Suh, Department of Radiology, Gangnam Severance Hospital, Yonsei University, 146-92 Dogok-dong Gangnam-gu, Seoul 135-720, Korea; [email protected] Received 18 September 2014 Revised 21 October 2014 Accepted 23 October 2014 Published Online First 10 November 2014
ABSTRACT Objective Several definitions have been proposed to distinguish the daughter sac when treating unruptured intracranial aneurysms. The aim of this study was to evaluate interobserver variability of aneurysm morphology, including the daughter sac, using criteria from the International Study of Unruptured Intracranial Aneurysms (ISUIA) and the Unruptured Cerebral Aneurysm Study of Japan (UCAS). Materials and methods After approval by the institutional review board, we analyzed three morphological features (daughter sac, lobulation, and irregular margin) from the ISUIA and UCAS using angiographic images from 102 saccular aneurysms. Four independent readers interpreted each morphological criterion using dichotomized scales (existence or not). The κ statistic was used to measure interobserver agreement, and κ>0.6 was considered substantial agreement. Results For discrimination of the daughter sac, interobserver agreement among the four readers was substantial using the UCAS criteria (k=0.626 for twodimensional (2D) and 0.659 for three-dimensional (3D) images) but not for the ISUIA criteria (k=0.487 for 2D and 0.473 for 3D images; significant difference). Irrespective of the images used, pairwise pooled κ values for the UCAS were >0.6, except for one case (score of 0.54 between readers A and B). Regarding the proportion of positive reads, there was a significant difference between reads for the daughter sac using the UCAS and ISUIA criteria. Conclusions For discrimination of the daughter sac, the UCAS definition showed a higher reliability than the ISUIA. However, a further prospective study is necessary to validate this definition as the treatment standard for unruptured intracranial aneurysms.
INTRODUCTION
To cite: Suh SH, Cloft HJ, Huston J, et al. J NeuroIntervent Surg 2016;8:38–41. 38
With the advance of imaging technology, unruptured intracranial aneurysms (UIAs) can be detected more frequently, and treatment of these previously undetected aneurysms is rapidly increasing. However, the natural course of UIAs is still poorly understood and their management remains controversial. The International Study of Unruptured Intracranial Aneurysms (ISUIA)1 has proposed the size and location of the aneurysm as independent risk factors predicting rupture, and recently the Unruptured Cerebral Aneurysm Study of Japan (UCAS)2 has also reported that the natural course of UIAs varies with aneurysm morphology, especially the presence of a daughter sac (DS). However, the definition of aneurysm morphology
is still not clearly defined and is subject to interobserver variation. Although some quantitative or mixed descriptions have been used in previous studies3–7 to overcome this ambiguity, these descriptions have yet to be standardized. Therefore, the purpose of this study was to assess interobserver agreement of aneurysm morphology, including the presence of a DS, using the different morphological criteria from the UCAS and ISUIA.
MATERIALS AND METHODS Patients After approval by the institutional review board for this retrospective study, 100 patients with 102 saccular aneurysms (men:women 20:80; mean age 55.6 years, range 31–89 years), who underwent aneurysm coiling from January 2001 through March 2010, were randomly selected from the institutional database of the Mayo Clinic (Rochester, Minnesota, USA). All selected patients had undergone two-dimensional (2D) DSA and three-dimensional (3D) rotational angiography (3DRA) during their management. Of 102 aneurysms, the location was the anterior cerebral artery in 18 (18%), the internal carotid artery–posterior communicating artery in 21 (21%), the cavernous internal carotid artery in 40 (39%), the middle cerebral artery in 7 (7%), and the vertebrobasilar artery in 16 (15%). All aneurysms were saccular type.
Image acquisition Acquisition of 2D images was accomplished with angiograms of the internal carotid artery or vertebral artery using a 5 or 6 F catheter via femoral artery access. Three-dimensional as well as 2D images, including conventional anteroposterior and lateral projection and working projection, were obtained before the procedure. The working projection was decided based on the image that provided the best discrimination between the aneurysm and parent artery after 3DRA. Three-dimensional images were acquired using a biplane C arm digital angiography suite (Integris; Philips Medical Systems, Best, The Netherlands) with an FOV of 17.78 cm and a frame rate of 30 f/s. Images were obtained with a head end propeller C arm orientation at a rotational speed of 55°/s, covering +120° to -120°. Image acquisition was started 1–3 s after contrast injection (16 mL of non-ionic contrast medium via an injector with a velocity of 4 mL/s); image acquisition time was 4.4 s. Volume rendered reconstruction was performed with 100%
Suh SH, et al. J NeuroIntervent Surg 2016;8:38–41. doi:10.1136/neurintsurg-2014-011471
Downloaded from http://jnis.bmj.com/ on December 17, 2015 - Published by group.bmj.com
Hemorrhagic stroke magnification (a field of 37.56 cm2) and a matrix of 256 pixels3 using the 3DRA volumetric measurement of the system software (Philips Medical Systems). The threshold for the volume rendered image was fixed at the default value provided by the manufacturer. In all cases, the 2D and 3D images were obtained at the initial diagnosis. A total of 618 images (314 3D and 304 2D images) in JPEG format were selected from a Mayo Clinic image database by one interventional neuroradiologist (SHS). A file for online review and evaluation was made in Adobe PDF format, where 2D and 3D images of the appropriate projections were arrayed side by side for comparison. Two different files, one of 2D images and the other of 3D images, were made for interpretation.
Table 2 Pairwise difference in κ values between the morphological features
DS-UCAS vs margin DS-UCAS vs lobulation DS-UCAS vs DS-ISUIA Margin vs lobulation Margin vs DS-ISUIA Lobulation vs DS-ISUIA
Readers and image interpretation All images of 102 aneurysms were evaluated by four experienced readers who worked in two different centers in different countries (USA and Korea). There were three interventional neuroradiologists (HJC and DFK, with more than 10 years of experience; SHS, with 8 years of experience) and one neuroradiologist ( JH, with more than 10 years of experience). For aneurysm morphology, three morphological features were identified from the UCAS2 and ISUIA8: discrimination of a ‘DS’, ‘lobulation’, and ‘margin’ of the aneurysm. ‘DS’ was defined as “an irregular protrusion of the aneurysm wall on 2D or 3D images from MR angiography (MRA), 3D CT angiography (CTA), or DSA” in the UCAS (DS-UCAS), and as “a separate protuberance arising from the margin sac that is less than 25% of the total volume of the sac” in the ISUIA (DS-ISUIA). ‘Lobulation’ was defined as “a protuberance arising directly from the primary neck of the aneurysm or from the main body and representing 25% or more of the apparent volume of the main sac”. The ‘margin’ of the aneurysm was defined as a ‘smooth’ or ‘irregular’ shape of the aneurysm. Each reader made his own morphological assessment independently using a dichotomized scale ( present or not) for the three features, and repeated the second assessment using the same criteria and materials 2 months after the first reading for measurement of intraobserver variability. No additional training or knowledge was provided to the readers.
Statistical analysis Statistical analyses were performed using SAS (V.9.2; SAS Institute Inc, Cary, North Carolina, USA). To determine the degree of intraobserver and interobserver agreement, the κ statistic was used for each morphological feature, using 2D and 3D images. The κ value was interpreted as follows9: slight agreement,
