IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS,
VOL. 20,
NO. 3,
MARCH 2014
391
An Evaluation of Depth Enhancing Perceptual Cues for Vascular Volume Visualization in Neurosurgery Marta Kersten-Oertel, Sean Jy-Shyang Chen, and D. Louis Collins Abstract—Cerebral vascular images obtained through angiography are used by neurosurgeons for diagnosis, surgical planning, and intraoperative guidance. The intricate branching of the vessels and furcations, however, make the task of understanding the spatial three-dimensional layout of these images challenging. In this paper, we present empirical studies on the effect of different perceptual cues (fog, pseudo-chromadepth, kinetic depth, and depicting edges) both individually and in combination on the depth perception of cerebral vascular volumes and compare these to the cue of stereopsis. Two experiments with novices and one experiment with experts were performed. The results with novices showed that the pseudo-chromadepth and fog cues were stronger cues than that of stereopsis. Furthermore, the addition of the stereopsis cue to the other cues did not improve relative depth perception in cerebral vascular volumes. In contrast to novices, the experts also performed well with the edge cue. In terms of both novice and expert subjects, pseudo-chromadepth and fog allow for the best relative depth perception. By using such cues to improve depth perception of cerebral vasculature, we may improve diagnosis, surgical planning, and intraoperative guidance. Index Terms—Depth cues, stereo, chromadepth, fog, volume rendering, vascular data, vessels, angiography
Ç 1
INTRODUCTION
I
N many forms of angiographic imaging, a contrast substance is injected into a patient to enable visualization of vascular structures. Until recently, two-dimensional (2D) projective X-ray angiography was the gold standard for the visualization of blood vessels (Stancanello et al. [37]). Such 2D images by themselves, however, are insufficient to provide the three-dimensional (3D) information that is needed to get a good understanding of the structure and spatial layout of the blood vessels. At our institute, stereoscopic angiograms were used for over 50 years, but these only gave a qualitative impression of 3D (Peters et al. [30]). Furthermore, projective angiograms do not allow for examination of the vasculature from an arbitrary point of view. For these reasons, 3D imaging techniques (e.g., magnetic resonance angiography (MRA), computed tomography angiography (CTA), and 3D X-ray angiography (3DXA)), which overcome these shortcomings, are now more commonly used. In our work, we focus on cerebral angiograms. Cerebral angiography provides images of the vessels in and around the brain and is used for identifying and detecting such abnormalities as: cranial stenosis (abnormal narrowing or constriction in blood vessels), aneurysms (localized expansions of a blood vessel), and arteriovenous malformations (abnormal connections between arteries and veins).
. The authors are with the Department of Biomedical Engineering at McGill University in the McConnell Brain Imaging Center, Montreal Neurological Institute, WB-317, 3801 University Street, Montreal, QC H3A 2B4, Canada. E-mail: [email protected], {sjschen, louis}@bic.mni.mcgill.ca. Manuscript received 27 Nov. 2012; revised 12 Apr. 2013; accepted 26 Sept. 2013; published online 3 Oct. 2013. Recommended for acceptance by D. Bowman. For information on obtaining reprints of this article, please send e-mail to: [email protected], and reference IEEECS Log Number TVCG-2012-11-0265. Digital Object Identifier no. 10.1109/TVCG.2013.240. 1077-2626/14/$31.00 ß 2014 IEEE
Cerebral angiography data is difficult to understand spatially due to the intricate branching of the vessels and the overlapping and interweaving of the vessels at many different depths. As well, spatial depth perception of angiography data are difficult because 1) surgeons may not be able to rely only on previous knowledge and experience in the perception process of such images (due to the high variability of vessel structures and their abnormalities), 2) the perspective cue is not typically used due to the possible introduction of distortions in measurements taken from the angiogram (Osborn [28]), and 3) the lack of the binocular disparity cue, common to all viewing of 3D objects on a 2D screen (Ropinski et al. [35]). There has been a shortcoming of research examining the use of different visualization techniques in the context of image-guided surgery (Kersten-Oertel et al. [25]). In particular, there is a strong need for new and existing methods to be validated and evaluated to determine whether they are effective in providing fast, intuitive, and accurate understanding of the data, with better spatial and depth perception. With this in mind, the goal of our work is to study whether a number of perceptually driven visualization methods may provide good spatial understanding and proper depth perception of vessel volumes. To this effect, we have looked at the effectiveness of a number of monoscopic depth cues, in addition to the stereopsis cue. While some groups have proposed stereoscopic visualization in terms of a surgical stereoscopic microscope (e.g., Aschke et al. [2] and Edwards et al. [10]), autostereoscopic monitor (e.g., Paloc et al. [29] and Jing et al. [22]), stereoscopic head mounted display (HMD) microscope (e.g., Birkfellner et al. [5] and Figl et al. [12]), or HMD (e.g., Fuchs et al. [14] and Bichlmeier et al. [4]) many surgeons find the additional hardware cumbersome Published by the IEEE Computer Society
392
IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS,
Fig. 1. The perceptive cues used in our experiments are shown.
and/or difficult to use within the constraints of the operating room. In this paper, we present the results of three psychophysical experiments, two completed with novice viewers and one with expert viewers. The novice subjects were unfamiliar with cerebral vasculature, whereas the experts were neurosurgical residents and surgeons. In the experiments, we evaluated and quantified the effectiveness of a number of perceptually driven visualization methods. In particular, we tried to answer the following two questions: 1) What depth cues work best to give a good spatial percept of cerebral vascular data? and 2) Are there monoscopic cues that can be used for rendering that give us the same or a better perception of relative depth compared to stereoscopic rendering? The results of these studies will give us a better understanding of how to display volumetric angiography data for diagnosis, surgical planning, and intraoperative guidance.
2
CUES FOR DEPTH PERCEPTION
The cues used to perceive depth by humans in everyday life have been thoroughly studied. For computer-generated images, however, what cues and the way in which these cues should be combined to convey depth remains an ongoing topic of research. In the following section, we briefly describe each of the cues that were tested. The cues are depicted in Fig. 1.
2.1 Stereopsis Stereopsis is the perception of depth due to the viewing of an object or scene with both eyes. The separation of our left and right eyes (of about 6 cm) creates two slightly different images of the world. These differences (binocular disparity) are used by the brain to calculate the depth of objects within a scene. Here, we use the term stereoscopic to refer to viewing a scene on a 2D monitor with 3D glasses. Numerous works have looked at the use of stereopsis for enhanced perception of computer generated scenes. Some of these are [18], [19], [20], [24], [26], [11], [41], [42], [44], [43].
VOL. 20,
NO. 3,
MARCH 2014
2.2 Edges Edges are boundaries between objects that help us to determine the structure and shape of objects. The depiction of edges also helps us to perceive local occlusions. Occlusion, one of the strongest cues, refers to the ability to perceive an object that partially obscures another in view as being closer to us. With regular volume rendering techniques edges may be depicted, however, occlusions may be difficult to perceive when there is low contrast between the overlapping objects caused by similar surface shading. Thus, distinguishing whether one object is closer or further than another can be difficult. A number of researchers have examined the use of edges or lines for enhanced perception. Some examples are: Ma et al. [45] who looked at a gradient method for enhanced depth-ordering in volumes; Svakhine et al. [39] who developed a number of outlining techniques for providing perceptual cues of object importance and enhanced spatial understanding; Interrante et al. [21] and Fischeret al. [13] who looked at silhouettes and outlines for enhanced perception of volumes and Bruckner and Gro¨ller [7], [8] for enhanced depth perception of volumetric data. 2.3 Kinetic Depth Kinetic depth is a cue whereby motion is used to give threedimensional structural information about an object. A number of works have shown that motion can aid in structural and depth perception. For example, Ware and Frank [41] showed that motion, regardless of type, can increase the size of a graph that can be perceived and that motion and stereo together increase the size of the structure of graph that can be understood by an order of magnitude (Ware and Mitchell [42]). More recently, Boucheny et al. [6] showed that motion may aid in depth perception within transparent volumetric data when appropriate transfer functions are found. In our experiments, we use a slight continuous rotation of the object about the axis between the center of the object and the viewer. This movement also introduces the cue of occlusion as vessels shift providing the viewer with the ability to distinguish the depth of objects due to the movement of the vessels. A video of the kinetic depth cue is included with the supplemental material of this paper, which can be found on the Computer Society Digital Library at http://doi.ieeecomputersociety.org/10.1109/ TVCG.2013.240. 2.4 Aerial Perspective Aerial Perspective is the perception of depth due to the scattering of light in the atmosphere. Light that reflects off a close object does not scatter as much before reaching the eye, but the same light that reflects off of a distant object undergoes much scattering, causing the darker portions of the object to be reduced in contrast (Gibson [15]). Objects that exhibit more contrast are assumed to be closer than objects that do not. Aerial perspective was shown to be as effective a cue for visualization of digitally reconstructed radiographs (DRRs) as stereopsis (Kersten et al. [24]). We modeled the aerial perspective cue by decreasing contrast as distance from the viewer increased. This has the effect of adding an
KERSTEN-OERTEL ET AL.: AN EVALUATION OF DEPTH ENHANCING PERCEPTUAL CUES FOR VASCULAR VOLUME VISUALIZATION IN...
approximate “uniform scattering” effect, as is present in the atmosphere. Aerial perspective is implemented using the OpenGL fog model and therefore, we refer to it as fog hence forth.
2.5 Chromadepth A chromadepth rendering, developed by Steenblik [38], uses color to represent distances in a way that follows the visible light spectrum. Near objects are red, going to orange, yellow, green, and blue, which are far away. The advantage of chromadepth over 3D stereoscopic viewing is that the depth information is encoded within a single image (Toutin [40], Bailey and Clark [3]); therefore ghosting, sensitivity to the viewer’s head position, and other issues that are present when using 3D glasses or stereoscopic hardware are avoided. Based on the results of Ropinski et al. [35] and Joshi et al. [23], we use a pseudo-chromadepth rendering that limits the hues to go from red (close to the viewer) through magenta, purple, and to blue (far away from the viewer), rather than the entire visible light spectrum.
3
RELATED WORK
Monoscopic depth cues including chromadepth, occlusion, depth of focus, and simulated perspective projection were studied to enhance perception of cerebral angiography images by Ropinski et al. [35]. In their psychophysical study, where subjects were to determine which of two vessels was closer to them, the authors found that a pseudo chromadepth cue (which uses fewer hues than chromadepth) allowed subjects to more accurately and more quickly determine the correct answer. However, occlusion, which is often considered the strongest cue, and in this case manifested by using overlaid edges such that the nearer a vessel the thicker the edge of that vessel, was found to be most effective in allowing users to determine which vessel was closest to them. The depth of focus cue that assumes that a viewer focuses on vessels nearer to them than those more distant was implemented by rendering vessels that were close in focus while those further were rendered out of focus. Although the cue gave good results in terms of correctness, response time increased. In general, blurring or reducing focus of medical images may not a good idea as it can be confusing for users causing them to take longer to determine if the given image is in focus, and frustrating that they cannot bring parts of the object into focus. A number of shortcomings were pointed out by the authors in this previous work. The authors believed that the subjects had problems perceiving stereoscopic pairs, and were not aware of how the chromadepth color coding worked due to an insufficient explanation and lack of training. As well as testing a number of new cues, in this work we have tried to improve on the work of Ropinski et al. [35] by ensuring all subjects could fuse and perceive the stereoscopic pairs and training the subjects so that they understood how each of the cues worked beforehand. Joshi et al. [23] developed a vessel segmentation method that used the intensity profile of a voxel to compute the likelihood that the voxel belongs to the vessel. The authors evaluated different visualization techniques of
393
their segmented vessel volumes by performing a user study with neurosurgeons, radiologists, and other expert users. Twelve experts performed an online survey comparing volume rendered images, volume rendering images with lighting, distance color blended images, tone shaded images, and images with halos (Joshi et al. [23]). The results showed that distance color blending which the authors term chromadepth, was preferred 74 percent of the time over other techniques. Another approach to enhancing vessel perception has been to use nonphotorealistic rendering. Ritter et al. [33] and Hansen et al. [16] used distance-encoded silhouettes with varying stroke textures and edge thickness to allow better perception of important vessels and easier and more accurate judgment of distances between vessels. In particular, they used texture techniques such as hatching, distance-encoded surfaces (where strokes on vessels become less visible with distance), and shadows to enhance shape and topology information of the vessels. In our work, we focus on volume rendered data as such rendering provides all of the information within the blood vessels not simply a representative solid surface. Abhari et al. [1] studied the use of enhanced contours for the display of cerebral MR angiography data. The results of their study showed that when vessel contours are enhanced, subjects have both a greater level of accuracy and perform faster in judging the connectivity of different vessels. For an overview of 3D visualization of vasculature, focused on model-based and model-free approaches to visualization, the readers are directed to Preim and Oeltze [32].
4
METHODOLOGY
Our test data consisted of a 512 512 512 volume acquired through 3DXA. The vessels were segmented from the volume, using region growing from the MINC toolkit1 with a predefined threshold. The segmented vascular data set was volume rendered under orthographic projection using each of the five cues (kinetic depth, stereo, edges, chromadepth, and fog) in addition to using no cue. With no depth cues, the ability to distinguish the depth at which two vessels lie is ambiguous, and subjects should have an accuracy of 50 percent. A volume rendering with no edges and no cues is not unusual for vascular data sets, as it may provide a smoother, cleaner image which some surgeons prefer due to its similarity to traditional X-ray angiography images. In this latter case, surgeons depend on their anatomical knowledge to determine depth.
4.1 Apparatus The volume rendered images were generated on a 2.8-GHz Pentium 4 processor. They were displayed on a ViewSonic FuHzion VX2268wm 120-Hz 22-inch V3D series LCD monitor and viewed by the subjects using NVidia 3D Vision Proactive shutter glasses.2 A full screen window (1; 680 1; 050) was used. Subjects sat approximately 75 cm away from the monitor and an interpupilary distance of 1. http://www.bic.mni.mcgill.ca/ServicesSoftware/MINC. 2. http://www.nvidia.ca/object/3d-vision-professional-users.html.
394
IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS,
VOL. 20,
NO. 3,
MARCH 2014
6 cm was used to create the stereo pairs. Subjects wore the 3D glasses at all times in the experiment to ensure constant luminosity across trials. To present flicker-free images to subjects, two things were done. First, the maximum monitor refresh rate 120 Hz (60 Hz per eye) was used. Second, each experiment was conducted in a darkened room with no fluorescent lights that may have interfered with the infrared receptor of the glasses and caused flickering. No subjects reported flickering.
4.2 Stimuli For all tests, volume rendering was done using OpenGL 3D texture mapping. With the exception of the kinetic depth cue, implementation was done using OpenGL Architecture Review Board (ARB) vertex and fragment shader programs. To implement fog or simulated aerial perspective, we used the general volume rendering integral: Rs Z s1 ðuÞ du CðsÞ ðsÞ ds e s0 ; ð1Þ Lð!Þ ¼ s0
where CðsÞ is the radiance density and ðsÞ is the probability density (i.e., the probability per unit distance) of light being absorbed at location s. We then reduce contrast of distant parts of the volume by shifting radiance densities, CðsÞ, toward white (the background color). At distance d from the viewer, normalized so that d 2 ½0; . . . ; 1 for all points in the volume, we set CðsÞ ¼ ð1 kÞd;
ð2Þ
where k is the factor by which the contrast is reduced. For k ¼ 0, there is a complete loss of contrast at the back, whereas for k ¼ 1 there is no change in contrast. We achieve contrast reduction by using the OpenGL linear fog model and a fragment shader program. In our previous work [24], we found that relative depth perception contrast factors of between 0.75 and 0.8 were best. Based on these results and on visual inspection, we set k ¼ 0:77. A pseudo-chromadepth rendering was obtained in a similar fashion to fog rendering. Radiance densities were shifted based on their distance; red for the closer parts of the volume and to blue for further parts. A smooth interpolation from red to blue through hues of magenta and purple was done throughout the volume. The following equation, which was implemented using a fragment shader program, was used for chromadepth rendering: 8 d < dstart ; < kred ð3Þ CðsÞ ¼ ð1 kred Þd þ kblue d dstart d dend ; : kblue d > dend ; where kred is the red color component set to ð1:0; 0:0; 0:2Þ and kblue is the blue color component set to ð0:2; 0:0; 1:0Þ. The coefficients dstart and dend determine the extent of the color blending. For our experiments, they were set to 0.02 and 0.84, respectively. For the kinetic depth cue, we use an arcball implemented rotation based on Ken Shoemake’s ArcBall controller [36]. The volume is rotated along the circumference of a sphere perpendicular to the view axis while the focal point of the camera or viewer remains at the center of the segmented vessels. The movement mimics a user clicking
Fig. 2. An example of the experiment task. Subjects determined which vessel (indicated in green or orange) was closer to them. Here, the fog and edge cue is shown from the combined cue novice experiment and the vessel indicated with green is closer.
down on a point at the center of the volume and dragging the cursor to rotate the volume in circles (i.e., the arc between the two points is converted into a rotation using quaternions, where the axis of rotation is perpendicular to the plane containing the two vectors representing the points). In our implementation, we set the radius that adjusts the magnitude of the rotation to 10-OpenGL units within the OpenGL unit sphere/cube and the speed of the rotation to =10 radians/frame. This results in a constant and continuous rotation around the z-axis.
4.3 Experimental Task The effectiveness of a vascular visualization highly depends on accentuating spatial depth and increasing the ability of a viewer to separate important individual features such as vascular branches (Hansen et al. [16] and Ritter et al. [33]). Therefore, the task of our subjects was to differentiate the relative depth of two separate vessels. We looked at cues both that should allow for good spatial understanding (such as stereo viewing) and those that may reduce the task to be 2D solvable (i.e., not requiring 3D information by using hue and/or saturation differentiation). In the context of neurovascular surgery, visualizations that reduce the task to simple comparisons can be very useful. Although for some tasks connectivity and spatial layout are necessary, for others it is simply important to know the relative depth of a vessel with respect to another vessel or structure. The two separate vessels that subjects had to differentiate in depth were manually selected for each trial (prior to the experiment). Vessels were selected in such a way as to ensure a good distribution in both depth and xy separation for each cue. More specifically, vessel pairs were chosen to be either near, medium or far apart both on the screen and in depth. The subject was asked to determine which of two vessels that were indicated (using either orange or green) was closer to them, see Fig. 2. Subjects pressed “O” if the vessel pointed to with orange was closer and “G” if the green one was closer. Pressing the key recorded the response and, after a time interval of 5 seconds during which time the subjects were shown a blank screen, the next stimulus was presented. Subjects were timed and were asked to respond both as accurately and as quickly, as possible. Response time was considered as visualizations that provide quick
KERSTEN-OERTEL ET AL.: AN EVALUATION OF DEPTH ENHANCING PERCEPTUAL CUES FOR VASCULAR VOLUME VISUALIZATION IN...
395
relative depth information to help in localization of vessels under the pressure and constrains of the OR are needed. All subjects were first trained on how each of the cues worked and on the task. In all of the experiments, the stimuli were presented randomly, interleaving all cues (including stereoscopic and monoscopic images). The order in which cues were presented to each of the subjects was randomized. After the experiment, each subject rated the ease with which they could determine which vessel was closer to them on a 6-point Likert scale, with 1 being very difficult and 6 being very easy.
3. chromadepth, edges, and stereo, 4. fog and edges (see Fig. 2), 5. fog and stereo, and 6. fog, edges, and stereo. The task was the same as that of the novice single depth cue experiment. A power analysis showed that for an expected medium impact of cue on correctness, 18 subjects should be tested. We ran 19 subjects (seven female, 12 male, age 25-48 year). No subjects were familiar with cerebral vasculature and the experience with stereo viewing varied from little to very experienced.
5
5.3 Expert Experiment: Single Depth Cues Studies with experts that examine the relative depth of vessels can help determine effective visualization techniques for the operating room. In the operating room (with its particular time and interaction constraints), visualizations which provide quick relative depth information may help in localization of vessels when patient variability and vascular anatomies make prior knowledge of vessel locations less reliable. In the expert experiment, we had three neurosurgeons and three neurosurgical residents (all male, aged 33-59 year) perform a similar study as above in Section 5.1. Based on the results of the first two experiments with novice subjects, in this experiment only single depth cues with the exception of the kinetic depth cue were tested. Ten novel views of the data, which did not allow the subject to rely on anatomical knowledge (due to large zoom factors and unconventional angiography view angles), were used. It was confirmed with a neurosurgeon that with the chosen views, there was no anatomical contextual information and the surgeons were unable to tell that one data set was used. Each subject viewed each cue 10 times.
EXPERIMENTS
In the following section, a description of the three experiments is given, results are described in Section 6 and a discussion of the results is presented in Section 7. The first two experiments were run with novices; subjects without knowledge of cerebral vasculature. The novice subjects were graduate students, postdoctoral fellows, and software engineers in the Biomedical Engineering Department at our university. These subjects all had a background in medical imaging and processing, however, not in cerebral vascular anatomy or three-dimensional visualization. Experiments with novices were done because availability of neurosurgeons is limited, and we wanted to run a large number of experimental parameters. Therefore, we ran our first studies with novices, and used these results to fine-tune our shorter expert experiment with the most promising techniques (i.e., only single depth cues minus the kinetic depth cue).
5.1 Novice Experiment: Single Depth Cues In this experiment, we looked at the effect of each of the cues mentioned in Section 2 individually, in addition to providing no cue. Sixteen different views of the data set were used randomly, and each of the cues was presented to the subject 10 times for a total of 160 stimulus presentations for each subject. Due to the very different viewpoints, it was not possible to tell that only one data set was used (even for a subject somewhat familiar with cerebral vasculature). The distance at which two vessels were chosen for each of the cues and views was also varied. The order in which a subject viewed each of the cues and views was varied. Given the large size of the expected effect, a power analysis showed that we needed 13 subjects. For the first experiment, we ran 13 subjects (three female, 10 male, age 22-45 year). No subjects were familiar with cerebral vasculature and their experience with stereo viewing varied from little to very experienced. 5.2 Novice Experiment: Combined Depth Cues Based on the results of our first experiment, we looked at the effect of combining cues in the second experiment. We no longer considered the kinetic depth cue due to its poor performance and the fact that although subjects liked it, some mentioned they would have liked to control the motion themselves. We looked at the following combined cues: 1. 2.
chromadepth and edges, chromadepth and stereo,
6
RESULTS
For the experiments, we measured each subjects’ classification correctness and decision response times, and analyzed the data using analysis of variance (ANOVA) and posthoc Tukey honestly significant difference (HSD) tests. The JMP Statistical Software package3 and R4 were used. As well as looking at correctness and decision response time, we looked at the effect of the distance between the two vessels chosen on the correctness of identifying which vessel was closer. We consider two distance measures: distance in z or depth, and distance in xy or on the screen. We found no correlation between xy and z distance (r2 ¼ 0:01042, p ¼ 0:4377). Both xy distance and z distances were recoded into categorical variables where numerical distances fell into one of three levels: near, medium, and far.5 This encoding 3. http://www.jmp.com. 4. http://www.r-project.org. 5. For the novice experiments, the values of xy based on OpenGL unit cube were equally divided into one of the following OpenGL-defined ranges: near [0.027-0.182], medium [0.185-0.349] and far [0.375-0.638]. For z, the ranges were: near [0-0.06], medium [0.0730-234], and far [0.238-0.44]. For xy distance, the ranges were: near [0.027-0.182], medium [0.184-0.35], and far [0.375-0.64]. For the expert, experiment values for z fell in the range of near [0.0006-0.028], medium [0.031-0.079], and far [0.089-0.31].
396
IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS,
Fig. 3. Mean correctness results as a function of the depth cue used. Error bars represent the standard error. Chromadepth and fog resulted in the highest correctness percentage. The lines below the graph show the results of a Tukey HSD test. Cues that are not connected by same a line are significantly different at p
VOL. 20,
NO. 3,
MARCH 2014
391
An Evaluation of Depth Enhancing Perceptual Cues for Vascular Volume Visualization in Neurosurgery Marta Kersten-Oertel, Sean Jy-Shyang Chen, and D. Louis Collins Abstract—Cerebral vascular images obtained through angiography are used by neurosurgeons for diagnosis, surgical planning, and intraoperative guidance. The intricate branching of the vessels and furcations, however, make the task of understanding the spatial three-dimensional layout of these images challenging. In this paper, we present empirical studies on the effect of different perceptual cues (fog, pseudo-chromadepth, kinetic depth, and depicting edges) both individually and in combination on the depth perception of cerebral vascular volumes and compare these to the cue of stereopsis. Two experiments with novices and one experiment with experts were performed. The results with novices showed that the pseudo-chromadepth and fog cues were stronger cues than that of stereopsis. Furthermore, the addition of the stereopsis cue to the other cues did not improve relative depth perception in cerebral vascular volumes. In contrast to novices, the experts also performed well with the edge cue. In terms of both novice and expert subjects, pseudo-chromadepth and fog allow for the best relative depth perception. By using such cues to improve depth perception of cerebral vasculature, we may improve diagnosis, surgical planning, and intraoperative guidance. Index Terms—Depth cues, stereo, chromadepth, fog, volume rendering, vascular data, vessels, angiography
Ç 1
INTRODUCTION
I
N many forms of angiographic imaging, a contrast substance is injected into a patient to enable visualization of vascular structures. Until recently, two-dimensional (2D) projective X-ray angiography was the gold standard for the visualization of blood vessels (Stancanello et al. [37]). Such 2D images by themselves, however, are insufficient to provide the three-dimensional (3D) information that is needed to get a good understanding of the structure and spatial layout of the blood vessels. At our institute, stereoscopic angiograms were used for over 50 years, but these only gave a qualitative impression of 3D (Peters et al. [30]). Furthermore, projective angiograms do not allow for examination of the vasculature from an arbitrary point of view. For these reasons, 3D imaging techniques (e.g., magnetic resonance angiography (MRA), computed tomography angiography (CTA), and 3D X-ray angiography (3DXA)), which overcome these shortcomings, are now more commonly used. In our work, we focus on cerebral angiograms. Cerebral angiography provides images of the vessels in and around the brain and is used for identifying and detecting such abnormalities as: cranial stenosis (abnormal narrowing or constriction in blood vessels), aneurysms (localized expansions of a blood vessel), and arteriovenous malformations (abnormal connections between arteries and veins).
. The authors are with the Department of Biomedical Engineering at McGill University in the McConnell Brain Imaging Center, Montreal Neurological Institute, WB-317, 3801 University Street, Montreal, QC H3A 2B4, Canada. E-mail: [email protected], {sjschen, louis}@bic.mni.mcgill.ca. Manuscript received 27 Nov. 2012; revised 12 Apr. 2013; accepted 26 Sept. 2013; published online 3 Oct. 2013. Recommended for acceptance by D. Bowman. For information on obtaining reprints of this article, please send e-mail to: [email protected], and reference IEEECS Log Number TVCG-2012-11-0265. Digital Object Identifier no. 10.1109/TVCG.2013.240. 1077-2626/14/$31.00 ß 2014 IEEE
Cerebral angiography data is difficult to understand spatially due to the intricate branching of the vessels and the overlapping and interweaving of the vessels at many different depths. As well, spatial depth perception of angiography data are difficult because 1) surgeons may not be able to rely only on previous knowledge and experience in the perception process of such images (due to the high variability of vessel structures and their abnormalities), 2) the perspective cue is not typically used due to the possible introduction of distortions in measurements taken from the angiogram (Osborn [28]), and 3) the lack of the binocular disparity cue, common to all viewing of 3D objects on a 2D screen (Ropinski et al. [35]). There has been a shortcoming of research examining the use of different visualization techniques in the context of image-guided surgery (Kersten-Oertel et al. [25]). In particular, there is a strong need for new and existing methods to be validated and evaluated to determine whether they are effective in providing fast, intuitive, and accurate understanding of the data, with better spatial and depth perception. With this in mind, the goal of our work is to study whether a number of perceptually driven visualization methods may provide good spatial understanding and proper depth perception of vessel volumes. To this effect, we have looked at the effectiveness of a number of monoscopic depth cues, in addition to the stereopsis cue. While some groups have proposed stereoscopic visualization in terms of a surgical stereoscopic microscope (e.g., Aschke et al. [2] and Edwards et al. [10]), autostereoscopic monitor (e.g., Paloc et al. [29] and Jing et al. [22]), stereoscopic head mounted display (HMD) microscope (e.g., Birkfellner et al. [5] and Figl et al. [12]), or HMD (e.g., Fuchs et al. [14] and Bichlmeier et al. [4]) many surgeons find the additional hardware cumbersome Published by the IEEE Computer Society
392
IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS,
Fig. 1. The perceptive cues used in our experiments are shown.
and/or difficult to use within the constraints of the operating room. In this paper, we present the results of three psychophysical experiments, two completed with novice viewers and one with expert viewers. The novice subjects were unfamiliar with cerebral vasculature, whereas the experts were neurosurgical residents and surgeons. In the experiments, we evaluated and quantified the effectiveness of a number of perceptually driven visualization methods. In particular, we tried to answer the following two questions: 1) What depth cues work best to give a good spatial percept of cerebral vascular data? and 2) Are there monoscopic cues that can be used for rendering that give us the same or a better perception of relative depth compared to stereoscopic rendering? The results of these studies will give us a better understanding of how to display volumetric angiography data for diagnosis, surgical planning, and intraoperative guidance.
2
CUES FOR DEPTH PERCEPTION
The cues used to perceive depth by humans in everyday life have been thoroughly studied. For computer-generated images, however, what cues and the way in which these cues should be combined to convey depth remains an ongoing topic of research. In the following section, we briefly describe each of the cues that were tested. The cues are depicted in Fig. 1.
2.1 Stereopsis Stereopsis is the perception of depth due to the viewing of an object or scene with both eyes. The separation of our left and right eyes (of about 6 cm) creates two slightly different images of the world. These differences (binocular disparity) are used by the brain to calculate the depth of objects within a scene. Here, we use the term stereoscopic to refer to viewing a scene on a 2D monitor with 3D glasses. Numerous works have looked at the use of stereopsis for enhanced perception of computer generated scenes. Some of these are [18], [19], [20], [24], [26], [11], [41], [42], [44], [43].
VOL. 20,
NO. 3,
MARCH 2014
2.2 Edges Edges are boundaries between objects that help us to determine the structure and shape of objects. The depiction of edges also helps us to perceive local occlusions. Occlusion, one of the strongest cues, refers to the ability to perceive an object that partially obscures another in view as being closer to us. With regular volume rendering techniques edges may be depicted, however, occlusions may be difficult to perceive when there is low contrast between the overlapping objects caused by similar surface shading. Thus, distinguishing whether one object is closer or further than another can be difficult. A number of researchers have examined the use of edges or lines for enhanced perception. Some examples are: Ma et al. [45] who looked at a gradient method for enhanced depth-ordering in volumes; Svakhine et al. [39] who developed a number of outlining techniques for providing perceptual cues of object importance and enhanced spatial understanding; Interrante et al. [21] and Fischeret al. [13] who looked at silhouettes and outlines for enhanced perception of volumes and Bruckner and Gro¨ller [7], [8] for enhanced depth perception of volumetric data. 2.3 Kinetic Depth Kinetic depth is a cue whereby motion is used to give threedimensional structural information about an object. A number of works have shown that motion can aid in structural and depth perception. For example, Ware and Frank [41] showed that motion, regardless of type, can increase the size of a graph that can be perceived and that motion and stereo together increase the size of the structure of graph that can be understood by an order of magnitude (Ware and Mitchell [42]). More recently, Boucheny et al. [6] showed that motion may aid in depth perception within transparent volumetric data when appropriate transfer functions are found. In our experiments, we use a slight continuous rotation of the object about the axis between the center of the object and the viewer. This movement also introduces the cue of occlusion as vessels shift providing the viewer with the ability to distinguish the depth of objects due to the movement of the vessels. A video of the kinetic depth cue is included with the supplemental material of this paper, which can be found on the Computer Society Digital Library at http://doi.ieeecomputersociety.org/10.1109/ TVCG.2013.240. 2.4 Aerial Perspective Aerial Perspective is the perception of depth due to the scattering of light in the atmosphere. Light that reflects off a close object does not scatter as much before reaching the eye, but the same light that reflects off of a distant object undergoes much scattering, causing the darker portions of the object to be reduced in contrast (Gibson [15]). Objects that exhibit more contrast are assumed to be closer than objects that do not. Aerial perspective was shown to be as effective a cue for visualization of digitally reconstructed radiographs (DRRs) as stereopsis (Kersten et al. [24]). We modeled the aerial perspective cue by decreasing contrast as distance from the viewer increased. This has the effect of adding an
KERSTEN-OERTEL ET AL.: AN EVALUATION OF DEPTH ENHANCING PERCEPTUAL CUES FOR VASCULAR VOLUME VISUALIZATION IN...
approximate “uniform scattering” effect, as is present in the atmosphere. Aerial perspective is implemented using the OpenGL fog model and therefore, we refer to it as fog hence forth.
2.5 Chromadepth A chromadepth rendering, developed by Steenblik [38], uses color to represent distances in a way that follows the visible light spectrum. Near objects are red, going to orange, yellow, green, and blue, which are far away. The advantage of chromadepth over 3D stereoscopic viewing is that the depth information is encoded within a single image (Toutin [40], Bailey and Clark [3]); therefore ghosting, sensitivity to the viewer’s head position, and other issues that are present when using 3D glasses or stereoscopic hardware are avoided. Based on the results of Ropinski et al. [35] and Joshi et al. [23], we use a pseudo-chromadepth rendering that limits the hues to go from red (close to the viewer) through magenta, purple, and to blue (far away from the viewer), rather than the entire visible light spectrum.
3
RELATED WORK
Monoscopic depth cues including chromadepth, occlusion, depth of focus, and simulated perspective projection were studied to enhance perception of cerebral angiography images by Ropinski et al. [35]. In their psychophysical study, where subjects were to determine which of two vessels was closer to them, the authors found that a pseudo chromadepth cue (which uses fewer hues than chromadepth) allowed subjects to more accurately and more quickly determine the correct answer. However, occlusion, which is often considered the strongest cue, and in this case manifested by using overlaid edges such that the nearer a vessel the thicker the edge of that vessel, was found to be most effective in allowing users to determine which vessel was closest to them. The depth of focus cue that assumes that a viewer focuses on vessels nearer to them than those more distant was implemented by rendering vessels that were close in focus while those further were rendered out of focus. Although the cue gave good results in terms of correctness, response time increased. In general, blurring or reducing focus of medical images may not a good idea as it can be confusing for users causing them to take longer to determine if the given image is in focus, and frustrating that they cannot bring parts of the object into focus. A number of shortcomings were pointed out by the authors in this previous work. The authors believed that the subjects had problems perceiving stereoscopic pairs, and were not aware of how the chromadepth color coding worked due to an insufficient explanation and lack of training. As well as testing a number of new cues, in this work we have tried to improve on the work of Ropinski et al. [35] by ensuring all subjects could fuse and perceive the stereoscopic pairs and training the subjects so that they understood how each of the cues worked beforehand. Joshi et al. [23] developed a vessel segmentation method that used the intensity profile of a voxel to compute the likelihood that the voxel belongs to the vessel. The authors evaluated different visualization techniques of
393
their segmented vessel volumes by performing a user study with neurosurgeons, radiologists, and other expert users. Twelve experts performed an online survey comparing volume rendered images, volume rendering images with lighting, distance color blended images, tone shaded images, and images with halos (Joshi et al. [23]). The results showed that distance color blending which the authors term chromadepth, was preferred 74 percent of the time over other techniques. Another approach to enhancing vessel perception has been to use nonphotorealistic rendering. Ritter et al. [33] and Hansen et al. [16] used distance-encoded silhouettes with varying stroke textures and edge thickness to allow better perception of important vessels and easier and more accurate judgment of distances between vessels. In particular, they used texture techniques such as hatching, distance-encoded surfaces (where strokes on vessels become less visible with distance), and shadows to enhance shape and topology information of the vessels. In our work, we focus on volume rendered data as such rendering provides all of the information within the blood vessels not simply a representative solid surface. Abhari et al. [1] studied the use of enhanced contours for the display of cerebral MR angiography data. The results of their study showed that when vessel contours are enhanced, subjects have both a greater level of accuracy and perform faster in judging the connectivity of different vessels. For an overview of 3D visualization of vasculature, focused on model-based and model-free approaches to visualization, the readers are directed to Preim and Oeltze [32].
4
METHODOLOGY
Our test data consisted of a 512 512 512 volume acquired through 3DXA. The vessels were segmented from the volume, using region growing from the MINC toolkit1 with a predefined threshold. The segmented vascular data set was volume rendered under orthographic projection using each of the five cues (kinetic depth, stereo, edges, chromadepth, and fog) in addition to using no cue. With no depth cues, the ability to distinguish the depth at which two vessels lie is ambiguous, and subjects should have an accuracy of 50 percent. A volume rendering with no edges and no cues is not unusual for vascular data sets, as it may provide a smoother, cleaner image which some surgeons prefer due to its similarity to traditional X-ray angiography images. In this latter case, surgeons depend on their anatomical knowledge to determine depth.
4.1 Apparatus The volume rendered images were generated on a 2.8-GHz Pentium 4 processor. They were displayed on a ViewSonic FuHzion VX2268wm 120-Hz 22-inch V3D series LCD monitor and viewed by the subjects using NVidia 3D Vision Proactive shutter glasses.2 A full screen window (1; 680 1; 050) was used. Subjects sat approximately 75 cm away from the monitor and an interpupilary distance of 1. http://www.bic.mni.mcgill.ca/ServicesSoftware/MINC. 2. http://www.nvidia.ca/object/3d-vision-professional-users.html.
394
IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS,
VOL. 20,
NO. 3,
MARCH 2014
6 cm was used to create the stereo pairs. Subjects wore the 3D glasses at all times in the experiment to ensure constant luminosity across trials. To present flicker-free images to subjects, two things were done. First, the maximum monitor refresh rate 120 Hz (60 Hz per eye) was used. Second, each experiment was conducted in a darkened room with no fluorescent lights that may have interfered with the infrared receptor of the glasses and caused flickering. No subjects reported flickering.
4.2 Stimuli For all tests, volume rendering was done using OpenGL 3D texture mapping. With the exception of the kinetic depth cue, implementation was done using OpenGL Architecture Review Board (ARB) vertex and fragment shader programs. To implement fog or simulated aerial perspective, we used the general volume rendering integral: Rs Z s1 ðuÞ du CðsÞ ðsÞ ds e s0 ; ð1Þ Lð!Þ ¼ s0
where CðsÞ is the radiance density and ðsÞ is the probability density (i.e., the probability per unit distance) of light being absorbed at location s. We then reduce contrast of distant parts of the volume by shifting radiance densities, CðsÞ, toward white (the background color). At distance d from the viewer, normalized so that d 2 ½0; . . . ; 1 for all points in the volume, we set CðsÞ ¼ ð1 kÞd;
ð2Þ
where k is the factor by which the contrast is reduced. For k ¼ 0, there is a complete loss of contrast at the back, whereas for k ¼ 1 there is no change in contrast. We achieve contrast reduction by using the OpenGL linear fog model and a fragment shader program. In our previous work [24], we found that relative depth perception contrast factors of between 0.75 and 0.8 were best. Based on these results and on visual inspection, we set k ¼ 0:77. A pseudo-chromadepth rendering was obtained in a similar fashion to fog rendering. Radiance densities were shifted based on their distance; red for the closer parts of the volume and to blue for further parts. A smooth interpolation from red to blue through hues of magenta and purple was done throughout the volume. The following equation, which was implemented using a fragment shader program, was used for chromadepth rendering: 8 d < dstart ; < kred ð3Þ CðsÞ ¼ ð1 kred Þd þ kblue d dstart d dend ; : kblue d > dend ; where kred is the red color component set to ð1:0; 0:0; 0:2Þ and kblue is the blue color component set to ð0:2; 0:0; 1:0Þ. The coefficients dstart and dend determine the extent of the color blending. For our experiments, they were set to 0.02 and 0.84, respectively. For the kinetic depth cue, we use an arcball implemented rotation based on Ken Shoemake’s ArcBall controller [36]. The volume is rotated along the circumference of a sphere perpendicular to the view axis while the focal point of the camera or viewer remains at the center of the segmented vessels. The movement mimics a user clicking
Fig. 2. An example of the experiment task. Subjects determined which vessel (indicated in green or orange) was closer to them. Here, the fog and edge cue is shown from the combined cue novice experiment and the vessel indicated with green is closer.
down on a point at the center of the volume and dragging the cursor to rotate the volume in circles (i.e., the arc between the two points is converted into a rotation using quaternions, where the axis of rotation is perpendicular to the plane containing the two vectors representing the points). In our implementation, we set the radius that adjusts the magnitude of the rotation to 10-OpenGL units within the OpenGL unit sphere/cube and the speed of the rotation to =10 radians/frame. This results in a constant and continuous rotation around the z-axis.
4.3 Experimental Task The effectiveness of a vascular visualization highly depends on accentuating spatial depth and increasing the ability of a viewer to separate important individual features such as vascular branches (Hansen et al. [16] and Ritter et al. [33]). Therefore, the task of our subjects was to differentiate the relative depth of two separate vessels. We looked at cues both that should allow for good spatial understanding (such as stereo viewing) and those that may reduce the task to be 2D solvable (i.e., not requiring 3D information by using hue and/or saturation differentiation). In the context of neurovascular surgery, visualizations that reduce the task to simple comparisons can be very useful. Although for some tasks connectivity and spatial layout are necessary, for others it is simply important to know the relative depth of a vessel with respect to another vessel or structure. The two separate vessels that subjects had to differentiate in depth were manually selected for each trial (prior to the experiment). Vessels were selected in such a way as to ensure a good distribution in both depth and xy separation for each cue. More specifically, vessel pairs were chosen to be either near, medium or far apart both on the screen and in depth. The subject was asked to determine which of two vessels that were indicated (using either orange or green) was closer to them, see Fig. 2. Subjects pressed “O” if the vessel pointed to with orange was closer and “G” if the green one was closer. Pressing the key recorded the response and, after a time interval of 5 seconds during which time the subjects were shown a blank screen, the next stimulus was presented. Subjects were timed and were asked to respond both as accurately and as quickly, as possible. Response time was considered as visualizations that provide quick
KERSTEN-OERTEL ET AL.: AN EVALUATION OF DEPTH ENHANCING PERCEPTUAL CUES FOR VASCULAR VOLUME VISUALIZATION IN...
395
relative depth information to help in localization of vessels under the pressure and constrains of the OR are needed. All subjects were first trained on how each of the cues worked and on the task. In all of the experiments, the stimuli were presented randomly, interleaving all cues (including stereoscopic and monoscopic images). The order in which cues were presented to each of the subjects was randomized. After the experiment, each subject rated the ease with which they could determine which vessel was closer to them on a 6-point Likert scale, with 1 being very difficult and 6 being very easy.
3. chromadepth, edges, and stereo, 4. fog and edges (see Fig. 2), 5. fog and stereo, and 6. fog, edges, and stereo. The task was the same as that of the novice single depth cue experiment. A power analysis showed that for an expected medium impact of cue on correctness, 18 subjects should be tested. We ran 19 subjects (seven female, 12 male, age 25-48 year). No subjects were familiar with cerebral vasculature and the experience with stereo viewing varied from little to very experienced.
5
5.3 Expert Experiment: Single Depth Cues Studies with experts that examine the relative depth of vessels can help determine effective visualization techniques for the operating room. In the operating room (with its particular time and interaction constraints), visualizations which provide quick relative depth information may help in localization of vessels when patient variability and vascular anatomies make prior knowledge of vessel locations less reliable. In the expert experiment, we had three neurosurgeons and three neurosurgical residents (all male, aged 33-59 year) perform a similar study as above in Section 5.1. Based on the results of the first two experiments with novice subjects, in this experiment only single depth cues with the exception of the kinetic depth cue were tested. Ten novel views of the data, which did not allow the subject to rely on anatomical knowledge (due to large zoom factors and unconventional angiography view angles), were used. It was confirmed with a neurosurgeon that with the chosen views, there was no anatomical contextual information and the surgeons were unable to tell that one data set was used. Each subject viewed each cue 10 times.
EXPERIMENTS
In the following section, a description of the three experiments is given, results are described in Section 6 and a discussion of the results is presented in Section 7. The first two experiments were run with novices; subjects without knowledge of cerebral vasculature. The novice subjects were graduate students, postdoctoral fellows, and software engineers in the Biomedical Engineering Department at our university. These subjects all had a background in medical imaging and processing, however, not in cerebral vascular anatomy or three-dimensional visualization. Experiments with novices were done because availability of neurosurgeons is limited, and we wanted to run a large number of experimental parameters. Therefore, we ran our first studies with novices, and used these results to fine-tune our shorter expert experiment with the most promising techniques (i.e., only single depth cues minus the kinetic depth cue).
5.1 Novice Experiment: Single Depth Cues In this experiment, we looked at the effect of each of the cues mentioned in Section 2 individually, in addition to providing no cue. Sixteen different views of the data set were used randomly, and each of the cues was presented to the subject 10 times for a total of 160 stimulus presentations for each subject. Due to the very different viewpoints, it was not possible to tell that only one data set was used (even for a subject somewhat familiar with cerebral vasculature). The distance at which two vessels were chosen for each of the cues and views was also varied. The order in which a subject viewed each of the cues and views was varied. Given the large size of the expected effect, a power analysis showed that we needed 13 subjects. For the first experiment, we ran 13 subjects (three female, 10 male, age 22-45 year). No subjects were familiar with cerebral vasculature and their experience with stereo viewing varied from little to very experienced. 5.2 Novice Experiment: Combined Depth Cues Based on the results of our first experiment, we looked at the effect of combining cues in the second experiment. We no longer considered the kinetic depth cue due to its poor performance and the fact that although subjects liked it, some mentioned they would have liked to control the motion themselves. We looked at the following combined cues: 1. 2.
chromadepth and edges, chromadepth and stereo,
6
RESULTS
For the experiments, we measured each subjects’ classification correctness and decision response times, and analyzed the data using analysis of variance (ANOVA) and posthoc Tukey honestly significant difference (HSD) tests. The JMP Statistical Software package3 and R4 were used. As well as looking at correctness and decision response time, we looked at the effect of the distance between the two vessels chosen on the correctness of identifying which vessel was closer. We consider two distance measures: distance in z or depth, and distance in xy or on the screen. We found no correlation between xy and z distance (r2 ¼ 0:01042, p ¼ 0:4377). Both xy distance and z distances were recoded into categorical variables where numerical distances fell into one of three levels: near, medium, and far.5 This encoding 3. http://www.jmp.com. 4. http://www.r-project.org. 5. For the novice experiments, the values of xy based on OpenGL unit cube were equally divided into one of the following OpenGL-defined ranges: near [0.027-0.182], medium [0.185-0.349] and far [0.375-0.638]. For z, the ranges were: near [0-0.06], medium [0.0730-234], and far [0.238-0.44]. For xy distance, the ranges were: near [0.027-0.182], medium [0.184-0.35], and far [0.375-0.64]. For the expert, experiment values for z fell in the range of near [0.0006-0.028], medium [0.031-0.079], and far [0.089-0.31].
396
IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS,
Fig. 3. Mean correctness results as a function of the depth cue used. Error bars represent the standard error. Chromadepth and fog resulted in the highest correctness percentage. The lines below the graph show the results of a Tukey HSD test. Cues that are not connected by same a line are significantly different at p
