Brain Topography, Volume 5, Number 2, 1992
153
Geometry Driven Multimodality Matching of Brain Images Petra A. van den Elsen*, J.B. Antoine Maintz*, and Max A. Viergever*
Summary: Clinical diagnosis, as well as therapy planningand evaluation, are increasingly supported by multimodalimages. There are many instances desiring integration of the information obtained by various imaging devices. This paper describes a new approach to match images of different modalities. Differential operators are used in combinationwith Gaussian blurring to extract geometric features from the images that correspond to similar structures. The resulting 'feature' images may be used with existing matching techniques that minimize the distance between the features in the images to be matched. Our first application of this new approach concerns matching of MRI and CT brain images. The so-called Lou operator produces a ridge-like feature image from which in CT and MRI the center curve of the cranium is easily extracted. First results of this operator's performance in matching tasks are shown. Another promising operator is the 'mnbilicity'operator, which is presented in combinationwith SPECT images. Key words: Brain Imaging; CT; Geometrical Features; Image Matching; Image Processing; Image Registration; MRI; SPECT.
Introduction Images from different devices usually provide complementary information. SPECT (single photon emission computed tomography), PET (positron emission tomography) and MRS (magnetic resonance spectroscopy) provide functional information but delineate anatomy poorly, whereas MRI (magnetic resonance imaging), ultrasound, and X-ray imaging including CT (computed tomography) depict aspects of anatomy but provide little functional information. There are many instances in which it would be desirable to integrate the information obtained from two or more studies of the same patient. Application areas include radiation therapy planning, where a CT scan is needed for dose distribution calculations while the contours of the target lesion are often best outlined on MRI. In nuclear medicine integration of functional and morphological information facilitates determination of the anatomical location of dysfunction-
*Computer Vision Research Group, University Hospital Utrecht, The Netherlands. Accepted for publication:July 30, 1992. This research was supported in part by the Netherlands ministries of Education & Science and Economic Affairs through a SPIN grant, and by the industrial companies Agfa-Gevaert, Philips Medical Systems, and KEMA. The valuable discussions with E. J. D. Pol, B. M. ter Haar Romeny, and L. M. J. Florack are gratefully acknowledged. Correspondence and reprint requests should be addressed to Petra van den Elsen, Computer Vision Research Group, Room E.02.222, UniversityHospital Utrecht, Heidelberglaan 100, 3584 CX Utrecht, The Netherlands. Copyright © 1992 Human Sciences Press, Inc.
al areas and studies on functional-structural relationships. Image matching is not an easy task. Not only because of the distinct physical reality represented by each technique, but also because of the variations in patient positioning during the various studies and the use of different parameters for slice thickness, interslice gap, pixel size, and angulation. Matching methods may determine the transformation relating the images from corresponding features that can be extracted from the images. There are a great number of ad hoc 'feature' extracting operators. It is impossible to choose between them on fundamental grounds. The combination of differential operators -basic tools for local image structure analysis- with Gaussian blurring, leads to a group of operationally defined differential operators. These pure mathematical operators may be used to extract features from images (Ter Haar Romeny and Florack 1992). There seems to be a close resemblance between these operators and the primary stages of human vision. The work presented here addresses the application of differential operators and Gaussian blurring to match images of multiple modalities. We will focus on the matching of CT and MR brain images using the L~o operator, a detector that produces a ridge-like feature image from which in CT and MRI the center curve of the cranium is easily extracted. For the matching of functional (PET, SPECT) images with CT or MRI the umbilicity operator shows promise. Examples of this operator's performance will be shown as well.
154
Figure 1. The CT slice of figure 2a depicted as intensity landscape (top) (pixels with high intensity are forming mountains). Very noisy skull, skin, and headholder ridges are present. Upon blurring the image to the right extent (G = 4 pixels) (bottom), a smooth skull ridge becomes clear which can be detected by the L~ ridge operator (see figure 2c).
Methods We started our search for differential operators suited for matching with 2D operators on 2D tomographic slices, for reasons of limited computational speed and memory usage. Because tomographic image matching is a 3D problem, the tomographic images of different modality are first matched in three dimensions by means of skin markers (Van den EDen and Viergever 1991b). From the matched data sets representative pairs of matching slices are selected. Promising differential operators can be tested by artificially translating and rotating one slice of each pair, after which feature extraction takes place in both images, followed by thresholding. The threshold operation replaces each image pixel with a grey value below the chosen threshold level by a black pixel, and each image pixel with a grey value above the threshold level by a white pixel. The thresholded feature images can be used by existing matching techniques that minimize the sum of the squared distances between the features in the images to be matched (Borgefors 1988). The transformation found by the matching procedure is compared to the inverse of the artificially applied transformation. Since the match may differ from the marker-guided match and still be equally accurate
van den Elsenet al.
(marker matching is not without error), the result is checked visually as well. Visual inspection is performed by overlaying discernable structures (e.g., skull contour, longitudinal fissure, ventricles) from one image onto the other. Image features derived by means of straightforward differentiation only reflect local image properties and are very sensitive to noise. These problems can be avoided by an appropriate amount of blurring. Figure I shows a CT slice depicted as an intensity landscape (in which pixels with high intensity form mountains) of both the original image and a blurred replica. The figure shows that the desired cranial (mountain) ridges can be detected better in the blurred image. The group of differential operators, consisting of the Gaussian function and all of its partial derivatives, extract features from an image blurred to a certain extent. A parameter ~, indicating the width of the Gaussian, controls the level of blurring (the image is more blurred with increasing ~). The Lu~ operator, i.e., the second order derivative of the image intensity in the direction normal to the (local) gradient, produces a ridge-feature image (Ter Haar Romeny at al. 1991). Under correct blurring, the L~v operator yields a sharp skull ridge, signed negative in CT where bone is white, and positive in MRI where bone is dark (Van den Elsen et al. 1992b). In MRI some other structures are extracted as well. In the used matching method, this does not deteriorate the matching results, as long as the features extracted from one image are a subset of the features extracted from the other image.
Results Skull ridges have been extracted from MRI and CT images by means of the L~u operator. Results were very clear under correct blurring. Of several pairs of representative CT and MR image slices that had already been matched using a skin marker based method (Van den Elsen and Viergever 1991b), one image was artificially rotated and translated. After ridge detection using the L~o operator and thresholding, a matching algorithm was applied, that produces the transformation minimizing the sum of the squared distances between the ridge features. When visually checked, most results were comparable to the marker-based matching results, yielding an error of maximally 2 mm. An example is shown in figure 2; the maximum distance between corresponding points from the feature match and marker match is less than I mm. Skull-based matching can not be applied to functional (PET or SPECT) images, because skull ridges are not present in these images. The ridges extracted from SPECT and PET images closest to the skull are situated in the cortex. Figure 3 shows an original SPECT slice of
Geometry Driven Image Matching
155
r
Figure 2. (left to right, a = top left, h = bottom right) Matching results for a CT slice and an MRI slice using the L~ operator. The original images are a pair of marker matched CT (a) and MRI (b) slices, of which the CT slice is artificially rotated. Application of the Lu~ operator at ~ = 4 pixels gives a pair of feature images. The center curve of the cranium shows up dark in the CT feature image (c) and light in the MR feature image (d), and can be extracted from these feature images by proper thresholding (e, f). These thresholded images are used as input for matching software. The calculated transformation is a very good approximation of the inverse of the artificially applied transformation. After applying the calculated transformation to the CT image, again a matching pair is formed. The brain and skin contours obtained from the MRI slice are superimposed on the CT slice (g), and skull contours from CT are overlayed on the MRI slice (h), for visual inspection of the quality of the match.
Figure 3. (left to right) A transversal SPECTslice (a) and its L~ feature image with ~ = 1.6 pixels (b). The dark ridges follow intensity extrema in the cortex. The white ventricle ridge is thresholded and overlayed on the SPECT image (c).
156
van den Elsenet al.
Figure 4. (left to right) Matching transverse MRI (a) and SPECT (b) brain images, and the umbilicity feature images obtained from the MRI (c) and SPECT (d) (a = 12 pixels).
the brain and an L ~ feature image obtained from that slice. Note that a strong ventricle ridge is present, which can also be extracted from MR images at a higher level of b l u r r i n g . W h e t h e r this ridge has p o t e n t i a l for MRI/SPECT matching is currently under investigation.
Discussion Matching of tomographic images has become a major research topic in medical image processing. An extensive classification of matching methods supplemented with a survey of the literature is presented by Van den Elsen et al. (1992a). A comparative overview of imaging techniques, matching methods, and approaches for integrated multimodality display applied to brain images is given by Viergever et al. (1992). Matching algorithms may either use artificial marker information (e.g., head frames (Peters et al. 1989; Zhang et al. 1990) or skin markers (Hawkes et al. 1990; Van den Elsen et al. 1991a; Van den Elsen and Viergever 1991b)), or patient related image properties, (e.g., anatomical landmark points (Maguire et al. 1986; Evans et al. 1989) or surfaces (Levin et al. 1988)). Marker-based matching methods have the advantage that any modality can be matched, as long as a marker can be constructed that is visible in the images. Extracting marker positions from medical images is often easier than extracting patient related image properties, because the marker is designed such that it stands out in the image. The task of extracting patient related image properties is difficult, since the same properties should be extracted from both images to be fused. On the other hand, matching methods using patient related image properties have distinct advantages over marker-based methods. Since the imaging protocol does not need any adjustments, the methods are fully retrospective, optimal in patient friendliness, and no special provisions are
necessary if the studies to be matched are not done successively. Furthermore, patient related image properties may not only be used in rigid body transformations, but also in elastic matching methods. We have shown the usefulness of the L ~ operator in 2D matching tasks. However, tomographic image matching is a 3D problem. 2D operators, working on a slice-byslice basis, canbe used for 3D matching tasks for example with a straightforward 3D extension of the 2D chamfer matching algorithm employed in this paper. A better approach, currently under investigation, is to extend the differential operator to 3 dimensions, so as to take the local image properties in all 3 dimensions into account. The extracted bony ridge is particularly suited for the fusion of CT and MR images of the brain because of its rigidity; bone does not deform like, for example, skin might. Some problems however still have to be solved. Sometimes unwanted structures are detected, offsetting the resulting transformation, especially when the unwanted features form thick clusters of feature points. We expect that postprocessing steps to be performed after thresholding will solve these problems. For example, a skeletonizing step that thins thick ridges would equalize the influence of all ridges, thus diminishing the adverse effect of unwanted thick ridges. These and other postprocessing steps are under investigation. A fully detailed error analysis of the present method is not relevant; the 2D method has shown the feasibility of geometry driven multimodality matching, and we will concentrate our efforts on extension of the method to 3D. A differential operator that particularly shows potential for functional images is the umbilicity operator. This operator detects both ellipsoid and hyperboloid patches in images, which can be seen as smooth hills and valleys in an intensity landscape. Figure 4 shows umbilicity feature images for a matching pair of transverse MRI and
Geometry Driven Image Matching
SPECT brain images. Note that the ventricles are preserved in the feature images. A small infarction, just marginally visible as a focal area of diminished flow in the original SPECT, stands out clearly in the SPECT feature image.
Conclusion We have initiated research on the application of differential operators for multimodality matching. The approach proposed in this paper produced some promising results; however, much work still needs to be done. The 3D version of the Luu operator will be investigated shortly on CT and MR images of the brain. For matching of functional images with CT and MRI several approaches are being investigated, including extraction of ventricle ridges and using differential operators other than Luu.
References Borgefors, G. Hierarchical Chamfer matching: a parametric edge matching algorithm. IEEE Trans. PAMI, 1988, 10: 849865. Evans, A.C., Marret, S., Collins, L. and Peters, T.M. Anatomicalfunctional correlative analysis of the human brain using three dimensional systems. Proc. SPIE Medical Imaging III: Image Processing, 1989, 1092: 264-274. Hawkes, D.J., Hill, D.L.G., Lehmann, E.D., Robinson, G.P., Maisey, M.N. and Colchester, A.C.F. Preliminary work on the interpretation of SPECT images with the aid of registered MR images and an MR derived 3D neuro-anatomical atlas. In: K.H. Hthne, H. Fuchs, S.M. Pizer (Eds.) 3D Imaging in Medicine. Springer, Berlin, 1990: 241-251. Levin, D.N., Pelizarri, C.A., Chen, G.T.Y., Chen, C.T. and Cooper, M.D. Retrospective geometric correlation of MR, CT, and PET Images. Radiology, 1988, 169: 817-823. Maguire Jr, G.Q., Noz, M.E., Lee, E.M. and Schimpf, J.H. Cor-
157
relation methods for tomographic images using two and three dimensional techniques. In: S. Bacharach (Ed.) Information Processing in Medical Imaging. Martinus Nijhoff Publishers, Dordrecht, 1986: 266-279. Peters, T.M., Clark, J.A., Pike, G.B., Henri, C., Collins, L., Leksell, D. and Jeppsson, O. Stereotactic neurosurgery planning on a personal-computer-based work station. J. Dig. Ira., 1989, 2: 75-81. Ter Haar Romeny, B.M., Florack, L.M.J., Koenderink, J.J. and Viergever, M.A. Scale space: its natural operators and differential invariants. In: A.C.F. Colchester, D.J. Hawkes (Eds.) Information Processing in Medical Imaging. Springer, Berlin, 1991, 239-255. Ter Haar Romeny, B.M. and Florack, L.M.J. A multiscale geometric model of human vision. In: B. Hendee, P.N.T. Wells (Eds.) Perception of Visual Information. SpringerVerlag, Berlin, 1992 (in press). Van den Elsen, P.A., Viergever, M.A., Van Huffelen, A.C., Van der Meij, W. and Wieneke, G.H. Accurate matching of electromagnetic dipole data with CT and MR images. Brain Topography, 1991a, 3: 425-432. Van den Elsen, P.A. and Viergever, M.A. Marker-guided multimodality matching of the brain. Report 3DCV 91-26, Utrecht University, 1991b. Van den Elsen, P.A., Pol, E.J.D. and Viergever, M.A. Medical image matching - A review with classification. IEEE Engineering in Medicine and Biology, 1992a (in press). Van den Elsen, P.A., Maintz, J.B.A. Pol, E.J.D. and Viergever, M.A. Image fusion using geometrical features. In: Visualization in Biomedical Computing. Proc. SPIE. SPIE Press, Bellingham, 1992b (in press). Viergever, M.A., Van den Elsen, P.A. and Stokking, R. Integrated presentation of multimodal brain images. Brain Topography, 1992, 5: 135-145. Zhang, J., Levesque, M.F., Wilson, C.L., Harper, R.M., Engel, Jr., J., Lufkin, R. and Behnke, E.J. Multimodality imaging of brain structures for stereotactic surgery. Radiology, 1990, 175: 435-441.
153
Geometry Driven Multimodality Matching of Brain Images Petra A. van den Elsen*, J.B. Antoine Maintz*, and Max A. Viergever*
Summary: Clinical diagnosis, as well as therapy planningand evaluation, are increasingly supported by multimodalimages. There are many instances desiring integration of the information obtained by various imaging devices. This paper describes a new approach to match images of different modalities. Differential operators are used in combinationwith Gaussian blurring to extract geometric features from the images that correspond to similar structures. The resulting 'feature' images may be used with existing matching techniques that minimize the distance between the features in the images to be matched. Our first application of this new approach concerns matching of MRI and CT brain images. The so-called Lou operator produces a ridge-like feature image from which in CT and MRI the center curve of the cranium is easily extracted. First results of this operator's performance in matching tasks are shown. Another promising operator is the 'mnbilicity'operator, which is presented in combinationwith SPECT images. Key words: Brain Imaging; CT; Geometrical Features; Image Matching; Image Processing; Image Registration; MRI; SPECT.
Introduction Images from different devices usually provide complementary information. SPECT (single photon emission computed tomography), PET (positron emission tomography) and MRS (magnetic resonance spectroscopy) provide functional information but delineate anatomy poorly, whereas MRI (magnetic resonance imaging), ultrasound, and X-ray imaging including CT (computed tomography) depict aspects of anatomy but provide little functional information. There are many instances in which it would be desirable to integrate the information obtained from two or more studies of the same patient. Application areas include radiation therapy planning, where a CT scan is needed for dose distribution calculations while the contours of the target lesion are often best outlined on MRI. In nuclear medicine integration of functional and morphological information facilitates determination of the anatomical location of dysfunction-
*Computer Vision Research Group, University Hospital Utrecht, The Netherlands. Accepted for publication:July 30, 1992. This research was supported in part by the Netherlands ministries of Education & Science and Economic Affairs through a SPIN grant, and by the industrial companies Agfa-Gevaert, Philips Medical Systems, and KEMA. The valuable discussions with E. J. D. Pol, B. M. ter Haar Romeny, and L. M. J. Florack are gratefully acknowledged. Correspondence and reprint requests should be addressed to Petra van den Elsen, Computer Vision Research Group, Room E.02.222, UniversityHospital Utrecht, Heidelberglaan 100, 3584 CX Utrecht, The Netherlands. Copyright © 1992 Human Sciences Press, Inc.
al areas and studies on functional-structural relationships. Image matching is not an easy task. Not only because of the distinct physical reality represented by each technique, but also because of the variations in patient positioning during the various studies and the use of different parameters for slice thickness, interslice gap, pixel size, and angulation. Matching methods may determine the transformation relating the images from corresponding features that can be extracted from the images. There are a great number of ad hoc 'feature' extracting operators. It is impossible to choose between them on fundamental grounds. The combination of differential operators -basic tools for local image structure analysis- with Gaussian blurring, leads to a group of operationally defined differential operators. These pure mathematical operators may be used to extract features from images (Ter Haar Romeny and Florack 1992). There seems to be a close resemblance between these operators and the primary stages of human vision. The work presented here addresses the application of differential operators and Gaussian blurring to match images of multiple modalities. We will focus on the matching of CT and MR brain images using the L~o operator, a detector that produces a ridge-like feature image from which in CT and MRI the center curve of the cranium is easily extracted. For the matching of functional (PET, SPECT) images with CT or MRI the umbilicity operator shows promise. Examples of this operator's performance will be shown as well.
154
Figure 1. The CT slice of figure 2a depicted as intensity landscape (top) (pixels with high intensity are forming mountains). Very noisy skull, skin, and headholder ridges are present. Upon blurring the image to the right extent (G = 4 pixels) (bottom), a smooth skull ridge becomes clear which can be detected by the L~ ridge operator (see figure 2c).
Methods We started our search for differential operators suited for matching with 2D operators on 2D tomographic slices, for reasons of limited computational speed and memory usage. Because tomographic image matching is a 3D problem, the tomographic images of different modality are first matched in three dimensions by means of skin markers (Van den EDen and Viergever 1991b). From the matched data sets representative pairs of matching slices are selected. Promising differential operators can be tested by artificially translating and rotating one slice of each pair, after which feature extraction takes place in both images, followed by thresholding. The threshold operation replaces each image pixel with a grey value below the chosen threshold level by a black pixel, and each image pixel with a grey value above the threshold level by a white pixel. The thresholded feature images can be used by existing matching techniques that minimize the sum of the squared distances between the features in the images to be matched (Borgefors 1988). The transformation found by the matching procedure is compared to the inverse of the artificially applied transformation. Since the match may differ from the marker-guided match and still be equally accurate
van den Elsenet al.
(marker matching is not without error), the result is checked visually as well. Visual inspection is performed by overlaying discernable structures (e.g., skull contour, longitudinal fissure, ventricles) from one image onto the other. Image features derived by means of straightforward differentiation only reflect local image properties and are very sensitive to noise. These problems can be avoided by an appropriate amount of blurring. Figure I shows a CT slice depicted as an intensity landscape (in which pixels with high intensity form mountains) of both the original image and a blurred replica. The figure shows that the desired cranial (mountain) ridges can be detected better in the blurred image. The group of differential operators, consisting of the Gaussian function and all of its partial derivatives, extract features from an image blurred to a certain extent. A parameter ~, indicating the width of the Gaussian, controls the level of blurring (the image is more blurred with increasing ~). The Lu~ operator, i.e., the second order derivative of the image intensity in the direction normal to the (local) gradient, produces a ridge-feature image (Ter Haar Romeny at al. 1991). Under correct blurring, the L~v operator yields a sharp skull ridge, signed negative in CT where bone is white, and positive in MRI where bone is dark (Van den Elsen et al. 1992b). In MRI some other structures are extracted as well. In the used matching method, this does not deteriorate the matching results, as long as the features extracted from one image are a subset of the features extracted from the other image.
Results Skull ridges have been extracted from MRI and CT images by means of the L~u operator. Results were very clear under correct blurring. Of several pairs of representative CT and MR image slices that had already been matched using a skin marker based method (Van den Elsen and Viergever 1991b), one image was artificially rotated and translated. After ridge detection using the L~o operator and thresholding, a matching algorithm was applied, that produces the transformation minimizing the sum of the squared distances between the ridge features. When visually checked, most results were comparable to the marker-based matching results, yielding an error of maximally 2 mm. An example is shown in figure 2; the maximum distance between corresponding points from the feature match and marker match is less than I mm. Skull-based matching can not be applied to functional (PET or SPECT) images, because skull ridges are not present in these images. The ridges extracted from SPECT and PET images closest to the skull are situated in the cortex. Figure 3 shows an original SPECT slice of
Geometry Driven Image Matching
155
r
Figure 2. (left to right, a = top left, h = bottom right) Matching results for a CT slice and an MRI slice using the L~ operator. The original images are a pair of marker matched CT (a) and MRI (b) slices, of which the CT slice is artificially rotated. Application of the Lu~ operator at ~ = 4 pixels gives a pair of feature images. The center curve of the cranium shows up dark in the CT feature image (c) and light in the MR feature image (d), and can be extracted from these feature images by proper thresholding (e, f). These thresholded images are used as input for matching software. The calculated transformation is a very good approximation of the inverse of the artificially applied transformation. After applying the calculated transformation to the CT image, again a matching pair is formed. The brain and skin contours obtained from the MRI slice are superimposed on the CT slice (g), and skull contours from CT are overlayed on the MRI slice (h), for visual inspection of the quality of the match.
Figure 3. (left to right) A transversal SPECTslice (a) and its L~ feature image with ~ = 1.6 pixels (b). The dark ridges follow intensity extrema in the cortex. The white ventricle ridge is thresholded and overlayed on the SPECT image (c).
156
van den Elsenet al.
Figure 4. (left to right) Matching transverse MRI (a) and SPECT (b) brain images, and the umbilicity feature images obtained from the MRI (c) and SPECT (d) (a = 12 pixels).
the brain and an L ~ feature image obtained from that slice. Note that a strong ventricle ridge is present, which can also be extracted from MR images at a higher level of b l u r r i n g . W h e t h e r this ridge has p o t e n t i a l for MRI/SPECT matching is currently under investigation.
Discussion Matching of tomographic images has become a major research topic in medical image processing. An extensive classification of matching methods supplemented with a survey of the literature is presented by Van den Elsen et al. (1992a). A comparative overview of imaging techniques, matching methods, and approaches for integrated multimodality display applied to brain images is given by Viergever et al. (1992). Matching algorithms may either use artificial marker information (e.g., head frames (Peters et al. 1989; Zhang et al. 1990) or skin markers (Hawkes et al. 1990; Van den Elsen et al. 1991a; Van den Elsen and Viergever 1991b)), or patient related image properties, (e.g., anatomical landmark points (Maguire et al. 1986; Evans et al. 1989) or surfaces (Levin et al. 1988)). Marker-based matching methods have the advantage that any modality can be matched, as long as a marker can be constructed that is visible in the images. Extracting marker positions from medical images is often easier than extracting patient related image properties, because the marker is designed such that it stands out in the image. The task of extracting patient related image properties is difficult, since the same properties should be extracted from both images to be fused. On the other hand, matching methods using patient related image properties have distinct advantages over marker-based methods. Since the imaging protocol does not need any adjustments, the methods are fully retrospective, optimal in patient friendliness, and no special provisions are
necessary if the studies to be matched are not done successively. Furthermore, patient related image properties may not only be used in rigid body transformations, but also in elastic matching methods. We have shown the usefulness of the L ~ operator in 2D matching tasks. However, tomographic image matching is a 3D problem. 2D operators, working on a slice-byslice basis, canbe used for 3D matching tasks for example with a straightforward 3D extension of the 2D chamfer matching algorithm employed in this paper. A better approach, currently under investigation, is to extend the differential operator to 3 dimensions, so as to take the local image properties in all 3 dimensions into account. The extracted bony ridge is particularly suited for the fusion of CT and MR images of the brain because of its rigidity; bone does not deform like, for example, skin might. Some problems however still have to be solved. Sometimes unwanted structures are detected, offsetting the resulting transformation, especially when the unwanted features form thick clusters of feature points. We expect that postprocessing steps to be performed after thresholding will solve these problems. For example, a skeletonizing step that thins thick ridges would equalize the influence of all ridges, thus diminishing the adverse effect of unwanted thick ridges. These and other postprocessing steps are under investigation. A fully detailed error analysis of the present method is not relevant; the 2D method has shown the feasibility of geometry driven multimodality matching, and we will concentrate our efforts on extension of the method to 3D. A differential operator that particularly shows potential for functional images is the umbilicity operator. This operator detects both ellipsoid and hyperboloid patches in images, which can be seen as smooth hills and valleys in an intensity landscape. Figure 4 shows umbilicity feature images for a matching pair of transverse MRI and
Geometry Driven Image Matching
SPECT brain images. Note that the ventricles are preserved in the feature images. A small infarction, just marginally visible as a focal area of diminished flow in the original SPECT, stands out clearly in the SPECT feature image.
Conclusion We have initiated research on the application of differential operators for multimodality matching. The approach proposed in this paper produced some promising results; however, much work still needs to be done. The 3D version of the Luu operator will be investigated shortly on CT and MR images of the brain. For matching of functional images with CT and MRI several approaches are being investigated, including extraction of ventricle ridges and using differential operators other than Luu.
References Borgefors, G. Hierarchical Chamfer matching: a parametric edge matching algorithm. IEEE Trans. PAMI, 1988, 10: 849865. Evans, A.C., Marret, S., Collins, L. and Peters, T.M. Anatomicalfunctional correlative analysis of the human brain using three dimensional systems. Proc. SPIE Medical Imaging III: Image Processing, 1989, 1092: 264-274. Hawkes, D.J., Hill, D.L.G., Lehmann, E.D., Robinson, G.P., Maisey, M.N. and Colchester, A.C.F. Preliminary work on the interpretation of SPECT images with the aid of registered MR images and an MR derived 3D neuro-anatomical atlas. In: K.H. Hthne, H. Fuchs, S.M. Pizer (Eds.) 3D Imaging in Medicine. Springer, Berlin, 1990: 241-251. Levin, D.N., Pelizarri, C.A., Chen, G.T.Y., Chen, C.T. and Cooper, M.D. Retrospective geometric correlation of MR, CT, and PET Images. Radiology, 1988, 169: 817-823. Maguire Jr, G.Q., Noz, M.E., Lee, E.M. and Schimpf, J.H. Cor-
157
relation methods for tomographic images using two and three dimensional techniques. In: S. Bacharach (Ed.) Information Processing in Medical Imaging. Martinus Nijhoff Publishers, Dordrecht, 1986: 266-279. Peters, T.M., Clark, J.A., Pike, G.B., Henri, C., Collins, L., Leksell, D. and Jeppsson, O. Stereotactic neurosurgery planning on a personal-computer-based work station. J. Dig. Ira., 1989, 2: 75-81. Ter Haar Romeny, B.M., Florack, L.M.J., Koenderink, J.J. and Viergever, M.A. Scale space: its natural operators and differential invariants. In: A.C.F. Colchester, D.J. Hawkes (Eds.) Information Processing in Medical Imaging. Springer, Berlin, 1991, 239-255. Ter Haar Romeny, B.M. and Florack, L.M.J. A multiscale geometric model of human vision. In: B. Hendee, P.N.T. Wells (Eds.) Perception of Visual Information. SpringerVerlag, Berlin, 1992 (in press). Van den Elsen, P.A., Viergever, M.A., Van Huffelen, A.C., Van der Meij, W. and Wieneke, G.H. Accurate matching of electromagnetic dipole data with CT and MR images. Brain Topography, 1991a, 3: 425-432. Van den Elsen, P.A. and Viergever, M.A. Marker-guided multimodality matching of the brain. Report 3DCV 91-26, Utrecht University, 1991b. Van den Elsen, P.A., Pol, E.J.D. and Viergever, M.A. Medical image matching - A review with classification. IEEE Engineering in Medicine and Biology, 1992a (in press). Van den Elsen, P.A., Maintz, J.B.A. Pol, E.J.D. and Viergever, M.A. Image fusion using geometrical features. In: Visualization in Biomedical Computing. Proc. SPIE. SPIE Press, Bellingham, 1992b (in press). Viergever, M.A., Van den Elsen, P.A. and Stokking, R. Integrated presentation of multimodal brain images. Brain Topography, 1992, 5: 135-145. Zhang, J., Levesque, M.F., Wilson, C.L., Harper, R.M., Engel, Jr., J., Lufkin, R. and Behnke, E.J. Multimodality imaging of brain structures for stereotactic surgery. Radiology, 1990, 175: 435-441.
