hf. J. Radrorion Oncology Biol. Phu.. Vol. 21. pp. 1653-1667 Printed I” the U.S A. All rights reserved.
Copyright
0360.3016/91 $3.00 + .W 0 1991 Pergamon Press plc
??Technical Innovations and Notes
INTEGRATION OF MULTIMODALITY IMAGING DATA FOR RADIOTHERAPY TREATMENT PLANNING M.
L. KESSLER, PH.D.
,‘,2 S. PITLUCK, PH.D.
,2 PAULA PET-II, PH.D.*
AND J. R. CASTRO, M.D.2y3
‘Graduate Group in Biophysics and Medical Physics, University of California; 2Research Medicine and Radiation Biophysics Division, Lawrence Berkeley Laboratory, Berkeley, CA 94720; and 3Department of Radiation Oncology, University of California, San Francisco, CA 94714 This paper describes computational techniques to permit the quantitative integration of magnetic resonance (MR), positron emission tomography (PET), and x-ray computed tomography (CT) imaging data sets. These methods are used to incorporate unique diagnostic information provided by PET and MR imaging into CTbased treatment planning for radiotherapy of intracranial tumors and vascular malformations. Integration of information from the different imaging modalities is treated as a two-step process. The first step is to determine the set of geometric parameters relating the coordinates of two imaging data sets. No universal method for determining these parameters is appropriate because of the diversity of contemporary imaging methods and data formats. Most situations can be handled by one of the four different techniques described. These four methods make use of specific geometric objects contained in the two data sets to determine the parameters. These objects are: (a) anatomical and/or fiducial points, (b) attached line markers, (c) anatomical surfaces, and (d) outlines of anatomical structures. The second step involves using the derived transformation to transfer outlines of treatment volumes and/or anatomical structures drawn on the images of one imaging study to the images of another study, usually the treatment planning CT. Solid modelling and image processing techniques have been adapted and developed further to accomplish this task. Clinical examples and phantom studies are presented which verify the different aspects of these techniques and demonstrate the accuracy with which they can be applied. Clinical use of these techniques for treatment planning has resulted in improvements in localization of treatment volumes and critical structures in the brain. These improvements have allowed greater sparing of normal tissues and more precise delivery of energy to the desired irradiation volume. It is believed that these improvements will have a positive impact on the outcome of radiation therapy. Multimodality imaging, Image registration, Treatment planning. sign and carry out a successful course of therapy and more closely follow the progress of the patient post-therapy. Imaging data from MRI and PET do not, however, provide the necessary geometric and physical information required in CT-based treatment planning systems. These modalities do not provide the requisite information such as the electron density or stopping power of body tissues and are unable to image complex bone/air inhomogeneities. This information is essential for calculation of beam penetration into the patient and in the design of compensators and collimators to shape the profile of the beam in three dimensions (2). Therefore, the unique information provided by MRI and PET must be transferred to and integrated with the CT study used for treatment planning. To perform the transfer of information in an objective manner, it is necessary to determine the 3-dimensional transformation which relates the coordinates of a particular MR or PET imaging study
INTRODUCTION Since its introduction in 1972, x-ray computed tomography (CT) has become the principal source of imaging data for 3-D treatment planning in radiotherapy. The information provided by CT is used to establish the extent of disease, to design the optimal therapy, and to monitor the patient post-treatment. There is now a growing demand to incorporate the diagnostic information currently available from magnetic resonance imaging (MRI) and positron emission tomography (PET) into the treatment planning process (4, 8, 10). MRI provides anatomical information superior to CT, allowing more precise delineation of normal critical structures and more accurate definition of treatment volumes. PET imaging provides unique functional information concerning tissue metabolism and structural integrity. This additional and complementary information can better enable the radiotherapist and the medical physicist to deReprint requests to: Marc L. Kessler, Ph.D., UH-B2C490, Box 0010, 1500 E. Medical Center Drive, Ann Arbor, MI 48109. Acknowledgments - The authors would like to thank Mark Phillips, Ron Huesman, and John Baker for many helpful discussions,
James Judnick for performing the imaging studies, and George T.Y. Chen for initiating the work on image registration. Supported in part by NIH Grant lpO1 CA19138. Accepted for publication 1653
24 May 199 1.
1654
I. J. Radiation Oncology 0 Biology 0 Physics
and the coordinates of the planning CT. Once determined, the transformation can be used to map information such as the outlines of anatomical structures or regions of interest from the PET or MR imaging study to the planning CT. In the same manner, CT-derived information, such as the dose delivered to the patient, can be transferred to the PET or MR imaging study. This paper describes computational techniques developed to allow the quantitative integration of information from PET and MR imaging data into CT-based radiotherapy treatment planning systems. This process is divided into two separate tasks. The first task is to determine the transformation which relates the coordinates of a particular imaging study to the coordinates of the CT study used for treatment planning. Four different techniques have been developed to determine this transformation: (a) point matching, (b) line matching, (c) surface matching, and (d) interactive matching. The second task involves using the resultant transformation to map information unique to one data set to another data set. Solid modelling and image processing techniques have been adapted and developed further to accomplish this task. The algorithms described in this work have been implemented in VAX-l 1 Fortran 4.0* and integrated with a CTbased treatment planning system used for the design and execution of treatment plans for radiation therapy and stereotactic radiosurgery with accelerated heavy charged particles (1, 2). Phantom studies performed to validate the different aspects of the algorithms are described. Examples of the use of these techniques in clinical practice are also presented. METHODS
AND MATERIALS
The process of integrating diagnostic information from a particular imaging study (study A) and another imaging study (study B) is divided into two main tasks. The first task is to estimate the parameters of the transformation that relate the coordinates of the two imaging studies. For intracranial imaging studies, where movement of the anatomy is governed by the motion of the skull, this transformation can be represented as a linear, spatially invariant function of the form: xA
=
Ax,
+ b,
(1)
where x, is the coordinate of a point in study A and xB is the coordinate of the corresponding point in study B. The matrix A incorporates the operations of rotation, differential scaling, and plane reflection. The vector b describes the operation of translation. The second task is to apply the resultant transformation to map structures or features of interest from one imaging study to another or to reformat the images from one study to match the orientation and scale of the images of the other. *Digital Equipment Corporation,
Maynard, MA.
November 1991, Volume 21, Number 6
Estimating the transformation parameters The set of nine parameters {p}, that must be estimated to establish the necessary transformations are: three rotation angles: three translation values: and three scaling factors:
(e,, 8,, f3,), (t,, tY, tZ), (s,, s,, sJ.
The rotation and translation parameters account for differences in the orientation and location of the patient in the different imaging devices. The scaling parameters are included to account for possible mis-calibrations of the imaging devices. In theory, all machine calibration parameters should be determined a priori and used to correct the imaging data before integration or be made available as known parameters to incorporate into the integration process. In practice, it is often the case (particularly with MR) that the available imaging data contain scaling (and possibly higher order) distortions (12). There is no universal method to determine these parameters because of the diversity of contemporary imaging methods and data formats in use. Most situations can be handled by one of four different techniques that were developed: (a) point matching, (b) line matching, (c) surface matching, and (d) interactive matching. Figure 1 illustrates these techniques. Each of these techniques involves using specific geometric information extracted from study A and study B to determine the transformation parameters. This information is either intrinsic to the imaging studies (e.g., anatomic information) or is specifically incorporated into the imaging studies (e.g., external markers) in order to calculate the transformation parameters. In each of the techniques except the interactive approach, geometric information is used to design a meritfunction, R, which provides a quantitative measure of the geometric misregistration between study A and study B. This function depends on a set of coordinates derived from the two imaging studies {x,} and {x,} and the desired parameters {fi}, that is: R = R({x,}, (~~13 CD>.
(2)
An estimate of the parameters is determined by manipulating them to minimize R. The manipulation is carried out using a non-linear optimization technique which employs a quasi-Newton (variable metric) algorithm to search the parameter space (5). Once the parameter estimates are determined, the covariance matrix of the estimates is computed by taking the inverse of the Hessian (11) of the merit-function evaluated at the minimum of R. Point matching In the point matching method, the transformation rameters are determined by establishing a one-to-one
pacor-
Integration of multimodality imaging
data
??M.
1655
L. KESSLERet al.
/nrflqc~
P‘rtwlrrMU\!T
Srrf
Copyright
0360.3016/91 $3.00 + .W 0 1991 Pergamon Press plc
??Technical Innovations and Notes
INTEGRATION OF MULTIMODALITY IMAGING DATA FOR RADIOTHERAPY TREATMENT PLANNING M.
L. KESSLER, PH.D.
,‘,2 S. PITLUCK, PH.D.
,2 PAULA PET-II, PH.D.*
AND J. R. CASTRO, M.D.2y3
‘Graduate Group in Biophysics and Medical Physics, University of California; 2Research Medicine and Radiation Biophysics Division, Lawrence Berkeley Laboratory, Berkeley, CA 94720; and 3Department of Radiation Oncology, University of California, San Francisco, CA 94714 This paper describes computational techniques to permit the quantitative integration of magnetic resonance (MR), positron emission tomography (PET), and x-ray computed tomography (CT) imaging data sets. These methods are used to incorporate unique diagnostic information provided by PET and MR imaging into CTbased treatment planning for radiotherapy of intracranial tumors and vascular malformations. Integration of information from the different imaging modalities is treated as a two-step process. The first step is to determine the set of geometric parameters relating the coordinates of two imaging data sets. No universal method for determining these parameters is appropriate because of the diversity of contemporary imaging methods and data formats. Most situations can be handled by one of the four different techniques described. These four methods make use of specific geometric objects contained in the two data sets to determine the parameters. These objects are: (a) anatomical and/or fiducial points, (b) attached line markers, (c) anatomical surfaces, and (d) outlines of anatomical structures. The second step involves using the derived transformation to transfer outlines of treatment volumes and/or anatomical structures drawn on the images of one imaging study to the images of another study, usually the treatment planning CT. Solid modelling and image processing techniques have been adapted and developed further to accomplish this task. Clinical examples and phantom studies are presented which verify the different aspects of these techniques and demonstrate the accuracy with which they can be applied. Clinical use of these techniques for treatment planning has resulted in improvements in localization of treatment volumes and critical structures in the brain. These improvements have allowed greater sparing of normal tissues and more precise delivery of energy to the desired irradiation volume. It is believed that these improvements will have a positive impact on the outcome of radiation therapy. Multimodality imaging, Image registration, Treatment planning. sign and carry out a successful course of therapy and more closely follow the progress of the patient post-therapy. Imaging data from MRI and PET do not, however, provide the necessary geometric and physical information required in CT-based treatment planning systems. These modalities do not provide the requisite information such as the electron density or stopping power of body tissues and are unable to image complex bone/air inhomogeneities. This information is essential for calculation of beam penetration into the patient and in the design of compensators and collimators to shape the profile of the beam in three dimensions (2). Therefore, the unique information provided by MRI and PET must be transferred to and integrated with the CT study used for treatment planning. To perform the transfer of information in an objective manner, it is necessary to determine the 3-dimensional transformation which relates the coordinates of a particular MR or PET imaging study
INTRODUCTION Since its introduction in 1972, x-ray computed tomography (CT) has become the principal source of imaging data for 3-D treatment planning in radiotherapy. The information provided by CT is used to establish the extent of disease, to design the optimal therapy, and to monitor the patient post-treatment. There is now a growing demand to incorporate the diagnostic information currently available from magnetic resonance imaging (MRI) and positron emission tomography (PET) into the treatment planning process (4, 8, 10). MRI provides anatomical information superior to CT, allowing more precise delineation of normal critical structures and more accurate definition of treatment volumes. PET imaging provides unique functional information concerning tissue metabolism and structural integrity. This additional and complementary information can better enable the radiotherapist and the medical physicist to deReprint requests to: Marc L. Kessler, Ph.D., UH-B2C490, Box 0010, 1500 E. Medical Center Drive, Ann Arbor, MI 48109. Acknowledgments - The authors would like to thank Mark Phillips, Ron Huesman, and John Baker for many helpful discussions,
James Judnick for performing the imaging studies, and George T.Y. Chen for initiating the work on image registration. Supported in part by NIH Grant lpO1 CA19138. Accepted for publication 1653
24 May 199 1.
1654
I. J. Radiation Oncology 0 Biology 0 Physics
and the coordinates of the planning CT. Once determined, the transformation can be used to map information such as the outlines of anatomical structures or regions of interest from the PET or MR imaging study to the planning CT. In the same manner, CT-derived information, such as the dose delivered to the patient, can be transferred to the PET or MR imaging study. This paper describes computational techniques developed to allow the quantitative integration of information from PET and MR imaging data into CT-based radiotherapy treatment planning systems. This process is divided into two separate tasks. The first task is to determine the transformation which relates the coordinates of a particular imaging study to the coordinates of the CT study used for treatment planning. Four different techniques have been developed to determine this transformation: (a) point matching, (b) line matching, (c) surface matching, and (d) interactive matching. The second task involves using the resultant transformation to map information unique to one data set to another data set. Solid modelling and image processing techniques have been adapted and developed further to accomplish this task. The algorithms described in this work have been implemented in VAX-l 1 Fortran 4.0* and integrated with a CTbased treatment planning system used for the design and execution of treatment plans for radiation therapy and stereotactic radiosurgery with accelerated heavy charged particles (1, 2). Phantom studies performed to validate the different aspects of the algorithms are described. Examples of the use of these techniques in clinical practice are also presented. METHODS
AND MATERIALS
The process of integrating diagnostic information from a particular imaging study (study A) and another imaging study (study B) is divided into two main tasks. The first task is to estimate the parameters of the transformation that relate the coordinates of the two imaging studies. For intracranial imaging studies, where movement of the anatomy is governed by the motion of the skull, this transformation can be represented as a linear, spatially invariant function of the form: xA
=
Ax,
+ b,
(1)
where x, is the coordinate of a point in study A and xB is the coordinate of the corresponding point in study B. The matrix A incorporates the operations of rotation, differential scaling, and plane reflection. The vector b describes the operation of translation. The second task is to apply the resultant transformation to map structures or features of interest from one imaging study to another or to reformat the images from one study to match the orientation and scale of the images of the other. *Digital Equipment Corporation,
Maynard, MA.
November 1991, Volume 21, Number 6
Estimating the transformation parameters The set of nine parameters {p}, that must be estimated to establish the necessary transformations are: three rotation angles: three translation values: and three scaling factors:
(e,, 8,, f3,), (t,, tY, tZ), (s,, s,, sJ.
The rotation and translation parameters account for differences in the orientation and location of the patient in the different imaging devices. The scaling parameters are included to account for possible mis-calibrations of the imaging devices. In theory, all machine calibration parameters should be determined a priori and used to correct the imaging data before integration or be made available as known parameters to incorporate into the integration process. In practice, it is often the case (particularly with MR) that the available imaging data contain scaling (and possibly higher order) distortions (12). There is no universal method to determine these parameters because of the diversity of contemporary imaging methods and data formats in use. Most situations can be handled by one of four different techniques that were developed: (a) point matching, (b) line matching, (c) surface matching, and (d) interactive matching. Figure 1 illustrates these techniques. Each of these techniques involves using specific geometric information extracted from study A and study B to determine the transformation parameters. This information is either intrinsic to the imaging studies (e.g., anatomic information) or is specifically incorporated into the imaging studies (e.g., external markers) in order to calculate the transformation parameters. In each of the techniques except the interactive approach, geometric information is used to design a meritfunction, R, which provides a quantitative measure of the geometric misregistration between study A and study B. This function depends on a set of coordinates derived from the two imaging studies {x,} and {x,} and the desired parameters {fi}, that is: R = R({x,}, (~~13 CD>.
(2)
An estimate of the parameters is determined by manipulating them to minimize R. The manipulation is carried out using a non-linear optimization technique which employs a quasi-Newton (variable metric) algorithm to search the parameter space (5). Once the parameter estimates are determined, the covariance matrix of the estimates is computed by taking the inverse of the Hessian (11) of the merit-function evaluated at the minimum of R. Point matching In the point matching method, the transformation rameters are determined by establishing a one-to-one
pacor-
Integration of multimodality imaging
data
??M.
1655
L. KESSLERet al.
/nrflqc~
P‘rtwlrrMU\!T
Srrf
