NIH Public Access Author Manuscript Med Image Comput Comput Assist Interv. Author manuscript; available in PMC 2015 February 25.
NIH-PA Author Manuscript
Published in final edited form as: Med Image Comput Comput Assist Interv. 2014 ; 17(0 1): 440–447.
Efficient Stereo Image Geometrical Reconstruction at Arbitrary Camera Settings from a Single Calibration Songbai Ji1,2, Xiaoyao Fan1, David W. Roberts2,3, and Keith D. Paulsen1,2,3 1
Thayer School of Engineering, Dartmouth College, Hanover, NH 03755, USA
2
Geisel School of Medicine, Dartmouth College, Hanover, NH 03755, USA
3
Dartmouth Hitchcock Medical Center, Lebanon, NH 03756, USA
Abstract NIH-PA Author Manuscript
Camera calibration is central to obtaining a quantitative image-to-physical-space mapping from stereo images acquired in the operating room (OR). A practical challenge for cameras mounted to the operating microscope is maintenance of image calibration as the surgeon’s field-of-view is repeatedly changed (in terms of zoom and focal settings) throughout a procedure. Here, we present an efficient method for sustaining a quantitative image-to-physical space relationship for arbitrary image acquisition settings (S) without the need for camera re-calibration. Essentially, we warp images acquired at S into the equivalent data acquired at a reference setting, S0, using deformation fields obtained with optical flow by successively imaging a simple phantom. Closed-form expressions for the distortions were derived from which 3D surface reconstruction was performed based on the single calibration at S0. The accuracy of the reconstructed surface was 1.05 mm and 0.59 mm along and perpendicular to the optical axis of the operating microscope on average, respectively, for six phantom image pairs, and was 1.26 mm and 0.71 mm for images acquired with a total of 47 arbitrary settings during three clinical cases. The technique is presented in the context of stereovision; however, it may also be applicable to other types of video image acquisitions (e.g., endoscope) because it does not rely on any a priori knowledge about the camera system itself, suggesting the method is likely of considerable significance.
NIH-PA Author Manuscript
1 Introduction Camera images provide texture intensity from the surface of objects in the scene, and are an increasingly popular form of data in image-guided procedures such as neurosurgery [1]. Calibration is central to obtaining quantitative geometrical information from the camera system to project 2D image pixels into their 3D coordinates in physical space in the case of stereovision. Techniques for calibrating a camera system at fixed zoom and focal settings are well studied [2]. However, many cameras offer a wide range of zoom factors and focal lengths that can be arbitrarily varied to obtain an optimal view [3]; thus, maintenance of camera calibration becomes a practical challenge. Because these images depend on acquisition settings, recovering the camera calibration parameters efficiently and for an
© Springer International Publishing Switzerland 2014 {songbai.ji,xiaoyao.fan,david.w.roberts,keith.d.paulsen}@dartmouth.edu.
Ji et al.
Page 2
arbitrary setting is essential for applications like stereovision in the operating room (OR) where the surgeon is repeatedly altering the field-of-view through the operating microscope.
NIH-PA Author Manuscript
Existing techniques for camera calibration at an arbitrary setting either actively re-calibrate at a given setting on-demand [3] or interpolate camera parameters via bivariate fitting by explicitly modeling each as a polynomial function of zoom and focal length based on data from a dense set of pre-calibrations [4, 5]. Although calibration at a given setting can be fully automated with an on-demand approach [2], repeatedly imaging an instrumented calibration target [3] is inconvenient and cumbersome in the OR. While interpolation of predetermined camera parameters minimizes disruption of surgical workflow, a dense combination of zoom and focal length settings have to be calibrated (and re-calibrated for quality assurance and/or when camera extrinsic parameters are changed, e.g., from repositioning) which too adds to pre-operative activity and personnel time requirements. Consequently, suggestions of a fixed zoom and focus have been made to ensure optimal accuracy [5], but such restrictions significantly limit the effective OR use of camera systems.
NIH-PA Author Manuscript
In this study, we present a method to recover geometry from stereo images at arbitrary camera settings using a single calibration at a fixed (reference) zoom and focal setting. The approach is especially appealing for OR applications because it does not disrupt surgical workflow nor does it require tedious calibration at numerous zoom-focus combinations. The performance of the technique is evaluated on a physical phantom and in three clinical cases involving open cranial surgery with a microscope-mounted stereovision system. However, the general strategy appears to be applicable to other types of stereo/video images (e.g., endoscope).
2 Material and Methods
NIH-PA Author Manuscript
A custom-designed stereovision system consisting of two C-mount cameras (Flea2 model FL2G-50S5C-C, Point Grey Research Inc., Richmond, BC, Canada) was rigidly mounted to a Zeiss surgical microscope (OPMI® Pentero™, Carl Zeiss, Inc., Oberkochen, Germany) through a binocular port [6]. The position and orientation of the microscope was available from a StealthStation® navigation system via StealthLink (Medtronic, Inc., Louisville, CO) through a rigidly-attached tracker. In addition, the microscope zoom, m, and focal length, f, are also directly available from StealthLink, which eliminates the need to manually record the acquisition settings. The technical details of stereovision calibration at a single acquisition setting and subsequent surface reconstruction have been well studied [2]. Both the left (IL) and right (IR) camera images depend on image acquisition settings such as m and f. Conceptually, the following functional forms define the images acquired: (1)
For notational simplicity, we drop the subscripts throughout the rest of the paper when an image is not specifically associated with either the left (L) or right (R) camera. We also
Med Image Comput Comput Assist Interv. Author manuscript; available in PMC 2015 February 25.
Ji et al.
Page 3
NIH-PA Author Manuscript
denote the image acquired at a set of reference settings, S0, as IO = G(m0, f0), which represents the lowest magnification (m0) and shortest focal length (f0) that the microscope offers. The choice of S0 is selected for convenience because of the ease with which the microscope can be returned to these settings; however, a different set of reference settings or multiple settings could be utilized (see Discussion). An image obtained at an arbitrary setting, S, is referred to as a “deformed” image (I = G(m, f)). The “deformation field” relating the deformed image to the reference or “undeformed” image is found via optical flow (OF) motion-tracking, which has been well studied [7] and successfully employed in image-guided procedures [1] including stereovision in neurosurgery [6]. Essentially, our technique for stereovision reconstruction at S warps the deformed images into the reference image as if the data were acquired at S0 using deformation/distortion fields obtained from a series of phantom images. Because stereo images at S0 have been calibrated, the warped stereo images acquired at S can then be reconstructed with the same single calibration once the warping is complete. 2.1 Image Deformation due to the Change in Acquisition Settings
NIH-PA Author Manuscript
To determine image deformation due to the change in image acquisition settings, m and f, a phantom was created by printing squares in random positions and intensities on paper. The phantom was first imaged at the reference setting, S0, and then a series of images was acquired by successively changing either m or f (while maintaining the partnered parameter at its respective reference value). Image acquisitions at multiple m values at each f settings were unnecessary because the resulting 2D image deformation induced by a change in m was independent of f settings, at least for the Zeiss Pentero surgical microscope based on setting values from StealthLink.
NIH-PA Author Manuscript
Because the OF algorithm is designed to detect small displacements, deformation fields between images obtained from two adjacent m or f values (instead of relative to the reference values) were computed. The resulting displacement vectors were found to vary radially relative to the focal point along the optical axis (Fig. 1). Thus, a local cylindrical coordinate system was established with its origin at the focal point in order to fit the deformation field as a function of radial distance, r. Because the OF algorithm can produce artifacts especially in image corners with poor lighting conditions, regions near the boundary (
NIH-PA Author Manuscript
Published in final edited form as: Med Image Comput Comput Assist Interv. 2014 ; 17(0 1): 440–447.
Efficient Stereo Image Geometrical Reconstruction at Arbitrary Camera Settings from a Single Calibration Songbai Ji1,2, Xiaoyao Fan1, David W. Roberts2,3, and Keith D. Paulsen1,2,3 1
Thayer School of Engineering, Dartmouth College, Hanover, NH 03755, USA
2
Geisel School of Medicine, Dartmouth College, Hanover, NH 03755, USA
3
Dartmouth Hitchcock Medical Center, Lebanon, NH 03756, USA
Abstract NIH-PA Author Manuscript
Camera calibration is central to obtaining a quantitative image-to-physical-space mapping from stereo images acquired in the operating room (OR). A practical challenge for cameras mounted to the operating microscope is maintenance of image calibration as the surgeon’s field-of-view is repeatedly changed (in terms of zoom and focal settings) throughout a procedure. Here, we present an efficient method for sustaining a quantitative image-to-physical space relationship for arbitrary image acquisition settings (S) without the need for camera re-calibration. Essentially, we warp images acquired at S into the equivalent data acquired at a reference setting, S0, using deformation fields obtained with optical flow by successively imaging a simple phantom. Closed-form expressions for the distortions were derived from which 3D surface reconstruction was performed based on the single calibration at S0. The accuracy of the reconstructed surface was 1.05 mm and 0.59 mm along and perpendicular to the optical axis of the operating microscope on average, respectively, for six phantom image pairs, and was 1.26 mm and 0.71 mm for images acquired with a total of 47 arbitrary settings during three clinical cases. The technique is presented in the context of stereovision; however, it may also be applicable to other types of video image acquisitions (e.g., endoscope) because it does not rely on any a priori knowledge about the camera system itself, suggesting the method is likely of considerable significance.
NIH-PA Author Manuscript
1 Introduction Camera images provide texture intensity from the surface of objects in the scene, and are an increasingly popular form of data in image-guided procedures such as neurosurgery [1]. Calibration is central to obtaining quantitative geometrical information from the camera system to project 2D image pixels into their 3D coordinates in physical space in the case of stereovision. Techniques for calibrating a camera system at fixed zoom and focal settings are well studied [2]. However, many cameras offer a wide range of zoom factors and focal lengths that can be arbitrarily varied to obtain an optimal view [3]; thus, maintenance of camera calibration becomes a practical challenge. Because these images depend on acquisition settings, recovering the camera calibration parameters efficiently and for an
© Springer International Publishing Switzerland 2014 {songbai.ji,xiaoyao.fan,david.w.roberts,keith.d.paulsen}@dartmouth.edu.
Ji et al.
Page 2
arbitrary setting is essential for applications like stereovision in the operating room (OR) where the surgeon is repeatedly altering the field-of-view through the operating microscope.
NIH-PA Author Manuscript
Existing techniques for camera calibration at an arbitrary setting either actively re-calibrate at a given setting on-demand [3] or interpolate camera parameters via bivariate fitting by explicitly modeling each as a polynomial function of zoom and focal length based on data from a dense set of pre-calibrations [4, 5]. Although calibration at a given setting can be fully automated with an on-demand approach [2], repeatedly imaging an instrumented calibration target [3] is inconvenient and cumbersome in the OR. While interpolation of predetermined camera parameters minimizes disruption of surgical workflow, a dense combination of zoom and focal length settings have to be calibrated (and re-calibrated for quality assurance and/or when camera extrinsic parameters are changed, e.g., from repositioning) which too adds to pre-operative activity and personnel time requirements. Consequently, suggestions of a fixed zoom and focus have been made to ensure optimal accuracy [5], but such restrictions significantly limit the effective OR use of camera systems.
NIH-PA Author Manuscript
In this study, we present a method to recover geometry from stereo images at arbitrary camera settings using a single calibration at a fixed (reference) zoom and focal setting. The approach is especially appealing for OR applications because it does not disrupt surgical workflow nor does it require tedious calibration at numerous zoom-focus combinations. The performance of the technique is evaluated on a physical phantom and in three clinical cases involving open cranial surgery with a microscope-mounted stereovision system. However, the general strategy appears to be applicable to other types of stereo/video images (e.g., endoscope).
2 Material and Methods
NIH-PA Author Manuscript
A custom-designed stereovision system consisting of two C-mount cameras (Flea2 model FL2G-50S5C-C, Point Grey Research Inc., Richmond, BC, Canada) was rigidly mounted to a Zeiss surgical microscope (OPMI® Pentero™, Carl Zeiss, Inc., Oberkochen, Germany) through a binocular port [6]. The position and orientation of the microscope was available from a StealthStation® navigation system via StealthLink (Medtronic, Inc., Louisville, CO) through a rigidly-attached tracker. In addition, the microscope zoom, m, and focal length, f, are also directly available from StealthLink, which eliminates the need to manually record the acquisition settings. The technical details of stereovision calibration at a single acquisition setting and subsequent surface reconstruction have been well studied [2]. Both the left (IL) and right (IR) camera images depend on image acquisition settings such as m and f. Conceptually, the following functional forms define the images acquired: (1)
For notational simplicity, we drop the subscripts throughout the rest of the paper when an image is not specifically associated with either the left (L) or right (R) camera. We also
Med Image Comput Comput Assist Interv. Author manuscript; available in PMC 2015 February 25.
Ji et al.
Page 3
NIH-PA Author Manuscript
denote the image acquired at a set of reference settings, S0, as IO = G(m0, f0), which represents the lowest magnification (m0) and shortest focal length (f0) that the microscope offers. The choice of S0 is selected for convenience because of the ease with which the microscope can be returned to these settings; however, a different set of reference settings or multiple settings could be utilized (see Discussion). An image obtained at an arbitrary setting, S, is referred to as a “deformed” image (I = G(m, f)). The “deformation field” relating the deformed image to the reference or “undeformed” image is found via optical flow (OF) motion-tracking, which has been well studied [7] and successfully employed in image-guided procedures [1] including stereovision in neurosurgery [6]. Essentially, our technique for stereovision reconstruction at S warps the deformed images into the reference image as if the data were acquired at S0 using deformation/distortion fields obtained from a series of phantom images. Because stereo images at S0 have been calibrated, the warped stereo images acquired at S can then be reconstructed with the same single calibration once the warping is complete. 2.1 Image Deformation due to the Change in Acquisition Settings
NIH-PA Author Manuscript
To determine image deformation due to the change in image acquisition settings, m and f, a phantom was created by printing squares in random positions and intensities on paper. The phantom was first imaged at the reference setting, S0, and then a series of images was acquired by successively changing either m or f (while maintaining the partnered parameter at its respective reference value). Image acquisitions at multiple m values at each f settings were unnecessary because the resulting 2D image deformation induced by a change in m was independent of f settings, at least for the Zeiss Pentero surgical microscope based on setting values from StealthLink.
NIH-PA Author Manuscript
Because the OF algorithm is designed to detect small displacements, deformation fields between images obtained from two adjacent m or f values (instead of relative to the reference values) were computed. The resulting displacement vectors were found to vary radially relative to the focal point along the optical axis (Fig. 1). Thus, a local cylindrical coordinate system was established with its origin at the focal point in order to fit the deformation field as a function of radial distance, r. Because the OF algorithm can produce artifacts especially in image corners with poor lighting conditions, regions near the boundary (
