Advanced hyperspectral video imaging system using Amici prism Jiao Feng,1 Xiaojing Fang,1 Xun Cao,2,* Chenguang Ma,3 Qionghai Dai,3 Hongbo Zhu,1 and Yongjin Wang1,4 1
Grünberg Research Centre, Nanjing University of Posts and Telecommunications, Nanjing, 210003, China 2 School of Electronic Science and Engineering, Nanjing University, Nanjing, 210093, China 3 Department of Automation, Tsinghua University, Beijing, 100084, China 4 [email protected] *[email protected]
Abstract: In this paper, we propose an advanced hyperspectral video imaging system (AHVIS), which consists of an objective lens, an occlusion mask, a relay lens, an Amici prism and two cameras. An RGB camera is used for spatial reading and a gray scale camera is used for measuring the scene with spectral information. The objective lens collects more light energy from the observed scene and images the scene on an occlusion mask, which subsamples the image of the observed scene. Then, the subsampled image is sent to the gray scale camera through the relay lens and the Amici prism. The Amici prism that is used to realize spectral dispersion along the optical path reduces optical distortions and offers direct view of the scene. The main advantages of the proposed system are improved light throughput and less optical distortion. Furthermore, the presented configuration is more compact, robust and practicable. ©2014 Optical Society of America OCIS codes: (110.4234) Multispectral and hyperspectral imaging; (150.0155) Machine vision optics.
References and links 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13.
W. L. Smith, D. K. Zhou, F. W. Harrison, H. E. Revercomb, A. M. Larar, H. L. Huang, and B. Huang, “Hyperspectral remote sensing of atmospheric profiles from satellites and aircraft,” Proc. SPIE 4151, 94–102 (2001). C. M. Stellman, F. M. Olchowski, and J. V. Michalowicz, “WAR HORSE (wide-area reconnaissance: hyperspectral overhead real-time surveillance experiment),” Proc. SPIE 4379, 339–346 (2001). R. P. Lin, B. R. Dennis, and A. O. Benz, eds., The Reuven Ramaty High-Energy Solar Spectroscopic Imager(RHESSI)-Mission Description and Early Results (Springer, 2003). J. Murguia, G. Diaz, T. Reeves, R. Nelson, J. Mooney, F. Shepherd, G. Griffith, and D. Franco, “Applications of multispectral video,” Proc. SPIE 7780, 77800B (2010). Z. Pan, G. Healey, M. Prasad, and B. Tromberg, “Face Recognition in Hyperspectral Images,” IEEE Trans. Pattern Anal. Mach. Intell. 25(12), 1552–1560 (2003). W. M. Porter and H. T. Enmark, “A system overview of the airborne visible/infrared imaging spectrometer(AVIRIS),” Proc. SPIE 834, 22–31 (1987). R. W. Basedow, D. C. Carmer, and M. E. Anderson, “HYDICE system: Implementation and performance,” Proc. SPIE 2480, 258–267 (1995). N. Gat, “Imaging spectroscopy using tunable filters: a review,” Proc. SPIE 4056, 50–64 (2000). M. Yamaguchi, H. Haneishi, H. Fukuda, J. Kishimoto, H. Kanazawa, M. Tsuchida, R. Iwama, and N. Ohyama, “High-fidelity video and still-image communication based on spectral information: Natural vision system and its applications,” Proc. SPIE 6062, 60620G (2006). M. R. Descour and E. L. Dereniak, “Computed-tomography imaging spectrometer: experimental calibration and reconstruction results,” Appl. Opt. 34(22), 4817–4826 (1995). W. R. Johnson, D. W. Wilson, and G. Bearman, “Spatial-spectral modulating snapshot hyperspectral imager,” Appl. Opt. 45(9), 1898–1908 (2006). N. A. Hagen, E. L. Dereniak, and D. T. Sass, “Maximizing the resolution of a CTIS instrument,” Proc. SPIE 6302, 63020L (2006). N. A. Hagen and E. L. Dereniak, “Analysis of computed tomographic imaging spectrometers. I. Spatial and spectral resolution,” Appl. Opt. 47(28), F85–F95 (2008).
#213390 - $15.00 USD Received 4 Jun 2014; revised 20 Jul 2014; accepted 26 Jul 2014; published 4 Aug 2014 (C) 2014 OSA 11 August 2014 | Vol. 22, No. 16 | DOI:10.1364/OE.22.019348 | OPTICS EXPRESS 19348
14. W. R. Johnson, D. W. Wilson, and G. Bearman, “All-reflective snapshot hyperspectral imager for ultraviolet and infrared applications,” Opt. Lett. 30(12), 1464–1466 (2005). 15. D. J. Brady and M. E. Gehm, “Compressive imaging spectrometers using coded apertures,” Proc. SPIE 6246, 105–115 (2006). 16. M. E. Gehm, R. John, D. J. Brady, R. M. Willett, and T. J. Schulz, “Single-shot compressive spectral imaging with a dual-disperser architecture,” Opt. Express 15(21), 14013–14027 (2007). 17. A. Wagadarikar, R. John, R. Willett, and D. Brady, “Single disperser design for coded aperture snapshot spectral imaging,” Appl. Opt. 47(10), B44–B51 (2008). 18. A. A. Wagadarikar, N. P. Pitsianis, X. Sun, and D. J. Brady, “Video rate spectral imaging using a coded aperture snapshot spectral imager,” Opt. Express 17(8), 6368–6388 (2009). 19. H. H. Barrett, “Editorial: limited-angle tomography for the nineties,” J. Nucl. 31, 1688–1692 (1990). 20. A. Bodkin, A. Sheinis, A. Norton, J. Daly, S. Beaven, and J. Weinheimer, “Snapshot hyperspectral imaging: the hyperpixel array camera,” Proc. SPIE 7334, 73340H (2009). 21. X. Cao, H. Du, X. Tong, Q. Dai, and S. Lin, “A prism-mask system for multispectral video acquisition,” IEEE Trans. Pattern Anal. Mach. Intell. 33(12), 2423–2435 (2011). 22. X. Cao, X. Tong, Q. Dai, and S. Lin, “High resolution multispectral video capture with a hybrid camera system,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR, 2011), pp. 297–304.
1. Introduction Recently, hyperspectral imagers that acquire data within a single snapshot have gained increasing attention to image dynamic scenes. Hyperspectral capture technique generates a map of spectral information, making it an important tool in many diverse applications, including remote sensing [1], military target discrimination [2], astrophysics [3] and computer vision region [4, 5]. A hyperspectral imager measures a detailed spectral signature at each pixel in an image. The acquired spectra provides the detailed information of the captured scene. Each frame can be reconstructed based on its 3D data cube, which is comprised of two spatial dimensions and one spectral dimension, as illustrated in Fig. 1.
Fig. 1. Three-dimensional hyperspectral data cubes.
Since the huge volume of hyperspectral data is required, it is needed to develop the scanning techniques for traditional hyperspectral imagers to acquire all the data, including whiskbroom [6] or pushbroom [7] spatial scanning and tunable filter [8] or filter wheel [9] based on spectral scanning. The above methods can only capture one or two-dimensional subset of a data cube through temporal scanning of the remaining dimension(s). Moreover, the low light collection efficiency from incoherent leads to a poor signal-to-noise ratio (SNR). These methods may capture a static scene well, but they are impractical for dynamic scene capture, since considerable time is required during the scanning process. To capture hyperspectral image at video rate, reconstruction-based systems with snapshot ability such as computed tomographic imaging spectrometry (CTIS) [10–14] and coded aperture imaging spectrometry [15–18] have been presented. However, CTIS is a tomographic imaging system with limited angle [19]. Errors are unavoidable in coded aperture imaging spectrometry owing to the reconstruction limitation for compressive sampling and time complexity of the reconstruction.
#213390 - $15.00 USD Received 4 Jun 2014; revised 20 Jul 2014; accepted 26 Jul 2014; published 4 Aug 2014 (C) 2014 OSA 11 August 2014 | Vol. 22, No. 16 | DOI:10.1364/OE.22.019348 | OPTICS EXPRESS 19349
For direct capture of hyperspectral video, systems based on occlusion masks and prisms have been proposed [20, 21]. Since these systems sacrifice considerable spatial resolution to increase spectral resolution, the utility of captured data for video analysis applications is limited. Recently, a hybrid spectral video imaging system (HVIS) has been demonstrated to overcome these issues [22]. It uses two cameras to record video streams: one camera with spatial reading but only RGB spectral readings and another gray scale camera with direct spectral measurements at sampled scene points. The spectral readings are obtained by using a prism dispersing light into its spectrum. An occlusion mask subsamples the scene and prevents dispersed spectra from overlapping on the sensor. The two forms of video data are integrated by using an information propagation algorithm to yield video with both spatial and spectral information in real time. In this paper, we present an advanced hyperspectral video imaging system using AmiciPrism. With the introduction of an objective lens and an Amici prism, the AHVIS can capture objects at far distance away with improved light throughput and less optical distortion compared to [22]. Moreover, the proposed system is more compact. We focus on interpreting the optical path of AHVIS together with a detailed theoretical analysis on the light throughput and optical distortion of the imaging system. Besides, the other key performances as a spectrometer are also analyzed including spectral resolution, light throughput, spatial and temporal resolution. Additional experiments are presented with comparisons to the HVIS on the light throughput. 2. Optical design of AHVIS
Fig. 2. (a) Overview of the optical design. (b) Dispersion comparison of the traditional prism and Amici prism. (c) Subsample process of the occlusion mask and the occlusion mask’s configuration.
Figure 2(a) shows the optical design of the proposed AHVIS, where two cameras are used to record video streams. An RGB camera is used for spatial reading. A gray scale camera is used for measuring the scene with spectral information. The observed scene is placed beyond the double focal length of the objective lens. The objective lens projects the scene on an occlusion mask which subsamples the image of the observed scene. Then, the subsampled image is sent to the gray scale camera through a relay lens. An off-the-shelf Amici prism is used for realizing spectral dispersion along the optical path. The dispersed spectral information is recorded by the gray scale camera. Therefore, in association with video data captured from RGB camera, video with both spatial and spectral information can be yielded in real time.
#213390 - $15.00 USD Received 4 Jun 2014; revised 20 Jul 2014; accepted 26 Jul 2014; published 4 Aug 2014 (C) 2014 OSA 11 August 2014 | Vol. 22, No. 16 | DOI:10.1364/OE.22.019348 | OPTICS EXPRESS 19350
The proposed AHVIS uses an off-the-shelf Amici prism, which benefits our system design. In contrast to the traditional prism, an Amici prism consists of two prisms. The first prism is the crown glass with medium refractive index and the second one is the flint glass with high refractive index. A disadvantage of the traditional prism is the geometric distortion caused by dispersing the incident light. The main distortions are keystone distortion and smile distortion. Keystone distortion is caused by light rays of different spectra traveling unequal distances through a triangular prism. Smile distortion, named after its curved shape, is caused by different vertical incident directions of the light rays along vertical lines of the scene. These geometric distortions not only complicate calibration, but also necessitate an expanded camera's extra depth of field (DOF) requirement shown in Fig. 2(b), in order to accommodate the different focus distances. The increased extra DOF requirement consequently requires a smaller aperture size, which decreases light throughput. The uniform thickness of an Amici prism significantly reduces the keystone distortion. As a result, the aperture of the camera can be enlarged to improve light throughput, which is of great importance to the design of an optical system. The Amici prism provides less optical distortion and direct view of the scene. Since all the optical components can be placed in a line, the system can be made more compact and much easier to align and calibrate. Meanwhile, the smile distortion of the smaller mask is negligible. In AHVIS, the objective lens in association with occlusion mask is used to subsample the scene shown in Fig. 2(c). The occlusion mask is placed in the imaging plane of the objective lens. The objective lens captures objects at far distance away and images the observed scene on a smaller occlusion mask, which makes AHVIS more compact. Because the objective lens is exchangeable, the camera system is able to alter the field of view. With an occlusion mask, the spectra of the rays passing through the mask holes are nonoverlapping in the recorded frames. The occlusion mask is lithographically patterned as a chrome coating on a quartz substrate. The size of the occlusion mask is 4 cm × 6 cm. The pinhole array is optimized to achieve a balance among light throughput, spectral resolution and spatial resolution. Figure 2(c) shows the distribution pattern of the hole on the occlusion mask used in AHVIS. The width and length of the hole are 50μm and 150μm, respectively. The distances between neighboring holes are 375μm and 250μm, respectively. 3. Theoretical analysis of the system 3.1 Spectral resolution In direct capture of hyperspectral video system, the spectral resolution is determined by the sensor cell size and the dispersed spectrum width on the sensor plane. The optical path in an Amici prism is shown in Fig. 3(a), the outgoing ray angle α' is given in Eq. (1):
α ' = arc sin(n 2(λ ) sin( y − arc sin
n1(λ ) sin(ω − arc sin( n 2 (λ )
sin(α ) )) n1(λ )
))
(1)
where n1, n2 are the refractive index of the K7 prism and the F1 prism, respectively, with incident light wavelength λ, y is the F1 prism angle, ω is the K7 prism angle, α is the angle of incidence and β denotes the angle of the F1 prism’s surface with the image plane. Then the spectral resolution is given in Eq. (2): Rspec =
W
ε
=
f ⋅ (tan(α '(λ e) − β ) − tan(α '(λ s ) − β ))
ε
(2)
where ε is the sensor cell size, W is the dispersed spectrum width and f is the gray scale camera’s focal length. The spectral resolution Rspec curve from 400nm to 800nm (λe is the
#213390 - $15.00 USD Received 4 Jun 2014; revised 20 Jul 2014; accepted 26 Jul 2014; published 4 Aug 2014 (C) 2014 OSA 11 August 2014 | Vol. 22, No. 16 | DOI:10.1364/OE.22.019348 | OPTICS EXPRESS 19351
maximum wavelength in the angle of refraction of the outgoing ray and λs is the minimum wavelength) in AHVIS is shown in Fig. 3(b).
Fig. 3. (a) Optical path in an Amici prism: y is the F1 prism angle, ω is the K7 prism angle, α is the incident angle and β is the angle of the F1 prism’s surface with the image plane, α' is the outgoing ray angle, red line is the maximum wavelength in the angle of refraction of the outgoing ray and purple line is the minimum wavelength. (b) Spectrum width in our proposed system with different incident angle α.
3.2 Light throughput Light throughput is an important aspect of an optical system. Higher light throughput allows shorter exposure time and hence faster video frame rate. Moreover, it also improves the SNR of each intensity measurement.
Fig. 4. (a) Optical path of HVIS: D1 is the distance between the image plane and the mask plane, D2 is the distance between the spot light and the mask plane. (b) Optical path of AHVIS: the distance u between spot light and objective lens is 60 cm, then the image distance v is 20 cm, the focal length of the relay lens f2 is 10 cm, the image plane of objective lens and the focal plane of relay lens are coincident.
Figures 4(a) and 4(b) show optical paths of HVIS and AHVIS, respectively. In HVIS shown in Fig. 4(a), the occlusion mask size is 20 cm × 26 cm and the factors that determine the system’s photon gathering ability are the occlusion mask, prism and the aperture size of the gray scale camera. The emitted energy intensity of the spot light is assumed to be I(o). The total light intensity of the dispersed spectrum I'(c) is given in Eq. (3): I '(c ) = k ' 2 k ' 4k ' 5 I (o) = 0.38 × 1.46 × k ' 5 I (o) = 0.5548k ' 5 I (o)
(3)
where k'2, k'4, k'5 represent the reduced light energy ratio caused by the occlusion mask, prism and aperture, respectively.
#213390 - $15.00 USD Received 4 Jun 2014; revised 20 Jul 2014; accepted 26 Jul 2014; published 4 Aug 2014 (C) 2014 OSA 11 August 2014 | Vol. 22, No. 16 | DOI:10.1364/OE.22.019348 | OPTICS EXPRESS 19352
In AHVIS shown in Fig. 4(b), the occlusion mask size is 4 cm × 6 cm and the factors that determine the system’s photon gathering ability are the objective lens, occlusion mask, relay lens, Amici prism and the aperture size of the gray scale camera. k1, k2, k3, k4 and k5 represent the reduced light energy ratio caused by the objective lens, occlusion mask, relay lens, Amici prism and aperture, respectively. The total light intensity of the dispersed spectrum is I(c): 5
(4)
I (c) = I (o) ∏ ki i =1
Both the objective lens and the relay lens with anti-reflection coating demonstrate high transmission in the wavelength of interest. Both k1 and k3 are assumed to be 0.85. Figure 5(a) shows the schematic map of light transmitting through occlusion mask’s slit, where ds is the width of the mask’s slit and dl is the light spot diameter formed by the spot light. The light transmittance k2 of the slit in the occlusion mask is given in [21] as: k2 = 1− 2× (
arc cos(r ) − r 1 − r 2
π
)
(5)
where r = ds/d1. The curve of k2 and r is shown in Fig. 5(b). We assumed the diameter of the spot light is D. In our case, as the objective lens images the observed scene on the mask, dl is expressed by d1 =
D⋅v u
(6)
where dl is equal to ds. Then k2 is 1.
Fig. 5. (a) Schematic map of light transmitting through occlusion mask’s slit. (b) Curve of r and the reduced light energy ratio k2 of the occlusion mask, when r is 1, no light energy is occluded.
As Amici prism has a complex effect on light energy change, we simulate this effect by tracing rays from a spot light with and without an Amici prism in the optical path. The reduced ratio k4 experimented is 1.04. As existence of the closer virtual point, the reduced ratio k4 is larger than 1. The light loss due to the aperture (k5) is related to the f-number N, while N is related to the DOF: DOF ≈
2H ⋅ s 2 H 2 − s2
(7)
f2 N ⋅c
(8)
where H≈
#213390 - $15.00 USD Received 4 Jun 2014; revised 20 Jul 2014; accepted 26 Jul 2014; published 4 Aug 2014 (C) 2014 OSA 11 August 2014 | Vol. 22, No. 16 | DOI:10.1364/OE.22.019348 | OPTICS EXPRESS 19353
where s is the subjective distance, f is the gray scale camera’s focal length, N is the f-number, c is the circle of confusion and H is the hyperfocal distance. In our proposed system, as being placed at the image plane formed by the objective lens, the occlusion mask subsamples the scene’s image instead of subsampling the scene directly, then the DOF of the camera no longer needs to cover the mask. We derivate required f-number N from the required DOF in AHVIS from Eq. (9) to Eq. (12): DOF ≈
2H ⋅ s2 ≥0 H 2 − s2
H 2 − s2 ≥ 0
(9)
(10)
2
f2 2 −s ≥0 N ⋅c N≤
f2 s⋅c
(11) (12)
the captured light energy I(c) is thus concluded: 5
I (c) = I (o) ∏ ki = 0.7514k 5 I (o) i =1
(13)
Under the same aperture size (k5 = k'5), the light throughput in AHVIS is 1.35 times as that in HVIS. 3.3 Spatial and temporal resolution In the hybrid camera system, spatial resolution is determined by the RGB camera and temporal resolution is determined by the cameras’ frame rate. In our proposed system, we used the 1024 × 768 pixels resolution RGB camera with maximum frame rate 30 fps and the 4096 × 3072 pixels resolution gray scale camera with maximum frame rate 58 fps. 1024 × 768 spatial resolution and 30 fps temporal resolution video can be captured with AHVIS. 4. System setup and experiment
Figure 6 illustrates the physical map of the proposed AHVIS. The objective lens with a focal length of 150 mm produces an image of the observed scene on the occlusion mask. The occlusion mask then subsamples the generated image of the scene. The transmitted light is collected through a relay lens with a focal length of 100 mm and focused onto an Amici prism. The Amici prism disperses light into spectrum. In our case, the first one is a medium dispersion K7 crown glass (refractive index n1 = 1.51) and the second is a higher-dispersion F1 flint glass (refractive index n2 = 1.6). The gray scale camera (Dalsa FA-80-12M1H) with a focal length of 50mm captures 8-bit video at a maximum resolution of 4096 × 3072.
#213390 - $15.00 USD Received 4 Jun 2014; revised 20 Jul 2014; accepted 26 Jul 2014; published 4 Aug 2014 (C) 2014 OSA 11 August 2014 | Vol. 22, No. 16 | DOI:10.1364/OE.22.019348 | OPTICS EXPRESS 19354
Fig. 6. Physical map of the proposed system, it consists of an objective lens, occlusion mask, a relay lens, an Amici prism, gray scale camera and an RGB camera.
4.1 Experiment on light throughput The light throughput of the HVIS and the AHVIS is measured for comparison. In our case, both the focus length and scene distance are fixed. The dispersed spectrum ranging from 400 nm to 800 nm is obtained with the spectral resolution of 5 nm/pixel. The exposure time is 20 ms and the gray scale camera captures the light passing through a hole of the occlusion mask. The measured light throughput in HVIS and AHVIS are shown in Fig. 7. The light throughput is measured by adding captured spectrum’s pixel value together. Figure 7 shows the light throughput increases with enlarging the aperture size of the gray scale camera. With the same aperture size, the light throughput is improved in AHVIS. The theoretical value is calculated according to the captured light throughput in HVIS and the multiple relationship between HVIS and AHVIS proved above. The measured light throughput in AHVIS coincides with the theoretical value well.
Fig. 7. The light throughput increases with enlarging the aperture size of the gray scale camera, with the same aperture size, the light throughput is improved in AHVIS, the measured light throughput in AHVIS coincides with the theoretical value well.
The exposure time also plays an important role on affecting the signal. The shorter exposure time allows faster frame rates, while leading to a bigger signal error. The error ratio is measured by sum-of-squared differences between the normalized captured spectra curve and the ground-truth fluorescent spectra curve. A fluorescent light source is captured by an Ocean Optics USB2000 spectrometer to form the ground-truth spectra. Table 1 shows the error ratio comparison with different exposure time of the HVIS and the AHVIS. It demonstrates that the error ratio decreases with the increase of the exposure time. In
#213390 - $15.00 USD Received 4 Jun 2014; revised 20 Jul 2014; accepted 26 Jul 2014; published 4 Aug 2014 (C) 2014 OSA 11 August 2014 | Vol. 22, No. 16 | DOI:10.1364/OE.22.019348 | OPTICS EXPRESS 19355
comparison with HVIS, AHVIS illustrates lower error ratio under the same exposure time. The results indicate that AHVIS can achieve faster frame rate. Table 1. Error ratio comparison of the two system
Exposure Time 30ms 40ms 50ms 60ms
HVIS 2.05% 1.91% 1.76% 1.71%
AHVIS 1.15% 1.11% 1.10% 1.09%
5. Conclusion
In this paper, the AHVIS is proposed and validated. The system consists of an objective lens, an occlusion mask, a relay lens, an Amici prism and two cameras. An RGB camera is used for spatial reading and a gray scale camera is used for measuring the scene with spectral information. The objective lens collects more light energy from the observed scene and images the scene on an occlusion mask, which subsamples the image of the observed scene. Then, the subsampled image is sent to the gray scale camera through the relay lens and the Amici prism. The Amici prism that is used to realize spectral dispersion along the optical path reduces optical distortions and offers direct view of the scene. The introduction of an objective lens and an Amici prism makes the proposed system throughput-enhanced, compact, direct view and the capability of capturing objects at far distance away. The camera system experimentally demonstrates improved light throughput and low error ratio. Many diverse applications including environmental monitoring, machine vision and scene modeling can benefit from this system. A limitation of this system is the fixed sampling points on the occlusion mask which leads to lack of flexibility and data loss. In future work, we are going to use a spatial light modulator to replace the occlusion mask and design subsample criterion where sampling points’ distribution is determined by the uniformity of color distribution, which can reduce data loss and increase reconstruction accuracy. Acknowledgments
This work is jointly supported by NSFC (11104147, 61322112), research project (NY211001, BJ211026, CXLX13_457).
#213390 - $15.00 USD Received 4 Jun 2014; revised 20 Jul 2014; accepted 26 Jul 2014; published 4 Aug 2014 (C) 2014 OSA 11 August 2014 | Vol. 22, No. 16 | DOI:10.1364/OE.22.019348 | OPTICS EXPRESS 19356
Grünberg Research Centre, Nanjing University of Posts and Telecommunications, Nanjing, 210003, China 2 School of Electronic Science and Engineering, Nanjing University, Nanjing, 210093, China 3 Department of Automation, Tsinghua University, Beijing, 100084, China 4 [email protected] *[email protected]
Abstract: In this paper, we propose an advanced hyperspectral video imaging system (AHVIS), which consists of an objective lens, an occlusion mask, a relay lens, an Amici prism and two cameras. An RGB camera is used for spatial reading and a gray scale camera is used for measuring the scene with spectral information. The objective lens collects more light energy from the observed scene and images the scene on an occlusion mask, which subsamples the image of the observed scene. Then, the subsampled image is sent to the gray scale camera through the relay lens and the Amici prism. The Amici prism that is used to realize spectral dispersion along the optical path reduces optical distortions and offers direct view of the scene. The main advantages of the proposed system are improved light throughput and less optical distortion. Furthermore, the presented configuration is more compact, robust and practicable. ©2014 Optical Society of America OCIS codes: (110.4234) Multispectral and hyperspectral imaging; (150.0155) Machine vision optics.
References and links 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13.
W. L. Smith, D. K. Zhou, F. W. Harrison, H. E. Revercomb, A. M. Larar, H. L. Huang, and B. Huang, “Hyperspectral remote sensing of atmospheric profiles from satellites and aircraft,” Proc. SPIE 4151, 94–102 (2001). C. M. Stellman, F. M. Olchowski, and J. V. Michalowicz, “WAR HORSE (wide-area reconnaissance: hyperspectral overhead real-time surveillance experiment),” Proc. SPIE 4379, 339–346 (2001). R. P. Lin, B. R. Dennis, and A. O. Benz, eds., The Reuven Ramaty High-Energy Solar Spectroscopic Imager(RHESSI)-Mission Description and Early Results (Springer, 2003). J. Murguia, G. Diaz, T. Reeves, R. Nelson, J. Mooney, F. Shepherd, G. Griffith, and D. Franco, “Applications of multispectral video,” Proc. SPIE 7780, 77800B (2010). Z. Pan, G. Healey, M. Prasad, and B. Tromberg, “Face Recognition in Hyperspectral Images,” IEEE Trans. Pattern Anal. Mach. Intell. 25(12), 1552–1560 (2003). W. M. Porter and H. T. Enmark, “A system overview of the airborne visible/infrared imaging spectrometer(AVIRIS),” Proc. SPIE 834, 22–31 (1987). R. W. Basedow, D. C. Carmer, and M. E. Anderson, “HYDICE system: Implementation and performance,” Proc. SPIE 2480, 258–267 (1995). N. Gat, “Imaging spectroscopy using tunable filters: a review,” Proc. SPIE 4056, 50–64 (2000). M. Yamaguchi, H. Haneishi, H. Fukuda, J. Kishimoto, H. Kanazawa, M. Tsuchida, R. Iwama, and N. Ohyama, “High-fidelity video and still-image communication based on spectral information: Natural vision system and its applications,” Proc. SPIE 6062, 60620G (2006). M. R. Descour and E. L. Dereniak, “Computed-tomography imaging spectrometer: experimental calibration and reconstruction results,” Appl. Opt. 34(22), 4817–4826 (1995). W. R. Johnson, D. W. Wilson, and G. Bearman, “Spatial-spectral modulating snapshot hyperspectral imager,” Appl. Opt. 45(9), 1898–1908 (2006). N. A. Hagen, E. L. Dereniak, and D. T. Sass, “Maximizing the resolution of a CTIS instrument,” Proc. SPIE 6302, 63020L (2006). N. A. Hagen and E. L. Dereniak, “Analysis of computed tomographic imaging spectrometers. I. Spatial and spectral resolution,” Appl. Opt. 47(28), F85–F95 (2008).
#213390 - $15.00 USD Received 4 Jun 2014; revised 20 Jul 2014; accepted 26 Jul 2014; published 4 Aug 2014 (C) 2014 OSA 11 August 2014 | Vol. 22, No. 16 | DOI:10.1364/OE.22.019348 | OPTICS EXPRESS 19348
14. W. R. Johnson, D. W. Wilson, and G. Bearman, “All-reflective snapshot hyperspectral imager for ultraviolet and infrared applications,” Opt. Lett. 30(12), 1464–1466 (2005). 15. D. J. Brady and M. E. Gehm, “Compressive imaging spectrometers using coded apertures,” Proc. SPIE 6246, 105–115 (2006). 16. M. E. Gehm, R. John, D. J. Brady, R. M. Willett, and T. J. Schulz, “Single-shot compressive spectral imaging with a dual-disperser architecture,” Opt. Express 15(21), 14013–14027 (2007). 17. A. Wagadarikar, R. John, R. Willett, and D. Brady, “Single disperser design for coded aperture snapshot spectral imaging,” Appl. Opt. 47(10), B44–B51 (2008). 18. A. A. Wagadarikar, N. P. Pitsianis, X. Sun, and D. J. Brady, “Video rate spectral imaging using a coded aperture snapshot spectral imager,” Opt. Express 17(8), 6368–6388 (2009). 19. H. H. Barrett, “Editorial: limited-angle tomography for the nineties,” J. Nucl. 31, 1688–1692 (1990). 20. A. Bodkin, A. Sheinis, A. Norton, J. Daly, S. Beaven, and J. Weinheimer, “Snapshot hyperspectral imaging: the hyperpixel array camera,” Proc. SPIE 7334, 73340H (2009). 21. X. Cao, H. Du, X. Tong, Q. Dai, and S. Lin, “A prism-mask system for multispectral video acquisition,” IEEE Trans. Pattern Anal. Mach. Intell. 33(12), 2423–2435 (2011). 22. X. Cao, X. Tong, Q. Dai, and S. Lin, “High resolution multispectral video capture with a hybrid camera system,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR, 2011), pp. 297–304.
1. Introduction Recently, hyperspectral imagers that acquire data within a single snapshot have gained increasing attention to image dynamic scenes. Hyperspectral capture technique generates a map of spectral information, making it an important tool in many diverse applications, including remote sensing [1], military target discrimination [2], astrophysics [3] and computer vision region [4, 5]. A hyperspectral imager measures a detailed spectral signature at each pixel in an image. The acquired spectra provides the detailed information of the captured scene. Each frame can be reconstructed based on its 3D data cube, which is comprised of two spatial dimensions and one spectral dimension, as illustrated in Fig. 1.
Fig. 1. Three-dimensional hyperspectral data cubes.
Since the huge volume of hyperspectral data is required, it is needed to develop the scanning techniques for traditional hyperspectral imagers to acquire all the data, including whiskbroom [6] or pushbroom [7] spatial scanning and tunable filter [8] or filter wheel [9] based on spectral scanning. The above methods can only capture one or two-dimensional subset of a data cube through temporal scanning of the remaining dimension(s). Moreover, the low light collection efficiency from incoherent leads to a poor signal-to-noise ratio (SNR). These methods may capture a static scene well, but they are impractical for dynamic scene capture, since considerable time is required during the scanning process. To capture hyperspectral image at video rate, reconstruction-based systems with snapshot ability such as computed tomographic imaging spectrometry (CTIS) [10–14] and coded aperture imaging spectrometry [15–18] have been presented. However, CTIS is a tomographic imaging system with limited angle [19]. Errors are unavoidable in coded aperture imaging spectrometry owing to the reconstruction limitation for compressive sampling and time complexity of the reconstruction.
#213390 - $15.00 USD Received 4 Jun 2014; revised 20 Jul 2014; accepted 26 Jul 2014; published 4 Aug 2014 (C) 2014 OSA 11 August 2014 | Vol. 22, No. 16 | DOI:10.1364/OE.22.019348 | OPTICS EXPRESS 19349
For direct capture of hyperspectral video, systems based on occlusion masks and prisms have been proposed [20, 21]. Since these systems sacrifice considerable spatial resolution to increase spectral resolution, the utility of captured data for video analysis applications is limited. Recently, a hybrid spectral video imaging system (HVIS) has been demonstrated to overcome these issues [22]. It uses two cameras to record video streams: one camera with spatial reading but only RGB spectral readings and another gray scale camera with direct spectral measurements at sampled scene points. The spectral readings are obtained by using a prism dispersing light into its spectrum. An occlusion mask subsamples the scene and prevents dispersed spectra from overlapping on the sensor. The two forms of video data are integrated by using an information propagation algorithm to yield video with both spatial and spectral information in real time. In this paper, we present an advanced hyperspectral video imaging system using AmiciPrism. With the introduction of an objective lens and an Amici prism, the AHVIS can capture objects at far distance away with improved light throughput and less optical distortion compared to [22]. Moreover, the proposed system is more compact. We focus on interpreting the optical path of AHVIS together with a detailed theoretical analysis on the light throughput and optical distortion of the imaging system. Besides, the other key performances as a spectrometer are also analyzed including spectral resolution, light throughput, spatial and temporal resolution. Additional experiments are presented with comparisons to the HVIS on the light throughput. 2. Optical design of AHVIS
Fig. 2. (a) Overview of the optical design. (b) Dispersion comparison of the traditional prism and Amici prism. (c) Subsample process of the occlusion mask and the occlusion mask’s configuration.
Figure 2(a) shows the optical design of the proposed AHVIS, where two cameras are used to record video streams. An RGB camera is used for spatial reading. A gray scale camera is used for measuring the scene with spectral information. The observed scene is placed beyond the double focal length of the objective lens. The objective lens projects the scene on an occlusion mask which subsamples the image of the observed scene. Then, the subsampled image is sent to the gray scale camera through a relay lens. An off-the-shelf Amici prism is used for realizing spectral dispersion along the optical path. The dispersed spectral information is recorded by the gray scale camera. Therefore, in association with video data captured from RGB camera, video with both spatial and spectral information can be yielded in real time.
#213390 - $15.00 USD Received 4 Jun 2014; revised 20 Jul 2014; accepted 26 Jul 2014; published 4 Aug 2014 (C) 2014 OSA 11 August 2014 | Vol. 22, No. 16 | DOI:10.1364/OE.22.019348 | OPTICS EXPRESS 19350
The proposed AHVIS uses an off-the-shelf Amici prism, which benefits our system design. In contrast to the traditional prism, an Amici prism consists of two prisms. The first prism is the crown glass with medium refractive index and the second one is the flint glass with high refractive index. A disadvantage of the traditional prism is the geometric distortion caused by dispersing the incident light. The main distortions are keystone distortion and smile distortion. Keystone distortion is caused by light rays of different spectra traveling unequal distances through a triangular prism. Smile distortion, named after its curved shape, is caused by different vertical incident directions of the light rays along vertical lines of the scene. These geometric distortions not only complicate calibration, but also necessitate an expanded camera's extra depth of field (DOF) requirement shown in Fig. 2(b), in order to accommodate the different focus distances. The increased extra DOF requirement consequently requires a smaller aperture size, which decreases light throughput. The uniform thickness of an Amici prism significantly reduces the keystone distortion. As a result, the aperture of the camera can be enlarged to improve light throughput, which is of great importance to the design of an optical system. The Amici prism provides less optical distortion and direct view of the scene. Since all the optical components can be placed in a line, the system can be made more compact and much easier to align and calibrate. Meanwhile, the smile distortion of the smaller mask is negligible. In AHVIS, the objective lens in association with occlusion mask is used to subsample the scene shown in Fig. 2(c). The occlusion mask is placed in the imaging plane of the objective lens. The objective lens captures objects at far distance away and images the observed scene on a smaller occlusion mask, which makes AHVIS more compact. Because the objective lens is exchangeable, the camera system is able to alter the field of view. With an occlusion mask, the spectra of the rays passing through the mask holes are nonoverlapping in the recorded frames. The occlusion mask is lithographically patterned as a chrome coating on a quartz substrate. The size of the occlusion mask is 4 cm × 6 cm. The pinhole array is optimized to achieve a balance among light throughput, spectral resolution and spatial resolution. Figure 2(c) shows the distribution pattern of the hole on the occlusion mask used in AHVIS. The width and length of the hole are 50μm and 150μm, respectively. The distances between neighboring holes are 375μm and 250μm, respectively. 3. Theoretical analysis of the system 3.1 Spectral resolution In direct capture of hyperspectral video system, the spectral resolution is determined by the sensor cell size and the dispersed spectrum width on the sensor plane. The optical path in an Amici prism is shown in Fig. 3(a), the outgoing ray angle α' is given in Eq. (1):
α ' = arc sin(n 2(λ ) sin( y − arc sin
n1(λ ) sin(ω − arc sin( n 2 (λ )
sin(α ) )) n1(λ )
))
(1)
where n1, n2 are the refractive index of the K7 prism and the F1 prism, respectively, with incident light wavelength λ, y is the F1 prism angle, ω is the K7 prism angle, α is the angle of incidence and β denotes the angle of the F1 prism’s surface with the image plane. Then the spectral resolution is given in Eq. (2): Rspec =
W
ε
=
f ⋅ (tan(α '(λ e) − β ) − tan(α '(λ s ) − β ))
ε
(2)
where ε is the sensor cell size, W is the dispersed spectrum width and f is the gray scale camera’s focal length. The spectral resolution Rspec curve from 400nm to 800nm (λe is the
#213390 - $15.00 USD Received 4 Jun 2014; revised 20 Jul 2014; accepted 26 Jul 2014; published 4 Aug 2014 (C) 2014 OSA 11 August 2014 | Vol. 22, No. 16 | DOI:10.1364/OE.22.019348 | OPTICS EXPRESS 19351
maximum wavelength in the angle of refraction of the outgoing ray and λs is the minimum wavelength) in AHVIS is shown in Fig. 3(b).
Fig. 3. (a) Optical path in an Amici prism: y is the F1 prism angle, ω is the K7 prism angle, α is the incident angle and β is the angle of the F1 prism’s surface with the image plane, α' is the outgoing ray angle, red line is the maximum wavelength in the angle of refraction of the outgoing ray and purple line is the minimum wavelength. (b) Spectrum width in our proposed system with different incident angle α.
3.2 Light throughput Light throughput is an important aspect of an optical system. Higher light throughput allows shorter exposure time and hence faster video frame rate. Moreover, it also improves the SNR of each intensity measurement.
Fig. 4. (a) Optical path of HVIS: D1 is the distance between the image plane and the mask plane, D2 is the distance between the spot light and the mask plane. (b) Optical path of AHVIS: the distance u between spot light and objective lens is 60 cm, then the image distance v is 20 cm, the focal length of the relay lens f2 is 10 cm, the image plane of objective lens and the focal plane of relay lens are coincident.
Figures 4(a) and 4(b) show optical paths of HVIS and AHVIS, respectively. In HVIS shown in Fig. 4(a), the occlusion mask size is 20 cm × 26 cm and the factors that determine the system’s photon gathering ability are the occlusion mask, prism and the aperture size of the gray scale camera. The emitted energy intensity of the spot light is assumed to be I(o). The total light intensity of the dispersed spectrum I'(c) is given in Eq. (3): I '(c ) = k ' 2 k ' 4k ' 5 I (o) = 0.38 × 1.46 × k ' 5 I (o) = 0.5548k ' 5 I (o)
(3)
where k'2, k'4, k'5 represent the reduced light energy ratio caused by the occlusion mask, prism and aperture, respectively.
#213390 - $15.00 USD Received 4 Jun 2014; revised 20 Jul 2014; accepted 26 Jul 2014; published 4 Aug 2014 (C) 2014 OSA 11 August 2014 | Vol. 22, No. 16 | DOI:10.1364/OE.22.019348 | OPTICS EXPRESS 19352
In AHVIS shown in Fig. 4(b), the occlusion mask size is 4 cm × 6 cm and the factors that determine the system’s photon gathering ability are the objective lens, occlusion mask, relay lens, Amici prism and the aperture size of the gray scale camera. k1, k2, k3, k4 and k5 represent the reduced light energy ratio caused by the objective lens, occlusion mask, relay lens, Amici prism and aperture, respectively. The total light intensity of the dispersed spectrum is I(c): 5
(4)
I (c) = I (o) ∏ ki i =1
Both the objective lens and the relay lens with anti-reflection coating demonstrate high transmission in the wavelength of interest. Both k1 and k3 are assumed to be 0.85. Figure 5(a) shows the schematic map of light transmitting through occlusion mask’s slit, where ds is the width of the mask’s slit and dl is the light spot diameter formed by the spot light. The light transmittance k2 of the slit in the occlusion mask is given in [21] as: k2 = 1− 2× (
arc cos(r ) − r 1 − r 2
π
)
(5)
where r = ds/d1. The curve of k2 and r is shown in Fig. 5(b). We assumed the diameter of the spot light is D. In our case, as the objective lens images the observed scene on the mask, dl is expressed by d1 =
D⋅v u
(6)
where dl is equal to ds. Then k2 is 1.
Fig. 5. (a) Schematic map of light transmitting through occlusion mask’s slit. (b) Curve of r and the reduced light energy ratio k2 of the occlusion mask, when r is 1, no light energy is occluded.
As Amici prism has a complex effect on light energy change, we simulate this effect by tracing rays from a spot light with and without an Amici prism in the optical path. The reduced ratio k4 experimented is 1.04. As existence of the closer virtual point, the reduced ratio k4 is larger than 1. The light loss due to the aperture (k5) is related to the f-number N, while N is related to the DOF: DOF ≈
2H ⋅ s 2 H 2 − s2
(7)
f2 N ⋅c
(8)
where H≈
#213390 - $15.00 USD Received 4 Jun 2014; revised 20 Jul 2014; accepted 26 Jul 2014; published 4 Aug 2014 (C) 2014 OSA 11 August 2014 | Vol. 22, No. 16 | DOI:10.1364/OE.22.019348 | OPTICS EXPRESS 19353
where s is the subjective distance, f is the gray scale camera’s focal length, N is the f-number, c is the circle of confusion and H is the hyperfocal distance. In our proposed system, as being placed at the image plane formed by the objective lens, the occlusion mask subsamples the scene’s image instead of subsampling the scene directly, then the DOF of the camera no longer needs to cover the mask. We derivate required f-number N from the required DOF in AHVIS from Eq. (9) to Eq. (12): DOF ≈
2H ⋅ s2 ≥0 H 2 − s2
H 2 − s2 ≥ 0
(9)
(10)
2
f2 2 −s ≥0 N ⋅c N≤
f2 s⋅c
(11) (12)
the captured light energy I(c) is thus concluded: 5
I (c) = I (o) ∏ ki = 0.7514k 5 I (o) i =1
(13)
Under the same aperture size (k5 = k'5), the light throughput in AHVIS is 1.35 times as that in HVIS. 3.3 Spatial and temporal resolution In the hybrid camera system, spatial resolution is determined by the RGB camera and temporal resolution is determined by the cameras’ frame rate. In our proposed system, we used the 1024 × 768 pixels resolution RGB camera with maximum frame rate 30 fps and the 4096 × 3072 pixels resolution gray scale camera with maximum frame rate 58 fps. 1024 × 768 spatial resolution and 30 fps temporal resolution video can be captured with AHVIS. 4. System setup and experiment
Figure 6 illustrates the physical map of the proposed AHVIS. The objective lens with a focal length of 150 mm produces an image of the observed scene on the occlusion mask. The occlusion mask then subsamples the generated image of the scene. The transmitted light is collected through a relay lens with a focal length of 100 mm and focused onto an Amici prism. The Amici prism disperses light into spectrum. In our case, the first one is a medium dispersion K7 crown glass (refractive index n1 = 1.51) and the second is a higher-dispersion F1 flint glass (refractive index n2 = 1.6). The gray scale camera (Dalsa FA-80-12M1H) with a focal length of 50mm captures 8-bit video at a maximum resolution of 4096 × 3072.
#213390 - $15.00 USD Received 4 Jun 2014; revised 20 Jul 2014; accepted 26 Jul 2014; published 4 Aug 2014 (C) 2014 OSA 11 August 2014 | Vol. 22, No. 16 | DOI:10.1364/OE.22.019348 | OPTICS EXPRESS 19354
Fig. 6. Physical map of the proposed system, it consists of an objective lens, occlusion mask, a relay lens, an Amici prism, gray scale camera and an RGB camera.
4.1 Experiment on light throughput The light throughput of the HVIS and the AHVIS is measured for comparison. In our case, both the focus length and scene distance are fixed. The dispersed spectrum ranging from 400 nm to 800 nm is obtained with the spectral resolution of 5 nm/pixel. The exposure time is 20 ms and the gray scale camera captures the light passing through a hole of the occlusion mask. The measured light throughput in HVIS and AHVIS are shown in Fig. 7. The light throughput is measured by adding captured spectrum’s pixel value together. Figure 7 shows the light throughput increases with enlarging the aperture size of the gray scale camera. With the same aperture size, the light throughput is improved in AHVIS. The theoretical value is calculated according to the captured light throughput in HVIS and the multiple relationship between HVIS and AHVIS proved above. The measured light throughput in AHVIS coincides with the theoretical value well.
Fig. 7. The light throughput increases with enlarging the aperture size of the gray scale camera, with the same aperture size, the light throughput is improved in AHVIS, the measured light throughput in AHVIS coincides with the theoretical value well.
The exposure time also plays an important role on affecting the signal. The shorter exposure time allows faster frame rates, while leading to a bigger signal error. The error ratio is measured by sum-of-squared differences between the normalized captured spectra curve and the ground-truth fluorescent spectra curve. A fluorescent light source is captured by an Ocean Optics USB2000 spectrometer to form the ground-truth spectra. Table 1 shows the error ratio comparison with different exposure time of the HVIS and the AHVIS. It demonstrates that the error ratio decreases with the increase of the exposure time. In
#213390 - $15.00 USD Received 4 Jun 2014; revised 20 Jul 2014; accepted 26 Jul 2014; published 4 Aug 2014 (C) 2014 OSA 11 August 2014 | Vol. 22, No. 16 | DOI:10.1364/OE.22.019348 | OPTICS EXPRESS 19355
comparison with HVIS, AHVIS illustrates lower error ratio under the same exposure time. The results indicate that AHVIS can achieve faster frame rate. Table 1. Error ratio comparison of the two system
Exposure Time 30ms 40ms 50ms 60ms
HVIS 2.05% 1.91% 1.76% 1.71%
AHVIS 1.15% 1.11% 1.10% 1.09%
5. Conclusion
In this paper, the AHVIS is proposed and validated. The system consists of an objective lens, an occlusion mask, a relay lens, an Amici prism and two cameras. An RGB camera is used for spatial reading and a gray scale camera is used for measuring the scene with spectral information. The objective lens collects more light energy from the observed scene and images the scene on an occlusion mask, which subsamples the image of the observed scene. Then, the subsampled image is sent to the gray scale camera through the relay lens and the Amici prism. The Amici prism that is used to realize spectral dispersion along the optical path reduces optical distortions and offers direct view of the scene. The introduction of an objective lens and an Amici prism makes the proposed system throughput-enhanced, compact, direct view and the capability of capturing objects at far distance away. The camera system experimentally demonstrates improved light throughput and low error ratio. Many diverse applications including environmental monitoring, machine vision and scene modeling can benefit from this system. A limitation of this system is the fixed sampling points on the occlusion mask which leads to lack of flexibility and data loss. In future work, we are going to use a spatial light modulator to replace the occlusion mask and design subsample criterion where sampling points’ distribution is determined by the uniformity of color distribution, which can reduce data loss and increase reconstruction accuracy. Acknowledgments
This work is jointly supported by NSFC (11104147, 61322112), research project (NY211001, BJ211026, CXLX13_457).
#213390 - $15.00 USD Received 4 Jun 2014; revised 20 Jul 2014; accepted 26 Jul 2014; published 4 Aug 2014 (C) 2014 OSA 11 August 2014 | Vol. 22, No. 16 | DOI:10.1364/OE.22.019348 | OPTICS EXPRESS 19356
