Design of a high-numerical-aperture digital micromirror device camera with high dynamic range Yang Qiao, Xiping Xu,* Tao Liu, and Yue Pan Department of Opto-Electronic Engineering, Changchun University of Science and Technology, Jilin 130022, China *Corresponding author: [email protected] Received 25 August 2014; accepted 31 October 2014; posted 14 November 2014 (Doc. ID 221675); published 23 December 2014
A high-NA imaging system with high dynamic range is presented based on a digital micromirror device (DMD). The DMD camera consists of an objective imaging system and a relay imaging system, connected by a DMD chip. With the introduction of a total internal reflection prism system, the objective imaging system is designed with a working F/# of 1.97, breaking through the F/2.45 limitation of conventional DMD projection lenses. As for the relay imaging system, an off-axis design that could correct off-axis aberrations of the tilt relay imaging system is developed. This structure has the advantage of increasing the NA of the imaging system while maintaining a compact size. Investigation revealed that the dynamic range of a DMD camera could be greatly increased, by 2.41 times. We built one prototype DMD camera with a working F/# of 1.23, and the field experiments proved the validity and reliability our work. © 2014 Optical Society of America OCIS codes: (080.0080) Geometric optics; (080.3620) Lens system design; (100.0100) Image processing; (110.0110) Imaging systems. http://dx.doi.org/10.1364/AO.54.000060
1. Introduction
Recent years have witnessed the emergence and development of lane departure warning (LDW), forward collision warning (FCW), and traffic sign recognition (TSR) using monocular cameras [1–3]. Although these driving assistance systems could work well in the daytime, nearly none of them could work effectively at night. In addition to the signal-tonoise ratio (SNR) problem under low illumination, the dynamic range is a big problem. When there are vehicles’ headlamps and other high-brightness light sources against the dark background, the picture of the imaging system will be incomplete because of the saturation area in CCD/CMOS chip. Since the dynamic range of an image sensor is not high enough to capture the scenes, this will lead to some serious consequences. 1559-128X/15/010060-11$15.00/0 © 2015 Optical Society of America 60
APPLIED OPTICS / Vol. 54, No. 1 / 1 January 2015
There are two challenges presenting problems in the development of LDW, FCW and TSR cameras. One involves improving the imaging quality in extremely low-illumination scenes. The other is increasing the dynamic range of digital cameras to record more scene details from shadows to highlights. As for the first method, it can be achieved by increasing both the size of the pixel and the aperture value of the imaging optical system. Together, these effects could generally benefit the SNR performance of the camera, and consequently enable the camera to capture more visual details. As for the high dynamic range, a lot of work has been done [4–6]. One approach is to use a beam splitter and two detectors to generate two copies of the optical image of the scene. Another approach is the spatially varying pixel exposure technique [7], which implements multiple capture by sacrificing spatial resolution. The recent technique of the digital pixel system (DPS) could also help to improve high dynamic range imaging. The DPS image sensor employs a self-reset
DPS architecture where each pixel is capable of resetting the exposure itself. In addition to the ideas proposed above, Nayar et al. [8] introduced a method of using digital micromirror devices (DMDs) for dynamic range enhancement. In Nayar et al.’s work, a DMD-based spatial light modulator has been implemented to vary the scene radiance received on each camera pixel. This has enabled highdynamic-range imaging at high speed. They have built a system with an off-the-shelf lens and modified one projector. However, in the system they propose, the separation of the objective and relay path, as far as we can tell, is just along the diagonal of the micromirrors, which is 45° to the array orthogonal. This will make the entire system bulky. Meanwhile, the tilt angle of the DMD mirror is 12°, which means that the working F/# will be no smaller than F/ 2.45 [2]. Also, the resolutions of both the DMD and CCD chips are relatively small. All those factors would limit the application of the DMD camera. In this paper, we design and build one high-NA and high-dynamic-range imaging DMD camera based on the work of Nayar et al. With the introduction of the DMD as a grayscale modulator in an optical imaging system, the exposure time of CCD pixels can be adjusted by changing the binary on–off pattern of corresponding DMD micromirrors. As a result, the images captured by a DMD camera can acquire the best exposure time compared with a conventional digital camera. Consequently, high-dynamic-range imaging can be achieved. The DMD camera proposed in this paper features strong light-collecting power with a working F/# of 1.23, and the dynamic range is 2.41 times higher than for a conventional camera. It could be an effective solution for camera driving assistance systems, where low-illumination imaging and a high dynamic imaging range are of great importance for detection. In our work, an innovative off-axis relay optical system is presented to correct the off-axis aberration of a relay imaging system. We decrease the working F/# to 1.23, which, to our knowledge, is the lowest working F/# presented for DMD cameras. With the introduction of the total internal reflection (TIR) prism system as the separator of the relay path and objective path, a very compact system can be designed. High resolution is also a merit of our system. The DMD used in this system model has 1024 × 768 pixels, and the CCD is 1360 × 1024. The working principle and design challenges of DMD cameras are described in Section 2. The optical system design and overall system specifications are presented in Section 3. The system evaluation and prototype testing results are given Section 4. Section 5 summarizes our work and research findings.
Fig. 1. Working principle of the DMD camera.
off-axis relay imaging lens, a TIR prism system, a DMD, a CCD detector, and a programmable controller. In general, it can be separated into three main parts: the objective imaging system, the relay imaging system, and the programmable controller. The main work of the objective imaging system is to capture the image of the scene. The relay imaging system reimages the image on the DMD to the CCD detector, and the image on the DMD is modified by the programmable controller. All of the three systems are synchronized to have real-time processing of the image. Figure 2 shows the optical function of the DMD. The DMD is a spatial light modulator consisting of an array of micromirrors that can be individually switched 12° or −12° to either direct or divert incident light corresponding to an “on” or “off” state.
2. Working Principle and Challenges of the DMD Camera
The working principle of the DMD camera is shown in Fig. 1. It can be seen in Fig. 1 that the whole system contains six parts: an objective imaging lens, an
Fig. 2. Optical function of the DMD. (a) Tilt incident light and (b) vertical incident light. 1 January 2015 / Vol. 54, No. 1 / APPLIED OPTICS
61
This can be used in projection systems, digital printers, optical spectrometers, and wavefront correction systems [9–13]. As illustrated in Fig. 2, the incident light could be modulated by a DMD due to its working principle. In the DMD camera, each individual DMD micromirror could operate as a controllable high-speed shutter. The relay imaging system of the DMD camera is well designed and calibrated. And the positions of the CCD sensor and the DMD micromirror array are in a conjugate relationship. The working process of the DMD camera can be expressed as follows. All of the micromirrors of the DMD are in the on state in the beginning, so the image will be reflected and reimaged on the CCD. The picture from the CCD detector will be delivered to a microprocessor. After image processing, the saturation area and the distribution of light illumination in the image can be calculated. Combining the correspondence of the DMD mirrors to the CCD pixels, the DMD camera program will turn corresponding individual DMD micromirrors to the off state, and the harmfully high light is modulated. In real practical work, the DMD works fast enough between the on and off states to be integrated by the CCD. Therefore, the saturation image area is depressed by the grayscale modulation. In such an application, the optimized exposure of the dark scene can be achieved and the high-dynamic-range imaging can be greatly improved. In order to design a high-speed and high-dynamicrange imaging DMD camera with good imaging quality, there are a lot of challenges. One important challenge is to increase the light collection power, or “speed,” of this camera, so that the imaging system can make details visible under low illumination. This can be done in one of two ways. The first is to increase the solid angle subtended from the image by increasing the aperture of the objective imaging system. The second way, which is deduced from the Abbe sine condition, is based on the magnification of the relay imaging system: β
hCCD n sin u ; hDMD n0 sin u0
(1)
where n sin u and n0 sin u0 are the object-side NA and image-side NA for the relay optical system, respectively, the refractive index of air n n0 1.0, and hCCD and hDMD are the size of the CCD sensor and the DMD active area, respectively. Then the image-side NA of the DMD camera NADC n0 sin u0 can be deduced by NADC NA ×
hDMD ; hCCD
(2)
where NA n sin u. Equation (2) shows that NADC can be increased with a large ratio of hDMD to hCCD . In our work, both of the two ways are utilized to increase the NA of our DMD camera. 62
APPLIED OPTICS / Vol. 54, No. 1 / 1 January 2015
The DMD camera resembles a single-panel DMD projector but is more complicated because it combines two imaging systems. So the second challenge is the optomechanical system structure design. Considering that DMD is a reflective device, it is crucial to prevent overlap of the on-state and off-state light bundles of the two imaging paths. In our work, unlike the DMD cameras proposed by Nayar et al. [8] and Ri et al. [13], one cuboid TIR prism system is introduced to separate the incident light from the light reflected by the DMD. With the use of a TIR prism for the physical separation of the two paths, a very compact system can be achieved. In addition, the unique working principle of the DMD also leads to the off-axis aberration correction, which is the third challenge. The TIR prism system used in this DMD camera consists of one rectangular prism and one right-angled trapezoid prism. As for the objective imaging system, the optic axis achieves TIR by the rectangular prism and is perpendicular to the surface plane of the DMD. After being reflected by the on-state micromirrors, the optic axis of the relay imaging system is tilted 24° with the DMD plane. Therefore, the object surface and the image surface of the relay imaging system must be all tilt with the tilt optic axis. This will create a lot of issues concerning the off-axis aberrations and make it much more difficult for aberration correction in the relay optical system. As for our work, we introduced an innovative off-axis optical system with a tilt-and-decenter optical element to correct the off-axis aberrations. The optical system we propose was well designed, with high resolution and good image quality. This structure also has the advantage of increasing the speed or NA of the imaging system while maintaining a compact size. Thus, the whole optical system could collect more light from the scene, pass it to the CCD, and potentially improve the imaging contrast. In this paper we discuss the optical system design with a DMD camera that could be greatly used in DMD-based imaging systems. The fourth challenge is associated with the optical efficiency. The optical system consists of two imaging lenses, the TIR prism and the DMD chip. Each surface of the optical elements used in this system will cause surface reflections. These reflections will reduce the optical efficiency and diminish the contrast ratio. Therefore, the optical efficiency is an important parameter in evaluating this whole system, which will be discussed in Section 4. The last challenge is associated with the response time. As for the DMD camera, real-time imaging should be guaranteed in order to be used in driving assistant systems. Such a system has a set of devices connected to one programmable controller. Therefore, a short response time for each device, fast data processing, and efficient program control are major issues. In the prototype we propose that the frame frequency of the CCD camera is 100 Hz and the DMD is 100 Hz using the development kit proposed by Texas Instruments. The process of data
acquisition can be summarized as follows. The CCD acquires the first frame from the scene. Then the image will be sent to the on-board processor to generate a grayscale model image, and this model image will be sent to the DMD. These two steps will last about 20 ms. Then the DMD will modulate the image, and this process takes about 10 ms. Before the third step, the processor of the DMD will send a synchronization signal to the CCD, and the CCD will start to acquire this modulated image simultaneously. This image, with optimal exposure time for each pixel, is transmitted for the driving assistant system. The ultimate frame frequency of the whole system is 25 Hz for our DMD camera prototype. 3. Optical System Design A.
Optical Design of the Objective Imaging System
As discussed above, the main function of the objective imaging system is to collect as much light as possible from the scene and capture the image on the active area of the DMD. As illustrated in Fig. 3, the objective imaging system consists of an objective imaging lens, a rectangular TIR prism (unfolded), and a DMD chip. The light path of the objective imaging system is separated from the relay imaging path by the TIR surface of the rectangular prism. The unfolded rectangular prism is equivalent to plate glass, which means the optical axis of the objective imaging system is perpendicular to the surface plane of the DMD. The unique structure of the DMD camera means high resolution and good image quality can be required for the objective imaging system. Also, the structure is telecentric which is more like the projection optical system in single-DMD projectors. As for a conventional projection lens, the maximum usable NA of the optical is generally set by the device tilt angle of DMD micromirrors. Normally, the illumination angle is 2 times the 12° mirror tilt angle. The working F/# of the project lens could be given by F∕#Con
1 ; 2 NA
instead of the commonly used paraxial working F∕ 1∕2n tan u, we use the working F/# in this paper. That is because the working F/# directly relates the F# and NAs of the projector lens and relay optics. Also, the working F/# is based upon real ray data at the actual conjugates of the optical system. Thus it is generally much more convenient when evaluating the performance of our high-NA system. Compared with a conventional telecentric DMD projection lens, the limitation of twice the device tilt angle is broken in our DMD camera with the use of a nonsymmetrical NA. The analysis of the NA of our objective imaging optical systems involves two times the TIR at the TIR surface. Figure 4 illustrates the light path of the incident imaging light bundle with different F/# values. As can be seen in Fig. 4(a), when the imaging beam enters the prism with an incident angle larger than the critical angle, the whole light bundle will be totally internally reflected at the TIR surface and the
(3)
where NA n sin u is the image space NA of the project lens, which is also the object-side NA of the relay system. It is recommended by the DMD user manual [14] that a 12° device would utilize a cone angle of u 11.77° for illumination and projection. Therefore, the conventional telecentric project lens has a working F/# of F/2.45. One should note that
Fig. 3. Layout of the objective imaging system.
Fig. 4. Light path of the incident imaging light bundle with different F/# values of the objective imaging lens: (a) F/7.41, (b) F/7.41–F/2.49, and (c) F/1.9. 1 January 2015 / Vol. 54, No. 1 / APPLIED OPTICS
63
whole beam will reach the DMD active area. This is the first incidence of TIR. The material of the prism is BK7, whose refractive index is approximately nprism 1.5163. The refractive index of air is nair 1. Based on Snell’s law, the critical angle at surface 2 is n sin θC air sin 90° 0.66: (4) nprism It can be deduced that θC 41.26°:
(5)
Considering the rectangular prism has an oblique angle of 45°, the angle between the optic axis and the ray, which is incident on the TIR surface with the critical angle, is Δθ 45° − θC 3.74°:
(6)
As a result, only incident rays within a cone angle of 3.74° can be totally internally reflected and enter the system. The corresponding working F/# that meets this TIR requirement is F∕1stTIR 7.67:
(7)
This is quite a small aperture, which means very little light can be received by the camera. With an increase in the NA, as can be seen in Fig. 4(b), some of the beam, the shadowed light bundle A, will travel through the air gap of the prism. However, other rays within a cone angle bigger than 3.74° can be totally internally reflected. Thus, we can design an objective imaging optical system with nonsymmetrical apertures to increase the NA. Figure 4(b) also illustrates that the no-TIR rays that travel through the air gap are absorbed at the black deglossing painted prism surface. Further research into the working principle of the DMD shows that there is a limit to the NA for the ray bundle reflected by the micromirrors. It can be seen in Fig. 4(c) that by increasing the circular aperture, the incident angle of the ray reflected by the micromirrors will reach the critical angle again. In such a situation, the second incidence of TIR rays, the shadowed light bundle B, will be trapped and cannot enter the relay optical system. Considering that the material of the TIR prism is BK7, the corresponding working F/# for the second TIR condition of the reflected light bundle can be calculated as F∕2ndTIR 2.54:
APPLIED OPTICS / Vol. 54, No. 1 / 1 January 2015
application, the objective imaging optical system is designed with a working F/# of 1.97, with the cone angle being 14.74°. Note that Figs. 4(a)–4(c) are just theoretical illustrations of the light path in the X–Y plane. Considering the working principle of the DMD, the tilt direction of the micromirror is perpendicular to the hinge axis, which is positioned diagonally relative to the overall array. The directions of the incident illumination and reflected light bundles are illustrated in Fig. 5. Thus, the reflected rays at the plane that is rotated by 45° with respect to the X–Y plane will have TIR first. The footprint at the stop surface for the center field is shown in Fig. 6. It can be seen that there are two blank areas, A and B, corresponding to the no-TIR and second TIR losses, respectively. As can be seen in Fig. 6, the ratio of the rays through is 73.35% for the F/1.97 objective imaging lens. The no-TIR and second TIR losses, described as vignetting losses here, are 25.70% and 0.95%, respectively. Since the illumination in an image is inversely proportional to the square of the F/#, the ratio of the optical collection ability of the objective imaging lens to that of the conventional DMD projection lens can be given by 2 1 0.7335 × 1.97 2 ε 1.1345: (9) 1 2.45
It can be seen in Eq. (9) that compared with a conventional DMD projection optical design, the
(8)
With a decrease in the working F/#, more rays will have TIR, but most of the incident light will travel through the prism to the relay optical system. Those analyses showed that the aperture of the objective imaging system can break the F/2.45 limitation and increase the optical collection ability of the system. However, an increase in the NA will also increase the design difficulty of the relay imaging system, as described in Section 3.B. In our practiced 64
Fig. 5. Direction of the incident illumination and reflected light bundles.
Fig. 6. Footprint diagram of the center field at the stop surface.
illumination of the image has been increased by 13.45%. This proves that the nonsymmetrical NA breaks the limit of a conventional DMD projection lens and increases the optical collection ability of the objective imaging optical system. The half field of view of the objective imaging system is chosen to be ω 27.4°. The focus could be calculated by f
hDMD ; 2 tan ω
(10)
where hDMD is the 17.51 mm diagonal length of the DMD active area and f stands for the focal length of the objective imaging system. Then it can be calculated that f 16.92 mm:
(11)
Based on the discussion above, the objective imaging optical system, considering the vignetting losses, is illustrated in Fig. 7. The optical specification of this objective imaging system is shown in Table 1. The modulation transfer function (MTF) curve of the objective imaging lens is illustrated in Fig. 8. It is worth noting that the MTF in Fig. 8 is calculated without considering vignetting. The influence of the vignetting factors will be considered with the combination of the whole system, which is discussed in Section 4. B.
Optical Design of the Relay Imaging System
The function of the relay imaging system is to make sure all the light reflected by the DMD could be reimaged on the CCD. As illustrated in Fig. 9, the system consists of the DMD chip, the TIR prism, the relay imaging lens, and the CCD chip. It is worth mentioning that the vignetting factors discussed in Section 2 are not considered during the optimization process of the relay imaging lens. As we discussed above, the chief ray of the light bundle reflected from on-state mirrors on the DMD is not vertical to the DMD chip. That is because the relay light bundle is tilted 24° with the nominal of the DMD plane. One solution for the relay imaging lens is to use one axisymmetric optical system and place the optical axis perpendicular to the DMD plane. This means the center of the DMD device is below the optical axis. There is one main disadvantage for the axisymmetric structure. Only the lower
Fig. 7. Layout of the objective imaging system, considering the vignetting factors.
Table 1.
Optical Specifications of the Objective Imaging System
Parameter Number of lenses used Optical structure Working F/# Entrance pupil Focal length Field of view DMD active area (diagonal) DMD resolution Working wavelength The fraction of the rays through (vignetting factor) MTF (without vignetting)
Value 10 Telecentric in image space 1.97 8.9 mm 16.92 mm 2ω 54.8° 17.51 mm [0.7 in.] (17.78 mm) 1024 × 768 420–650 nm 73.35% >0.5 at 36.5 lp∕mm (line pairs per millimeter)
half of the lens can be used due to the 24° tilt of the relay light bundle. The optical elements at the front and rear parts of the optical system can grow quite large with an increase in the field of view. This is a great waste for the optical system. Also, the large NA of the objective imaging optical system and relay optical system should be matched. The optical system design of a large field of view and large NA is a great challenge. In this work, a relatively simplified off-axis relay optical system is presented. This off-axis relay system features a compact size, simplified design, and high-light-efficiency relay light bundle. Figure 9 shows the layout of the whole off-axis relay optical system. It can be seen in Fig. 9 that the rays reflected by the DMD would enter the TIR system first. Note that the TIR prism system used in this DMD camera consists of one rectangular prism and one right-angled trapezoid prism. In our design, the TIR prism system can be treated as one plane glass in the off-axis relay imaging path. And the front and back surfaces of the cuboid prism system are parallel to the DMD surface. Although this plane glass would introduce some aberrations such as coma and astigmatism, these aberrations are corrected well by the following decenter-and-tilt optical elements.
Fig. 8. MTF of the objective imaging lens. T and S, tangential and sagittal MTF respectively; DIFF LIMIT, diffraction limited; OTF, optical transfer function. 1 January 2015 / Vol. 54, No. 1 / APPLIED OPTICS
65
Table 2.
Optical Specifications of the Relay Imaging System
Parameter
Fig. 9. Layout of the relay imaging system without vignetting.
The diagonal size of the CCD we used in this system is 10.98 mm. Considering that the diagonal length of the DMD active area is 17.51 mm, the lateral magnification of the whole relay system is β
hCCD 10.98 0.627: hDMD 17.51
(12)
When one lens is used in the case of lateral magnification of nearly −1, the symmetrical lens construction has the great virtue of almost automatic correction for coma, distortion, and lateral chromatic aberration [15]. The double-Gauss lens is one commonly used symmetrical structure. Our relay lens group is derived from a modified double-Gauss structure. The relay imaging lens combines one decentered big field lens and one tilt double-Gauss lens group. The decentered field lens has two main functions. First, it is used to converge light reflected from the DMD and reduce the aperture of the optical system, so that the ray bundles would pass through the following lens groups without vignetting. Second, the decentration of the field lens could compensate some of the off-axis aberrations such as primary astigmatism and primary coma. After the field lens, 10 lenses have a group tilt and the axis of the lens group is nearly parallel to the chief ray of the center field. The tilted lens group is used for further balancing of the off-axis aberrations. It is worth noting that in the relay system, we regard the DMD active area as the object plane and the CCD sensor as the image plane. For simplicity, we take the tilted axis of the lens group as the relay optical axis. Then the slope object will form a slope image. Now let’s go back to the influence of the image-side NA with the lateral magnification of the relay optical system. It can be derived from Eq. (1) that sin u0
sin u ; β
APPLIED OPTICS / Vol. 54, No. 1 / 1 January 2015
u0 23.94°:
(14)
The corresponding working F/# for the relay system, which is also the working F/# for the DMD camera, can be deduced: F∕DC
1 1.23: 2n sin u0 0
(15)
To our knowledge, it is the lowest working F/#, i.e., the biggest image space NA, of the DMD cameras that has been presented. The optical specifications of this relay imaging system are shown in Table 2. The MTF curve of the relay imaging system for the whole DMD camera is illustrated in Fig. 10. 4. Optical System Evaluation
The combination of the imaging system and relay system is illustrated in Fig. 11. It can be seen that even though parts of the rays would be trapped in the TIR prism system, most of the light would transmit through the whole system and be well imaged on CCD. Meanwhile, the aperture of the objective imaging system and relay imaging system are almost all filled up. This demonstrates the advantages of compact size and high light efficiency for this off-axis relay optical system design. In order to adjust the exposure time of CCD pixels by changing the binary on–off pattern of corresponding DMD micromirrors, a correspondence adjustment between DMD and CCD pixels should be
(13)
where u is the angle of the marginal ray for the light bundle from the micromirror, which is the 14.74° cone angle of the incident light bundle on the micromirror, and u0 is the marginal ray angle on the CCD sensor. Substituting our values into Eq. (13), it could be calculated that 66
Value
Number of lenses used 11 Object space numerical aperture (NA) 0.25 CCD detector working F/# 1.23 CCD detector area 8.77 mm × 6.60 mm (2/3 in.) CCD resolution 1360 × 1024 Lateral magnification 0.627 Working wavelength 420–650 nm MTF (without vignetting) >0.48 at 77 lp∕mm
Fig. 10. MTF of the relay imaging system.
Fig. 11. Layout of the whole imaging optical system. (a) 3D layout and (b) shaded model.
Concerning our DMD camera, the value of A is
done. In this work, the adjustment is finished based on the method provided by Nayar et al. [8]. Because this system is an imaging system, we first evaluate it by analyzing the imaging quality using the MTF. Second, the dynamic range of the whole system is analyzed. We evaluate the optical efficiency of the entire system in the end.
and we choose the size of one pixel for an acceptable blur diameter. Then the hyperfocal distance can be calculated as
A. Optical Specifications and Imaging Quality of the DMD Camera
Dhyp 14.61m:
The optical specifications of this DMD camera are given in Table 3. The CCD chip used in this DMD camera is a 2/3 in. (16.9 mm) sensor with a resolution of 1360 × 1024, and the unite cell size is 6.45 μm× 6.45 μm. So the Niquest spatial frequency of the lens is 77 lines/mm. As shown in Fig. 12, the MTF of the designed optical system at the focal plane is higher than 0.35 within its entire field of view. Another important factor concerning the imaging quality is the depth of field. In order to evaluate depth of field, the hyperfocal distance should be calculated, which is simply [16]
Dhyp −
fA ; B
(16)
A 8.9 mm;
(17)
(18)
This is a rough estimation of the distance beyond which all objects are acceptably sharp for the DMD camera focused at infinity. This is a relatively large distance, which means at close distances the image quality tends to deteriorate. However, in the above calculation the acceptable blur diameter B is just one pixel, which is quite a strict requirement. In practice, the acceptable blur diameter B can be empirically 2 or 3 pixels based on the specific usage. Also, when used in systems like LDW and FCW, objects at a close distance turned to be big enough for detection, even though the image quality had deteriorated to some extent. Meanwhile, this depth of field has the advantage of reducing the influence of speckles on the windglass in front of the camera.
where A is the diameter of the entrance pupil of the lens, and B is the acceptable blur diameter. Table 3
Optical Specifications of the DMD Camera
Parameter Working F/# Entrance pupil Field of view Focal length Working wavelength The fraction of the rays through (vignetting factor) MTF (vignetting considered)
Value 1.23 8.9 mm 2ω 54.8° 10.59 mm 420–650 nm 73.35% >0.35 at 77 lp∕mm
Fig. 12. MTF of the whole imaging optical system 1 January 2015 / Vol. 54, No. 1 / APPLIED OPTICS
67
B.
Table 4.
Dynamic Range
Dynamic range quantifies the ability of a detector to adequately image both bright light and dark shadows in a scene. The dynamic range of a CCD or CMOS sensor is defined as the ratio of the largest nonsaturating input signal to the smallest detectable input signal [17,18]. The expression of dynamic range for CCD is DR 20 log
Cfull ; Nr
DRDMD− 20 log
Cfull I max ; × N r I min
(20)
where I max and I min are the maximum and minimum gray levels for the DMD, respectively. In our system, the modulation DMD can be controlled with 8 bits of precision. Thus, it can be seen that I max 28 − 1 255: I min
(21)
Cfull 20 log × 255 ≈ 2.41 × DR: Nr (22)
From Eq. (22), it can be seen that the former dynamic range has been greatly increased, by 2.41 times. It is a dramatic increase compared with a conventional digital camera. C.
Optical Efficiency
Our goal is to design a high-NA camera with good imaging quality even under low illumination circumstances. Thus, the total luminous flux delivered to the CCD is a primary concern for this DMD camera. Numerous key parameters of the optical components must be carefully determined to qualify the design goal. For this DMD camera, the light efficiency is determined by the optical efficiency of the DMD, the spectral efficiency of the optical elements, the optical efficiency of the TIR prism system, and the vignetting factors, which are ηDMD , ηSPEC , ηTIR , and ζ VIG , respectively. The total light transmission efficiency of the system K SYS is the product of these efficiencies: K SYS ηDMD × ηSPEC × ηTIR × ζVIG :
(23)
The optical efficiency of the DMD is given in Table 4 [14]. 68
APPLIED OPTICS / Vol. 54, No. 1 / 1 January 2015
Value (%)
Array fill factor η1 Window transmission η2 Micromirror reflectivity η3 Array diffraction efficiency η4
92.0 97.0 88.0 86.0
Then ηDMD could be calculated as ηDMD η1 × η2 × η3 × η4 67.54%:
(24)
All optical surfaces should be antireflection (AR) coated to maximize light efficiency. The objective imaging system consists of 16 lens surfaces and 1 cement surface. The relay imaging system consists of 16 lens surfaces, 3 cement surfaces, and 1 IR-cut filter. For an approximate calculation, the AR coating for the lens surface has 1% loss per surface and the cement surface has 1% loss per surface. The IR-cut filter has 98% transmission. The overall spectral efficiency is about ηSPEC 0.9936 × 0.98 68.25%:
(25)
As for TIR prism systems, due to the difficulty of having AR coatings in the air gap, the typical overall transmission is approximately ηTIR 92%:
Hence, the dynamic range of the modulated image is DRDMD−camera
Parameter
(19)
where Cfull represents the full-well capacity of the detector and N r is the RMS of the read noise of the CCD. Using DMD micromirrors as a grayscale modulator, the dynamic range of our DMD camera should be changed to
Optical Efficiency of the DMD
(26)
Substituting Eqs. (24)–(26) into Eq. (23), the total light transmission efficiency of the whole system is K SYS 31.11%:
(27)
Considering the DMD camera as an imaging system, the illumination of the light delivered to the CCD through the optical system is given by E0 K SYS πL sin2 U 0 n02 ∕n2 ;
(28)
where E0 is the illumination on CCD, K SYS 31.11%, L is the luminance of the object scene field, n0 and n are the refractive indices of the image and object, respectively, and n0 n 1, u0 is the solid angle subtended from the CCD sensor, and u0 23.94°. Substituting these parameters into Eq. (28), it can be calculated that the illumination on the CCD sensor is E0 0.161L: D.
(29)
Measured Performance
We have designed and fabricated a DMD camera prototype. Figure 13 shows a photograph of the DMD camera. With the folding of the light path, it
5. Conclusions
Fig. 13. Photograph of the DMD camera.
This paper proposes a design of a very high-NA DMD camera that also significantly enhances the dynamic range. With the DMD as a grayscale modulator, the exposure time of the CCD pixels can be adjusted and a high dynamic range can be achieved. Further discussion showed that with the addition of an 8 bit gray level of the DMD, the dynamic range was dramatically increase by 2.41 times. The greatly enhanced wide-dynamic-range DMD camera can record more scene details, from shadows to highlights, than normal cameras. Meanwhile, a working F/# of 1.23 has been realized, which will significantly increase the SNR performance of cameras in dark scenes. With the innovative off-axis relay optical system, the DMD camera has a compact size and good imaging quality. The design principles of the whole system are discussed in detail. A prototype DMD camera has been designed and fabricated. The imaging experiments have validated our work, as described in the paper. This DMD camera can also be used for other image processing functions, such as feature detection and shape measurement. References
Fig. 14. Images captured by a conventional digital camera and the DMD camera.
can be seen in Fig. 13 that the overall size of the DMD camera could be decreased. Figure 14 provides a comparison of images captured by a conventional digital camera and the DMD camera. In the top image, the scene dynamic range obviously exceeds the dynamic range of the camera, resulting in lost details in both the light and dark regions. In the bottom image, the grayscale modulation of the DMD enables the same scene to be captured with complete visual details even in the extremes of brightness and darkness.
1. O. Mano, G. Stein, E. Dagan, and A. Shashua, “Forward collision warning with a single camera,” in IEEE Intelligent Vehicles Symposium (IEEE, 2004). 2. C.-M. Chang and H.-P. D. Shieh, “Design of illumination and projection optics for projectors with single digital micromirror devices,” Appl. Opt. 39, 3202–3208 (2000). 3. Y. Lu and S. Chen, “Direct write of microlens array using digital projection photopolymerization,” Appl. Phys. Lett. 92, 041109 (2008). 4. M. D. Grossberg and S. K. Nayar, “High dynamic range from multiple images: which exposure to combine?” in Proceedings of ICCV Vorkshop on Color and Photometric Methods in Computer Vision (2003). 5. E. Reinhard, G. Ward, S. Pattanaik, and P. Debevec, High Dynamic Range Imaging: Acquisition, Display, and ImageBased Lighting (Morgan Kaufmann, 2005), Chap. 4. 6. A. A. Adeyemi, N. Barakat, and T. E. Darcie, “Applications of digital micro-mirror devices to digital optical microscope dynamic range enhancement,” Opt. Express 17, 1831–1843 (2009). 7. S. K. Nayar and T. Mitsunaga, “High dynamic imaging: spatially varying pixel exposures,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2000), pp. 472–479. 8. S. K. Nayar, V. Branzoi, and T. E. Boult, “Programmable imaging using a digital micromirror array,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2004), pp. 436–443. 9. M. R. Douglass and D. M. Kozuch, “DMD reliability assessment for large-area displays,” in Society for Information Display International Symposium Digest of Technical Papers (1995), Vol. 26, pp. 49–52. 10. M. Liang, R. L. Stehr, and A. W. Krause, “Confocal pattern period in multiple-aperture confocal imaging systems with coherent illumination,” Opt. Lett. 22, 751–753 (1997). 11. E. P. Wagner, B. W. Smith, S. Madden, J. D. Winefordner, and M. Mignardi, “Construction and evaluation of a visible spectrometer using digital micromirror spatial light modulator,” Appl. Spectrosc. 49, 1715–1719 (1995). 12. T. Ota, S. Kawata, T. Sugiura, M. J. Booth, M. A. A. Neil, R. Juskaitis, and T. Wilson, “Dynamic axial-position control of a laser-trapped particle by wavefront modification,” Opt. Lett. 28, 465–467 (2003). 1 January 2015 / Vol. 54, No. 1 / APPLIED OPTICS
69
13. S. Ri, M. Fujigaki, T. Matui, and Y. Morimoto, “Accurate pixelto-pixel correspondence adjustment in a digital micromirror device camera by using the phase-shifting moiré method,” Appl. Opt. 45, 6940–6946 (2006). 14. Texas Instruments Incorporated, DLP 0.7 XGA 2xLVDS Type A DMD, August 2012. 15. A. Cox, Photographic Optics (Focal, 1974), p. 241.
70
APPLIED OPTICS / Vol. 54, No. 1 / 1 January 2015
16. W. J. Smith, “Stops, apertures, pupils and diffraction,” in Modern Optical Engineering, 4th ed. (McGraw-Hill, 2007), Chap. 9. 17. A. J. P. Theuwissen, Solid State Imaging with Charge-Coupled Devices (Kluwer Academic, 1995). 18. G. Healey and R. Kondepudy, “Radiometric CCD camera calibration and noise estimation,” IEEE Trans. Pattern Anal. Mach. Intell. 16, 267–276 (1994).
A high-NA imaging system with high dynamic range is presented based on a digital micromirror device (DMD). The DMD camera consists of an objective imaging system and a relay imaging system, connected by a DMD chip. With the introduction of a total internal reflection prism system, the objective imaging system is designed with a working F/# of 1.97, breaking through the F/2.45 limitation of conventional DMD projection lenses. As for the relay imaging system, an off-axis design that could correct off-axis aberrations of the tilt relay imaging system is developed. This structure has the advantage of increasing the NA of the imaging system while maintaining a compact size. Investigation revealed that the dynamic range of a DMD camera could be greatly increased, by 2.41 times. We built one prototype DMD camera with a working F/# of 1.23, and the field experiments proved the validity and reliability our work. © 2014 Optical Society of America OCIS codes: (080.0080) Geometric optics; (080.3620) Lens system design; (100.0100) Image processing; (110.0110) Imaging systems. http://dx.doi.org/10.1364/AO.54.000060
1. Introduction
Recent years have witnessed the emergence and development of lane departure warning (LDW), forward collision warning (FCW), and traffic sign recognition (TSR) using monocular cameras [1–3]. Although these driving assistance systems could work well in the daytime, nearly none of them could work effectively at night. In addition to the signal-tonoise ratio (SNR) problem under low illumination, the dynamic range is a big problem. When there are vehicles’ headlamps and other high-brightness light sources against the dark background, the picture of the imaging system will be incomplete because of the saturation area in CCD/CMOS chip. Since the dynamic range of an image sensor is not high enough to capture the scenes, this will lead to some serious consequences. 1559-128X/15/010060-11$15.00/0 © 2015 Optical Society of America 60
APPLIED OPTICS / Vol. 54, No. 1 / 1 January 2015
There are two challenges presenting problems in the development of LDW, FCW and TSR cameras. One involves improving the imaging quality in extremely low-illumination scenes. The other is increasing the dynamic range of digital cameras to record more scene details from shadows to highlights. As for the first method, it can be achieved by increasing both the size of the pixel and the aperture value of the imaging optical system. Together, these effects could generally benefit the SNR performance of the camera, and consequently enable the camera to capture more visual details. As for the high dynamic range, a lot of work has been done [4–6]. One approach is to use a beam splitter and two detectors to generate two copies of the optical image of the scene. Another approach is the spatially varying pixel exposure technique [7], which implements multiple capture by sacrificing spatial resolution. The recent technique of the digital pixel system (DPS) could also help to improve high dynamic range imaging. The DPS image sensor employs a self-reset
DPS architecture where each pixel is capable of resetting the exposure itself. In addition to the ideas proposed above, Nayar et al. [8] introduced a method of using digital micromirror devices (DMDs) for dynamic range enhancement. In Nayar et al.’s work, a DMD-based spatial light modulator has been implemented to vary the scene radiance received on each camera pixel. This has enabled highdynamic-range imaging at high speed. They have built a system with an off-the-shelf lens and modified one projector. However, in the system they propose, the separation of the objective and relay path, as far as we can tell, is just along the diagonal of the micromirrors, which is 45° to the array orthogonal. This will make the entire system bulky. Meanwhile, the tilt angle of the DMD mirror is 12°, which means that the working F/# will be no smaller than F/ 2.45 [2]. Also, the resolutions of both the DMD and CCD chips are relatively small. All those factors would limit the application of the DMD camera. In this paper, we design and build one high-NA and high-dynamic-range imaging DMD camera based on the work of Nayar et al. With the introduction of the DMD as a grayscale modulator in an optical imaging system, the exposure time of CCD pixels can be adjusted by changing the binary on–off pattern of corresponding DMD micromirrors. As a result, the images captured by a DMD camera can acquire the best exposure time compared with a conventional digital camera. Consequently, high-dynamic-range imaging can be achieved. The DMD camera proposed in this paper features strong light-collecting power with a working F/# of 1.23, and the dynamic range is 2.41 times higher than for a conventional camera. It could be an effective solution for camera driving assistance systems, where low-illumination imaging and a high dynamic imaging range are of great importance for detection. In our work, an innovative off-axis relay optical system is presented to correct the off-axis aberration of a relay imaging system. We decrease the working F/# to 1.23, which, to our knowledge, is the lowest working F/# presented for DMD cameras. With the introduction of the total internal reflection (TIR) prism system as the separator of the relay path and objective path, a very compact system can be designed. High resolution is also a merit of our system. The DMD used in this system model has 1024 × 768 pixels, and the CCD is 1360 × 1024. The working principle and design challenges of DMD cameras are described in Section 2. The optical system design and overall system specifications are presented in Section 3. The system evaluation and prototype testing results are given Section 4. Section 5 summarizes our work and research findings.
Fig. 1. Working principle of the DMD camera.
off-axis relay imaging lens, a TIR prism system, a DMD, a CCD detector, and a programmable controller. In general, it can be separated into three main parts: the objective imaging system, the relay imaging system, and the programmable controller. The main work of the objective imaging system is to capture the image of the scene. The relay imaging system reimages the image on the DMD to the CCD detector, and the image on the DMD is modified by the programmable controller. All of the three systems are synchronized to have real-time processing of the image. Figure 2 shows the optical function of the DMD. The DMD is a spatial light modulator consisting of an array of micromirrors that can be individually switched 12° or −12° to either direct or divert incident light corresponding to an “on” or “off” state.
2. Working Principle and Challenges of the DMD Camera
The working principle of the DMD camera is shown in Fig. 1. It can be seen in Fig. 1 that the whole system contains six parts: an objective imaging lens, an
Fig. 2. Optical function of the DMD. (a) Tilt incident light and (b) vertical incident light. 1 January 2015 / Vol. 54, No. 1 / APPLIED OPTICS
61
This can be used in projection systems, digital printers, optical spectrometers, and wavefront correction systems [9–13]. As illustrated in Fig. 2, the incident light could be modulated by a DMD due to its working principle. In the DMD camera, each individual DMD micromirror could operate as a controllable high-speed shutter. The relay imaging system of the DMD camera is well designed and calibrated. And the positions of the CCD sensor and the DMD micromirror array are in a conjugate relationship. The working process of the DMD camera can be expressed as follows. All of the micromirrors of the DMD are in the on state in the beginning, so the image will be reflected and reimaged on the CCD. The picture from the CCD detector will be delivered to a microprocessor. After image processing, the saturation area and the distribution of light illumination in the image can be calculated. Combining the correspondence of the DMD mirrors to the CCD pixels, the DMD camera program will turn corresponding individual DMD micromirrors to the off state, and the harmfully high light is modulated. In real practical work, the DMD works fast enough between the on and off states to be integrated by the CCD. Therefore, the saturation image area is depressed by the grayscale modulation. In such an application, the optimized exposure of the dark scene can be achieved and the high-dynamic-range imaging can be greatly improved. In order to design a high-speed and high-dynamicrange imaging DMD camera with good imaging quality, there are a lot of challenges. One important challenge is to increase the light collection power, or “speed,” of this camera, so that the imaging system can make details visible under low illumination. This can be done in one of two ways. The first is to increase the solid angle subtended from the image by increasing the aperture of the objective imaging system. The second way, which is deduced from the Abbe sine condition, is based on the magnification of the relay imaging system: β
hCCD n sin u ; hDMD n0 sin u0
(1)
where n sin u and n0 sin u0 are the object-side NA and image-side NA for the relay optical system, respectively, the refractive index of air n n0 1.0, and hCCD and hDMD are the size of the CCD sensor and the DMD active area, respectively. Then the image-side NA of the DMD camera NADC n0 sin u0 can be deduced by NADC NA ×
hDMD ; hCCD
(2)
where NA n sin u. Equation (2) shows that NADC can be increased with a large ratio of hDMD to hCCD . In our work, both of the two ways are utilized to increase the NA of our DMD camera. 62
APPLIED OPTICS / Vol. 54, No. 1 / 1 January 2015
The DMD camera resembles a single-panel DMD projector but is more complicated because it combines two imaging systems. So the second challenge is the optomechanical system structure design. Considering that DMD is a reflective device, it is crucial to prevent overlap of the on-state and off-state light bundles of the two imaging paths. In our work, unlike the DMD cameras proposed by Nayar et al. [8] and Ri et al. [13], one cuboid TIR prism system is introduced to separate the incident light from the light reflected by the DMD. With the use of a TIR prism for the physical separation of the two paths, a very compact system can be achieved. In addition, the unique working principle of the DMD also leads to the off-axis aberration correction, which is the third challenge. The TIR prism system used in this DMD camera consists of one rectangular prism and one right-angled trapezoid prism. As for the objective imaging system, the optic axis achieves TIR by the rectangular prism and is perpendicular to the surface plane of the DMD. After being reflected by the on-state micromirrors, the optic axis of the relay imaging system is tilted 24° with the DMD plane. Therefore, the object surface and the image surface of the relay imaging system must be all tilt with the tilt optic axis. This will create a lot of issues concerning the off-axis aberrations and make it much more difficult for aberration correction in the relay optical system. As for our work, we introduced an innovative off-axis optical system with a tilt-and-decenter optical element to correct the off-axis aberrations. The optical system we propose was well designed, with high resolution and good image quality. This structure also has the advantage of increasing the speed or NA of the imaging system while maintaining a compact size. Thus, the whole optical system could collect more light from the scene, pass it to the CCD, and potentially improve the imaging contrast. In this paper we discuss the optical system design with a DMD camera that could be greatly used in DMD-based imaging systems. The fourth challenge is associated with the optical efficiency. The optical system consists of two imaging lenses, the TIR prism and the DMD chip. Each surface of the optical elements used in this system will cause surface reflections. These reflections will reduce the optical efficiency and diminish the contrast ratio. Therefore, the optical efficiency is an important parameter in evaluating this whole system, which will be discussed in Section 4. The last challenge is associated with the response time. As for the DMD camera, real-time imaging should be guaranteed in order to be used in driving assistant systems. Such a system has a set of devices connected to one programmable controller. Therefore, a short response time for each device, fast data processing, and efficient program control are major issues. In the prototype we propose that the frame frequency of the CCD camera is 100 Hz and the DMD is 100 Hz using the development kit proposed by Texas Instruments. The process of data
acquisition can be summarized as follows. The CCD acquires the first frame from the scene. Then the image will be sent to the on-board processor to generate a grayscale model image, and this model image will be sent to the DMD. These two steps will last about 20 ms. Then the DMD will modulate the image, and this process takes about 10 ms. Before the third step, the processor of the DMD will send a synchronization signal to the CCD, and the CCD will start to acquire this modulated image simultaneously. This image, with optimal exposure time for each pixel, is transmitted for the driving assistant system. The ultimate frame frequency of the whole system is 25 Hz for our DMD camera prototype. 3. Optical System Design A.
Optical Design of the Objective Imaging System
As discussed above, the main function of the objective imaging system is to collect as much light as possible from the scene and capture the image on the active area of the DMD. As illustrated in Fig. 3, the objective imaging system consists of an objective imaging lens, a rectangular TIR prism (unfolded), and a DMD chip. The light path of the objective imaging system is separated from the relay imaging path by the TIR surface of the rectangular prism. The unfolded rectangular prism is equivalent to plate glass, which means the optical axis of the objective imaging system is perpendicular to the surface plane of the DMD. The unique structure of the DMD camera means high resolution and good image quality can be required for the objective imaging system. Also, the structure is telecentric which is more like the projection optical system in single-DMD projectors. As for a conventional projection lens, the maximum usable NA of the optical is generally set by the device tilt angle of DMD micromirrors. Normally, the illumination angle is 2 times the 12° mirror tilt angle. The working F/# of the project lens could be given by F∕#Con
1 ; 2 NA
instead of the commonly used paraxial working F∕ 1∕2n tan u, we use the working F/# in this paper. That is because the working F/# directly relates the F# and NAs of the projector lens and relay optics. Also, the working F/# is based upon real ray data at the actual conjugates of the optical system. Thus it is generally much more convenient when evaluating the performance of our high-NA system. Compared with a conventional telecentric DMD projection lens, the limitation of twice the device tilt angle is broken in our DMD camera with the use of a nonsymmetrical NA. The analysis of the NA of our objective imaging optical systems involves two times the TIR at the TIR surface. Figure 4 illustrates the light path of the incident imaging light bundle with different F/# values. As can be seen in Fig. 4(a), when the imaging beam enters the prism with an incident angle larger than the critical angle, the whole light bundle will be totally internally reflected at the TIR surface and the
(3)
where NA n sin u is the image space NA of the project lens, which is also the object-side NA of the relay system. It is recommended by the DMD user manual [14] that a 12° device would utilize a cone angle of u 11.77° for illumination and projection. Therefore, the conventional telecentric project lens has a working F/# of F/2.45. One should note that
Fig. 3. Layout of the objective imaging system.
Fig. 4. Light path of the incident imaging light bundle with different F/# values of the objective imaging lens: (a) F/7.41, (b) F/7.41–F/2.49, and (c) F/1.9. 1 January 2015 / Vol. 54, No. 1 / APPLIED OPTICS
63
whole beam will reach the DMD active area. This is the first incidence of TIR. The material of the prism is BK7, whose refractive index is approximately nprism 1.5163. The refractive index of air is nair 1. Based on Snell’s law, the critical angle at surface 2 is n sin θC air sin 90° 0.66: (4) nprism It can be deduced that θC 41.26°:
(5)
Considering the rectangular prism has an oblique angle of 45°, the angle between the optic axis and the ray, which is incident on the TIR surface with the critical angle, is Δθ 45° − θC 3.74°:
(6)
As a result, only incident rays within a cone angle of 3.74° can be totally internally reflected and enter the system. The corresponding working F/# that meets this TIR requirement is F∕1stTIR 7.67:
(7)
This is quite a small aperture, which means very little light can be received by the camera. With an increase in the NA, as can be seen in Fig. 4(b), some of the beam, the shadowed light bundle A, will travel through the air gap of the prism. However, other rays within a cone angle bigger than 3.74° can be totally internally reflected. Thus, we can design an objective imaging optical system with nonsymmetrical apertures to increase the NA. Figure 4(b) also illustrates that the no-TIR rays that travel through the air gap are absorbed at the black deglossing painted prism surface. Further research into the working principle of the DMD shows that there is a limit to the NA for the ray bundle reflected by the micromirrors. It can be seen in Fig. 4(c) that by increasing the circular aperture, the incident angle of the ray reflected by the micromirrors will reach the critical angle again. In such a situation, the second incidence of TIR rays, the shadowed light bundle B, will be trapped and cannot enter the relay optical system. Considering that the material of the TIR prism is BK7, the corresponding working F/# for the second TIR condition of the reflected light bundle can be calculated as F∕2ndTIR 2.54:
APPLIED OPTICS / Vol. 54, No. 1 / 1 January 2015
application, the objective imaging optical system is designed with a working F/# of 1.97, with the cone angle being 14.74°. Note that Figs. 4(a)–4(c) are just theoretical illustrations of the light path in the X–Y plane. Considering the working principle of the DMD, the tilt direction of the micromirror is perpendicular to the hinge axis, which is positioned diagonally relative to the overall array. The directions of the incident illumination and reflected light bundles are illustrated in Fig. 5. Thus, the reflected rays at the plane that is rotated by 45° with respect to the X–Y plane will have TIR first. The footprint at the stop surface for the center field is shown in Fig. 6. It can be seen that there are two blank areas, A and B, corresponding to the no-TIR and second TIR losses, respectively. As can be seen in Fig. 6, the ratio of the rays through is 73.35% for the F/1.97 objective imaging lens. The no-TIR and second TIR losses, described as vignetting losses here, are 25.70% and 0.95%, respectively. Since the illumination in an image is inversely proportional to the square of the F/#, the ratio of the optical collection ability of the objective imaging lens to that of the conventional DMD projection lens can be given by 2 1 0.7335 × 1.97 2 ε 1.1345: (9) 1 2.45
It can be seen in Eq. (9) that compared with a conventional DMD projection optical design, the
(8)
With a decrease in the working F/#, more rays will have TIR, but most of the incident light will travel through the prism to the relay optical system. Those analyses showed that the aperture of the objective imaging system can break the F/2.45 limitation and increase the optical collection ability of the system. However, an increase in the NA will also increase the design difficulty of the relay imaging system, as described in Section 3.B. In our practiced 64
Fig. 5. Direction of the incident illumination and reflected light bundles.
Fig. 6. Footprint diagram of the center field at the stop surface.
illumination of the image has been increased by 13.45%. This proves that the nonsymmetrical NA breaks the limit of a conventional DMD projection lens and increases the optical collection ability of the objective imaging optical system. The half field of view of the objective imaging system is chosen to be ω 27.4°. The focus could be calculated by f
hDMD ; 2 tan ω
(10)
where hDMD is the 17.51 mm diagonal length of the DMD active area and f stands for the focal length of the objective imaging system. Then it can be calculated that f 16.92 mm:
(11)
Based on the discussion above, the objective imaging optical system, considering the vignetting losses, is illustrated in Fig. 7. The optical specification of this objective imaging system is shown in Table 1. The modulation transfer function (MTF) curve of the objective imaging lens is illustrated in Fig. 8. It is worth noting that the MTF in Fig. 8 is calculated without considering vignetting. The influence of the vignetting factors will be considered with the combination of the whole system, which is discussed in Section 4. B.
Optical Design of the Relay Imaging System
The function of the relay imaging system is to make sure all the light reflected by the DMD could be reimaged on the CCD. As illustrated in Fig. 9, the system consists of the DMD chip, the TIR prism, the relay imaging lens, and the CCD chip. It is worth mentioning that the vignetting factors discussed in Section 2 are not considered during the optimization process of the relay imaging lens. As we discussed above, the chief ray of the light bundle reflected from on-state mirrors on the DMD is not vertical to the DMD chip. That is because the relay light bundle is tilted 24° with the nominal of the DMD plane. One solution for the relay imaging lens is to use one axisymmetric optical system and place the optical axis perpendicular to the DMD plane. This means the center of the DMD device is below the optical axis. There is one main disadvantage for the axisymmetric structure. Only the lower
Fig. 7. Layout of the objective imaging system, considering the vignetting factors.
Table 1.
Optical Specifications of the Objective Imaging System
Parameter Number of lenses used Optical structure Working F/# Entrance pupil Focal length Field of view DMD active area (diagonal) DMD resolution Working wavelength The fraction of the rays through (vignetting factor) MTF (without vignetting)
Value 10 Telecentric in image space 1.97 8.9 mm 16.92 mm 2ω 54.8° 17.51 mm [0.7 in.] (17.78 mm) 1024 × 768 420–650 nm 73.35% >0.5 at 36.5 lp∕mm (line pairs per millimeter)
half of the lens can be used due to the 24° tilt of the relay light bundle. The optical elements at the front and rear parts of the optical system can grow quite large with an increase in the field of view. This is a great waste for the optical system. Also, the large NA of the objective imaging optical system and relay optical system should be matched. The optical system design of a large field of view and large NA is a great challenge. In this work, a relatively simplified off-axis relay optical system is presented. This off-axis relay system features a compact size, simplified design, and high-light-efficiency relay light bundle. Figure 9 shows the layout of the whole off-axis relay optical system. It can be seen in Fig. 9 that the rays reflected by the DMD would enter the TIR system first. Note that the TIR prism system used in this DMD camera consists of one rectangular prism and one right-angled trapezoid prism. In our design, the TIR prism system can be treated as one plane glass in the off-axis relay imaging path. And the front and back surfaces of the cuboid prism system are parallel to the DMD surface. Although this plane glass would introduce some aberrations such as coma and astigmatism, these aberrations are corrected well by the following decenter-and-tilt optical elements.
Fig. 8. MTF of the objective imaging lens. T and S, tangential and sagittal MTF respectively; DIFF LIMIT, diffraction limited; OTF, optical transfer function. 1 January 2015 / Vol. 54, No. 1 / APPLIED OPTICS
65
Table 2.
Optical Specifications of the Relay Imaging System
Parameter
Fig. 9. Layout of the relay imaging system without vignetting.
The diagonal size of the CCD we used in this system is 10.98 mm. Considering that the diagonal length of the DMD active area is 17.51 mm, the lateral magnification of the whole relay system is β
hCCD 10.98 0.627: hDMD 17.51
(12)
When one lens is used in the case of lateral magnification of nearly −1, the symmetrical lens construction has the great virtue of almost automatic correction for coma, distortion, and lateral chromatic aberration [15]. The double-Gauss lens is one commonly used symmetrical structure. Our relay lens group is derived from a modified double-Gauss structure. The relay imaging lens combines one decentered big field lens and one tilt double-Gauss lens group. The decentered field lens has two main functions. First, it is used to converge light reflected from the DMD and reduce the aperture of the optical system, so that the ray bundles would pass through the following lens groups without vignetting. Second, the decentration of the field lens could compensate some of the off-axis aberrations such as primary astigmatism and primary coma. After the field lens, 10 lenses have a group tilt and the axis of the lens group is nearly parallel to the chief ray of the center field. The tilted lens group is used for further balancing of the off-axis aberrations. It is worth noting that in the relay system, we regard the DMD active area as the object plane and the CCD sensor as the image plane. For simplicity, we take the tilted axis of the lens group as the relay optical axis. Then the slope object will form a slope image. Now let’s go back to the influence of the image-side NA with the lateral magnification of the relay optical system. It can be derived from Eq. (1) that sin u0
sin u ; β
APPLIED OPTICS / Vol. 54, No. 1 / 1 January 2015
u0 23.94°:
(14)
The corresponding working F/# for the relay system, which is also the working F/# for the DMD camera, can be deduced: F∕DC
1 1.23: 2n sin u0 0
(15)
To our knowledge, it is the lowest working F/#, i.e., the biggest image space NA, of the DMD cameras that has been presented. The optical specifications of this relay imaging system are shown in Table 2. The MTF curve of the relay imaging system for the whole DMD camera is illustrated in Fig. 10. 4. Optical System Evaluation
The combination of the imaging system and relay system is illustrated in Fig. 11. It can be seen that even though parts of the rays would be trapped in the TIR prism system, most of the light would transmit through the whole system and be well imaged on CCD. Meanwhile, the aperture of the objective imaging system and relay imaging system are almost all filled up. This demonstrates the advantages of compact size and high light efficiency for this off-axis relay optical system design. In order to adjust the exposure time of CCD pixels by changing the binary on–off pattern of corresponding DMD micromirrors, a correspondence adjustment between DMD and CCD pixels should be
(13)
where u is the angle of the marginal ray for the light bundle from the micromirror, which is the 14.74° cone angle of the incident light bundle on the micromirror, and u0 is the marginal ray angle on the CCD sensor. Substituting our values into Eq. (13), it could be calculated that 66
Value
Number of lenses used 11 Object space numerical aperture (NA) 0.25 CCD detector working F/# 1.23 CCD detector area 8.77 mm × 6.60 mm (2/3 in.) CCD resolution 1360 × 1024 Lateral magnification 0.627 Working wavelength 420–650 nm MTF (without vignetting) >0.48 at 77 lp∕mm
Fig. 10. MTF of the relay imaging system.
Fig. 11. Layout of the whole imaging optical system. (a) 3D layout and (b) shaded model.
Concerning our DMD camera, the value of A is
done. In this work, the adjustment is finished based on the method provided by Nayar et al. [8]. Because this system is an imaging system, we first evaluate it by analyzing the imaging quality using the MTF. Second, the dynamic range of the whole system is analyzed. We evaluate the optical efficiency of the entire system in the end.
and we choose the size of one pixel for an acceptable blur diameter. Then the hyperfocal distance can be calculated as
A. Optical Specifications and Imaging Quality of the DMD Camera
Dhyp 14.61m:
The optical specifications of this DMD camera are given in Table 3. The CCD chip used in this DMD camera is a 2/3 in. (16.9 mm) sensor with a resolution of 1360 × 1024, and the unite cell size is 6.45 μm× 6.45 μm. So the Niquest spatial frequency of the lens is 77 lines/mm. As shown in Fig. 12, the MTF of the designed optical system at the focal plane is higher than 0.35 within its entire field of view. Another important factor concerning the imaging quality is the depth of field. In order to evaluate depth of field, the hyperfocal distance should be calculated, which is simply [16]
Dhyp −
fA ; B
(16)
A 8.9 mm;
(17)
(18)
This is a rough estimation of the distance beyond which all objects are acceptably sharp for the DMD camera focused at infinity. This is a relatively large distance, which means at close distances the image quality tends to deteriorate. However, in the above calculation the acceptable blur diameter B is just one pixel, which is quite a strict requirement. In practice, the acceptable blur diameter B can be empirically 2 or 3 pixels based on the specific usage. Also, when used in systems like LDW and FCW, objects at a close distance turned to be big enough for detection, even though the image quality had deteriorated to some extent. Meanwhile, this depth of field has the advantage of reducing the influence of speckles on the windglass in front of the camera.
where A is the diameter of the entrance pupil of the lens, and B is the acceptable blur diameter. Table 3
Optical Specifications of the DMD Camera
Parameter Working F/# Entrance pupil Field of view Focal length Working wavelength The fraction of the rays through (vignetting factor) MTF (vignetting considered)
Value 1.23 8.9 mm 2ω 54.8° 10.59 mm 420–650 nm 73.35% >0.35 at 77 lp∕mm
Fig. 12. MTF of the whole imaging optical system 1 January 2015 / Vol. 54, No. 1 / APPLIED OPTICS
67
B.
Table 4.
Dynamic Range
Dynamic range quantifies the ability of a detector to adequately image both bright light and dark shadows in a scene. The dynamic range of a CCD or CMOS sensor is defined as the ratio of the largest nonsaturating input signal to the smallest detectable input signal [17,18]. The expression of dynamic range for CCD is DR 20 log
Cfull ; Nr
DRDMD− 20 log
Cfull I max ; × N r I min
(20)
where I max and I min are the maximum and minimum gray levels for the DMD, respectively. In our system, the modulation DMD can be controlled with 8 bits of precision. Thus, it can be seen that I max 28 − 1 255: I min
(21)
Cfull 20 log × 255 ≈ 2.41 × DR: Nr (22)
From Eq. (22), it can be seen that the former dynamic range has been greatly increased, by 2.41 times. It is a dramatic increase compared with a conventional digital camera. C.
Optical Efficiency
Our goal is to design a high-NA camera with good imaging quality even under low illumination circumstances. Thus, the total luminous flux delivered to the CCD is a primary concern for this DMD camera. Numerous key parameters of the optical components must be carefully determined to qualify the design goal. For this DMD camera, the light efficiency is determined by the optical efficiency of the DMD, the spectral efficiency of the optical elements, the optical efficiency of the TIR prism system, and the vignetting factors, which are ηDMD , ηSPEC , ηTIR , and ζ VIG , respectively. The total light transmission efficiency of the system K SYS is the product of these efficiencies: K SYS ηDMD × ηSPEC × ηTIR × ζVIG :
(23)
The optical efficiency of the DMD is given in Table 4 [14]. 68
APPLIED OPTICS / Vol. 54, No. 1 / 1 January 2015
Value (%)
Array fill factor η1 Window transmission η2 Micromirror reflectivity η3 Array diffraction efficiency η4
92.0 97.0 88.0 86.0
Then ηDMD could be calculated as ηDMD η1 × η2 × η3 × η4 67.54%:
(24)
All optical surfaces should be antireflection (AR) coated to maximize light efficiency. The objective imaging system consists of 16 lens surfaces and 1 cement surface. The relay imaging system consists of 16 lens surfaces, 3 cement surfaces, and 1 IR-cut filter. For an approximate calculation, the AR coating for the lens surface has 1% loss per surface and the cement surface has 1% loss per surface. The IR-cut filter has 98% transmission. The overall spectral efficiency is about ηSPEC 0.9936 × 0.98 68.25%:
(25)
As for TIR prism systems, due to the difficulty of having AR coatings in the air gap, the typical overall transmission is approximately ηTIR 92%:
Hence, the dynamic range of the modulated image is DRDMD−camera
Parameter
(19)
where Cfull represents the full-well capacity of the detector and N r is the RMS of the read noise of the CCD. Using DMD micromirrors as a grayscale modulator, the dynamic range of our DMD camera should be changed to
Optical Efficiency of the DMD
(26)
Substituting Eqs. (24)–(26) into Eq. (23), the total light transmission efficiency of the whole system is K SYS 31.11%:
(27)
Considering the DMD camera as an imaging system, the illumination of the light delivered to the CCD through the optical system is given by E0 K SYS πL sin2 U 0 n02 ∕n2 ;
(28)
where E0 is the illumination on CCD, K SYS 31.11%, L is the luminance of the object scene field, n0 and n are the refractive indices of the image and object, respectively, and n0 n 1, u0 is the solid angle subtended from the CCD sensor, and u0 23.94°. Substituting these parameters into Eq. (28), it can be calculated that the illumination on the CCD sensor is E0 0.161L: D.
(29)
Measured Performance
We have designed and fabricated a DMD camera prototype. Figure 13 shows a photograph of the DMD camera. With the folding of the light path, it
5. Conclusions
Fig. 13. Photograph of the DMD camera.
This paper proposes a design of a very high-NA DMD camera that also significantly enhances the dynamic range. With the DMD as a grayscale modulator, the exposure time of the CCD pixels can be adjusted and a high dynamic range can be achieved. Further discussion showed that with the addition of an 8 bit gray level of the DMD, the dynamic range was dramatically increase by 2.41 times. The greatly enhanced wide-dynamic-range DMD camera can record more scene details, from shadows to highlights, than normal cameras. Meanwhile, a working F/# of 1.23 has been realized, which will significantly increase the SNR performance of cameras in dark scenes. With the innovative off-axis relay optical system, the DMD camera has a compact size and good imaging quality. The design principles of the whole system are discussed in detail. A prototype DMD camera has been designed and fabricated. The imaging experiments have validated our work, as described in the paper. This DMD camera can also be used for other image processing functions, such as feature detection and shape measurement. References
Fig. 14. Images captured by a conventional digital camera and the DMD camera.
can be seen in Fig. 13 that the overall size of the DMD camera could be decreased. Figure 14 provides a comparison of images captured by a conventional digital camera and the DMD camera. In the top image, the scene dynamic range obviously exceeds the dynamic range of the camera, resulting in lost details in both the light and dark regions. In the bottom image, the grayscale modulation of the DMD enables the same scene to be captured with complete visual details even in the extremes of brightness and darkness.
1. O. Mano, G. Stein, E. Dagan, and A. Shashua, “Forward collision warning with a single camera,” in IEEE Intelligent Vehicles Symposium (IEEE, 2004). 2. C.-M. Chang and H.-P. D. Shieh, “Design of illumination and projection optics for projectors with single digital micromirror devices,” Appl. Opt. 39, 3202–3208 (2000). 3. Y. Lu and S. Chen, “Direct write of microlens array using digital projection photopolymerization,” Appl. Phys. Lett. 92, 041109 (2008). 4. M. D. Grossberg and S. K. Nayar, “High dynamic range from multiple images: which exposure to combine?” in Proceedings of ICCV Vorkshop on Color and Photometric Methods in Computer Vision (2003). 5. E. Reinhard, G. Ward, S. Pattanaik, and P. Debevec, High Dynamic Range Imaging: Acquisition, Display, and ImageBased Lighting (Morgan Kaufmann, 2005), Chap. 4. 6. A. A. Adeyemi, N. Barakat, and T. E. Darcie, “Applications of digital micro-mirror devices to digital optical microscope dynamic range enhancement,” Opt. Express 17, 1831–1843 (2009). 7. S. K. Nayar and T. Mitsunaga, “High dynamic imaging: spatially varying pixel exposures,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2000), pp. 472–479. 8. S. K. Nayar, V. Branzoi, and T. E. Boult, “Programmable imaging using a digital micromirror array,” in Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (IEEE, 2004), pp. 436–443. 9. M. R. Douglass and D. M. Kozuch, “DMD reliability assessment for large-area displays,” in Society for Information Display International Symposium Digest of Technical Papers (1995), Vol. 26, pp. 49–52. 10. M. Liang, R. L. Stehr, and A. W. Krause, “Confocal pattern period in multiple-aperture confocal imaging systems with coherent illumination,” Opt. Lett. 22, 751–753 (1997). 11. E. P. Wagner, B. W. Smith, S. Madden, J. D. Winefordner, and M. Mignardi, “Construction and evaluation of a visible spectrometer using digital micromirror spatial light modulator,” Appl. Spectrosc. 49, 1715–1719 (1995). 12. T. Ota, S. Kawata, T. Sugiura, M. J. Booth, M. A. A. Neil, R. Juskaitis, and T. Wilson, “Dynamic axial-position control of a laser-trapped particle by wavefront modification,” Opt. Lett. 28, 465–467 (2003). 1 January 2015 / Vol. 54, No. 1 / APPLIED OPTICS
69
13. S. Ri, M. Fujigaki, T. Matui, and Y. Morimoto, “Accurate pixelto-pixel correspondence adjustment in a digital micromirror device camera by using the phase-shifting moiré method,” Appl. Opt. 45, 6940–6946 (2006). 14. Texas Instruments Incorporated, DLP 0.7 XGA 2xLVDS Type A DMD, August 2012. 15. A. Cox, Photographic Optics (Focal, 1974), p. 241.
70
APPLIED OPTICS / Vol. 54, No. 1 / 1 January 2015
16. W. J. Smith, “Stops, apertures, pupils and diffraction,” in Modern Optical Engineering, 4th ed. (McGraw-Hill, 2007), Chap. 9. 17. A. J. P. Theuwissen, Solid State Imaging with Charge-Coupled Devices (Kluwer Academic, 1995). 18. G. Healey and R. Kondepudy, “Radiometric CCD camera calibration and noise estimation,” IEEE Trans. Pattern Anal. Mach. Intell. 16, 267–276 (1994).
