Computational phase modulation in light field imaging.

Computational phase modulation in light field imaging Tomoya Nakamura,1,2 Ryoichi Horisaki,1,∗ and Jun Tanida1 1 Department

of Information and Physical Sciences, Graduate School of Information Science and Technology,Osaka University, 1-5 Yamadaoka, Suita, Osaka 565-0871, Japan 2 Research Fellow of the Japan Society for the Promotion of Science, 5-3-1 Kojimachi, Chiyoda, Tokyo 102-0083, Japan ∗ [email protected]

Abstract: We propose a scheme for modulating phase computationally in light field imaging systems. In a camera system based on the scheme, light field (LF) data is obtained by array-based optics, and the data is computationally projected into a single image with arbitrary phase modulation. In a projector system based on the scheme, LF data with arbitrary phase modulation is computationally generated before optical projection, and the phase-modulated image is projected by array-based optics. We describe the system design and required conditions based on the sampling theorem. We experimentally verified the proposed scheme based on camera and projector systems. In the experiment, we demonstrated a super-resolution camera and projector with extended depth-of-field without estimating the object’s shape. © 2013 Optical Society of America OCIS codes: (110.1758) Computational imaging; (110.4190) Multiple imaging.

References and links 1. E. H. Adelson and J. Y. A. Wang, “Single lens stereo with a plenoptic camera,” IEEE Trans. Pattern Anal. Mach. Intell. 14, 99–106 (1992). 2. J. Tanida, T. Kumagai, K. Yamada, S. Miyatake, K. Ishida, T. Morimoto, N. Kondou, D. Miyazaki, and Y. Ichioka, “Thin observation module by bound optics (TOMBO): concept and experimental verification,” Appl. Opt. 40, 1806–1813 (2001). 3. M. Levoy and P. Hanrahan, “Light field rendering,” in Proc. ACM SIGGRAPH (1996), pp. 31–42. 4. A. Isaksen, L. McMillan, and S. J. Gortler, “Dynamically reparameterized light fields,” in Proc. ACM SIGGRAPH (2000), pp. 297–306. 5. R. Ng, M. Levoy, M. Brédif, G. Duval, M. Horowitz, and P. Hanrahan, “Light field photography with a hand-held plenoptic camera,” Stanford Tech. Report CTSR 2005-02 (2005). 6. R. Horisaki, S. Irie, Y. Ogura, and J. Tanida, “Three-dimensional information acquisition using a compound imaging system,” Opt. Rev. 14, 347–350 (2007). 7. T. E. Bishop and P. Favaro, “The light field camera: extended depth of field, aliasing, and superresolution,” IEEE Trans. Pattern Anal. Mach. Intell. 34, 972–986 (2012). 8. Y. Kitamura, R. Shogenji, K. Yamada, S. Miyatake, M. Miyamoto, T. Morimoto, Y. Masaki, N. Kondou, D. Miyazaki, J. Tanida, and Y. Ichioka, “Reconstruction of a high-resolution image on a compound-eye imagecapturing system,” Appl. Opt. 43, 1719–1727 (2004). 9. T. E. Bishop, S. Zenetti, and P. Favaro, “Light field superresolution,” in Proc. IEEE International Conference on Computational Photography (ICCP) (2009), pp. 1–9. 10. S. C. Park, M. K. Park, and M. G. Kang, “Super-resolution image reconstruction: a technical overview,” IEEE Signal Process. Mag. 20, 21–36 (2003). 11. S. A. Shroff and K. Berkner, “Image formation analysis and high resolution image reconstruction for plenoptic imaging systems,” Appl. Opt. 52, D22–D31 (2013).

#198703 - $15.00 USD Received 2 Oct 2013; revised 15 Nov 2013; accepted 15 Nov 2013; published 21 Nov 2013 (C) 2013 OSA 2 December 2013 | Vol. 21, No. 24 | DOI:10.1364/OE.21.029523 | OPTICS EXPRESS 29523

12. R. Horisaki, K. Kagawa, Y. Nakao, T. Toyoda, Y. Masaki, and J. Tanida, “Irregular lens arrangement design to improve imaging performance of compound-eye imaging systems,” Appl. Phys. Express 3, 022501 (2010). 13. R. Horisaki and J. Tanida, “Full-resolution light-field single-shot acquisition with spatial encoding,” in Imaging and Applied Optics, OSA Technical Digest (CD) (Optical Society of America, 2011), paper CTuB5. 14. Z. Xu, J. Ke, and E. Y. Lam, “High-resolution lightfield photography using two masks,” Opt. Express 20, 10971– 10983 (2012). 15. K. Marwah, G. Wetzstein, Y. Bando, and R. Raskar, “Compressive light field photography using overcomplete dictionaries and optimized projections,” ACM Trans. Graph. 32, 1–11 (2013). 16. E. R. Dowski and W. T. Cathey, “Extended depth of field through wave-front coding,” Appl. Opt. 34, 1859–1866 (1995). 17. P. Mouroulis, “Depth of field extension with spherical optics,” Opt. Express 16, 12995–13004 (2008). 18. T. Nakamura, R. Horisaki, and J. Tanida, “Computational superposition compound eye imaging for extended depth-of-field and field-of-view,” Opt. Express 20, 27482–27495 (2012). 19. O. Cossairt, C. Zhou, and S. K. Nayar, “Diffusion coded photography for extended depth of field,” ACM Trans. Graph. 29, 1–10 (2010). 20. P. Pentland, “A new sense for depth of field,” IEEE Trans. Pattern Anal. Mach. Intell. 9, 523–531 (1987). 21. G. E. Johnson, E. R. Dowski, and W. T. Cathey, “Passive ranging through wave-front coding: information and application,” Appl. Opt. 39, 1700–1710 (2000). 22. A. Greengard, Y. Schechner, and R. Piestun, “Depth from diffracted rotation,” Opt. Lett. 31, 181–183 (2006). 23. C. Zhou, O. Cossairt, and S. K. Nayar, “Depth from diffusion,” in IEEE International Conference on Computer Vision and Pattern Recognition (CVPR) (2010), pp.1–8. 24. A. Ashok and M. A. Neifeld, “Pseudorandom phase masks for superresolution imaging from subpixel shifting,” Appl. Opt. 46, 2256–2268 (2007). 25. A. Ashok and M. A. Neifeld, “Information-based analysis of simple incoherent imaging systems,” Opt. Express 11, 2153–2162 (2003). 26. J. Chai, X. Tong, S. Chan, and H. Shum, “Plenoptic sampling,” in Proc. ACM SIGGRAPH (2000), pp. 307–318. 27. J. W. Goodman, Introduction to Fourier Optics (McGraw-Hill, 1996). 28. S. S. Sherif, W. T. Cathey, and E. R. Dowski, “Phase plate to extend the depth of field of incoherent hybrid imaging systems,” Appl. Opt. 43, 2709–2721 (2004). 29. W. Zhang, Z. Ye, T. Zhao, Y. Chen, and F. Yu, “Point spread function characteristics analysis of the wavefront coding system,” Opt. Express 15, 1543–1552 (2007). 30. Y. Takahashi and S. Komatsu, “Optimized free-form phase mask for extension of depth of field in wavefrontcoded imaging,” Opt. Lett. 33, 1515–1517 (2008). 31. T. Nakamura, R. Horisaki, and J. Tanida, “Computational superposition projector for extended depth of field and field of view,” Opt. Lett. 9, 1560–1562 (2013). 32. M. Sieler, P. Schreiber, P. Dannberg, A. Bräuer, and A. Tünnermann, “Ultraslim fixed pattern projectors with inherent homogenization of illumination,” Appl. Opt. 51, 64–74 (2012). 33. M. Grosse, G. Wetztein, A. Grundhöfer, and O. Bimber, “Coded aperture projection,” ACM Trans. Graph. 29, 1–12 (2010). 34. R. Horisaki and J. Tanida, “Compact compound-eye projector using superresolved projection,” Opt. Lett. 36, 121–123 (2011). 35. W. H. Richardson, “Bayesian-based iterative method of image restoration,” J. Opt. Soc. Am. 62, 55–59 (1972). 36. L. B. Lucy, “An iterative technique for the rectification of observed distributions,” Astron. J. 79, 745–754 (1974).

1.

Introduction

1.1. Light field imaging Light field imaging systems using array-based optics and postprocessing have been proposed for capturing spatio-angular information of light rays in object space; this light ray information is called the light field (LF) [1–3]. The LF is expressed by the four-dimensional function L(s, t, u, v) determined by two parallel planes indicating the spatial and angular coordinates of a ray, as shown in Fig. 1 [3]. Figure 2 shows a schematic diagram of an LF imaging system, in which array-based optics capture the four-dimensional LF in the object space, and the acquired LF data is computationally projected to a single two-dimensional virtual image on the virtual image plane. The computational process in conventional LF imaging corresponds to an imaging process of a virtual camera in a virtual space, as shown in Fig. 2. The virtual imaging process is emulated by ray-tracing. As a result, arbitrary camera conditions, including the focusing distance, F#198703 - $15.00 USD Received 2 Oct 2013; revised 15 Nov 2013; accepted 15 Nov 2013; published 21 Nov 2013 (C) 2013 OSA 2 December 2013 | Vol. 21, No. 24 | DOI:10.1364/OE.21.029523 | OPTICS EXPRESS 29524

v

t

L(s, t, u, v) u

s

Fig. 1. Definition of light field (LF).

Virtual optics Virtual image plane Object

LF data Optical imaging

Virtual image

Computational projection

Fig. 2. Schematic diagram of LF imaging system.

number, camera position, etc., can be realized even after image capturing [4, 5]. An interesting application of LF imaging is so-called all in-focus imaging, or extended depthof-field (EDOF) imaging, where objects located at different distances can be brought into focus. In the case of Fig. 2, the focused object has to be located on the virtual image plane, and the depth-of-field (DOF) of the virtual image is limited. All in-focus imaging is realized by arbitrarily changing the shape of the virtual image plane for space-variant focusing. The shape of the virtual image plane has to be equivalent to the object’s shape. In general, however, it is difficult to obtain an accurate depth map in object space, and processing for estimating the depth map generally involves a large computational cost [6, 7]. As well as depth estimation, another important issue in LF imaging is super-resolution processing, because the spatial resolution is typically compromised in order to observe the angular information of the light rays. The pixel pitch in the virtual image plane can also be determined arbitrarily. A super-resolved image having a higher resolution than that captured by elemental optics, which is one of the sub-optics in the array-based optics, can be reconstructed by a computational process based on ray tracing with sub-pixel precision [6–12]. As an alternative to the image-based super-resolution in LF imaging, it can also be realized by providing additional optical elements [13–15]. 1.2. Phase-modulation imaging Imaging systems based on optical phase modulation and postprocessing have been studied with the aim of enhancing imaging performance and realizing highly functional imaging. Fig-


Phase modulation

Postprocessing Object

Image Optical imaging Fig. 3. Schematic diagram of phase-modulation imaging.

ure 3 schematic illustrates phase-modulation imaging using a phase plate. Examples of phasemodulation imaging are given below. 1.2.1.

Extended depth-of-field (EDOF)

Optical design employing phase modulation can realize highly depth-invariant point spread functions (PSFs). In this design, the focusing range is optically extended in single-shot imaging, as illustrated in Fig. 3, resulting in blurred but depth-invariant PSFs. The images captured by such systems can be deconvolved into a single EDOF image with a single PSF [16]. For example, a cubic phase mask, spherically aberrated optics, and a radially symmetric kinoform diffuser have been used to realize EDOF imaging [16–19]. 1.2.2.

Depth measurement

Phase modulation can also realize depth-variant PSFs. The depth map of an object is retrieved by estimating the defocused PSFs on a captured image [20]. PSFs in which depth-variance is enhanced have been implemented by using a phase plate or a diffuser in the imaging optics [21– 23]. 1.2.3.

Super-resolution

Phase modulation is also used for enhancing sub-pixel structures in PSFs to realize superresolution imaging. A super-resolved image, which has a higher resolution than the Nyquist sampling pitch of the image sensor, can be obtained via the inverse process of the imaging process [24]. A random phase mask has been proposed for realizing such PSFs [24, 25]. 1.3. Phase-modulated LF imaging In this paper, we present a framework for realizing arbitrary phase modulation in LF imaging systems to achieve the promising imaging functions mentioned above. Although amplitude modulation in LF imaging has been demonstrated by other researchers [4, 5], there has been little work on phase modulation. The computational projection used in the amplitude modulation schemes assumes a virtual aperture stop with an arbitrary shape to realize variable perspective view and variable DOF. In contrast, in the proposed method, virtual optics used in the computational process implements phase modulation virtually. The proposed method does not require specially designed physical phase modulating elements because these elements are computationally emulated. This means that the proposed method is advantageous in terms of its higher flexibility and lower implementation costs compared with conventional phase-modulation imaging systems. #198703 - $15.00 USD Received 2 Oct 2013; revised 15 Nov 2013; accepted 15 Nov 2013; published 21 Nov 2013 (C) 2013 OSA 2 December 2013 | Vol. 21, No. 24 | DOI:10.1364/OE.21.029523 | OPTICS EXPRESS 29526

Tilted optical axes in virtual optics

Postprocessing

Object

LF data Optical imaging

Virtual image Computational projection

Fig. 4. Schematic diagram of phase-modulated LF camera.

s u v t f

z

Fig. 5. Definitions of the system parameters.

We describe an application of the proposed method to a phase-modulated camera and projector, and we analyze the conditions for the system design to satisfy the sampling theorem. In the following section, we explain the concept with an LF camera. An LF projector is merely an inversion of an LF camera, and thus, the concept of the LF camera can be directly applied to an LF projector. Finally, as examples of phase-modulated LF imaging, we demonstrate EDOF and super-resolution imaging for a camera and projector. 2.

Proposed scheme

In the proposed scheme, the phase modulation is computationally realized by the modulated virtual optics used in the computational process of the LF camera, as shown in Fig. 4. From the perspective of geometric optics, phase modulation that changes the normals of the equiphase wavefront corresponds to changing the directions of the rays. In this paper, therefore, the phase modulation is implemented by tilting the optical axes of the virtual elemental optics in the virtual space. In this section, we formulate an expression for the required angles of the optical axes for achieving the desired phase modulation. #198703 - $15.00 USD Received 2 Oct 2013; revised 15 Nov 2013; accepted 15 Nov 2013; published 21 Nov 2013 (C) 2013 OSA 2 December 2013 | Vol. 21, No. 24 | DOI:10.1364/OE.21.029523 | OPTICS EXPRESS 29527

(a)

(b)

Fig. 6. Designs for implementing phase modulation in virtual optics. Phase modulation by (a) using a phase plate, and (b) tilting the optical axes of virtual elemental optics for achieving modulation equivalent to that of a phase plate.

Figure 5 shows definitions of the system parameters used for the analysis. In this paper, the global coordinates of the lens plane are (s, t), and the local coordinates of the sensor plane in each elemental optics are (u, v). The distance between each lens and sensor is the focal length f of the lens, and the axial coordinate from the lens plane is defined as z. For simplicity, the v-axis and t-axis are omitted in our analysis. The index k of the elemental optics is defined as shown in Fig. 5, and the order of k is along the global coordinate s. The total number of elemental optics is N. A phase plate for phase modulation is modeled as a glass plate with a refractive index n and a shape function z = g(s). The thickness of the phase plate is neglected in this paper. As the effect of refraction by the phase plate, the angle of an emerging ray is changed as shown in Fig. 6(a). The modulated angle m(s) of the optical axes of the virtual elemental optics for achieving the phase modulation is calculated as m(s) = φem − φin ,

(1)

where, based on Snell’s law, ! ∂g(s) , φin = arctan ∂s φem = arcsin (n sin (φin )) .

(2) (3)

Here, φin and φem are the angles of incidence and emergence measured from the normal. In the proposed scheme, the modulation m(s) is emulated by the tilt angle of the optical axes of the virtual elemental optics in the LF camera, as shown in Fig. 6(b). By changing the design of g(s), an arbitrary phase plate can be realized computationally. 3.

Sampling in phase-modulated LF imaging

In LF camera systems, the LF in the object space is sampled discretely by an array of imaging optics and an array of detector pixels. To avoid aliasing in the virtual images due to undersampling, the pitches of the elemental optics and detector pixels should be designed based on


the sampling theorem [26]. In this section, we introduce the sampling conditions for phasemodulated LF imaging. With the introduced conditions, the sampling properties of LF signals, PSFs, and image acquisition are simulated by emulating EDOF imaging as an example of phase-modulation imaging. 3.1. Formulation of sampling In this subsection, we formulate the Nyquist pixel pitch and the Nyquist pitch of elemental optics, which are used for the proposed system design. 3.1.1.

Nyquist pixel pitch

Here, we formulate the required pixel pitch ∆u of the elemental optics for sampling the LF signal without aliasing. The smallest structure in an image of an object on a sensor is defined as ∆o. In this paper, we assume Lambertian objects whose surfaces exhibit light reflection with no angular dependency. The resolution of an imaging system is limited by diffraction and aberrations in the optics, as well as the pixel pitch of the image sensor. The resolvable size due to diffraction and aberrations, namely, the diameter of the Airy disc, is defined as ∆a [27], and the resolvable size due to the image sensor, namely, the pixel pitch, is defined as ∆u. The latter resolution limit ∆u can be modified by applying super-resolution processing to give δu as follows [10]: δu =

∆u , nsr

(4)

where nsr is a natural number which can be considered as a resolution improvement factor resulting from the super-resolution processing. The maximum nsr is the total number N of elemental optics theoretically. In practice, however, nsr is lower than N because the resolution of the imaging system is determined by the larger of the wave optics-based resolution ∆a and the geometrical optics-based resolution δu, considering super-resolution [10]. To sample the LF signal with an over-sampling rate for the smallest structure ∆o in the object image, the required condition is max(∆a, δu) ≤

∆o . 2

(5)

Assuming that general imaging conditions satisfy δu > ∆a, the required condition for the pixel pitch ∆u can be formulated as ∆u = nsr δu ≤ nsr 3.1.2.

∆o . 2

(6)

Nyquist pitch of elemental optics

Now we determine the pitch of the elemental optics in the phase-modulated LF camera system. The pitch has to be appropriately set to generate a smaller disparity between neighboring elemental optics than the pixel pitch of the image sensor [26]. In the formulation of this condition, we consider the disparity of the center pixel of the image sensor, and the disparities of the other pixels are approximated as that of the center pixel based on the paraxial approximation. In the computational process, the LF data are projected onto the virtual image plane. In our scheme, the projection is performed by calculating the geometrical relation between the pixels on the sensor and the pixels on the virtual image plane, as shown in Fig. 7. By projecting from the k-th elemental optics located at s(k) , the center pixel in the virtual image plane detects a ray


Virtual lens array Virtual image plane

Fig. 7. Geometrical relation between pixels on a sensor and pixels on a virtual image plane in virtual space.

Object image plane

Virtual image plane

Fig. 8. Geometrical representation of the disparity.

from a pixel at us , as shown in the figure. The coordinate us of the sampled pixel in the LF data can be formulated as follows: us s(k) , zv , m (s) = f tan θray + θmod , (7) where, as shown in Fig. 7,

! s(k) , θray = arctan zv θmod = m s(k) .

(8) (9)

Here, zv is the distance between the virtual image plane and the virtual elemental optics, θray is the angle of the ray passing through the lens coordinate s(k) measured from the normal of the virtual image plane, and θmod is the tilt angle of the virtual lens for phase modulation. The geometrical relation between the pixels on the sensor and the object image plane can also be calculated by using Eq. (7) by replacing the distance zv with the distance zo between the object image plane and the real elemental optics, and the phase is not modulated, namely, m(s) = 0. The disparity d is the difference of the local coordinates us obtained by Eq. (7) of the neighboring elemental optics in the object and virtual spaces, as shown in Fig. 8. It also changes #198703 - $15.00 USD Received 2 Oct 2013; revised 15 Nov 2013; accepted 15 Nov 2013; published 21 Nov 2013 (C) 2013 OSA 2 December 2013 | Vol. 21, No. 24 | DOI:10.1364/OE.21.029523 | OPTICS EXPRESS 29530

depending on the phase modulation m(s) or the tilt of the virtual elemental optics, as shown in the figure. The disparity d between the k-th and (k − 1)-th elemental optics can be calculated as follows: d s(k) , ∆s, zo , zv , m (s) = |ωo − ωv | , (10) where, as illustrated in Fig. 8,

ωo = us s(k) , zo , 0 − us s(k−1) , zo , 0 , ωv = us s(k) , zv , m(s) − us s(k−1) , zv , m(s) , (k)

∆s = s

(k−1)

−s

.

(11) (12) (13)

Here, ωo is the difference of the local coordinates us of the two elemental optics in the object space, ωv is that in the virtual space, and ∆s is the pitch of the elemental optics (element pitch). The disparity d of the k-th elemental optics is defined as the absolute difference between ωo and ωv . To introduce the Nyquist element pitch, the mean disparity d of the system is defined as follows: PN (k) (s) d s , ∆s, z , z , m o v k=2 d (∆s, zo , zv , m (s)) = . (14) N −1 The condition for the pitch of the elemental optics for making the mean disparity less than the pixel pitch on the image sensor is as follows: d (∆s, zo , zv , m (s)) ≤

∆u = δu. nsr

(15)

The mean disparity d monotonically increases with the pitch of the elemental optics ∆s; therefore, the maximum element pitch satisfying the sampling condition in Eq. (15) can be found uniquely. To avoid aliasing in reconstructed images, the pitches of the image sensor pixels and the elemental optics in the phase-modulated LF imaging system should be designed to satisfy the conditions of Eqs. (6) and (15) for all virtual distances zv of the image plane within an assumed range. 3.2. Simulations based on EDOF imaging In this subsection, we analyze the sampling properties of the LF data in the computational projection based on Eq. (7), and we show the results of simulations performed to confirm the effect of the sampling condition of Eq. (15). In the simulations, EDOF imaging described in Section 1.2.1 is used as an example of phase-modulation imaging. 3.2.1.

Sampling properties

To analyze the EDOF properties, a normalized defocus parameter Ψ is introduced [16, 27]: ! 1 πA2 1 1 (16) Ψ= − − Ψ . 4λ fLF zo zv Here, zΨ v is the distance of a virtual image plane generating the defocus Ψ, λ is the wavelength, A is the diameter of the whole array-based optics, and fLF is the focal length of the system. A


Table 1. System parameters used in simulations.

Refractive index of phase plate Free-space wavelength Diameter of whole array-based optics Distance of object image plane Focal length of elemental optics Number of pixels of elemental optics Pitch of pixels Pitch of elemental optics Distance of virtual image plane

Parameter n λ A zo f ∆u ∆s zv

Input value 1.6 550 [nm] 100 [mm] ∞ 0.26 [mm] 256 × 256 1 [µm] variable variable

and fLF are given by A = (N − 1)∆s, 1 fLF = 1 . 1 zo + zΨ=0

(17) (18)

v

In this paper, the in-focus distance zΨ=0 of the virtual image plane is defined identically to the v distance zo of the assumed object plane as follows: zΨ=0 = zo . v

(19)

Assuming a system with the parameters shown in Table 1, the sampling pattern of the LF data by the center virtual pixel based on Eq. (7) is shown in Fig. 9. In the simulations, the distance zv of the virtual image plane was varied to evaluate the EDOF performance, and the element pitch ∆s was also varied to satisfy the condition in Eq. (15) at each zv . Figure 9(a) shows the sampling pattern of a conventional LF imaging system with no phase modulation. Figures 9(b)–9(d) show the sampling patterns of LF imaging systems with a cubic phase mask, spherical optics, and a radially symmetric kinoform diffuser. For emulating such phase plates, the shape functions g(s) in Eqs. (2) and (3) are given by [16, 17, 19]: g(s) = αs3 , s g(s) = β

d g(s) ∼ P, ds

1−

(20) !2

s , γ

(21) (22)

where α, β, and γ are arbitrary constants for adjusting the imaging conditions [28], and P is an arbitrary probabilistic function. Crosses in Fig. 9 are the LF data captured by the arraybased optics and the image sensor, and the LF data on the colored lines and within the colored regions are data sampled based on Eq. (7). Note that the LF data in Fig. 9 are sparser than the actually sampled data in the simulations in order to enhance the effect of discrete sampling. Red, Ψ=−15 , and zΨ=−30 , respectively. The green, and blue colors show the sampling patterns at zΨ=0 v , zv v computational projection in Fig. 4 corresponds to the integration of the LF data on the colored lines or in the colored regions. In the case of the diffuser shown in Fig. 9(d), the LF data in the colored regions are sampled probabilistically based on the designed probabilistic function [19]. #198703 - $15.00 USD Received 2 Oct 2013; revised 15 Nov 2013; accepted 15 Nov 2013; published 21 Nov 2013 (C) 2013 OSA 2 December 2013 | Vol. 21, No. 24 | DOI:10.1364/OE.21.029523 | OPTICS EXPRESS 29532

50

0

−50 −50

s (mm)

s (mm)

50

0

0

50

u (μm)

−50 −50

0

(a)

(b) 50

s (mm)

s (mm)

50

0

−50 −50

50

u (μm)

0

50

u (μm)

(c)

0

−50 −50

0

50

u (μm)

(d)

Fig. 9. Sampling patterns of LF data and PSFs in the systems (a) without and (b)–(d) with phase modulation. The modulations were designed by emulating (b) a cubic phase mask, (c) spherical optics, and (d) a radially symmetric kinoform diffuser.

In conventional LF imaging, the integral lines are straight, as in Fig. 9(a). The gradients of the lines correspond to the distances zv of the virtual image planes. In phase-modulated LF imaging on the other hand, the lines are curved or replaced with regions. The line profiles in the right-hand sides of each subfigure in Fig. 9 show each of the PSFs based on the integrations. In a phase-modulated LF imaging system with EDOF, the imaging is more depth-invariant compared with that of a conventional imaging system. Fig. 9(a) illustrates the pixel pitch ∆u, the element pitch ∆s, and the disparity d in the computational projection defined as in Eq. (10). The disparity d corresponds to the difference of the sampled u-coordinates in the neighboring elemental optics, as indicated in the figure, and the mean disparity d corresponds to the mean value of the disparities in the whole set of elemental optics. The sampling condition for the pitch of the elemental optics in Eq. (15) corresponds to a comparison between the mean disparity d and the pixel pitch δu after super-resolution processing. 3.2.2.

PSF analysis

To verify the sampling conditions in the proposed scheme, we analyzed the PSFs obtained by changing the pitch of the elemental optics. As an example of EDOF imaging, the cubic phase mask (CPM) expressed in Eq. (20) was emulated in the phase-modulated LF camera with α = 40 for one direction, along the u and s-axes.


−50

0

50

u (μm)

Intensity

Intensity

Intensity

Theoretically derived

−50

50

−50

0

u (μm)

50

(c)

Intensity

(b)

Intensity

(a)

0

u (μm)

−50

0

u (μm) (d)

50

−50

0

u (μm)

50

(e)

Fig. 10. The PSFs in CPM-based wavefront coding obtained by (a) analytical derivation and (b)–(e) numerical simulations. In the simulations, the pitch of the elemental optics was determined to make the mean disparity (b) half of the pixel pitch, (c) equal to the pixel pitch, (d) double the pixel pitch, and (e) five times larger than the pixel pitch at zΨ=0 v .

The PSF in CPM-based wavefront coding can be theoretically derived as follows: Z 2 1 +1 exp jαs3p + jkW20 s2p − j2πusp dsp , h (u, W20 ) = √ 2 −1 λΨ , W20 = 2π s sp = , A

(23) (24) (25)

where sp is the normalized pupil coordinate [16, 29]. The line profile of the theoretical PSF at Ψ = 0, which was derived by Eq. (23) with geometrical optics, is shown in Fig. 10(a). Note that, since the computational projection in the proposed scheme is performed based on geometrical optics, the PSF in Fig. 10(a) was calculated by neglecting the diffraction effects due to the finite pupil, for comparison with the proposed scheme. Figures 10(b)–10(e) show the PSFs of the proposed scheme with four different element pitches, in which the mean disparities d in Eq. (14) are one-half of the pixel pitch, equal to the pixel pitch, double the pixel pitch, and five times larger than the pixel pitch, respectively. With the over-sampling condition of the element pitch in Eq. (15), the PSFs were approximately the same as the theoretical one in Fig. 10(a). In contrast, in the PSFs with the under-sampling con#198703 - $15.00 USD Received 2 Oct 2013; revised 15 Nov 2013; accepted 15 Nov 2013; published 21 Nov 2013 (C) 2013 OSA 2 December 2013 | Vol. 21, No. 24 | DOI:10.1364/OE.21.029523 | OPTICS EXPRESS 29534

Table 2. The pitch of optics and the achieved sampling pitch.

zvΨ=0 Ψ=−30 zv

∆s = 0.26 mm d/δu = 0.50 d/δu = 0.55

∆s = 0.51 mm d/δu = 1.0 d/δu = 1.1

∆s = 1.0 mm d/δu = 2.0 d/δu = 2.2

∆s = 2.6 mm d/δu = 5.0 d/δu = 5.5

dition, some fluctuations appeared. The results also indicate that the change in the PSFs according to the element pitch is only slight around the bound of the sampling condition (d/δu = 1.0); however, the bound can be used as a good criterion for obtaining approximately the same PSF as the theoretically derived PSF. 3.2.3.

Analysis with two-dimensional image

The effect of the sampling condition in Eq. (15) was confirmed with a two-dimensional object using the system described by the parameters in Table 1. In the simulations, the finest structure ∆o of the object image on the image sensor was set to double the pixel pitch δu to satisfy Eq. (6), where nsr = 1. As well as the PSF analysis described above, CPM-based wavefront coding in one direction was chosen as an example of the phase modulation. Based on the proposed system emulating the CPM, the imaging process was simulated with virtual image planes at zΨ=0 and zΨ=−30 while changing the element pitch ∆s. The relation between the simulated v v element pitch ∆s and the achieved mean disparity d in the two virtual image planes is shown in Table 2. The images computationally projected onto the virtual image planes based on Eq. (7) are shown in Fig. 11(a). With the conventional LF imaging, a sharp image appeared at zΨ=0 v , which is the in-focus distance, and a defocused image appeared at zΨ=−30 . On the other hand, with the v phase-modulated LF camera, images blurred by the depth-invariant PSF were obtained at both distances. The computationally projected images for the under-sampling condition in Eq. (15) showed some aliasing artifacts. Figure 11(b) shows final images of the conventional and phase-modulated LF cameras. The final images obtained with the conventional method are the same as the images in Fig. 11(a), and those obtained with the proposed method are deconvolved versions of those in Fig. 11(a). In the deconvolution of all conditions, a single Wiener filter with a PSF at d/δu = 0.5 and zΨ=0 v , which can be approximated as the theoretically derived in-focus PSF as indicated in Fig. 10(b), was used to make clear the effect of each sampling condition [27]. Compared with the conventional LF imaging, the DOF of the proposed LF camera was successfully extended when the sampling condition in Eq. (15) was satisfied. The deconvolved images with the under-sampling condition have some artifacts even at the in-focus distance zΨ=0 v . Artifacts in the deconvolved image at zΨ=−30 in the over-sampling condition occurred due to a small depth invariance of the PSFs v caused by the CPM for EDOF imaging, not the LF imaging [30]. When the element pitch was designed to satisfy the under-sampling condition, artifacts appeared due to aliasing in the computational projection. The differences between the deconvolved images with d/δu = 0.5 and the others at each distance are shown in Fig. 12. The images of the differences and their corresponding peak signal-to-noise ratios (PSNRs) indicate that the artifacts increased in proportion to the element pitch at both distances. In conclusion, the system should be designed to satisfy the sampling conditions of Eq. (6) and Eq. (15) at all virtual image plane distances to avoid aliasing.


Conventional

Proposed

(a) Conventional

Proposed

(b)

Fig. 11. Simulations with two-dimensional image. (a) Computationally projected images and (b) final images obtained with the conventional and proposed LF cameras with the CPM while changing the element pitch and the object distance. Red rectangles and blue rectangles mean over-sampling and under-sampling conditions, respectively. #198703 - $15.00 USD Received 2 Oct 2013; revised 15 Nov 2013; accepted 15 Nov 2013; published 21 Nov 2013 (C) 2013 OSA 2 December 2013 | Vol. 21, No. 24 | DOI:10.1364/OE.21.029523 | OPTICS EXPRESS 29536

120 100 80 60 40 20 0

PSNR = ∞ dB

PSNR = 35.1 dB

PSNR = 32.2dB

PSNR = 22.2 dB 120 100 80 60 40 20 0

PSNR = ∞ dB

PSNR = 36.6 dB

PSNR = 31.7 dB

PSNR = 23.9 dB

Fig. 12. Differences between deconvolved images with d/δu = 0.5 and others at zΨ=0 and v zΨ=−30 , and their corresponding PSNRs. v Imaging by array-based optics

Computational projection with phase modulation

Optical imaging

Deconvolution

Computational reconstruction (a)

Deconvolution

Computational projection with phase modulation Computational processing

Imaging by array-based optics Optical imaging

(b)

Fig. 13. Schematic diagram of (a) EDOF camera and (b) EDOF projector based on phasemodulated LF imaging.

4.

Experimental demonstrations

We experimentally demonstrated the proposed scheme in camera and projection systems. In the experiments, we demonstrated super-resolution and EDOF based on the CPM as examples of phase-modulation imaging. The demonstrated camera and projector did not require estimation of a depth map of the three-dimensional object or screen, like CPM-based EDOF imaging. This is the main advantage of the proposed method over the conventional all in-focus LF imaging from the perspective of computational cost [6,7]. System flows of the experiments are shown in Fig. 13. In the camera system, the LF in the object space was captured by a camera array, and the captured LF data were computationally projected into a single image with phase modulation. The projected image was deconvolved into an EDOF image. In the projection system, on the other hand, the deconvolution process and computational projection were performed before the optical projection, as in [31–34]. The generated LF data were projected by a projector array. In


s z Capturing

t CCD camera

Paper 35

100

350

[mm]

Fig. 14. Setup used for experimental verification of the EDOF camera.

the proposed scheme, the phase modulation was achieved by tilting the virtual elemental optics in the computational projection process, as shown in Fig. 4. 4.1. Camera systems 4.1.1.

EDOF camera

EDOF imaging based on a camera array was demonstrated by use of the setup shown in Fig. 14. In this experiment, a monochrome CCD camera (PL-B953U, manufactured by PIXELINK, including 1200 × 768 pixels with a pixel pitch of 4.65 × 4.65 µm) was scanned mechanically to emulate a camera array. As in the simulations described in Section 3, scanning of the t-axis was omitted. A tilted sheet of paper was used as a three-dimensional object. In the experiment, the object was captured while scanning the camera along the s-axis at a fixed interval. The interval was set to 0.2 mm to satisfy the sampling condition of Eq. (15) within the range of the object distances used. Since the aim of this experiment was to verify the EDOF performance, super-resolution was not demonstrated, and thus, the improvement factor nsr in Eq. (15) was 1. The pixel pitch sampling condition in Eq. (6), where nsr = 1, was also satisfied. One of the captured images is shown in Fig. 15(a). The pixel values of the image were normalized according to the reconstructed images, and white Gaussian noise, with an SNR of 10 dB, was added to the captured images to demonstrate noise robustness in EDOF imaging. The captured images (LF data) were computationally projected. A projected image without the computational phase modulation emulating the CPM is shown in Fig. 15(b). The noise in the projected image was suppressed compared with that in the captured image by the integration process in the computational projection; however, the in-focus region in the image was restricted by the DOF of the conventional LF imaging. On the other hand, the computationally projected image with the computational phase modulation shown in Fig 15(c) was subjected to noise removal and was defocused by the depth-invariant PSF. The deconvolved image obtained by Wiener filtering using the phase-modulated image is shown in Fig. 15(d). The DOF was successfully extended compared with the computationally projected image without phase modulation in Fig. 15(b). The noise in the computationally projected image in Fig. 15(c) was slightly enhanced by deconvolution, as with general EDOF imaging based on depth-invariant PSFs; however, the amount of noise in the deconvolved image shown in Fig. 15(d) was obviously lower than that in the image captured by the single elemental optics shown in Fig. 15(a).


(a)

(b)

(c)

(d)

Fig. 15. Results obtained with the EDOF camera based on phase-modulated LF imaging. (a) A single captured image, the computationally projected images (b) without and (c) with computational phase modulation, and (d) the deconvolution of Fig. (c).

4.1.2.

Super-resolved EDOF camera

We also confirmed the effect of super-resolution in the EDOF imaging. The horizontal pixel count of captured images was reduced by 1/nsr , where nsr is the resolution improvement factor in Eq. (4); in this experiment, nsr = 3. The down-sampled pixel pitch in the low-resolution captured images was nsr ∆u. Noise was also added to them, with an SNR of 15 dB. One of the low-resolution captured images with nsr = 3 is shown in Fig. 16(a). The text is not recognizable due to the down-sampling and noise. Computationally projected images without and with phase modulation, obtained using ray tracing with sub-pixel precision for achieving super-resolution, are shown in Figs. 16(b) and 16(c), respectively. The resolutions were improved and noise was removed in both cases. The DOF in Fig. 16(b) is limited, and the PSF in Fig. 16(c) is depthinvariant. The deconvolution result of Fig. 16(c) is shown in Fig. 16(d), in which the resolution was improved and the noise was suppressed compared with Fig. 16(a). The results show that a super-resolved EDOF camera was successfully demonstrated. 4.2. Projector systems A projector based on the proposed scheme was also demonstrated. The experimental setup is illustrated in Fig. 17. Instead of an array projector, in this experiment, a three-panel LCD projector (EH-TW400 manufactured by EPSON, including 1280 × 800 pixels with a pixel pitch of 5.93 ×7.12 µm) was scanned along the s-axis. The projected images were captured by a CCD camera (PL-B953U manufactured by PIXELINK), and the captured images were superposed


(a)

(b)

(c)

(d)

Fig. 16. Results obtained with super-resolved EDOF camera based on phase-modulated LF imaging. (a) A single low-resolution captured image, the computationally projected images (b) without and (c) with computational phase modulation, and (d) the deconvolved image of Fig. (c).

CCD camera z Projection

s t 3-LCD projector

Screen 200

40

590

[mm]

Fig. 17. Setup used for experimental verification of EDOF projector.

by signal processing instead of the integration function performed by the human visual system. A tilted screen was placed in front of the projector.


(a)

(b)

Fig. 18. (a) An input Lena image and (b) its deconvolved image.

(a)

(b)

(c)

(d)

Fig. 19. Results obtained with the EDOF projector based on phase-modulated LF imaging. The optically projected images of the Lena image in Fig. 18(a) based on (a) the conventional single projector and the LF projector (b) without and (c) with computational phase modulation. (d) The optically projected image of the deconvolved Lena image in Fig. 18(b) based on the LF projector with computational phase modulation.

4.2.1.

EDOF projector

First, input Lena images like that shown in Fig. 18(a) were computationally projected to generate the LF data without and with the phase modulation emulating the CPM. To verify the EDOF performance, the deconvolution of the Lena image in Fig. 18(b) was also projected into the LF data with phase modulation, as shown Fig. 13(b). Considering the constraint of the maximum grayscale value of pixels in incoherent projectors, deconvolution was performed by solving the #198703 - $15.00 USD Received 2 Oct 2013; revised 15 Nov 2013; accepted 15 Nov 2013; published 21 Nov 2013 (C) 2013 OSA 2 December 2013 | Vol. 21, No. 24 | DOI:10.1364/OE.21.029523 | OPTICS EXPRESS 29541

following inverse problem instead of the frequency filtering process as in [31]: î = arg max L[ii|o¯ = t ] i

(26)

subject to 0 ≤ î (p) ≤ c, ∀p, where î is the deconvolution result used as an input image i , L[·] is the likelihood function, o¯ is a projected image of the EDOF projector, t is a target image, î (p) is the pixel value of the p-th pixel of î, and c is the maximum pixel value of the incoherent projector. In this experiment, the target image t was the Lena image shown in Fig. 18(a). In the experiment, the inversion problem of Eq. (26) was solved by using the Richardson and Lucy algorithm [35, 36]. The generated LF data were projected onto the tilted screen by scanning the projector at fixed spatial intervals. The interval was set to 1.3 mm to satisfy the sampling condition of Eq. (15). Since super-resolution was not demonstrated in this experiment, the improvement factor nsr was 1. Noise with an SNR of 1 dB was added to the captured images to demonstrate noise robustness in the EDOF imaging. An image optically projected by the conventional single projector is shown in Fig. 19(a). The pixel values were normalized according to the images obtained with the LF projector. Images optically projected by the LF projector using the original Lena image in Fig. 18(a) without and with phase modulation are shown in Figs. 19(b) and 19(c). Similarly to the camera system experiments described above, a noise-suppressed image with limited-DOF and a noisesuppressed image defocused with the depth-invariant PSF were generated. The resulting image obtained with the LF projector using the deconvolved Lena image in Fig. 18(b) and the phase modulation is shown in Fig. 19(d). The noiseless EDOF image was optically projected onto a three-dimensional screen. 4.2.2.

Super-resolved EDOF projector

We demonstrated a super-resolved EDOF projector based on the proposed scheme. Similar to the camera system experiments described above, the elemental input image in the LF data was down-sampled horizontally, with an improvement factor nsr = 3 in Eq. (4). Noise was also added to the captured images, with an SNR of 5 dB. The optically projected low-resolution image generated by the conventional single projector is shown in Fig. 20(a). Using ray tracing with sub-pixel precision to achieve super-resolution, the optically projected images obtained with the LF projector using the original Lena image in Fig. 18(a) generated without and with computational phase modulation for the CPM are shown in Figs. 20(b) and 20(c). The resolutions were improved and noise was removed in both cases. The DOF in Fig. 20(b) is limited, and the PSF in Fig. 20(c) is depth-invariant. The optically projected image obtained with the super-resolved LF projector using the deconvolved Lena image in Fig. 18(b) with computational phase modulation is shown in Fig. 20(d). The noise-suppressed and super-resolved EDOF image was optically projected onto the three-dimensional screen successfully. 5.

Conclusion

In the research described in this paper, we have proposed a scheme for realizing arbitrary phase modulation in light field (LF) imaging. In a camera system based on our scheme, array-based optics capture the LF of an object, and the captured LF data is computationally projected into a single image with phase modulation. In a projector system based on the scheme, an input image is computationally projected into the LF data with phase modulation, and the generated LF data is optically projected by array-based optics. Phase modulation was realized by tilting the optical axes of the virtual elemental optics in the computational projection process in LF imaging. #198703 - $15.00 USD Received 2 Oct 2013; revised 15 Nov 2013; accepted 15 Nov 2013; published 21 Nov 2013 (C) 2013 OSA 2 December 2013 | Vol. 21, No. 24 | DOI:10.1364/OE.21.029523 | OPTICS EXPRESS 29542

(a)

(b)

(c)

(d)

Fig. 20. Results obtained with the super-resolved EDOF projector based on phasemodulated LF imaging. Optically projected images of the Lena image in Fig. 18(a) based on (a) the conventional single projector and the LF projector (b) without and (c) with computational phase modulation. (d) An optically projected image of the deconvolved Lena image in Fig. 18(b) based on the LF projector with computational phase modulation.

The required conditions in the system design were also described. Expressions for the pitches of image sensor pixels and an array of elemental optics in the proposed scheme were formulated, and the conditions were verified by simulation. As examples of phase-modulation imaging, a super-resolved camera and projector with EDOF based on the proposed LF imaging scheme were numerically and experimentally demonstrated. In these systems, it was not necessary to estimate the shapes of the three-dimensional object and screen, like general EDOF techniques based on depth-invariant PSFs. Although the proposed method was demonstrated by scanning a camera or a projector in the experiments, the method is applicable to any type of LF imaging system so long as the sampling conditions derived in Section 3.1 are satisfied. Our proposed scheme realizes arbitrary phase modulation in a single imaging system. Such an imaging system is a promising platform for computational imaging with phase coding. Acknowledgment This research was supported by Grant-in-Aid for JSPS Fellows from the Japan Society for the Promotion of Science.


Direct imaging of phase objects enables conventional deconvolution in bright field light microscopy.

Computational photography with plenoptic camera and light field capture: tutorial.

Spectral modulation interferometry for quantitative phase imaging.

Light field shaping by tailoring both phase and polarization.

In-vivo dark-field and phase-contrast x-ray imaging.

Modulation efficiency of double-phase hologram complex light modulation macro-pixels.

Light Field Imaging Based Accurate Image Specular Highlight Removal.

Real-time integral imaging system for light field microscopy.

Surface measurements by white light spatial-phase-shift imaging interferometry.

Subnanoradian X-ray phase-contrast imaging using a far-field interferometer of nanometric phase gratings.

Liquid crystal spatial light modulator with very large phase modulation operating in high harmonic orders.

Wide-field computational imaging of pathology slides using lens-free on-chip microscopy.

Computational superoscillation imaging beyond the Rayleigh limit from far-field measurements.

Wide-field in vivo background free imaging by selective magnetic modulation of nanodiamond fluorescence.

Light output measurements and computational models of microcolumnar CsI scintillators for x-ray imaging.

Compressive light field displays.

Computational On-Chip Imaging of Nanoparticles and Biomolecules using Ultraviolet Light.

Volumetric Light-Field Excitation.

Investigation of mechanosensation in C. elegans using light field calcium imaging.

Light field otoscope design for 3D in vivo imaging of the middle ear.

Optical dissection of brain circuits with patterned illumination through the phase modulation of light.

Computational error of irregular field dosimetry.

Optimized asymmetrical tangent phase mask to obtain defocus invariant modulation transfer function in incoherent imaging systems.

General analytic solution for far-field phase and amplitude control, with a phase-only spatial light modulator.