Experimental verification of compressive reflectance field acquisition Yusuke Tampa, Ryoichi Horisaki,* and Jun Tanida Department of Information and Physical Sciences, Graduate School of Information Science and Technology, Osaka University, 1-5 Yamadaoka, Suita, Osaka 565-0871, Japan *Corresponding author: [email protected]‐u.ac.jp Received 3 December 2013; revised 22 April 2014; accepted 22 April 2014; posted 22 April 2014 (Doc. ID 202245); published 9 May 2014

We demonstrate compressive sensing (CS) of the eight-dimensional reflectance field (RF), which describes spatial and angular information of light rays toward and from an object. The RF is also known as the bidirectional scattering surface reflectance distribution function. In this method, incident rays and reflected rays to/from the object are modulated by variable coding masks, and the modulated rays are multiplexed onto an image sensor. The images captured with multiple mask patterns are decoded by a CS algorithm. The RF of the object was successfully reconstructed from less than half of the number of measurements required with conventional methods. © 2014 Optical Society of America OCIS codes: (110.1758) Computational imaging; (100.3010) Image reconstruction techniques; (290.1483) BSDF, BRDF, and BTDF. http://dx.doi.org/10.1364/AO.53.003157

1. Introduction

Computer graphics rendering is a technique for representing the physical world virtually. Various rendering methods have been proposed in this field, and image-based rendering is one promising method [1,2]. This method can represent a realistic scene without any geometrical information, which normally involves a heavy computational cost. The rendering process uses light ray information called the light field (LF) [3]. The LF is a four-dimensional function representing light rays with two parallel planes, as shown in Fig. 1. The LF enables us to generate an arbitrary view and to refocus an object computationally even after the object’s LF is acquired [4–6]. The LF has also been used to represent illumination coming toward an object. In this case, an object illuminated by a light source having an arbitrary shape and light direction can be computationally rendered [7,8]. These LFs toward and from the object have been integrated into an eight-dimensional 1559-128X/14/143157-07$15.00/0 © 2014 Optical Society of America

function called the eight-dimensional reflectance field (RF) [8]. An object under arbitrary illumination and camera conditions can be rendered based on this function. On the other hand, the cost of measuring the RF is extremely high because the positions and angles of the illumination source and camera must be scanned in eight dimensions. To overcome this problem, we have proposed a method for achieving fast RF acquisition using multiplexed measurements of the RF and computational reconstruction with a sparsity constraint based on compressive sensing (CS) [9]. In this paper, we experimentally demonstrate the method and show a reduction in observation time compared with the conventional methods. 2. Light Field and Reflectance Field

The LF Ls; t; u; v of a light ray is defined by the spatial positions on two parallel planes, as shown in Fig. 1, where s; t and u; v are the coordinates through which the ray passes. By using the LF model, both the spatial position and the angle of the ray can be described. The LF can be measured by 10 May 2014 / Vol. 53, No. 14 / APPLIED OPTICS

3157

An alternative model is the RF, which is also known as the bidirectional scattering surface reflectance distribution function (BSSRDF) [8]. The RF Rs; t; u; v; s0 ; t0 ; u0 ; v0  is an eight-dimensional function that is expressed as Rs; t; u; v; s0 ; t0 ; u0 ; v0  

Fig. 1. Definition of the LF.

using a microlens array, a camera array, or a camera with a coded aperture mask [3,10–12]. To express the scene reflectance, a model combining the LFs of incoming rays from an illumination source toward an object and reflected outgoing rays from the object is necessary. One such model is the bidirectional reflectance distribution function (BRDF), Bs; t; u; v; u0 ; v0 , which is a six-dimensional function defined as follows: Bs; t; u; v; u0 ; v0  

dLout s; t; u; v ; dLin s; t; u0 ; v0 

(1)

where Lin s; t; u0 ; v0  and Lout s; t; u; v are the incoming LF toward the object and the outgoing LF from the object, respectively, as illustrated in Fig. 2(a) [13]. This model assumes reflection and scattering on the surface of opaque materials.

dLout s; t; u; v : dLin s0 ; t0 ; u0 ; v0 

(2)

Contrary to the BRDF model, the RF can handle internal reflections and subsurface scattering, where the incident point of the incoming light ray on the object surface is different from the exiting point on the surface, as shown in Fig. 2(b). The RF is more general than the BRDF, and it can represent translucent materials such as human skin and gemstones. Therefore, the RF is promising for photorealistic rendering. A display representing the complete RF has also been demonstrated [14]. Several systems have been proposed for measuring these reflectance functions. For example, mechanical scanning of an illumination source and a camera, a hemispherical arrangement of multiple illumination sources and cameras, and an ellipsoidal mirror with a fixed illumination source and a camera have been used for measuring reflectance functions [15–17]. However, these systems require a long observation time and have a trade-off between spatial and angular resolutions [8,18–20]. 3. Compressive Reflectance Field Acquisition

We have proposed a method for observing the RF based on the CS framework [9]. A. Compressive Sensing

CS is a framework for retrieving a large amount of object data with a few measurements [21–23]. In other words, CS can solve the inverse problem of an ill-posed linear system. Various CS-based optical systems have been proposed for LF imaging and acquisition of some reflectance functions, but these functions have fewer dimensions than the eightdimensional RF [24–28]. Here, we apply CS to complete RF acquisition. Any linear system can be written in the following form: g  Hr;

(3)

where g ∈ RM×1 is vectored measurement data, H ∈ RM×N is a system matrix, and r ∈ RN×1 is vectored object data, which is the RF data in this paper. CS assumes an ill-posed system, where M < N. If the system matrix H; satisfies a condition referred to as the restricted isometry property (RIP), CS solves the inversion of Eq. (3) with a sparsity constraint as follows: rˆ  argmin‖g − Hr‖l2  τSr; r

Fig. 2. Definition of parameters in reflectance functions. (a) BRDF model and (b) RF model. 3158

APPLIED OPTICS / Vol. 53, No. 14 / 10 May 2014

(4)

where ‖ · ‖l2 is the l2 norm, τ is a regularization parameter, and S· is a regularizer for the sparsity

constraint. Some random matrices, such as Gaussian and Bernoulli matrices, satisfy the RIP. We realize CS-based RF acquisition with a pseudo-random system matrix. B.

System Model

We have developed a system for compressive RF acquisition, as shown in Fig. 3. The system uses confocal optics to control the illumination and camera conditions with variable coded masks for light intensity modulation. The projector generates controllable light source arrays on the diffuser, which illuminates the spatial light modulator 1 (SLM1) to angularly modulate the illumination toward the object based on the paraxial approximation as shown in Fig. 3. The intensity of the reflected and/or scattered light from the object is modulated angularly by SLM2 also based on the paraxial approximation. Finally, the modulated light is integrated on the image sensor. Thus, the projector and the two SLMs intensity modulate the RF and multiplex the modulated signals. The multiplexed images are captured multiple times with different patterns on the projector and the SLMs. The RF is reconstructed based on Eq. (4). The system that we developed is modeled as follows: Gm s; t 

XXXXXX u

×

v

s0

t0

s0 ; t0  P SLM1 m

u0

v0

P SLM2 u; v m

0 0 × P proj m u ; v 

× Rs; t; u; v; s0 ; t0 ; u0 ; v0 ;

where Gm · is the mth captured image, and P SLM2 ·, P SLM1 ·, and P proj m · are the mth patterns m m on SLM2 (u − v plane), SLM1 (s0 − t0 plane), and the projector (u0 − v0 plane), respectively. On the object surface, s; t and s0 ; t0  correspond to the spatial coordinates of the outgoing and incoming LFs, and u; v and u0 ; v0  correspond to the angular coordinates of the outgoing and incoming LFs. Equation (5) shows that the RF R of the object is modulated spatially and angularly with the projector and the two SLMs, and then the modulated signals are integrated onto the image sensor as the captured images Gm ·. In this case, the system matrix H ∈ RN s ×N t ×N m ×N s ×N t ×N u ×N v ×N s0 ×N t0 ×N u0 ×N v0  in Eq. (3), where M  N s × N t × N m and N  N s × N t × N u × N v × N s0 × N t0 × N u0 × N v0 , is written as 2 6 6 H6 4

A1;1;1;1;1;1;1 I



A1;N u ;N v ;N s0 ;N t0 ;N u0 ;N v0 I

.. .

..

.. .

.

AN m ;1;1;1;1;1;1 I    AN m ;N u ;N v ;N s0 ;N t0 ;N u0 ;N v0 I

3 7 7 7; (6) 5

where I ∈ RN s ×N t ×N s ×N t  is an identity matrix, where N s and N t correspond to the pixel count of the image sensor; N m is the number of measurements (exposures); and N s , N t , N u , N v , N s0 , N t0 , N u0 , and N v0 are the numbers of elements along the s, t, u, v, s0 , t0 , u0 , and v0 dimensions. The coefficient A is calculated as

(5) Am;u;v;s0 ;t0 ;u0 ;v0  P SLM2 u; v × P SLM1 s0 ; t0  m m 0 0 × P proj m u ; v :

(7)

As indicated in Eqs. (6) and (7), a pseudo-random system matrix can be implemented with the randomized modulations performed by the projector and the two SLMs to employ the CS framework. 4. Experimental Verification

Fig. 3. Optical setup for compressive RF acquisition.

We experimentally demonstrated the system shown in Fig. 3. Confocal imaging was constructed by the imaging lenses 1 and 2, the beam splitter (BS), and the image sensor (Cool SNAP cf, manufactured by Photometrics). In this experiment, 4 × 4-pixel neighborhoods of each pixel on the image sensor were combined into single pixels to enhance the sensitivity of the image sensor elements. In this case, the number of pixels of the captured image was 348 × 260. SLM1 (LC2012, manufactured by HOLOEYE, 1024 × 768 pixels) and SLM2 (LC2002, manufactured by HOLOEYE, 832 × 634 pixels) were placed at the focusing plane of Lens 1 and in front of Lens 2, respectively. The projector (EMP-745, manufactured by EPSON, 1024 × 768 pixels) was focused on the diffuser. In each measurement, all of the patterns on the projector, SLM1, and SLM2 were designed as binary 2 × 2-pixel rectangular regions. The patterns were 10 May 2014 / Vol. 53, No. 14 / APPLIED OPTICS

3159

multiplied with the RF data, as shown in Eq. (5). In this setup, the total size of the acquirable RF was 348 × 260 × 2 × 2 × 2 × 2 × 2 × 2 N s × N t  × N u × N v  × N s0 × N t0  × N u0 × N v0 . The RF data were retrieved by solving the optimization problem in Eq. (4) with two-step iterative shrinkage/ thresholding (TwIST) [29]. The eight-dimensional discrete cosine transform (8-D DCT) was used as the regularizer S·. The regularization parameter τ in Eq. (4) was chosen experimentally. The experiments were designed to compare the conventional RF measurement with sequential changes in the u, v, s0 , t0 , u0 , and v0 dimensions and the proposed method. In the first experiment, an origami crane was used as a scattering object, as shown in Fig. 4(a). Examples of patterns on the projector, SLM1, and SLM2 are illustrated in Fig. 4(b). Two of the 2 × 2 regions in each pattern were set to ones (open), and the other two regions were set to zeros (closed) in each measurement. The mask patterns for the projector and SLMs were designed with numerical trials to achieve a high reconstruction fidelity for simulated RF data. In the trials, 100 sets of patterns were generated randomly. The image captured using these patterns is shown in Fig. 4(c). Figure 5 shows the experimental results. The right subfigures show 64 captured or reconstructed RF images, which are the s − t data at each u, v, s0 , t0 , u0 , and v0 coordinate, and the left subfigures show two close-up RF images. In the conventional method, in the patterns on the projector and the two SLMs at each measurement, one 2 × 2 region was set to ones, and the other regions were set to

Fig. 5. RF reconstruction of the scattering object. Reconstructions with (a) conventional method using 64 measurements, (b) proposed method using 24 measurements, and (c) proposed method using eight measurements.

Fig. 4. Single image capture of a scattering object. (a) Object (origami crane), (b) example patterns on the projector, SLM1, and SLM2, and (c) captured image. 3160

APPLIED OPTICS / Vol. 53, No. 14 / 10 May 2014

zeros. The open regions were sequentially scanned. In other words, the conventional method does not multiplex the RF, and it requires 64 measurements. The captured RF images are shown in Fig. 5(a). The RF images obtained with the proposed method were reconstructed from 24 and eight multiplexed measurements N m  with TwIST, as shown in Figs. 5(b) and 5(c), respectively. Some reconstruction errors appeared in Fig. 5(c). From a comparison of Figs. 5(a) and 5(b), the proposed method successfully reduced the number of measurements by about 60% while maintaining the image quality. In the second experiment, a bead was used as a translucent object. The object had surface and internal reflections, as shown in Fig. 6(a). Such an object cannot be expressed with the BRDF model. Its RF data was observed with the conventional and proposed methods, as in the first experiment. The 64 RF images captured by the conventional method

In the second experiment, 32 measurements were needed for successful reconstruction. The number of measurements was larger than in the first experiment because the reconstruction fidelity in CS depends on the sparsity of the object in the regularizer’s domain, which is the 8-D DCT domain in this paper [21–23]. The histograms of the 8-D DCT coefficients of the RFs in Fig. 5(a) (the scattering object: origami crane) and Fig. 6(b) (the translucent object: bead) are shown in Figs. 7(a) and 7(b), respectively. The histograms indicate that the scattering object in the first experiment is sparser in the 8-D DCT domain or more compressive than the translucent object in the second experiment. This means that the second object requires more measurements than the first one. The relationship between the number of measurements and the reconstruction fidelity in a numerical 5

10

x 10

9

Number of coefficients

8 7 6 5 4 3 2 1 0 0

1

2

3

Intensity in the DCT domain

4 −4

x 10

5

10

x 10

9

using 64 measurements are shown in Fig. 6(b). The 64 RF images reconstructed by the proposed method using 32 and eight multiplexed measurements are shown in Figs. 6(c) and 6(d). The RF images obtained with the conventional method and those obtained with the proposed method with 32 measurements were comparable. On the other hand, those obtained with the proposed method with eight measurements had some artifacts. The results suggest that the proposed method successfully reduced the number of measurements for acquiring the RF of a translucent object.

Number of coefficients

8

Fig. 6. RF reconstruction of a translucent object. (a) Translucent object (bead), the reconstructions with (b) conventional method using 64 measurements, (c) proposed method using 32 measurements, and (d) proposed method using eight measurements.

7 6 5 4 3 2 1 0 0

1

2

3

Intensity in the DCT domain

4 −4

x 10

Fig. 7. Histograms of the 8-D DCT coefficients. (a) Scattering object (origami crane) in Figs. 5(a), and 5(b) translucent object (bead) in Fig. 6(b). 10 May 2014 / Vol. 53, No. 14 / APPLIED OPTICS

3161

Reconstruction PSNR (dB)

45

method and to integrate multiple projectors and cameras. Improvement of the reconstruction algorithm is also important; for example, it will be important to find a better basis for compressive RF acquisition.

40

35

References 30

25

20

0

8

16 24 32 40 48 Number of measurements

56

64

Fig. 8. Plot of reconstruction fidelity versus number of measurements.

experiment is plotted in Fig. 8. In this evaluation, the images captured with the conventional method in the first experiment were used as the reference. For the proposed method, the expected captured images were computationally generated by a combination of the reference images according to the assumed measurement numbers. Then, the RF was reconstructed with the generated measurements. The reconstruction fidelity was calculated based on the peak signal-to-noise ratio (PSNR) between reference images and the reconstructed RFs. The plot shows that at least 24 measurements were needed for the reconstruction in this case, because a PSNR of more than 30 dB means good reconstruction fidelity in general. The reconstruction quality from the smaller number of measurements is not sufficient, as shown in Fig. 5. 5. Conclusion

We have experimentally demonstrated RF acquisition based on CS. The demonstrated system uses a confocal optical system consisting of coded illumination and a camera, where a projector and two SLMs are used for the coding masks. These components angularly and spatially modulate the RF, and the modulated signals are multiplexed on the image sensor. The RF was measured multiple times by using the optical system with different randomized intensity modulations. The original RF was reconstructed from the captured images with TwIST using the 8-D DCT regularization. In the experiments, RF acquisition was demonstrated with a scattering object and a translucent object. The RFs of the two objects were reconstructed from less than half the number of measurements required with the conventional method, where the multiplexed approach is not used. Future issues to be addressed include increasing the total size of the acquirable RF data and extending the field of view and the viewing angle of the imaging system. Promising ways to achieve these requirements are to develop an accurate calibration 3162

APPLIED OPTICS / Vol. 53, No. 14 / 10 May 2014

1. R. Szeliski, Computer Vision: Algorithms and Applications, 1st ed. (Springer-Verlag, 2010). 2. H. Shum and S. B. Kang, “Review of image-based rendering techniques,” Proc. SPIE 4067, 2 (2000). 3. M. Levoy and P. Hanrahan, “Light field rendering,” in Proceedings of the ACM Special Interest Group on Computer Graphics and Interactive Techniques (ACM, 1996), pp. 31–42. 4. E. H. Adelson and J. Y. A. Wang, “Single lens stereo with a plenoptic camera,” IEEE Trans. Pattern Anal. Mach. Intell. 14, 99–106 (1992). 5. A. Isaksen, L. McMillan, and S. J. Gortler, “Dynamically reparameterized light fields,” in Proceedings of the ACM Special Interest Group on Computer Graphics and Interactive Techniques (ACM, 2000), pp. 297–306. 6. R. Ng, M. Levoy, M. Brédif, G. Duval, M. Horowitz, and P. Hanrahan, “Light field photography with a hand-held plenoptic camera,” Tech. Rep. CSTR 2005-02 (Stanford University, 2005). 7. V. Masselus, P. Peers, P. Dutré, and Y. D. Willems, “Relighting with 4D incident light fields,” ACM Trans. Graph. 22, 613–620 (2003). 8. P. Debevec, T. Hawkins, C. Tchou, H.-P. Duiker, W. Sarokin, and M. Sagar, “Acquiring the reflectance field of a human face,” in Proceedings of the ACM Special Interest Group on Computer Graphics and Interactive Techniques (ACM, 2000), pp. 145–156. 9. R. Horisaki, Y. Tampa, and J. Tanida, “Compressive reflectance field acquisition using confocal imaging with variable coded apertures,” in Computational Optical Sensing and Imaging (Optical Society of America, 2012), paper CTu3B.4. 10. J. Tanida, T. Kumagai, K. Yamada, S. Miyatake, K. Ishida, T. Morimoto, N. Kondou, D. Miyazaki, and Y. Ichioka, “Thin observation module by bound optics (TOMBO): concept and experimental verification,” Appl. Opt. 40, 1806–1813 (2001). 11. B. Wilburn, N. Joshi, V. Vaish, E.-V. Talvala, E. Antunez, A. Barth, A. Adams, M. Horowitz, and M. Levoy, “High performance imaging using large camera arrays,” ACM Trans. Graph. 24, 765–776 (2005). 12. A. Veeraraghavan, R. Raskar, A. Agrawal, A. Mohan, and J. Tumblin, “Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing,” ACM Trans. Graph. 26, 69 (2007). 13. F. E. Nicodemus, “Directional reflectance and emissivity of an opaque surface,” Appl. Opt. 4, 767–775 (1965). 14. R. Horisaki and J. Tanida, “Reflectance field display,” Opt. Express 21, 11181–11186 (2013). 15. H. Li, S. C. Foo, K. E. Torrance, and S. H. Westin, “Automated three-axis gonioreflectometer for computer graphics applications,” Proc. SPIE 5878, 221–231 (2005). 16. G. Müller, G. H. Bendels, and R. Klein, “Rapid synchronous acquisition of geometry and appearance of cultural heritage artefacts,” in Proceedings of the 6th International Conference on Virtual Reality, Archaeology and Intelligent Cultural Heritage (Eurographics Association, 2005), pp. 13–20. 17. Y. Mukaigawa, K. Sumino, and Y. Yagi, “Rapid BRDF measurement using an ellipsoidal mirror and a projector,” IPSJ Trans. Comput. Vis. Appl. 1, 21–32 (2009). 18. G. Garg, E.-V. Talvala, M. Levoy, and H. P. A. Lensch, “Symmetric photography: exploiting data-sparseness in reflectance fields,” in Rendering Techniques 2006: Eurographics Symposium on Rendering, T. Akenine-Möller and W. Heidrich, eds. (Eurographics Association, 2006), pp. 251–262. 19. M. Levoy, Z. Zhang, and I. Mcdowall, “Recording and controlling the 4D light field in a microscope using microlens arrays,” J. Microsc. 235, 144–162 (2009).

20. S. Tagawa, Y. Mukaigawa, and Y. Yagi, “8-D reflectance field for computational photography,” in Proceedings of the International Conference on Pattern Recognition (IEEE, 2012), pp. 2181–2185. 21. D. L. Donoho, “Compressed sensing,” IEEE Trans. Inf. Theory 52, 1289–1306 (2006). 22. E. J. Candes and M. B. Wakin, “An introduction to compressive sampling,” IEEE Signal Process. Mag. 25(2), 21–30 (2008). 23. R. Baraniuk, “Compressive sensing,” IEEE Signal Process. Mag. 24(4), 118–121 (2007). 24. S. D. Babacan, R. Ansorge, M. Luessi, R. Molina, and A. K. Katsaggelos, “Compressive sensing of light fields,” in IEEE International Conference on Image Processing (IEEE, 2009), pp. 2313–2316. 25. A. Ashok and M. A. Neifeld, “Compressive light field imaging,” Proc. SPIE 7690, 76900Q (2010).

26. R. Horisaki and J. Tanida, “Full-resolution light-field singleshot acquisition with spatial encoding,” in Computational Optical Sensing and Imaging (Optical Society of America, 2011), p. CTuB5. 27. K. Marwah, G. Wetzstein, Y. Bando, and R. Raskar, “Compressive light field photography using overcomplete dictionaries and optimized projections,” in Proceedings of the ACM Special Interest Group on Computer Graphics and Interactive Techniques (ACM, 2013), pp. 1–11. 28. P. Peers, D. K. Mahajan, B. Lamond, A. Ghosh, W. Matusik, R. Ramamoorthi, and P. Debevec, “Compressive light transport sensing,” ACM Trans. Graph. 28, 1–18 (2009). 29. J. M. Bioucas-Dias and M. A. T. Figueiredo, “A new TwIST: two-step iterative shrinkage/thresholding algorithms for image restoration,” IEEE Trans. Image Process. 16, 2992–3004 (2007).

10 May 2014 / Vol. 53, No. 14 / APPLIED OPTICS

3163

Experimental verification of compressive reflectance field acquisition.

We demonstrate compressive sensing (CS) of the eight-dimensional reflectance field (RF), which describes spatial and angular information of light rays...
1MB Sizes 0 Downloads 3 Views