A144
Vol. 55, No. 3 / January 20 2016 / Applied Optics
Research Article
See-through integral imaging display with background occlusion capability YUTA YAMAGUCHI*
AND
YASUHIRO TAKAKI
Institute of Engineering, Tokyo University of Agriculture and Technology, 2-24-16, Nakacho, Koganei, Tokyo 184-8588, Japan *Corresponding author: [email protected] Received 7 September 2015; revised 2 December 2015; accepted 3 December 2015; posted 7 December 2015 (Doc. ID 247458); published 12 January 2016
Background occlusion capability is provided to a flat-panel-type integral imaging display that has a transparent screen and can superimpose three-dimensional (3D) images on real scenes. A symmetric integral imaging system that comprises two integral imaging systems connected by an additional lens array, is proposed. Elementary images are displayed on a flat-panel display on one integral imaging system to generate 3D images, and the occlusion mask patterns are displayed on a flat-panel display on the other integral imaging system to selectively block rays from background scenes. The proposed system was constructed and experimentally verified. © 2016 Optical Society of America OCIS codes: (110.0110) Imaging systems; (120.2040) Displays. http://dx.doi.org/10.1364/AO.55.00A144
1. INTRODUCTION An optical see-through three-dimensional (3D) display that can superimpose digital information on real objects directly and at the same depths is a key device for augmented reality (AR) technology [1,2]. When 3D images are generated on the basis of the integral imaging technique [3,4], the 3D images have horizontal and vertical parallaxes (full parallax) so that they can be stably superimposed on the real world. We have previously proposed a flat-panel-type optical see-through integral imaging display [5]. The background occlusion capability, by which background scenes are occluded by 3D images, is not supported by most optical see-through 3D displays; 3D images themselves are see through, and real objects behind the 3D images are also visible. As illustrated in Fig. 1(a), without the background occlusion capability, the visibility of 3D images decreases, making the perception of 3D structures difficult. Thus, the lack of background occlusion capability reduces the reality of 3D images. As illustrated in Fig. 1(b), the background occlusion capability increases the presence of 3D images. The importance of the background occlusion capability in AR applications has been explained in Ref. [6]. In the current study, we propose a technique to provide background occlusion capability to the flatpanel-type optical see-through integral imaging display that we have previously described [5]. Most optical see-through displays employ projection optics and need a relatively high system volume. Monocular seethrough head-mounted displays (HMDs) such as Google Glass employ a single projector to superimpose two-dimensional 1559-128X/16/03A144-06$15/0$15.00 © 2016 Optical Society of America
(2D) images on real scenes. Binocular see-through HMDs employ two projectors to superimpose 3D images at various depths on real scenes. However, precise superposition of the 3D images on real objects is impossible because the perceived positions of the 3D images depend on the eye positions as well as the interpupillary distances (IPDs). Integral-imaging-based optical see-through displays that provide motion parallax in such a way that the perceived positions are independent of the eye position and IPD have been developed [7,8]. However, these displays also make use of projection optics. We have developed a flat-panel-type see-through display based on integral imaging without the use of projection optics [5]. Video see-through HMDs [9] support background occlusion capability wherein the 3D images are electronically superimposed on the left and right views of the real scenes captured by video cameras. However, the resolution of the background images is limited by the video system, and a delay in the display of the background images is unavoidable.
Fig. 1. Effect of the background occlusion capability on AR applications: (a) without occlusion capability and (b) with occlusion capability.
Research Article Kiyokawa et al. [10] proposed a binocular optical seethrough HMD with background occlusion capability. This device has left and right channels consisting of a series of imaging systems containing an image-showing liquid crystal display (LCD) panel and a masking LCD panel. The system is complicated and requires a considerable volume. A projection-type integral imaging display having background occlusion capability has been proposed [11]. It contains an LCD panel displaying an occluded background image so that it is based on the video see-through technique. Maimone and Fuchs [12] proposed a flat-panel-type seethrough 3D display having a background occlusion capability consisting of a multilayer 3D display and an optical shutter. To provide the background occlusion capability, the 3D images and background mask patterns are sequentially shown; in this case, the time-multiplexing technique is used. Another type of background occlusion capable flat-panel-type see-through display has been proposed [13] with a LCD panel and a point light source array that produces 2D images with a wide field of view. A transparent backlight with a dot array is used to provide the point light source array, and the time-multiplexing technique is used to provide the background occlusion capability. However, without an optical shutter, only an incomplete background occlusion capability can be achieved. Here we propose a flat-panel-type implementation of the optical see-through integral imaging display that provides background occlusion capability without using the time-multiplexing technique. In Section 2, after a brief description of the optical see-through integral imaging display that we have previously proposed, a method to provide the background occlusion capability is presented. In Section 3, the experimental verification is shown. Discussion and conclusions are presented in Section 4 and Section 5, respectively. 2. PROPOSED SYSTEM A. Flat-Panel-Type Optical See-Through 3D Display
Figure 2 illustrates a schematic diagram of the flat-panel-type optical see-through integral imaging display that has been described in detail in our previous study [5]. The imaging system comprises three lens arrays, a light blocking wall (LBW), and a transparent display for displaying elementary images. The screen of display 1 is located on the focal plane of lens array
Vol. 55, No. 3 / January 20 2016 / Applied Optics
A145
Fig. 3. Elementary imaging system of the see-through system shown in Fig. 2.
1. Instead of the refractive lens arrays depicted in Fig. 2, gradient index lens arrays can also be used. Here, the see-through function is described by ray transfer matrix analysis as used in geometric optics. This aspect has not been described in Ref. [5]. An elementary imaging system of the see-through display is illustrated in Fig. 3. The matrix of this elementary imaging system is given as follows: S e T 2f Lf T 2f Lf ∕2 T 2f Lf T 2f ; where
T 2f Lf
2f 1
1 −1∕f
Lf ∕2
1 0
1 −2∕f
(1)
;
(2)
0 ; 1
(3)
0 : 1
(4)
T 2f is a transfer matrix for ray propagation with a distance of 2f , Lf is a lens matrix for lens arrays 1 and 2, having a focal length of f , and Lf ∕2 is a lens matrix for lens array 3, having a focal length of f ∕2. Because S e is a unit matrix, each elementary imaging system produces an upright image with unit magnification. The condition of rays on the object plane is reconstructed on the image plane. Thus, the 2D array of elementary imaging systems provides the see-through function. LBW prevents the generation of multiple background images. The resolution of the background images is not limited by the lens pitch because each elementary imaging system produces its own image. When elementary images are displayed on display 1, 3D images formed by integral imaging are produced by virtue of lens array 1, which is superimposed on the background scenes. However, the background scenes are not occluded by the 3D images. B. Background Occlusion Capability
Fig. 2. Flat-panel-type optical see-through 3D display.
Figure 4 illustrates a schematic diagram of the modified optical see-through integral imaging display proposed in this study, which provides background occlusion capability. Transparent display 2 is newly added, and it has a screen located on the focal plane of lens array 2. The proposed system is a symmetric integral imaging system in which two integral imaging displays are connected by the additional lens array. The calculated ray transfer matrix from the screen of display 2 to that of display 1 is given as follows:
A146
Research Article
Vol. 55, No. 3 / January 20 2016 / Applied Optics
Fig. 4. Symmetric integral imaging system for an optical seethrough 3D display with background occlusion capability.
S c T f Lf ∕2 T f ; where
Tf
1 0
(5) 3. EXPERIMENTS
f : 1
(6)
T f is a transfer matrix for ray propagation with a distance of f . The matrix S c is calculated as follows: −1 0 Sc : (7) −2∕f −1 Because the upper left element is −1 and the upper right element is 0, the screen of display 2 is inversely imaged on the screen of display 1 with unit magnification. To achieve background occlusion capability, display 2 displays occlusion mask patterns that selectively eliminate rays from the background scene. As explained in the previous paragraph, the screen of display 1 is a conjugate of that of display 2 with respect to lens array 3. To generate the occlusion mask pattern, each elementary image is inverted in a 2D manner, and the non-transparent pixels are replaced by black pixels that block rays from the background, as shown in Fig. 5. Rays from the background scenes that are superimposed on the rays that generate the 3D images are eliminated selectively. When a passive display such as an LCD panel is used as display 1, a transparent backlight should be attached to it. Because rays from the background scene are blocked by the occlusion mask patterns in display 2, no light illuminates the pixels of display 1, which generate the 3D images. A transparent
Fig. 5. Generation of occlusion mask patterns.
frontlight can also be used; it has already been commercialized for e-book readers (e.g., Amazon Kindle). A typical structure for both the transparent frontlights and backlights is a transparent plate having a microstructure on one surface that prevents total internal reflection inside the transparent plate. When an active display, such as an organic light-emitting diode, is used as display 1, no illumination component is required. When LCD panels are used for both displays 1 and 2, the direction of the input polarizer of display 1 has to be identical to that of the output polarizer of display 2. In this case, more than 50% light is lost because of the polarizers. To generate color 3D images, display 1 should generate color images on the basis of the time-sequential technique to avoid loss of light. The frontlight or backlight illuminates the display sequentially with R, G, and B lights. When a commonly used color display containing color filters is used, more than one-third of the light is lost because of the color filters.
A. Experimental System
The proposed technique for providing background occlusion capability was experimentally verified. The experimental system is illustrated in Fig. 6. Four identical plano–convex lens arrays were used, two of which were combined to obtain the central lens array (lens array 3) shown in Fig. 4, which has half of the focal length of the outer two lens arrays (lens arrays 1 and 2). Printed transparent films were used as displays 1 and 2 instead of transparent displays because highresolution transparent flat-panel displays have not been commercialized. A frontlight was used to illuminate the transparent film containing elementary images. Commercial lens arrays (Fresnel technologies Inc., #630) were used; the lens pitch p was 1.0 mm, the focal length f was 3.3 mm, and the size was 152 mm × 152 mm. Antireflection coating was applied to the lens arrays to increase transmittance. The gap between the central combined lens arrays and the outer lens arrays was equal to the focal length of each lens array, because one of the focal planes of each lens array was located on its surface. Elementary images and mask patterns were printed using a thermal transfer printer with a resolution of 600 dpi. Each elementary image as well as each mask pattern consisted of 24 × 24 dots. The frontlight was a polymethyl methacrylate plate with a thickness of 0.4 mm with dense small transparent dots on one surface. A one-dimensional array of white light emitting
Fig. 6. Experimental system.
Research Article
Vol. 55, No. 3 / January 20 2016 / Applied Optics
A147
Fig. 7. Components constituting the experimental system: (a) frontlight and (b) LBW.
diodes was attached to one edge of the transparent plate. The fabricated frontlight is shown in Fig. 7(a). LBW was fabricated using a stereolithography 3D printer. The wall height was 3.0 mm, and the wall pitch was 1.0 mm. The wall width should be as thin as possible to maximize light transmittance. The wall width cannot be reduced below 0.2 mm because this is the minimum feature size of the 3D printer. Thus, the aperture ratio of LBW was 64%. Figure 7(b) shows the magnified structure of the constructed LBW. A holder was designed and fabricated to ensure the precise alignment of the four lens arrays and the other components. Small aluminum plates with 5 × 3 circular holes that had a pitch of 1.0 mm were used to align the two central lens arrays; these were placed at the four corners of the combined arrays. The holder had a fine alignment mechanism for the outer two lens arrays. Because the lens arrays were supported by the holder using their four sides, the effective screen size became 146 mm × 146 mm. Figure 8 shows the constructed experimental system.
Fig. 9. See-through image obtained by the experimental system and captured from the (a) upper, (b) left, (c) center, (d) right, and (e) lower positions, and (f ) the magnified image.
had both horizontal and vertical parallaxes. In the magnified image shown in Fig. 9(f ), an image was observed in each lens; thus, the resolution of the background image was not limited by the number of lenses in the lens arrays. Figure 10 shows the background image obtained with the elementary images that are used to generate a 3D image. The
B. Experimental Results
Figure 9 shows a background image observed through the experimental system without the elementary images and the occlusion mask patterns. A toy car and a paper with printed characters were placed at distances of 50 and 150 mm, respectively, behind the experimental system. These were captured from five different viewing positions. The see-through image
Fig. 8. Constructed experimental system (LBW removed).
Fig. 10. Superposition of the 3D image on the background scene without occlusion mask patterns and captured from the (a) upper, (b) left, (c) center, (d) right, and (e) lower positions (see Visualization 1).
A148
Research Article
Vol. 55, No. 3 / January 20 2016 / Applied Optics
Fig. 13. Examples of see-through images when real objects are apple [(a) and (b)] and banana [(c) and (d)]; (a) and (c) are see-through images without occlusion mask patterns, and (b) and (d) are those with occlusion mask patterns.
Fig. 11. See-through images with black regions generated by occlusion mask patterns and captured from the (a) upper, (b) left, (c) center, (d) right, and (e) lower positions.
word “3D” was produced at a distance of 50 mm in front of the experimental system, and the word “IMAGE” was produced at a distance of 50 mm behind it. Although the 3D image was generated and was superposed on the background scene, the background scene remained visible.
Figure 11 shows the background image using the occlusion mask patterns and without the elementary images. Black regions were generated in the background image at the location where the 3D image was displayed. Finally, the verification of the background occlusion capability is shown in Fig. 12. Both the elementary images and the occlusion mask patterns were displayed. A 3D image was produced that occluded the portion of the background image appearing behind it. Thus, the background occlusion capability was successfully demonstrated. Figure 13 shows other two examples of the superposition of 3D images on real objects. Figures 13(a) and 13(c) show see-through images without occlusion mask patterns, and Figs. 13(b) and 13(d) show those with occlusion mask patterns. The background images were successfully occluded by the 3D images in both cases. 4. DISCUSSION
Fig. 12. Superposition of 3D images with background occlusion capability and captured from the (a) upper, (b) left, (c) center, (d) right, and (e) lower positions (see Visualization 2).
As shown in Fig. 12, the 3D images could not be precisely overlapped on the black regions. In the proposed system, the occlusion mask patterns should be imaged on the elementary images with unit magnification. However, the actual magnification was less than unity, because the two gaps between the central combined lens arrays and the outer lens arrays were maintained by several thin spacers whose minimum thickness was 0.1 mm. The magnification changes by approximately 6% when the gap spacing changes by 0.1 mm. Because the occlusion mask patterns were smaller than the elementary images, the black regions were generated behind the 3D images generated by the elementary images. Thus, they could not be overlapped precisely. The measured viewing zone angle for the background images was 4.3°, whereas the calculated angle was 8.7°. This is because the actual aperture pitch of the LBW was smaller than the lens pitch of the lens arrays [5]. The light intensity of the 3D images was not high. The density of the dots on the surface of the frontlight should be increased for higher light intensity. We observed low-contrast,
Research Article coarse, regular patterns on the screen of the experimental system. This was due to the dots on the surface of the frontlight, which were magnified by the lens arrays. The diameter of the dots was 33 μm. The diameter of the dots should be reduced to eliminate the low-contrast regular patterns. The measured transmittance of the experimental system without the two transparent films was 47.9%. The low aperture ratio of the LBW mainly caused the low transmittance. Light reflection and scattering on the frontlight also decreased the transmittance. From the experimental results, it is clear that the proposed technique enabled the 3D images to completely occlude the background scenes using two separate displays: one for generating 3D images and the other for masking background scenes. The symmetric integral imaging system allows employing two displays located in conjugate positions. The previously proposed flat-panel-type see-through displays [12,13] used the time-multiplexing technique to realize the background occlusion capability, i.e., a single display was used for image generation and background masking. When the time-multiplexing technique is used, an optical shutter is required to provide the complete background occlusion capability [12]. Without the optical shutter, the background scene is still observed during the image generation process. 5. CONCLUSIONS We presented an optical see-through integral imaging display having background occlusion capability. A symmetric integral imaging system that consists of three lens arrays and two transparent displays was proposed. The proposed technique was experimentally verified by constructing a system using two transparent films instead of two transparent displays. We confirmed that the 3D images were generated by one transparent film, and the background scenes were selectively masked by the other transparent film. The background scenes were successfully occluded by the 3D images, thereby verifying the background occlusion capability.
Vol. 55, No. 3 / January 20 2016 / Applied Optics
Funding.
A149
SCOPE.
REFERENCES 1. R. Azuma, Y. Baillot, R. Behringer, S. Feiber, S. Julier, and B. MacIntyre, “Recent advances in augmented reality,” IEEE Comput. Graph. Appl. 21, 34–47 (2001). 2. D. W. F. van Krevelen and R. Poelman, “A survey of augmented reality technologies, applications and limitations,” Int. J. Virtual Reality 9, 1–19 (2010). 3. B. Javidi, “Three-dimensional holographic image sensing and integral imaging display,” J. Disp. Technol. 1, 341–346 (2005). 4. J. Hong, Y. Kim, H.-J. Choi, J. Hahn, J.-H. Park, H. Kim, S.-W. Min, N. Chen, and B. Lee, “Three-dimensional display technologies of recent interest: principles, status, and issues,” Appl. Opt. 50, H87–H115 (2011). 5. Y. Takaki and Y. Yamaguchi, “Flat-panel see-through threedimensional display based on integral imaging,” Opt. Lett. 40, 1873–1876 (2015). 6. R. Azuma, “A survey of augmented reality,” Presence 6, 355–385 (1997). 7. K. Hong, J. Yeom, C. Jang, J. Hong, and B. Lee, “Full-color lens-array holographic optical element for three-dimensional optical see-through augmented reality,” Opt. Lett. 39, 127–130 (2014). 8. H. Hua and B. Javidi, “A 3D integral imaging optical see-through headmounted display,” Opt. Express 22, 13484–13491 (2014). 9. J. P. Rolland and H. Fuchs, “Optical versus video see-through headmounted displays in medical visualization,” Presence 9, 287–309 (2000). 10. K. Kiyokawa, M. Billinghurst, B. Campbell, and E. Woods, “An occlusion-capable optical see-through head mounted display for supporting co-located collaboration,” in Proceedings of IEEE and ACM International Symposium on Mixed and Augmented Reality (IEEE, 2003), pp. 133–141. 11. H. Kakeya, S. Ishizuka, and Y. Sato, “Realization of an aerial 3D image that occludes the background scenery,” Opt. Express 22, 24491–24496 (2014). 12. A. Maimone and H. Fuchs, “Computational augmented reality eyeglasses,” in Proceedings of IEEE International Symposium on Mixed and Augmented Reality (IEEE, 2013), pp. 29–38. 13. A. Maimone, D. Lanman, K. Rathinavel, K. Keller, D. Luebke, and H. Fuchs, “Pinlight displays: Wide field of view augmented eyeglasses using defocused point light source,” ACM Trans. Graph. 33, 89 (2014).
Vol. 55, No. 3 / January 20 2016 / Applied Optics
Research Article
See-through integral imaging display with background occlusion capability YUTA YAMAGUCHI*
AND
YASUHIRO TAKAKI
Institute of Engineering, Tokyo University of Agriculture and Technology, 2-24-16, Nakacho, Koganei, Tokyo 184-8588, Japan *Corresponding author: [email protected] Received 7 September 2015; revised 2 December 2015; accepted 3 December 2015; posted 7 December 2015 (Doc. ID 247458); published 12 January 2016
Background occlusion capability is provided to a flat-panel-type integral imaging display that has a transparent screen and can superimpose three-dimensional (3D) images on real scenes. A symmetric integral imaging system that comprises two integral imaging systems connected by an additional lens array, is proposed. Elementary images are displayed on a flat-panel display on one integral imaging system to generate 3D images, and the occlusion mask patterns are displayed on a flat-panel display on the other integral imaging system to selectively block rays from background scenes. The proposed system was constructed and experimentally verified. © 2016 Optical Society of America OCIS codes: (110.0110) Imaging systems; (120.2040) Displays. http://dx.doi.org/10.1364/AO.55.00A144
1. INTRODUCTION An optical see-through three-dimensional (3D) display that can superimpose digital information on real objects directly and at the same depths is a key device for augmented reality (AR) technology [1,2]. When 3D images are generated on the basis of the integral imaging technique [3,4], the 3D images have horizontal and vertical parallaxes (full parallax) so that they can be stably superimposed on the real world. We have previously proposed a flat-panel-type optical see-through integral imaging display [5]. The background occlusion capability, by which background scenes are occluded by 3D images, is not supported by most optical see-through 3D displays; 3D images themselves are see through, and real objects behind the 3D images are also visible. As illustrated in Fig. 1(a), without the background occlusion capability, the visibility of 3D images decreases, making the perception of 3D structures difficult. Thus, the lack of background occlusion capability reduces the reality of 3D images. As illustrated in Fig. 1(b), the background occlusion capability increases the presence of 3D images. The importance of the background occlusion capability in AR applications has been explained in Ref. [6]. In the current study, we propose a technique to provide background occlusion capability to the flatpanel-type optical see-through integral imaging display that we have previously described [5]. Most optical see-through displays employ projection optics and need a relatively high system volume. Monocular seethrough head-mounted displays (HMDs) such as Google Glass employ a single projector to superimpose two-dimensional 1559-128X/16/03A144-06$15/0$15.00 © 2016 Optical Society of America
(2D) images on real scenes. Binocular see-through HMDs employ two projectors to superimpose 3D images at various depths on real scenes. However, precise superposition of the 3D images on real objects is impossible because the perceived positions of the 3D images depend on the eye positions as well as the interpupillary distances (IPDs). Integral-imaging-based optical see-through displays that provide motion parallax in such a way that the perceived positions are independent of the eye position and IPD have been developed [7,8]. However, these displays also make use of projection optics. We have developed a flat-panel-type see-through display based on integral imaging without the use of projection optics [5]. Video see-through HMDs [9] support background occlusion capability wherein the 3D images are electronically superimposed on the left and right views of the real scenes captured by video cameras. However, the resolution of the background images is limited by the video system, and a delay in the display of the background images is unavoidable.
Fig. 1. Effect of the background occlusion capability on AR applications: (a) without occlusion capability and (b) with occlusion capability.
Research Article Kiyokawa et al. [10] proposed a binocular optical seethrough HMD with background occlusion capability. This device has left and right channels consisting of a series of imaging systems containing an image-showing liquid crystal display (LCD) panel and a masking LCD panel. The system is complicated and requires a considerable volume. A projection-type integral imaging display having background occlusion capability has been proposed [11]. It contains an LCD panel displaying an occluded background image so that it is based on the video see-through technique. Maimone and Fuchs [12] proposed a flat-panel-type seethrough 3D display having a background occlusion capability consisting of a multilayer 3D display and an optical shutter. To provide the background occlusion capability, the 3D images and background mask patterns are sequentially shown; in this case, the time-multiplexing technique is used. Another type of background occlusion capable flat-panel-type see-through display has been proposed [13] with a LCD panel and a point light source array that produces 2D images with a wide field of view. A transparent backlight with a dot array is used to provide the point light source array, and the time-multiplexing technique is used to provide the background occlusion capability. However, without an optical shutter, only an incomplete background occlusion capability can be achieved. Here we propose a flat-panel-type implementation of the optical see-through integral imaging display that provides background occlusion capability without using the time-multiplexing technique. In Section 2, after a brief description of the optical see-through integral imaging display that we have previously proposed, a method to provide the background occlusion capability is presented. In Section 3, the experimental verification is shown. Discussion and conclusions are presented in Section 4 and Section 5, respectively. 2. PROPOSED SYSTEM A. Flat-Panel-Type Optical See-Through 3D Display
Figure 2 illustrates a schematic diagram of the flat-panel-type optical see-through integral imaging display that has been described in detail in our previous study [5]. The imaging system comprises three lens arrays, a light blocking wall (LBW), and a transparent display for displaying elementary images. The screen of display 1 is located on the focal plane of lens array
Vol. 55, No. 3 / January 20 2016 / Applied Optics
A145
Fig. 3. Elementary imaging system of the see-through system shown in Fig. 2.
1. Instead of the refractive lens arrays depicted in Fig. 2, gradient index lens arrays can also be used. Here, the see-through function is described by ray transfer matrix analysis as used in geometric optics. This aspect has not been described in Ref. [5]. An elementary imaging system of the see-through display is illustrated in Fig. 3. The matrix of this elementary imaging system is given as follows: S e T 2f Lf T 2f Lf ∕2 T 2f Lf T 2f ; where
T 2f Lf
2f 1
1 −1∕f
Lf ∕2
1 0
1 −2∕f
(1)
;
(2)
0 ; 1
(3)
0 : 1
(4)
T 2f is a transfer matrix for ray propagation with a distance of 2f , Lf is a lens matrix for lens arrays 1 and 2, having a focal length of f , and Lf ∕2 is a lens matrix for lens array 3, having a focal length of f ∕2. Because S e is a unit matrix, each elementary imaging system produces an upright image with unit magnification. The condition of rays on the object plane is reconstructed on the image plane. Thus, the 2D array of elementary imaging systems provides the see-through function. LBW prevents the generation of multiple background images. The resolution of the background images is not limited by the lens pitch because each elementary imaging system produces its own image. When elementary images are displayed on display 1, 3D images formed by integral imaging are produced by virtue of lens array 1, which is superimposed on the background scenes. However, the background scenes are not occluded by the 3D images. B. Background Occlusion Capability
Fig. 2. Flat-panel-type optical see-through 3D display.
Figure 4 illustrates a schematic diagram of the modified optical see-through integral imaging display proposed in this study, which provides background occlusion capability. Transparent display 2 is newly added, and it has a screen located on the focal plane of lens array 2. The proposed system is a symmetric integral imaging system in which two integral imaging displays are connected by the additional lens array. The calculated ray transfer matrix from the screen of display 2 to that of display 1 is given as follows:
A146
Research Article
Vol. 55, No. 3 / January 20 2016 / Applied Optics
Fig. 4. Symmetric integral imaging system for an optical seethrough 3D display with background occlusion capability.
S c T f Lf ∕2 T f ; where
Tf
1 0
(5) 3. EXPERIMENTS
f : 1
(6)
T f is a transfer matrix for ray propagation with a distance of f . The matrix S c is calculated as follows: −1 0 Sc : (7) −2∕f −1 Because the upper left element is −1 and the upper right element is 0, the screen of display 2 is inversely imaged on the screen of display 1 with unit magnification. To achieve background occlusion capability, display 2 displays occlusion mask patterns that selectively eliminate rays from the background scene. As explained in the previous paragraph, the screen of display 1 is a conjugate of that of display 2 with respect to lens array 3. To generate the occlusion mask pattern, each elementary image is inverted in a 2D manner, and the non-transparent pixels are replaced by black pixels that block rays from the background, as shown in Fig. 5. Rays from the background scenes that are superimposed on the rays that generate the 3D images are eliminated selectively. When a passive display such as an LCD panel is used as display 1, a transparent backlight should be attached to it. Because rays from the background scene are blocked by the occlusion mask patterns in display 2, no light illuminates the pixels of display 1, which generate the 3D images. A transparent
Fig. 5. Generation of occlusion mask patterns.
frontlight can also be used; it has already been commercialized for e-book readers (e.g., Amazon Kindle). A typical structure for both the transparent frontlights and backlights is a transparent plate having a microstructure on one surface that prevents total internal reflection inside the transparent plate. When an active display, such as an organic light-emitting diode, is used as display 1, no illumination component is required. When LCD panels are used for both displays 1 and 2, the direction of the input polarizer of display 1 has to be identical to that of the output polarizer of display 2. In this case, more than 50% light is lost because of the polarizers. To generate color 3D images, display 1 should generate color images on the basis of the time-sequential technique to avoid loss of light. The frontlight or backlight illuminates the display sequentially with R, G, and B lights. When a commonly used color display containing color filters is used, more than one-third of the light is lost because of the color filters.
A. Experimental System
The proposed technique for providing background occlusion capability was experimentally verified. The experimental system is illustrated in Fig. 6. Four identical plano–convex lens arrays were used, two of which were combined to obtain the central lens array (lens array 3) shown in Fig. 4, which has half of the focal length of the outer two lens arrays (lens arrays 1 and 2). Printed transparent films were used as displays 1 and 2 instead of transparent displays because highresolution transparent flat-panel displays have not been commercialized. A frontlight was used to illuminate the transparent film containing elementary images. Commercial lens arrays (Fresnel technologies Inc., #630) were used; the lens pitch p was 1.0 mm, the focal length f was 3.3 mm, and the size was 152 mm × 152 mm. Antireflection coating was applied to the lens arrays to increase transmittance. The gap between the central combined lens arrays and the outer lens arrays was equal to the focal length of each lens array, because one of the focal planes of each lens array was located on its surface. Elementary images and mask patterns were printed using a thermal transfer printer with a resolution of 600 dpi. Each elementary image as well as each mask pattern consisted of 24 × 24 dots. The frontlight was a polymethyl methacrylate plate with a thickness of 0.4 mm with dense small transparent dots on one surface. A one-dimensional array of white light emitting
Fig. 6. Experimental system.
Research Article
Vol. 55, No. 3 / January 20 2016 / Applied Optics
A147
Fig. 7. Components constituting the experimental system: (a) frontlight and (b) LBW.
diodes was attached to one edge of the transparent plate. The fabricated frontlight is shown in Fig. 7(a). LBW was fabricated using a stereolithography 3D printer. The wall height was 3.0 mm, and the wall pitch was 1.0 mm. The wall width should be as thin as possible to maximize light transmittance. The wall width cannot be reduced below 0.2 mm because this is the minimum feature size of the 3D printer. Thus, the aperture ratio of LBW was 64%. Figure 7(b) shows the magnified structure of the constructed LBW. A holder was designed and fabricated to ensure the precise alignment of the four lens arrays and the other components. Small aluminum plates with 5 × 3 circular holes that had a pitch of 1.0 mm were used to align the two central lens arrays; these were placed at the four corners of the combined arrays. The holder had a fine alignment mechanism for the outer two lens arrays. Because the lens arrays were supported by the holder using their four sides, the effective screen size became 146 mm × 146 mm. Figure 8 shows the constructed experimental system.
Fig. 9. See-through image obtained by the experimental system and captured from the (a) upper, (b) left, (c) center, (d) right, and (e) lower positions, and (f ) the magnified image.
had both horizontal and vertical parallaxes. In the magnified image shown in Fig. 9(f ), an image was observed in each lens; thus, the resolution of the background image was not limited by the number of lenses in the lens arrays. Figure 10 shows the background image obtained with the elementary images that are used to generate a 3D image. The
B. Experimental Results
Figure 9 shows a background image observed through the experimental system without the elementary images and the occlusion mask patterns. A toy car and a paper with printed characters were placed at distances of 50 and 150 mm, respectively, behind the experimental system. These were captured from five different viewing positions. The see-through image
Fig. 8. Constructed experimental system (LBW removed).
Fig. 10. Superposition of the 3D image on the background scene without occlusion mask patterns and captured from the (a) upper, (b) left, (c) center, (d) right, and (e) lower positions (see Visualization 1).
A148
Research Article
Vol. 55, No. 3 / January 20 2016 / Applied Optics
Fig. 13. Examples of see-through images when real objects are apple [(a) and (b)] and banana [(c) and (d)]; (a) and (c) are see-through images without occlusion mask patterns, and (b) and (d) are those with occlusion mask patterns.
Fig. 11. See-through images with black regions generated by occlusion mask patterns and captured from the (a) upper, (b) left, (c) center, (d) right, and (e) lower positions.
word “3D” was produced at a distance of 50 mm in front of the experimental system, and the word “IMAGE” was produced at a distance of 50 mm behind it. Although the 3D image was generated and was superposed on the background scene, the background scene remained visible.
Figure 11 shows the background image using the occlusion mask patterns and without the elementary images. Black regions were generated in the background image at the location where the 3D image was displayed. Finally, the verification of the background occlusion capability is shown in Fig. 12. Both the elementary images and the occlusion mask patterns were displayed. A 3D image was produced that occluded the portion of the background image appearing behind it. Thus, the background occlusion capability was successfully demonstrated. Figure 13 shows other two examples of the superposition of 3D images on real objects. Figures 13(a) and 13(c) show see-through images without occlusion mask patterns, and Figs. 13(b) and 13(d) show those with occlusion mask patterns. The background images were successfully occluded by the 3D images in both cases. 4. DISCUSSION
Fig. 12. Superposition of 3D images with background occlusion capability and captured from the (a) upper, (b) left, (c) center, (d) right, and (e) lower positions (see Visualization 2).
As shown in Fig. 12, the 3D images could not be precisely overlapped on the black regions. In the proposed system, the occlusion mask patterns should be imaged on the elementary images with unit magnification. However, the actual magnification was less than unity, because the two gaps between the central combined lens arrays and the outer lens arrays were maintained by several thin spacers whose minimum thickness was 0.1 mm. The magnification changes by approximately 6% when the gap spacing changes by 0.1 mm. Because the occlusion mask patterns were smaller than the elementary images, the black regions were generated behind the 3D images generated by the elementary images. Thus, they could not be overlapped precisely. The measured viewing zone angle for the background images was 4.3°, whereas the calculated angle was 8.7°. This is because the actual aperture pitch of the LBW was smaller than the lens pitch of the lens arrays [5]. The light intensity of the 3D images was not high. The density of the dots on the surface of the frontlight should be increased for higher light intensity. We observed low-contrast,
Research Article coarse, regular patterns on the screen of the experimental system. This was due to the dots on the surface of the frontlight, which were magnified by the lens arrays. The diameter of the dots was 33 μm. The diameter of the dots should be reduced to eliminate the low-contrast regular patterns. The measured transmittance of the experimental system without the two transparent films was 47.9%. The low aperture ratio of the LBW mainly caused the low transmittance. Light reflection and scattering on the frontlight also decreased the transmittance. From the experimental results, it is clear that the proposed technique enabled the 3D images to completely occlude the background scenes using two separate displays: one for generating 3D images and the other for masking background scenes. The symmetric integral imaging system allows employing two displays located in conjugate positions. The previously proposed flat-panel-type see-through displays [12,13] used the time-multiplexing technique to realize the background occlusion capability, i.e., a single display was used for image generation and background masking. When the time-multiplexing technique is used, an optical shutter is required to provide the complete background occlusion capability [12]. Without the optical shutter, the background scene is still observed during the image generation process. 5. CONCLUSIONS We presented an optical see-through integral imaging display having background occlusion capability. A symmetric integral imaging system that consists of three lens arrays and two transparent displays was proposed. The proposed technique was experimentally verified by constructing a system using two transparent films instead of two transparent displays. We confirmed that the 3D images were generated by one transparent film, and the background scenes were selectively masked by the other transparent film. The background scenes were successfully occluded by the 3D images, thereby verifying the background occlusion capability.
Vol. 55, No. 3 / January 20 2016 / Applied Optics
Funding.
A149
SCOPE.
REFERENCES 1. R. Azuma, Y. Baillot, R. Behringer, S. Feiber, S. Julier, and B. MacIntyre, “Recent advances in augmented reality,” IEEE Comput. Graph. Appl. 21, 34–47 (2001). 2. D. W. F. van Krevelen and R. Poelman, “A survey of augmented reality technologies, applications and limitations,” Int. J. Virtual Reality 9, 1–19 (2010). 3. B. Javidi, “Three-dimensional holographic image sensing and integral imaging display,” J. Disp. Technol. 1, 341–346 (2005). 4. J. Hong, Y. Kim, H.-J. Choi, J. Hahn, J.-H. Park, H. Kim, S.-W. Min, N. Chen, and B. Lee, “Three-dimensional display technologies of recent interest: principles, status, and issues,” Appl. Opt. 50, H87–H115 (2011). 5. Y. Takaki and Y. Yamaguchi, “Flat-panel see-through threedimensional display based on integral imaging,” Opt. Lett. 40, 1873–1876 (2015). 6. R. Azuma, “A survey of augmented reality,” Presence 6, 355–385 (1997). 7. K. Hong, J. Yeom, C. Jang, J. Hong, and B. Lee, “Full-color lens-array holographic optical element for three-dimensional optical see-through augmented reality,” Opt. Lett. 39, 127–130 (2014). 8. H. Hua and B. Javidi, “A 3D integral imaging optical see-through headmounted display,” Opt. Express 22, 13484–13491 (2014). 9. J. P. Rolland and H. Fuchs, “Optical versus video see-through headmounted displays in medical visualization,” Presence 9, 287–309 (2000). 10. K. Kiyokawa, M. Billinghurst, B. Campbell, and E. Woods, “An occlusion-capable optical see-through head mounted display for supporting co-located collaboration,” in Proceedings of IEEE and ACM International Symposium on Mixed and Augmented Reality (IEEE, 2003), pp. 133–141. 11. H. Kakeya, S. Ishizuka, and Y. Sato, “Realization of an aerial 3D image that occludes the background scenery,” Opt. Express 22, 24491–24496 (2014). 12. A. Maimone and H. Fuchs, “Computational augmented reality eyeglasses,” in Proceedings of IEEE International Symposium on Mixed and Augmented Reality (IEEE, 2013), pp. 29–38. 13. A. Maimone, D. Lanman, K. Rathinavel, K. Keller, D. Luebke, and H. Fuchs, “Pinlight displays: Wide field of view augmented eyeglasses using defocused point light source,” ACM Trans. Graph. 33, 89 (2014).
