www.nature.com/scientificreports
OPEN
Image deformation as a cue to material category judgment Takahiro Kawabe1,* & Rok Kogovšek1,2,*
received: 23 August 2016 accepted: 06 February 2017 Published: 09 March 2017
Human observers easily recognize complex natural phenomena, such as flowing water, which often generate highly chaotic dynamic arrays of light on the retina. It has not been clarified how the visual system discerns the source of a fluid flow. Here we show that the magnitude of image deformation caused by light refraction is a critical factor for the visual system to determine the perceptual category of fluid flows. Employing a physics engine, we created computer-rendered scenes of water and hot air flows. For each flow, we manipulated the rendering parameters (distortion factors and the index of refraction) that strongly influence the magnitude of image deformation. The observers rated how strongly they felt impressions of water and hot air in the video clips of the flows. The ratings showed that the water and hot air impressions were positively and negatively related to the magnitude of image deformation. Based on the results, we discuss how the visual system heuristically utilizes image deformation to discern non-rigid materials such as water and hot air flows. Vision scientists have recently attempted to unveil the perceptual and cognitive mechanisms underlying the recognition of materials1. Most of them have focused on the perception of material surface properties. In particular, previous studies have shown that human observers can decode surface reflectance properties such as specularity2–6 and subsurface scattering7,8 from complex retinal images. In addition to surface reflectance, surface refraction is an important phenomenon determining perceptual appearance of materials. When light penetrates a transparent material with an index of refraction (IOR) of more (or less) than 1, the path of the light is bent at the material’s surface, where the direction of the bending depends on the IOR. The bent light path often causes image deformation of the scene behind a curved material, wherein the larger (smaller) IOR of the material creates a larger (smaller) magnitude of image deformation on the retina. It has been reported that image deformation due to refraction is a vital cue for the visual system to judge several aspects of material properties. For example, the magnitude of image deformation serves as a cue to judge the thickness of rigid materials such as glass [ref. 9, but also see ref. 10]. Moreover, the human visual system can utilize the dynamic pattern of image deformation due to refraction as a cue to a material’s elasticity11. Importantly, dynamic image deformation gives observers the impression of a transparent water-like layer. A previous study12 reported that human observers can recognize a transparent water flow from dynamic image deformation alone; no other cues, such as specular reflection or luminance reduction due to absorption, were necessary for the transparent water recognition. It also showed that the critical component in dynamic image deformation is the spatiotemporal frequency of deformation and suggested that the specific band of the spatiotemporal deformation frequency is critical for perceiving the deformation as arising from the intervention of a transparent water layer. Besides a water flow, a hot air flow also optically deforms the image behind it. A temperature change physically causes the spatiotemporal variation of the density of air, which eventually alters the IOR. Because the density change spatiotemporally occurs on the basis of the Rayleigh–Taylor instability13, the image deformation caused by refraction can be dynamic. In our everyday experience, we easily notice the existence of hot air just by seeing the dynamic image deformation. On the other hand, how the visual system recognizes hot air and how the system differentiates between hot air and water flows on the basis of dynamic image deformation are still open questions. Clarifying how the visual system categorizes transparent non-rigid material will be an important step to understanding how the mind generates rich and detailed visual representations of the very complex real world from a handful of visual inputs.
1
NTT Communication Science Laboratories, Nippon Telegraph and Telephone Corporation, Japan. 2Faculty of Computer and Information Science, University of Ljubljana, Slovenia. *These authors contributed equally to this work. Correspondence and requests for materials should be addressed to T.K. (email: [email protected]. co.jp) Scientific Reports | 7:44274 | DOI: 10.1038/srep44274
1
www.nature.com/scientificreports/ (a)
(d)
(g)
(e)
Rightward
Horizontal vectors
(b)
Real hot air clip
Mo on v ectors (pixels)
Simulated hot air clip
Le ward
(f)
Ver cal vectors
(c)
Mo on v ectors (pixels)
Upward
Downward
(i)
N = 12 (Error bars: ±1 SEM) Dynamic Sta c 5
5
4.5
4.5
4
4
Ra ng scores
Ra ng scores
(h)
3.5 3 2.5
3.5 3 2.5
2
2
1.5
1.5
1
Simulated Real Type of video clips
N = 11 (Error bars: ±1 SEM) Hot air simula on Water simula on
1
Water impression Hot air impression Types of impression
Figure 1. Simulation geometry, rendered scene, motion vector analysis, and experimental results. (a) The geometry of water simulation seen from an oblique angle. (b) A rendered scene of (a), which was viewed from an oblique angle. (c) A rendered scene of water flow. (d) The geometry of hot air simulation seen from an oblique angle. (e) A rendered scene of (d), which was viewed from an oblique angle. (f) A rendered scene of hot air. (g) Motion vector comparison between clips of simulated and real hot air for horizontal (upper row) and vertical (lower row) vectors, which were calculated between two video frames. (h) Experiment 1 results. (i) Experiment 2 results.
In this work, we examined what the critical image information is for human observers to discern the source of a transparent material flow. We used computer graphics with a physics engine to create video clips of simulated hot air flows and water flows. We noticed that the former generated a smaller magnitude of image deformation than the latter. Thus, we parametrically manipulated the magnitude of image deformation by changing rendering parameters. We then found that the hot air and water impressions were negatively and positively related to the magnitude of image deformation under the condition where other aspects of the flows, such as flow direction and turbulence, were intact. On the basis of the results, we propose a visual mechanism underlying the recognition of fluid materials from image deformation.
Results and Discussion
In Experiment 1, we first tried to determine whether our simulation of hot air (Fig. 1a–f) could produce a reasonable level of the desired impression. The observers were asked to rate their hot air impression after they had watched the video clips of simulated and real hot air flows. As shown in Fig. 1g, the range of motion vectors was similar between the simulated hot air and real hot air. Two styles of video clip presentation were tested. One was a dynamic style in which dynamic sequences of video frames were presented. The other was a static style in which a single static video frame was randomly extracted from a sequence of video frames and presented to the observer. Scientific Reports | 7:44274 | DOI: 10.1038/srep44274
2
Standard devia on of mo on vector (pixels)
www.nature.com/scientificreports/ Water clip
Hot air clip
12 10 8 6 4 2 0 Horizontal vector
Ver cal vector
Figure 2. Standard deviation of motion vectors in the optical flow fields of simulated water and hot air. The simulation video clips with the background of a ‘Rocks’ scene (see Method section for details) were used for the calculation of optical flow fields. The standard deviation is separately shown for horizontal and vertical motion vectors. Error bars denote the standard deviation across 29 optical flow field patterns (which were obtained from each successive pair in 30 video frames).
The results (shown in Fig. 1h) showed that the hot air impression in the simulation could produce a comparable level of impression to the clips of real hot air. A two-way repeated measures ANOVA with clip types and presentation style as within-subject factors showed that there were no significant differences in rating scores between the simulation clips and real hot air ones [F(1, 11) = 0.000, p > 0.05, partial η2 = 0.000] (their average scores were just identical to each other by accident). The rating scores were strongly affected by the presentation style [F(1, 11) = 13.885, p 0.05, partial η2 = 0.260]. The results indicate that our hot air simulation clips could give the observers the intended impression, whose strength was comparable to their impression of the clips of real hot air. In Experiment 2, we wanted to check whether the observers could discern hot air from water in our simulation. To see the pure effect of image deformation on our task, our water flow simulation omitted specular reflection and caustics. In each trial, the observers were asked to view the clips of simulated water or hot air flows (Video 1) and successively rate their impressions of water and hot air. The results (Fig. 1i) showed that the water impression was stronger for water simulation clips [t(11) = 2.49, p
OPEN
Image deformation as a cue to material category judgment Takahiro Kawabe1,* & Rok Kogovšek1,2,*
received: 23 August 2016 accepted: 06 February 2017 Published: 09 March 2017
Human observers easily recognize complex natural phenomena, such as flowing water, which often generate highly chaotic dynamic arrays of light on the retina. It has not been clarified how the visual system discerns the source of a fluid flow. Here we show that the magnitude of image deformation caused by light refraction is a critical factor for the visual system to determine the perceptual category of fluid flows. Employing a physics engine, we created computer-rendered scenes of water and hot air flows. For each flow, we manipulated the rendering parameters (distortion factors and the index of refraction) that strongly influence the magnitude of image deformation. The observers rated how strongly they felt impressions of water and hot air in the video clips of the flows. The ratings showed that the water and hot air impressions were positively and negatively related to the magnitude of image deformation. Based on the results, we discuss how the visual system heuristically utilizes image deformation to discern non-rigid materials such as water and hot air flows. Vision scientists have recently attempted to unveil the perceptual and cognitive mechanisms underlying the recognition of materials1. Most of them have focused on the perception of material surface properties. In particular, previous studies have shown that human observers can decode surface reflectance properties such as specularity2–6 and subsurface scattering7,8 from complex retinal images. In addition to surface reflectance, surface refraction is an important phenomenon determining perceptual appearance of materials. When light penetrates a transparent material with an index of refraction (IOR) of more (or less) than 1, the path of the light is bent at the material’s surface, where the direction of the bending depends on the IOR. The bent light path often causes image deformation of the scene behind a curved material, wherein the larger (smaller) IOR of the material creates a larger (smaller) magnitude of image deformation on the retina. It has been reported that image deformation due to refraction is a vital cue for the visual system to judge several aspects of material properties. For example, the magnitude of image deformation serves as a cue to judge the thickness of rigid materials such as glass [ref. 9, but also see ref. 10]. Moreover, the human visual system can utilize the dynamic pattern of image deformation due to refraction as a cue to a material’s elasticity11. Importantly, dynamic image deformation gives observers the impression of a transparent water-like layer. A previous study12 reported that human observers can recognize a transparent water flow from dynamic image deformation alone; no other cues, such as specular reflection or luminance reduction due to absorption, were necessary for the transparent water recognition. It also showed that the critical component in dynamic image deformation is the spatiotemporal frequency of deformation and suggested that the specific band of the spatiotemporal deformation frequency is critical for perceiving the deformation as arising from the intervention of a transparent water layer. Besides a water flow, a hot air flow also optically deforms the image behind it. A temperature change physically causes the spatiotemporal variation of the density of air, which eventually alters the IOR. Because the density change spatiotemporally occurs on the basis of the Rayleigh–Taylor instability13, the image deformation caused by refraction can be dynamic. In our everyday experience, we easily notice the existence of hot air just by seeing the dynamic image deformation. On the other hand, how the visual system recognizes hot air and how the system differentiates between hot air and water flows on the basis of dynamic image deformation are still open questions. Clarifying how the visual system categorizes transparent non-rigid material will be an important step to understanding how the mind generates rich and detailed visual representations of the very complex real world from a handful of visual inputs.
1
NTT Communication Science Laboratories, Nippon Telegraph and Telephone Corporation, Japan. 2Faculty of Computer and Information Science, University of Ljubljana, Slovenia. *These authors contributed equally to this work. Correspondence and requests for materials should be addressed to T.K. (email: [email protected]. co.jp) Scientific Reports | 7:44274 | DOI: 10.1038/srep44274
1
www.nature.com/scientificreports/ (a)
(d)
(g)
(e)
Rightward
Horizontal vectors
(b)
Real hot air clip
Mo on v ectors (pixels)
Simulated hot air clip
Le ward
(f)
Ver cal vectors
(c)
Mo on v ectors (pixels)
Upward
Downward
(i)
N = 12 (Error bars: ±1 SEM) Dynamic Sta c 5
5
4.5
4.5
4
4
Ra ng scores
Ra ng scores
(h)
3.5 3 2.5
3.5 3 2.5
2
2
1.5
1.5
1
Simulated Real Type of video clips
N = 11 (Error bars: ±1 SEM) Hot air simula on Water simula on
1
Water impression Hot air impression Types of impression
Figure 1. Simulation geometry, rendered scene, motion vector analysis, and experimental results. (a) The geometry of water simulation seen from an oblique angle. (b) A rendered scene of (a), which was viewed from an oblique angle. (c) A rendered scene of water flow. (d) The geometry of hot air simulation seen from an oblique angle. (e) A rendered scene of (d), which was viewed from an oblique angle. (f) A rendered scene of hot air. (g) Motion vector comparison between clips of simulated and real hot air for horizontal (upper row) and vertical (lower row) vectors, which were calculated between two video frames. (h) Experiment 1 results. (i) Experiment 2 results.
In this work, we examined what the critical image information is for human observers to discern the source of a transparent material flow. We used computer graphics with a physics engine to create video clips of simulated hot air flows and water flows. We noticed that the former generated a smaller magnitude of image deformation than the latter. Thus, we parametrically manipulated the magnitude of image deformation by changing rendering parameters. We then found that the hot air and water impressions were negatively and positively related to the magnitude of image deformation under the condition where other aspects of the flows, such as flow direction and turbulence, were intact. On the basis of the results, we propose a visual mechanism underlying the recognition of fluid materials from image deformation.
Results and Discussion
In Experiment 1, we first tried to determine whether our simulation of hot air (Fig. 1a–f) could produce a reasonable level of the desired impression. The observers were asked to rate their hot air impression after they had watched the video clips of simulated and real hot air flows. As shown in Fig. 1g, the range of motion vectors was similar between the simulated hot air and real hot air. Two styles of video clip presentation were tested. One was a dynamic style in which dynamic sequences of video frames were presented. The other was a static style in which a single static video frame was randomly extracted from a sequence of video frames and presented to the observer. Scientific Reports | 7:44274 | DOI: 10.1038/srep44274
2
Standard devia on of mo on vector (pixels)
www.nature.com/scientificreports/ Water clip
Hot air clip
12 10 8 6 4 2 0 Horizontal vector
Ver cal vector
Figure 2. Standard deviation of motion vectors in the optical flow fields of simulated water and hot air. The simulation video clips with the background of a ‘Rocks’ scene (see Method section for details) were used for the calculation of optical flow fields. The standard deviation is separately shown for horizontal and vertical motion vectors. Error bars denote the standard deviation across 29 optical flow field patterns (which were obtained from each successive pair in 30 video frames).
The results (shown in Fig. 1h) showed that the hot air impression in the simulation could produce a comparable level of impression to the clips of real hot air. A two-way repeated measures ANOVA with clip types and presentation style as within-subject factors showed that there were no significant differences in rating scores between the simulation clips and real hot air ones [F(1, 11) = 0.000, p > 0.05, partial η2 = 0.000] (their average scores were just identical to each other by accident). The rating scores were strongly affected by the presentation style [F(1, 11) = 13.885, p 0.05, partial η2 = 0.260]. The results indicate that our hot air simulation clips could give the observers the intended impression, whose strength was comparable to their impression of the clips of real hot air. In Experiment 2, we wanted to check whether the observers could discern hot air from water in our simulation. To see the pure effect of image deformation on our task, our water flow simulation omitted specular reflection and caustics. In each trial, the observers were asked to view the clips of simulated water or hot air flows (Video 1) and successively rate their impressions of water and hot air. The results (Fig. 1i) showed that the water impression was stronger for water simulation clips [t(11) = 2.49, p
