Perception, 1978, volume 7, pages 707-715
An additional dimension to grating perception
Christopher W Tyler Smith-Kettlewell Institute of Visual Sciences, 2232 Webster Street, San Francisco, California 94115, USA Received 15 January 1976, in revised form 20 May 1978
Abstract. Visual acuity (A) for a two-dimensional multiplicative sinusoidal contrast grid (sinusoidal chessboard) was measured as a function of spatial frequency (co) of modulation in one of the dimensions. The reduction in acuity as frequency was increased was well described by the equation: A = -27rAxL> + c, where k and c are constants. At suprathreshold contrasts, the frequency of modulation appeared to be doubled relative to the underlying modulation frequency. This doubling is not related to the frequency of the harmonic components of the chessboard, but suggests the existence of a perceptual mechanism sensitive to areas of sinusoidal modulation which are seen at twice the frequency of the variation in contrast. 1 Visual acuity for two-dimensional modulations Grating targets with a periodic variation of luminance are standard test objects for the measurements of visual acuity. For grating stimuli it is convenient to define visual acuity as the reciprocal of the angular distance in arc minutes between the centres of adjacent dark (or light) bars at detection threshold. By this definition grating acuity has a maximum value of about 1 • 1 under optimal conditions (Schlaer 1937). The present study was designed to quantify the effects of luminance variations in the direction, orthogonal to those in the grating for which threshold is being measured. There are several ways in which such orthogonal variations may be combined with the original grating. For example, the two may be added, in which case the orthogonal grating may be considered as a simultaneous mask. Campbell and Kulikowski (1966) report the effect of a simultaneous mask to be slight, but they did not measure visual acuity under these conditions. An alternative approach is to examine the effect of multiplying the test grating by an orthogonal grating (see figure 1). Both gratings are sinusoidal and the resultant configuration will be described as a multiplicative sinusoidal grid. This is the sinusoidal equivalent of the chessboard grid which is widely used as a stimulus in psychophysical and electrophysiological studies. In this case it is the sinusoidal modulations which are multiplied (as opposed to multiplication of the luminances, for example). The luminance of the grid, L(x, y), is therefore specified at any point by the equation L(x,y)
= L 0 + mcos(27rcox)cos(27rco^),
(1)
where L0 is the mean luminance level, co is the spatial frequency in radians, and m is the modulation depth and is defined by the relationship m
=
2L0
•
The demonstration grid shown in figure 1 has m = 1 and a logarithmic variation in spatial frequency of two log units along both axes so as to show a full range of combinations of spatial frequency. It is generated by varying the dot density of a random-noise field produced by a Fortran program on a Calcomp plotter.
708
C W Tyler
Visual acuity in one direction as a function of spatial frequency in the orthogonal direction was measured in two observers, both of whom had 20/20 vision with full refractive correction. The sinusoidal grid was viewed in an upright orientation as displayed in figure 1, from two distances of 2 m and 4 m. At the greater distance the noise background was close to threshold visibility and therefore should have little effect on perception of the grid. It had a luminance of 3 cd m~2 and a contrast of 0-6. For each reading the experimenter moved a pointer orthogonally to the test modulation until the subject reported that it indicated threshold visibility of the grid at that spatial frequency. Two readings at each combination of frequencies were taken in fully randomised order. The results (figure 2) show that visual acuity 04) for modulation in the test orientation is strongly dependent on spatial frequency of modulation in the direction orthogonal to the test orientation. The measurements from the two observation distances do not appear to differ significantly for either observer, so the combined points were fitted by inspection by the linear function (3)
—lirkoo + c ,
• t M M
"X/XAAAAAAIlliWFigure 1. Demonstration two-dimensional multiplicative sinusoidal grid, with logarithmic increase in spatial frequency on both axes. The figure is generated by modulation of dot density in a random-noise field. subject NV
subject CWT
10 15 20 0 5 10 Orthogonal spatial frequency (cycles deg-1)
Figure 2. The dependence of visual acuity on spatial frequency in the orthogonal dimension of a multiplicative sinusoidal grid, measured for two subjects at two viewing distances (open circles, 2 m; filled circles, 4 m). Mean standard error for all points is shown as the vertical bar to the right of each graph.
An additional dimension to grating perception
709
where the constants k and c have quite similar values for the two observers (NV: & = 0-028, c = 0-68; CWT: k = 0-022, c = 0-64). The demonstration of a negative linear relationship between visual acuity and orthogonal spatial frequency in a multiplicative grid has interesting physiological implications. In monkey and therefore probably in man the receptive fields of the retina are circularly symmetric (Hubel and Wiesel 1960) whereas those of the visual cortex are predominantly elongated (Hubel and Wiesel 1968). The elements of the multiplicative grid are approximately circular where the spatial frequencies in the two directions are equal as on the main diagonal of figure 1, but they are elongated where the spatial frequency at threshold is high; there is the opportunity for cortical units to summate over long stimulus elements when the orthogonal spatial frequency is low. At high orthogonal spatial frequencies, where the spatial frequencies are similar in the two directions, such summation is not possible in the same way. The elongation is in the most sensitive meridian (horizontal or vertical) when the grating is viewed in an upright orientation. The effects of viewing in the oblique meridian may be understood from a consideration of the two-dimensional Fourier spectrum of the stimulus, and are considered in the following section. 2 Implications of two-dimensional Fourier analysis The discussion of the two-dimensional acuity data in the previous section was derived from consideration of the individual elements of the pattern. It is possible that the pattern is processed in a more distributed manner. One way in which this might occur is by some visual approximation to a two-dimensional Fourier analysis. Since this kind of process has a current popularity, it is important to analyse its implications in the two-dimensional acuity task. The two-dimensional Fourier spectrum, F(r\, v) of a stimulus defined in twodimensional space as having a luminance distribution of L(x, y) is given by the expression F(T?, V)
=
L(x, y) exp[27ri(7?x+ vy)] dx dy .
(4)
This spectrum is defined in terms of the luminous energy in a stimulus. For the present purposes it is more convenient to describe the stimulus and its spectrum in terms of the contrast variations around a mean luminance level L0. In this framework the Fourier contrast spectrum Fc(rj, v) of the stimulus contrast distribution C(x,y) is similarly Fein, v) =
C(x, y) exp [2m{r]x + vy)] dx dy .
(5)
This formulation avoids specification of a zero-frequency harmonic component and its complications when two stimuli are multiplied. The two-dimensional surface represented by the Fourier contrast spectrum FC(T7, V) is shown for a multiplicative sinusoidal grid in figure 3a. In this space each point may be conceptualised as a one-dimensional grating whose frequency is given by the distance from the centre and whose orientation is given by the angular position, 0, which represents the meridian along which the grating is modulated. Modulation amplitude may be represented by the size of the dot in each location, and the spatial phase of the modulation is not specified. The Fourier contrast spectrum has centric symmetry since modulation along one meridian implies modulation along the same meridian in the opposite direction, or 180° opposite in the Fourier plane, so that for a real stimulus each point has a conjugate pair diametrically opposite.
710
CW Tyler
A multiplicative sinusoidal grid is obtained by multiplying a horizontal modulation (horizontal bars in figure 3 a) by a vertical modulation of the same frequency (vertical bars in figure 3a). The resultant components are shown as the four black spots. It is immediately evident that each component, and its conjugate pair, represent one-dimensional gratings. This means that a multiplicative sinusoidal grid obtained by multiplying horizontal and vertical modulation is equivalent to an additive sinusoidal grid obtained by adding two oblique modulations, each of half the amplitude. This follows from the well-known trigonometric relationship: cosxcosj; =
cos(x+y) ~
1
cos(x-y) 2
•
w
These oblique components become visible in the suprathreshold portion of figure 1 as the diagonal bars seen where the two orthogonal spatial frequencies are approximately equal (along the main diagonal). The question now arises whether these oblique components derived from Fourier considerations are a determining factor in the two-dimensional acuity task described in the previous section. The conclusion I shall reach is that the Fourier analysis provides a lower bound for the two-dimensional acuity data that does not invalidate the physiological analysis presented above. In order to derive the Fourier contrast spectra for the two-dimensional acuity task, envisage stimuli in which the characteristics of any local region of figure 1 are extended infinitely through space in all directions. Then the Fourier contrast spectrum of the lower left-hand corner, if infinitely extended, would appear as the full circles in figure 3 a. The spectrum of the upper right-hand corner, which is similar in that the multiplied modulations are of equal frequency, but have a much higher frequency, would appear as figure 3b. The lower right-hand corner, which contains low-frequency modulation vertically but high-frequency modulation horizontally, would be represented by figure 3c. Intermediate cases may be imagined by extrapolation. What can be said about the visibility of the stimulus in these different cases? To answer this the Fourier contrast spectrum must be compared with the visibility limitations of the human visual system. These are represented schematically in figure 3. The outer boundary shows the upper limit of one-dimensional grating resolution, typically in the region of 50 cycles deg -1 and reduced in the oblique
(a) (b) (c) Figure 3. (a) Fourier contrast harmonics (black dots) generated by multiplying a horizontal and a vertical sinusoidal modulation of low spatial frequency (short bars). The resultant components are at 45° orientation, representing the sum of two oblique modulations, (b) Fourier contrast harmonics obtained by multiplying similar orthogonal modulations but with a high spatial frequency, (c) Fourier contrast harmonics obtained when the two orthogonal modulations are of very different spatial frequencies. The upper spatial frequency visibility limit is shown as the enclosing 'bubble' in each graph. The circles represent one neural 'channel' tuned in spatial frequency and orientation.
An additional dimension to grating perception
711
meridians (Campbell et al 1966). The inner circle represents the narrowest twodimensional Fourier limitations of a typical pattern-detection mechanism best stimulated by one of the Fourier components, estimated from human psychophysics (Blakemore and Campbell 1969; Stromeyer and Julesz 1972) and from monkey neurophysiology (Schiller et al 1976), as described by Tyler and Chang (1977). The vertical limitations have two important consequences. The size of the pattern mechanism in Fourier space is such that for multiplicative gratings of equal spatial frequency in the two directions (figure 3b) each Fourier component can stimulate a separate pattern mechanism, whereas when the orthogonal spatial frequency is very low (figure 3c) the two components must stimulate the same pattern mechanism. If the effectiveness of each component is additive within a mechanism, and if the mechanisms act independently to determine threshold acuity, this analysis would suggest that threshold modulation for perception of a sinusoidal grid should be the same as the sensitivity of one of its components, and twice the modulation required for perception of a one-dimensional grating, according to equation (6). It is unnecessary to review all the evidence supporting the two assumptions made to crystallise this prediction, as the prediction itself has been amply validated by Carlson et al (1977). The Fourier description therefore suggests that a two-dimensional visual acuity should be reduced by a factor of two as the orthogonal spatial frequency varies from zero to equality with the spatial frequency in the test direction. To account for the further loss as shown in figure 2, the orientation of the Fourier components on the retina must be considered. 3 The 'oblique effect' in two-dimensional visual acuity The 'oblique effect' is the reduction in visual acuity for many types of target when they are viewed in oblique orientations on the retina as opposed to horizontal and vertical orientations (French 1920; Campbell et al 1966; Tyler and Mitchell 1977). It has a straightforward implication for the two-dimensional acuity in that when the Fourier components are oriented towards the oblique meridians, visibility should be reduced. The data of Campbell et al (1966) suggest that the maximum loss should be about 0-5 octaves or a factor of 1 -4 for grating acuity. Taking the effect of orientation together with the effect of the changing distribution of Fourier components suggests that two-dimensional acuity should be reduced by a factor of 2.x l - 4 « 3 , where the orthogonal frequency is equal to the test frequency as compared to acuity when the orthogonal modulation is of a much lower frequency. This is a good fit to the total loss seen in figure 1 to within experimental error. The success of this prediction in terms of the Fourier description may be consolidated by an experiment in which the sinusoidal grid stimulus is viewed in oblique retinal orientations (45° or 135° from upright). Visibility of the previously oblique Fourier components (where two orthogonal modulations are approximately equal) should be enhanced by a factor of 1-6, while visibility of the previously horizontal and vertical components (where the orthogonal modulations are very different) should be reduced by the same amount. In this way the effects of Fourier component amplitude and of retinal orientation can be dissociated (on the assumption that they are independent and additive). The experiment was performed by two observers (one of whom participated in the first experiment) under the same conditions as the first experiment, except that the viewing distance was increased to 5 m so that thresholds would occur in different absolute positions in relation to the noise background. Two readings were obtained for a range of spatial frequencies in each of four stimulus orientations (0°, 45°, 90°, 135°). The mean data for the 0° and 90° orientations replicated the data for the first experiment (figure 1). On the other hand the mean data for the 45° and 135°
712
C W Tyler
orientations showed a great reduction in the effect of orthogonal spatial frequency (not shown here). In order to separate the effects of orientation and spatial distribution of the Fourier components on two-dimensional acuity I shall assume they are independent and additive. The mean of all orientations gives the effect of spatial frequency of the orthogonal modulation. The difference between vertical/horizontal and oblique thresholds gives the effect of orientation alone. The difference curves (given as the mean for the two observers in figure 4a) show the total effect of varying the retinal orientation from horizontal/vertical to oblique. The maximum difference is close to a factor of 2 (1 octave), as would be predicted by the fact that the orientation change should increase sensitivity at one end of the curve while decreasing it at the other by the same amount, both factors of 1-4. If it is assumed that the effect of retinal orientation is linear through each octant of rotation (which has been validated to a first approximation by Mitchell et al 1967), then the mean of each pair of curves averaged over orientation in figure 4a gives the effect of distribution of the components without contamination by orientation. The mean for both observers (figure 4b) again confirms the main features of figure 1, but shows that the total effect is now close to a factor of 2, as opposed to a factor of 3 when orientation was not controlled. As demonstrated in the previous section, change of a factor of 2 is what was predicted from the Fourier description of the stimulus. Both predictions are therefore confirmed by the data of figure 4.
3 2 o
0-6
2-0
3.S 0-4 i 13 a
1-0
g& °'2
£ S £ 0-5
10 15 20 25 30 25 30 0 5 -1 Spatial frequency (cycles deg ) „ „ — , ,..-_„, (a) (b) Figure 4. (a) Ratio of acuity in the oblique meridians to that in the horizontal and vertical meridians, as a function of the orthogonal spatial frequency. (Open and filled points are for two observers.) (b) Mean visual acuity averaged over all orientations as a function of orthogonal spatial frequency.
10
15
4 Two-dimensional acuity:
20
conclusion
It should be noted that this confirmation of the predictions from Fourier analysis does not imply that the visual system is performing a process that should be regarded as an approximation to a spatial Fourier analysis of the visual image. Instead it shows that the Fourier description of the stimuli and the neural responses provides a useful framework within which to unravel neural image processing. The predicted relationships may be implemented in any number of processes on a physiological level. However, the sinusoidal grid stimulus was originally designed with the object of creating a continuous array of elements of variable length/width ratio. The Fourier analysis of the previous section shows that it is difficult to avoid generating oblique Fourier components under these conditions. A stimulus consisting solely of circular light and dark areas is logically impossible. The sinusoidal grid used in the experiments reported here is one compromise solution to the problem of generating a stimulus with no long elements. It succeeds to the extent that acuity is reduced when the spatial frequencies are equal in the two directions.
An additional dimension to grating perception
The results shown in figure 2 therefore represent an upper bound to the loss of acuity that would be obtained if all unwanted Fourier components had been eliminated. The fact that the results are explicable in terms of the Fourier description does not detract from this point. Indeed, this fact emphasises the extent to which visual perception is determined by elongated rather than punctate aspects of the stimulus array. The equifrequency region of the sinusoidal grid has the same contrast as the rest of the display, so if detection occurred by point-to-point integration of contrast over some fixed area, acuity should be unaffected by the orthogonal spatial frequency, or orientation of the grid. The results in figure 2 show that both factors are important, and that elongated stimuli in the optimal orientation can be detected with an acuity about three times greater than in the equifrequency region of the sinusoidal grid. Koenderink and van Doom (1974) have measured sensitivity for two-dimensional band-limited noise, which is another way of minimising elongated elements. At the high mean spatial frequency they used, sensitivity was reduced by a factor of 10 in comparison with sinusoidal gratings of the same frequency. It is not clear whether this reduction is due to the dearth of elongated elements or to the masking effects of having many similar frequencies present in the stimulus. In the former case, their results would provide strong confirmation of the importance of the elongated elements in visual perception. 5 Spatiospatial frequency doubling Turning attention to the subjective appearance of the grid at suprathreshold contrast, one soon notices that a new phenomenon is present. In regions where the frequency in one dimension is much greater than in the other, the apparent frequency on the low-frequency dimension is doubled relative to the basic modulation in that dimension (figure 1). The basic modulation is depicted along the edge of the figure, and the doubling may be observed as modulated stripes. The arrows indicate the peaks of the modulated stripes, and the troughs appear in this display as random noise. It is clear that the arrows occur at twice the spatial frequency of the modulation waveform. This spatiospatial frequency doubling is analogous to the spatiotemporal frequency doubling observed in flickering gratings (Kelly 1966; Tyler 1974) in that multiplying the pairs of dimensions gives rise to a doubling percept. It should be emphasised that in the region of doubling, the frequency appears as a kind of amplitude modulation rather than in terms of simple contrast. Thus, it is not that the contrast variations are at twice the frequency, but that a new percept appears (low-frequency modulation of a high-frequency grating) which has twice the frequency of the input modulation (figure 5a). This spatiospatial frequency doubling should not be confused with the fact that the harmonic components (coh) of a sinusoidal grid have a frequency higher than those of the multiplied one-dimensional modulations (co*, cjy) as was shown in figure 3. The frequency of the harmonic components is given by:
which has a maximum of 1 -4o; h when GJX =
An additional dimension to grating perception
Christopher W Tyler Smith-Kettlewell Institute of Visual Sciences, 2232 Webster Street, San Francisco, California 94115, USA Received 15 January 1976, in revised form 20 May 1978
Abstract. Visual acuity (A) for a two-dimensional multiplicative sinusoidal contrast grid (sinusoidal chessboard) was measured as a function of spatial frequency (co) of modulation in one of the dimensions. The reduction in acuity as frequency was increased was well described by the equation: A = -27rAxL> + c, where k and c are constants. At suprathreshold contrasts, the frequency of modulation appeared to be doubled relative to the underlying modulation frequency. This doubling is not related to the frequency of the harmonic components of the chessboard, but suggests the existence of a perceptual mechanism sensitive to areas of sinusoidal modulation which are seen at twice the frequency of the variation in contrast. 1 Visual acuity for two-dimensional modulations Grating targets with a periodic variation of luminance are standard test objects for the measurements of visual acuity. For grating stimuli it is convenient to define visual acuity as the reciprocal of the angular distance in arc minutes between the centres of adjacent dark (or light) bars at detection threshold. By this definition grating acuity has a maximum value of about 1 • 1 under optimal conditions (Schlaer 1937). The present study was designed to quantify the effects of luminance variations in the direction, orthogonal to those in the grating for which threshold is being measured. There are several ways in which such orthogonal variations may be combined with the original grating. For example, the two may be added, in which case the orthogonal grating may be considered as a simultaneous mask. Campbell and Kulikowski (1966) report the effect of a simultaneous mask to be slight, but they did not measure visual acuity under these conditions. An alternative approach is to examine the effect of multiplying the test grating by an orthogonal grating (see figure 1). Both gratings are sinusoidal and the resultant configuration will be described as a multiplicative sinusoidal grid. This is the sinusoidal equivalent of the chessboard grid which is widely used as a stimulus in psychophysical and electrophysiological studies. In this case it is the sinusoidal modulations which are multiplied (as opposed to multiplication of the luminances, for example). The luminance of the grid, L(x, y), is therefore specified at any point by the equation L(x,y)
= L 0 + mcos(27rcox)cos(27rco^),
(1)
where L0 is the mean luminance level, co is the spatial frequency in radians, and m is the modulation depth and is defined by the relationship m
=
2L0
•
The demonstration grid shown in figure 1 has m = 1 and a logarithmic variation in spatial frequency of two log units along both axes so as to show a full range of combinations of spatial frequency. It is generated by varying the dot density of a random-noise field produced by a Fortran program on a Calcomp plotter.
708
C W Tyler
Visual acuity in one direction as a function of spatial frequency in the orthogonal direction was measured in two observers, both of whom had 20/20 vision with full refractive correction. The sinusoidal grid was viewed in an upright orientation as displayed in figure 1, from two distances of 2 m and 4 m. At the greater distance the noise background was close to threshold visibility and therefore should have little effect on perception of the grid. It had a luminance of 3 cd m~2 and a contrast of 0-6. For each reading the experimenter moved a pointer orthogonally to the test modulation until the subject reported that it indicated threshold visibility of the grid at that spatial frequency. Two readings at each combination of frequencies were taken in fully randomised order. The results (figure 2) show that visual acuity 04) for modulation in the test orientation is strongly dependent on spatial frequency of modulation in the direction orthogonal to the test orientation. The measurements from the two observation distances do not appear to differ significantly for either observer, so the combined points were fitted by inspection by the linear function (3)
—lirkoo + c ,
• t M M
"X/XAAAAAAIlliWFigure 1. Demonstration two-dimensional multiplicative sinusoidal grid, with logarithmic increase in spatial frequency on both axes. The figure is generated by modulation of dot density in a random-noise field. subject NV
subject CWT
10 15 20 0 5 10 Orthogonal spatial frequency (cycles deg-1)
Figure 2. The dependence of visual acuity on spatial frequency in the orthogonal dimension of a multiplicative sinusoidal grid, measured for two subjects at two viewing distances (open circles, 2 m; filled circles, 4 m). Mean standard error for all points is shown as the vertical bar to the right of each graph.
An additional dimension to grating perception
709
where the constants k and c have quite similar values for the two observers (NV: & = 0-028, c = 0-68; CWT: k = 0-022, c = 0-64). The demonstration of a negative linear relationship between visual acuity and orthogonal spatial frequency in a multiplicative grid has interesting physiological implications. In monkey and therefore probably in man the receptive fields of the retina are circularly symmetric (Hubel and Wiesel 1960) whereas those of the visual cortex are predominantly elongated (Hubel and Wiesel 1968). The elements of the multiplicative grid are approximately circular where the spatial frequencies in the two directions are equal as on the main diagonal of figure 1, but they are elongated where the spatial frequency at threshold is high; there is the opportunity for cortical units to summate over long stimulus elements when the orthogonal spatial frequency is low. At high orthogonal spatial frequencies, where the spatial frequencies are similar in the two directions, such summation is not possible in the same way. The elongation is in the most sensitive meridian (horizontal or vertical) when the grating is viewed in an upright orientation. The effects of viewing in the oblique meridian may be understood from a consideration of the two-dimensional Fourier spectrum of the stimulus, and are considered in the following section. 2 Implications of two-dimensional Fourier analysis The discussion of the two-dimensional acuity data in the previous section was derived from consideration of the individual elements of the pattern. It is possible that the pattern is processed in a more distributed manner. One way in which this might occur is by some visual approximation to a two-dimensional Fourier analysis. Since this kind of process has a current popularity, it is important to analyse its implications in the two-dimensional acuity task. The two-dimensional Fourier spectrum, F(r\, v) of a stimulus defined in twodimensional space as having a luminance distribution of L(x, y) is given by the expression F(T?, V)
=
L(x, y) exp[27ri(7?x+ vy)] dx dy .
(4)
This spectrum is defined in terms of the luminous energy in a stimulus. For the present purposes it is more convenient to describe the stimulus and its spectrum in terms of the contrast variations around a mean luminance level L0. In this framework the Fourier contrast spectrum Fc(rj, v) of the stimulus contrast distribution C(x,y) is similarly Fein, v) =
C(x, y) exp [2m{r]x + vy)] dx dy .
(5)
This formulation avoids specification of a zero-frequency harmonic component and its complications when two stimuli are multiplied. The two-dimensional surface represented by the Fourier contrast spectrum FC(T7, V) is shown for a multiplicative sinusoidal grid in figure 3a. In this space each point may be conceptualised as a one-dimensional grating whose frequency is given by the distance from the centre and whose orientation is given by the angular position, 0, which represents the meridian along which the grating is modulated. Modulation amplitude may be represented by the size of the dot in each location, and the spatial phase of the modulation is not specified. The Fourier contrast spectrum has centric symmetry since modulation along one meridian implies modulation along the same meridian in the opposite direction, or 180° opposite in the Fourier plane, so that for a real stimulus each point has a conjugate pair diametrically opposite.
710
CW Tyler
A multiplicative sinusoidal grid is obtained by multiplying a horizontal modulation (horizontal bars in figure 3 a) by a vertical modulation of the same frequency (vertical bars in figure 3a). The resultant components are shown as the four black spots. It is immediately evident that each component, and its conjugate pair, represent one-dimensional gratings. This means that a multiplicative sinusoidal grid obtained by multiplying horizontal and vertical modulation is equivalent to an additive sinusoidal grid obtained by adding two oblique modulations, each of half the amplitude. This follows from the well-known trigonometric relationship: cosxcosj; =
cos(x+y) ~
1
cos(x-y) 2
•
w
These oblique components become visible in the suprathreshold portion of figure 1 as the diagonal bars seen where the two orthogonal spatial frequencies are approximately equal (along the main diagonal). The question now arises whether these oblique components derived from Fourier considerations are a determining factor in the two-dimensional acuity task described in the previous section. The conclusion I shall reach is that the Fourier analysis provides a lower bound for the two-dimensional acuity data that does not invalidate the physiological analysis presented above. In order to derive the Fourier contrast spectra for the two-dimensional acuity task, envisage stimuli in which the characteristics of any local region of figure 1 are extended infinitely through space in all directions. Then the Fourier contrast spectrum of the lower left-hand corner, if infinitely extended, would appear as the full circles in figure 3 a. The spectrum of the upper right-hand corner, which is similar in that the multiplied modulations are of equal frequency, but have a much higher frequency, would appear as figure 3b. The lower right-hand corner, which contains low-frequency modulation vertically but high-frequency modulation horizontally, would be represented by figure 3c. Intermediate cases may be imagined by extrapolation. What can be said about the visibility of the stimulus in these different cases? To answer this the Fourier contrast spectrum must be compared with the visibility limitations of the human visual system. These are represented schematically in figure 3. The outer boundary shows the upper limit of one-dimensional grating resolution, typically in the region of 50 cycles deg -1 and reduced in the oblique
(a) (b) (c) Figure 3. (a) Fourier contrast harmonics (black dots) generated by multiplying a horizontal and a vertical sinusoidal modulation of low spatial frequency (short bars). The resultant components are at 45° orientation, representing the sum of two oblique modulations, (b) Fourier contrast harmonics obtained by multiplying similar orthogonal modulations but with a high spatial frequency, (c) Fourier contrast harmonics obtained when the two orthogonal modulations are of very different spatial frequencies. The upper spatial frequency visibility limit is shown as the enclosing 'bubble' in each graph. The circles represent one neural 'channel' tuned in spatial frequency and orientation.
An additional dimension to grating perception
711
meridians (Campbell et al 1966). The inner circle represents the narrowest twodimensional Fourier limitations of a typical pattern-detection mechanism best stimulated by one of the Fourier components, estimated from human psychophysics (Blakemore and Campbell 1969; Stromeyer and Julesz 1972) and from monkey neurophysiology (Schiller et al 1976), as described by Tyler and Chang (1977). The vertical limitations have two important consequences. The size of the pattern mechanism in Fourier space is such that for multiplicative gratings of equal spatial frequency in the two directions (figure 3b) each Fourier component can stimulate a separate pattern mechanism, whereas when the orthogonal spatial frequency is very low (figure 3c) the two components must stimulate the same pattern mechanism. If the effectiveness of each component is additive within a mechanism, and if the mechanisms act independently to determine threshold acuity, this analysis would suggest that threshold modulation for perception of a sinusoidal grid should be the same as the sensitivity of one of its components, and twice the modulation required for perception of a one-dimensional grating, according to equation (6). It is unnecessary to review all the evidence supporting the two assumptions made to crystallise this prediction, as the prediction itself has been amply validated by Carlson et al (1977). The Fourier description therefore suggests that a two-dimensional visual acuity should be reduced by a factor of two as the orthogonal spatial frequency varies from zero to equality with the spatial frequency in the test direction. To account for the further loss as shown in figure 2, the orientation of the Fourier components on the retina must be considered. 3 The 'oblique effect' in two-dimensional visual acuity The 'oblique effect' is the reduction in visual acuity for many types of target when they are viewed in oblique orientations on the retina as opposed to horizontal and vertical orientations (French 1920; Campbell et al 1966; Tyler and Mitchell 1977). It has a straightforward implication for the two-dimensional acuity in that when the Fourier components are oriented towards the oblique meridians, visibility should be reduced. The data of Campbell et al (1966) suggest that the maximum loss should be about 0-5 octaves or a factor of 1 -4 for grating acuity. Taking the effect of orientation together with the effect of the changing distribution of Fourier components suggests that two-dimensional acuity should be reduced by a factor of 2.x l - 4 « 3 , where the orthogonal frequency is equal to the test frequency as compared to acuity when the orthogonal modulation is of a much lower frequency. This is a good fit to the total loss seen in figure 1 to within experimental error. The success of this prediction in terms of the Fourier description may be consolidated by an experiment in which the sinusoidal grid stimulus is viewed in oblique retinal orientations (45° or 135° from upright). Visibility of the previously oblique Fourier components (where two orthogonal modulations are approximately equal) should be enhanced by a factor of 1-6, while visibility of the previously horizontal and vertical components (where the orthogonal modulations are very different) should be reduced by the same amount. In this way the effects of Fourier component amplitude and of retinal orientation can be dissociated (on the assumption that they are independent and additive). The experiment was performed by two observers (one of whom participated in the first experiment) under the same conditions as the first experiment, except that the viewing distance was increased to 5 m so that thresholds would occur in different absolute positions in relation to the noise background. Two readings were obtained for a range of spatial frequencies in each of four stimulus orientations (0°, 45°, 90°, 135°). The mean data for the 0° and 90° orientations replicated the data for the first experiment (figure 1). On the other hand the mean data for the 45° and 135°
712
C W Tyler
orientations showed a great reduction in the effect of orthogonal spatial frequency (not shown here). In order to separate the effects of orientation and spatial distribution of the Fourier components on two-dimensional acuity I shall assume they are independent and additive. The mean of all orientations gives the effect of spatial frequency of the orthogonal modulation. The difference between vertical/horizontal and oblique thresholds gives the effect of orientation alone. The difference curves (given as the mean for the two observers in figure 4a) show the total effect of varying the retinal orientation from horizontal/vertical to oblique. The maximum difference is close to a factor of 2 (1 octave), as would be predicted by the fact that the orientation change should increase sensitivity at one end of the curve while decreasing it at the other by the same amount, both factors of 1-4. If it is assumed that the effect of retinal orientation is linear through each octant of rotation (which has been validated to a first approximation by Mitchell et al 1967), then the mean of each pair of curves averaged over orientation in figure 4a gives the effect of distribution of the components without contamination by orientation. The mean for both observers (figure 4b) again confirms the main features of figure 1, but shows that the total effect is now close to a factor of 2, as opposed to a factor of 3 when orientation was not controlled. As demonstrated in the previous section, change of a factor of 2 is what was predicted from the Fourier description of the stimulus. Both predictions are therefore confirmed by the data of figure 4.
3 2 o
0-6
2-0
3.S 0-4 i 13 a
1-0
g& °'2
£ S £ 0-5
10 15 20 25 30 25 30 0 5 -1 Spatial frequency (cycles deg ) „ „ — , ,..-_„, (a) (b) Figure 4. (a) Ratio of acuity in the oblique meridians to that in the horizontal and vertical meridians, as a function of the orthogonal spatial frequency. (Open and filled points are for two observers.) (b) Mean visual acuity averaged over all orientations as a function of orthogonal spatial frequency.
10
15
4 Two-dimensional acuity:
20
conclusion
It should be noted that this confirmation of the predictions from Fourier analysis does not imply that the visual system is performing a process that should be regarded as an approximation to a spatial Fourier analysis of the visual image. Instead it shows that the Fourier description of the stimuli and the neural responses provides a useful framework within which to unravel neural image processing. The predicted relationships may be implemented in any number of processes on a physiological level. However, the sinusoidal grid stimulus was originally designed with the object of creating a continuous array of elements of variable length/width ratio. The Fourier analysis of the previous section shows that it is difficult to avoid generating oblique Fourier components under these conditions. A stimulus consisting solely of circular light and dark areas is logically impossible. The sinusoidal grid used in the experiments reported here is one compromise solution to the problem of generating a stimulus with no long elements. It succeeds to the extent that acuity is reduced when the spatial frequencies are equal in the two directions.
An additional dimension to grating perception
The results shown in figure 2 therefore represent an upper bound to the loss of acuity that would be obtained if all unwanted Fourier components had been eliminated. The fact that the results are explicable in terms of the Fourier description does not detract from this point. Indeed, this fact emphasises the extent to which visual perception is determined by elongated rather than punctate aspects of the stimulus array. The equifrequency region of the sinusoidal grid has the same contrast as the rest of the display, so if detection occurred by point-to-point integration of contrast over some fixed area, acuity should be unaffected by the orthogonal spatial frequency, or orientation of the grid. The results in figure 2 show that both factors are important, and that elongated stimuli in the optimal orientation can be detected with an acuity about three times greater than in the equifrequency region of the sinusoidal grid. Koenderink and van Doom (1974) have measured sensitivity for two-dimensional band-limited noise, which is another way of minimising elongated elements. At the high mean spatial frequency they used, sensitivity was reduced by a factor of 10 in comparison with sinusoidal gratings of the same frequency. It is not clear whether this reduction is due to the dearth of elongated elements or to the masking effects of having many similar frequencies present in the stimulus. In the former case, their results would provide strong confirmation of the importance of the elongated elements in visual perception. 5 Spatiospatial frequency doubling Turning attention to the subjective appearance of the grid at suprathreshold contrast, one soon notices that a new phenomenon is present. In regions where the frequency in one dimension is much greater than in the other, the apparent frequency on the low-frequency dimension is doubled relative to the basic modulation in that dimension (figure 1). The basic modulation is depicted along the edge of the figure, and the doubling may be observed as modulated stripes. The arrows indicate the peaks of the modulated stripes, and the troughs appear in this display as random noise. It is clear that the arrows occur at twice the spatial frequency of the modulation waveform. This spatiospatial frequency doubling is analogous to the spatiotemporal frequency doubling observed in flickering gratings (Kelly 1966; Tyler 1974) in that multiplying the pairs of dimensions gives rise to a doubling percept. It should be emphasised that in the region of doubling, the frequency appears as a kind of amplitude modulation rather than in terms of simple contrast. Thus, it is not that the contrast variations are at twice the frequency, but that a new percept appears (low-frequency modulation of a high-frequency grating) which has twice the frequency of the input modulation (figure 5a). This spatiospatial frequency doubling should not be confused with the fact that the harmonic components (coh) of a sinusoidal grid have a frequency higher than those of the multiplied one-dimensional modulations (co*, cjy) as was shown in figure 3. The frequency of the harmonic components is given by:
which has a maximum of 1 -4o; h when GJX =
