NeuroImage 110 (2015) 219–222
Contents lists available at ScienceDirect
NeuroImage journal homepage: www.elsevier.com/locate/ynimg
Comments and Controversies
Orientation decoding: Sense in spirals? Colin W.G. Clifford ⁎, Damien J. Mannion School of Psychology, UNSW Australia, Sydney, NSW, Australia
a r t i c l e
i n f o
Article history: Accepted 19 December 2014 Available online 27 December 2014 Keywords: fMRI Visual cortex Computational neuroimaging Multivariate analysis Spatial vision
a b s t r a c t The orientation of a visual stimulus can be successfully decoded from the multivariate pattern of fMRI activity in human visual cortex. Whether this capacity requires coarse-scale orientation biases is controversial. We and others have advocated the use of spiral stimuli to eliminate a potential coarse-scale bias—the radial bias toward local orientations that are collinear with the centre of gaze—and hence narrow down the potential coarse-scale biases that could contribute to orientation decoding. The usefulness of this strategy is challenged by the computational simulations of Carlson (2014), who reported the ability to successfully decode spirals of opposite sense (opening clockwise or counter-clockwise) from the pooled output of purportedly unbiased orientation filters. Here, we elaborate the mathematical relationship between spirals of opposite sense to confirm that they cannot be discriminated on the basis of the pooled output of unbiased or radially biased orientation filters. We then demonstrate that Carlson's (2014) reported decoding ability is consistent with the presence of inadvertent biases in the set of orientation filters; biases introduced by their digital implementation and unrelated to the brain's processing of orientation. These analyses demonstrate that spirals must be processed with an orientation bias other than the radial bias for successful decoding of spiral sense. © 2014 Elsevier Inc. All rights reserved.
Functional magnetic resonance imaging (fMRI) of the blood oxygenation level-dependent (BOLD) signal allows us to observe patterns of activity in the brain (Logothetis, 2008). In human early visual cortex, these activity patterns convey information about characteristics of images presented to the subject such as their form, color, and motion (Kamitani and Tong, 2005, 2006; Haynes and Rees, 2005; Mannion et al., 2009; Freeman et al., 2011, 2013; Alink et al., 2013; Brouwer and Heeger, 2009; Parkes et al., 2009; Goddard et al., 2010), as well as to conjunctions of these attributes (Sumner et al., 2008; Seymour et al., 2009, 2010; Zhang et al., 2014). However, the source of the information in these activity patterns remains a subject of ongoing debate (Op de Beeck, 2010; Kamitani and Sawahata, 2010; Clifford et al., 2011; Freeman et al., 2011, 2013; Alink et al., 2013). Data acquired during fMRI are represented in spatial samples (voxels) whose volume is generally of the order of a few cubic millimeters. This spatial resolution is rather coarser than the scale at which features such as orientation and color have been observed to be mapped onto the surface of mammalian visual cortex (Vanduffel et al., 2002; Yacoub et al., 2008; Xiao et al., 2003, 2007). It has been suggested that information about features such as orientation is nonetheless available at the spatial scale of voxels by virtue of random variability in the spatial distribution of these fine-scale feature maps or their supporting
⁎ Corresponding author at: School of Psychology, UNSW Australia, Sydney, NSW 2052, Australia. E-mail address: [email protected] (C.W.G. Clifford).
http://dx.doi.org/10.1016/j.neuroimage.2014.12.055 1053-8119/© 2014 Elsevier Inc. All rights reserved.
vasculature (Boynton, 2005; Kamitani and Tong, 2005, 2006; Haynes and Rees, 2005; Swisher et al., 2010). However, it has also been claimed that only much coarser scale biases in the way orientation is mapped to the cortical surface allow information about oriented image structure to be recovered from patterns of activity in early visual areas (Freeman et al., 2011, 2013). One such coarse-scale bias that has been observed in the response of early visual cortex to orientation is radial bias: a preference for orientations radial to the point of fixation. Radial bias has been observed in the topographic representations of the visual field that characterize the early areas of visual cortex (Sasaki et al., 2006; Clifford et al., 2009; Mannion et al., 2010; Freeman et al., 2011, 2013), although it is likely that a component of it is inherited from earlier in the visual pathway (Levick and Thibos, 1982; Schall et al., 1986; Shou and Leventhal, 1989; Smith et al., 1990). When subjects are presented with a widefield, obliquely oriented grating, for example, the response in the two quadrants of the visual field where the orientation is close to radial is typically larger than the response to the opposite oblique orientation (Sasaki et al., 2006). Thus, simply observing the pattern of activity across the four quadrants of early visual areas would be sufficient to discriminate which of two oblique orientations is being viewed by the subject (Fig. 1). To remove the confounding effect of radial bias on information about spatial image structure, several studies have used spirals as stimuli (Mannion et al., 2009; Seymour et al., 2010; Alink et al., 2013; Freeman et al., 2013). The logic of this manipulation is that, in a pair of
220
C.W.G. Clifford, D.J. Mannion / NeuroImage 110 (2015) 219–222
Fig. 1. Local orientation structure of oblique gratings relative to fixation. (A) For a centrally fixating observer viewing an oblique grating, two diagonally opposite quadrants of the visual field (outlined in white) contain predominantly radial orientations while the orientations in the other two quadrants (outlined in black) are predominantly tangential. (B) This configuration is reversed for an orthogonal grating.
Then, for any meridian orientation θ0:
spirals of opposite sense, the local orientation at corresponding points in the two images is always at equal but opposite angles from radial (Fig. 2A). Formally, let I+ and I− be two similar spirals of opposite sense:
I− ðr; θ−θ0 Þ ¼ expði ðkr r−kθ ðθ−θ0 Þ þ φ0 ÞÞ W ðr Þ
ð3Þ
Iþ ðr; θÞ ¼ expði ðkr r þ kθ θ þ φ0 ÞÞ W ðr Þ
ð1Þ
I− ðr; θ−θ0 Þ ¼ expði ðkr r þ kθ ðθ0 − θÞ þ φ0 ÞÞ W ðr Þ
ð4Þ
I− ðr; θÞ ¼ expði ðkr r−kθ θ þ φ0 ÞÞ W ðr Þ
ð2Þ
I− ðr; θ−θ0 Þ ¼ Iþ ðr; θ0 − θÞ
ð5Þ
where kr and kθ are the radial and angular frequency, respectively, of the spiral, W(r) is any circularly symmetric stimulus window, and φ0 determines the spatial phase.
Thus, spirals of opposite sense are mirror images of one another over any meridian (Fig. 2B) and a bias to response more strongly to radial orientations will provide no information upon which to discriminate which of two opposite spirals is being viewed. It is also evident that a
Fig. 2. Symmetry between spirals of opposite sense. (A) In a pair of spirals of opposite sense, local image orientation at corresponding points in the two images is always at equal but opposite angles from radial. (B) Spirals of opposite sense are mirror images of one another over any meridian.
C.W.G. Clifford, D.J. Mannion / NeuroImage 110 (2015) 219–222
221
response that is completely invariant over orientation will be similarly uninformative with regard to spiral sense. Despite this, Carlson (2014) has recently claimed that spirals of opposite sense can be discriminated from a representation of local image structure that is unbiased with respect to orientation. On this basis, it was argued that “there is no need to posit a bias either at fine grain or coarse-scale representation to account for decoding spiral sense” (Carlson, 2014). Here, we refute this claim. Specifically, we demonstrate that Carlson's (2014) reported decoding ability is consistent with biases introduced by the digital implementation of discrete orientation filters. Such biases stem from the inevitable loss of rotational invariance when sampling a continuous function on a discrete grid of pixels (Perona, 1995) and are unrelated to the brain's processing of orientation.1 In his computational analysis, Carlson (2014) convolved stimulus images with banks of oriented Gabor filters. However, digital implementation of banks of Gabor filters inevitably results in a small but systematic bias in response power as a function of orientation (Fig. 3A). Specifically, although response power is equal at the orientation of each filter, it varies systematically across intermediate orientations (Freeman and Adelson, 1991; Perona, 1995). This is a problem for the analysis of spiral stimuli, as across the image they contain structure at all orientations. Formally, let P(⊖) be the response power of the filter bank in response to orientation ⊖. Then, for the response to spirals of opposite sense to be equal: P ðθ−θ0 Þ ¼ P ðθ0 − θÞ
ð6Þ
For this requirement to hold over the whole stimulus, it must apply for all meridian orientations, θ0: P ðθ−θ0 Þ ¼ P ðθ0 −θÞ; ∀θ0
ð7Þ
i:e:; P ðθÞ ¼ const
ð8Þ
Thus, for the bank of filters to be effectively unbiased in its response to spiral stimuli, it is necessary for its response power to be equal across all orientations, not just at the peak orientations of the individual filters (Fig. 3B, C). For the illustration of orientation bias in response to gratings in Fig. 3A, we used a bank of 8 Gabor filters with orientations of 0°, 22.5°, 45°, 67.5°, 90°, 112.5°, 135°, and 157.5°, a sine-phase carrier with a frequency of 0.025 cycles per pixel and a two-dimensional Gaussian envelope with σ = 20 pixels, implemented in a 121 × 121 pixel array. The spatial frequency of the grating stimuli was 0.025 cycles per pixel, matching the peak tuning of the filters. The response power of the filter bank varies systematically with the orientation of the stimulus grating. Specifically, power is maximal at the orientation of each of the 8 filters and minimal midway between filters (Fig. 3A). Thus, the response profile is invariant to reflection only about the preferred orientation of any one of the filters (Fig. 3B) and not to reflection about any intermediate orientation (Fig. 3C). Since spirals of opposite sense are mirror images of one another over any meridian (Fig. 2B), it follows that the difference in response power to spirals of opposite sense will only be zero along the meridians through the image corresponding to the orientations of the filters. To illustrate the consequences of orientation bias in the filter bank on the response to spirals, we convolved the same set of filters with complex spiral stimuli defined according to Eqs. (1) and (2) with radial frequency, kr = 0.1 cycles per pixel, and angular frequency, kθ = 10, in 1 Our critique applies specifically to the decoding of spiral sense. Carlson (2014) also demonstrated that the response to oriented gratings at the edge of the stimulus aperture could masquerade as radial bias. We believe that Carlson's analysis of the response to oriented gratings is sound in theory, although empirically our (Clifford et al., 2009) and others' retinotopic data (e.g. Sasaki et al., 2006) indicate that radial bias is not restricted to the edge of the aperture.
Fig. 3. Bias arising from digital implementation of filters. (A) The overall power of a bank of oriented Gabor filters convolved with a sinusoidal grating inevitably shows a small but systematic bias with the orientation of the grating. Here, response power is maximal at the orientation of each filter and minimal midway between filters. The response of the filter bank as a function of stimulus orientation is (B) invariant to reflection about the preferred orientation of the any one of the filters but (C) not to reflection about any intermediate orientation. Thus, the difference in response power to spirals of opposite sense will only be zero along the meridians through the image corresponding to the orientations of the filters.
an annular contrast window with raised cosine edges (Fig. 2A). The total response power of the filter bank (summed over all filter orientations) to a complex spiral stimulus shows a spatial distribution that essentially mimics the contrast window in which the stimulus is presented (Fig. 4A). The difference in total response power between spiral pairs of opposite sense (Fig. 4B) follows a distinctive spatial pattern that shows a marked similarity with the corresponding simulated data plots from Figs. 7B and D of Carlson (2014). Several characteristics of the spatial distribution of differential response power are notable: 1. The differential response power is roughly 3 orders of magnitude less than the overall response power (Fig. 4C) and is thus of the same order as the modulation of response power with grating orientation (Fig. 3A). 2. The spatial structure of the differential response power has an angular frequency of 8, characteristic of the filter bank, as opposed to the angular frequency of 10 in the stimuli. 3. The meridians of zero differential response power lie along the orientations of the filters. Each of these observations indicates that the difference between the response to spirals of opposite sense is a consequence of inadvertent orientation bias in the filter bank. These biases are introduced by the digital implementation of discrete orientation filters and are unrelated to the brain's processing of orientation. The analyses that we have presented demonstrate the utility of spirals in controlling for the radial bias and, contrary to the claim of Carlson (2014), the necessity of some other coarse- or fine-scale bias to account for decoding of spiral sense. Spirals remain sensitive to other identified coarse-scale biases, such as differences in the level of response to horizontal and vertical orientations (Mannion et al., 2010), which are likely to
222
C.W.G. Clifford, D.J. Mannion / NeuroImage 110 (2015) 219–222
Fig. 4. Consequences of orientation biases in filter bank for response to spirals. (A) Total response power summed over all filter orientations to a complex spiral stimulus. (B) Difference in total response power between spiral pairs of opposite sense. (C) Horizontal slice through total (magenta) and differential (cyan) response power. Location of slice is indicated by colored lines in (A) and (B). Responses are normalized to the maximum total response power and displayed on a logarithmic scale. Dotted lines show response powers differing by three orders of magnitude.
contribute to or explain the capacity to decode orientation from the multivariate pattern of fMRI activity in early visual cortex (Mannion, 2010; Alink et al., 2013; Freeman et al., 2013; Maloney, 2015).
Acknowledgment Colin Clifford is supported by Australian Research Council Future Fellowship FT110100150.
References Alink, A., Krugliak, A., Walther, A., Kriegeskorte, N., 2013. fMRI orientation decoding in V1 does not require global maps or globally coherent orientation stimuli. Front. Psychol. 4, 493. Boynton, G.M., 2005. Imaging orientation selectivity: decoding conscious perception in V1. Nat. Neurosci. 8, 541–542. Brouwer, G.J., Heeger, D.J., 2009. Decoding and reconstructing color from responses in human visual cortex. J. Neurosci. 29, 13992–14003. Carlson, T.A., 2014. Orientation decoding in human visual cortex: new insights from an unbiased perspective. J. Neurosci. 34 (24), 8373–8383. Clifford, C.W.G., Mannion, D.J., McDonald, J.S., 2009. Radial biases in the processing of motion and motion-defined contours by human visual cortex. J. Neurophysiol. 102, 2974–2981. Clifford, C.W.G., Mannion, D.J., Seymour, K.J., McDonald, J.S., Bartels, A., 2011. Are coarse-scale orientation maps really necessary for orientation decoding? J. Neurosci. (available at: http://www.jneurosci.org/content/31/13/4792/reply). Freeman, W., Adelson, E., 1991. The design and use of steerable filters. IEEE Trans. Pattern. Anal. Mach. Intell. 13, 891–906. Freeman, J., Brouwer, G.J., Heeger, D.J., Merriam, E.P., 2011. Orientation decoding depends on maps, not columns. J. Neurosci. 31, 4792–4804. Freeman, J., Heeger, D.J., Merriam, E.P., 2013. Coarse-scale biases for spirals and orientation in human visual cortex. J. Neurosci. 33, 19695–19703. Goddard, E., Mannion, D.J., McDonald, J.S., Solomon, S.G., Clifford, C.W.G., 2010. Combination of subcortical colour channels in human visual cortex. J. Vis. 10 (5), 1–17 (25). Haynes, J.D., Rees, G., 2005. Predicting the orientation of invisible stimuli from activity in human primary visual cortex. Nat. Neurosci. 8, 686–691. Kamitani, Y., Sawahata, Y., 2010. Spatial smoothing hurts localization but not information: pitfalls for brain mappers. Neuroimage 49, 1949–1952. Kamitani, Y., Tong, F., 2005. Decoding the visual and subjective contents of the human brain. Nat. Neurosci. 8, 679–685. Kamitani, Y., Tong, F., 2006. Decoding seen and attended motion directions from activity in the human visual cortex. Curr. Biol. 16, 1096–1102. Levick, W.R., Thibos, L.N., 1982. Analysis of orientation bias in cat retina. J. Physiol. 329, 243–261.
Logothetis, N.K., 2008. What we can do and what we cannot do with fMRI. Nature 453, 869–878. Maloney, R.T., 2015. The basis of orientation decoding in human primary visual cortex: fine- or coarse-scale biases? J. Neurophysiol. 113, 1–3. Mannion (2010) Orientation anisotropies in human visual cortex. Unpublished PhD Thesis, The University of Sydney. Mannion, D.J., McDonald, J.S., Clifford, C.W.G., 2009. Discrimination of the local orientation structure of spiral Glass patterns early in human visual cortex. Neuroimage 46, 511–515. Mannion, D.J., McDonald, J.S., Clifford, C.W.G., 2010. Orientation anisotropies in human visual cortex. J. Neurophysiol. 103, 3465–3471. Op de Beeck, H.P., 2010. Against hyperacuity in brain reading: spatial smoothing does not hurt multivariate fMRI analyses? Neuroimage 49, 1943–1948. Parkes, L., Marsman, J., Oxley, D., Goulermas, J., Wuerger, S., 2009. Multivoxel fMRI analysis of color tuning in human primary visual cortex. J. Vis. 9 (1), 1–13 (1). Perona, P., 1995. Deformable kernels for early vision. IEEE Trans. Pattern. Anal.Mach.Intell. 17, 488–499. Sasaki, Y., Rajimehr, R., Kim, B.W., Ekstrom, L.B., Vanduffel, W., Tootell, R.B., 2006. The radial bias: a different slant on visual orientation sensitivity in human and nonhuman primates. Neuron 51, 661–670. Schall, J.D., Perry, V.H., Leventhal, A.G., 1986. Retinal ganglion cell dendritic fields in oldworld monkeys are oriented radially. Brain Res. 368, 18–23. Seymour, K.J., Clifford, C.W.G., Logothetis, N.K., Bartels, A., 2009. The coding of color, motion and their conjunction in the human visual cortex. Curr. Biol. 19, 177–183. Seymour, K.J., Clifford, C.W.G., Logothetis, N.K., Bartels, A., 2010. Coding and binding of color and form in visual cortex. Cereb. Cortex 20, 1946–1954. Shou, T.D., Leventhal, A.G., 1989. Organized arrangement of orientation-sensitive relay cells in the cat's dorsal lateral geniculate nucleus. J. Neurosci. 9, 4287–4302. Smith 3rd, E.L., Chino, Y.M., Ridder 3rd, W.H., Kitagawa, K., Langston, A., 1990. Orientation bias of neurons in the lateral geniculate nucleus of macaque monkeys. Vis. Neurosci. 5, 525–545. Sumner, P., Anderson, E.J., Sylvester, R., Haynes, J.D., Rees, G., 2008. Combined orientation and colour information in human V1 for both L–M and S-cone chromatic axes. Neuroimage 39, 814–824. Swisher, J.D., Gatenby, J.G., Gore, J.C., Wolfe, B.A., Moon, C.-H., Kim, S.-G., Tong, F., 2010. Multiscale pattern analysis of orientation-selective activity in the primary visual cortex. J. Neurosci. 30, 325–330. Vanduffel, W., Tootell, R.B., Schoups, A.A., Orban, G.A., 2002. The organization of orientation selectivity throughout macaque visual cortex. Cereb. Cortex 12, 647–662. Xiao, Y., Wang, Y., Felleman, D.J., 2003. A spatially organized representation of colour in macaque cortical area V2. Nature 421, 535–539. Xiao, Y., Casti, A., Xiao, J., Kaplan, E., 2007. Hue maps in primate striate cortex. Neuroimage 35, 771–786. Yacoub, E., Harel, N., Ugurbil, K., 2008. High-field fMRI unveils orientation columns in humans. Proc. Natl. Acad. Sci. U. S. A. 105, 10607–10612. Zhang, X., Qiu, J., Zhang, Y., Han, S., Fang, F., 2014. Misbinding of color and motion in human visual cortex. Curr. Biol. 16, 1354–1360.
Contents lists available at ScienceDirect
NeuroImage journal homepage: www.elsevier.com/locate/ynimg
Comments and Controversies
Orientation decoding: Sense in spirals? Colin W.G. Clifford ⁎, Damien J. Mannion School of Psychology, UNSW Australia, Sydney, NSW, Australia
a r t i c l e
i n f o
Article history: Accepted 19 December 2014 Available online 27 December 2014 Keywords: fMRI Visual cortex Computational neuroimaging Multivariate analysis Spatial vision
a b s t r a c t The orientation of a visual stimulus can be successfully decoded from the multivariate pattern of fMRI activity in human visual cortex. Whether this capacity requires coarse-scale orientation biases is controversial. We and others have advocated the use of spiral stimuli to eliminate a potential coarse-scale bias—the radial bias toward local orientations that are collinear with the centre of gaze—and hence narrow down the potential coarse-scale biases that could contribute to orientation decoding. The usefulness of this strategy is challenged by the computational simulations of Carlson (2014), who reported the ability to successfully decode spirals of opposite sense (opening clockwise or counter-clockwise) from the pooled output of purportedly unbiased orientation filters. Here, we elaborate the mathematical relationship between spirals of opposite sense to confirm that they cannot be discriminated on the basis of the pooled output of unbiased or radially biased orientation filters. We then demonstrate that Carlson's (2014) reported decoding ability is consistent with the presence of inadvertent biases in the set of orientation filters; biases introduced by their digital implementation and unrelated to the brain's processing of orientation. These analyses demonstrate that spirals must be processed with an orientation bias other than the radial bias for successful decoding of spiral sense. © 2014 Elsevier Inc. All rights reserved.
Functional magnetic resonance imaging (fMRI) of the blood oxygenation level-dependent (BOLD) signal allows us to observe patterns of activity in the brain (Logothetis, 2008). In human early visual cortex, these activity patterns convey information about characteristics of images presented to the subject such as their form, color, and motion (Kamitani and Tong, 2005, 2006; Haynes and Rees, 2005; Mannion et al., 2009; Freeman et al., 2011, 2013; Alink et al., 2013; Brouwer and Heeger, 2009; Parkes et al., 2009; Goddard et al., 2010), as well as to conjunctions of these attributes (Sumner et al., 2008; Seymour et al., 2009, 2010; Zhang et al., 2014). However, the source of the information in these activity patterns remains a subject of ongoing debate (Op de Beeck, 2010; Kamitani and Sawahata, 2010; Clifford et al., 2011; Freeman et al., 2011, 2013; Alink et al., 2013). Data acquired during fMRI are represented in spatial samples (voxels) whose volume is generally of the order of a few cubic millimeters. This spatial resolution is rather coarser than the scale at which features such as orientation and color have been observed to be mapped onto the surface of mammalian visual cortex (Vanduffel et al., 2002; Yacoub et al., 2008; Xiao et al., 2003, 2007). It has been suggested that information about features such as orientation is nonetheless available at the spatial scale of voxels by virtue of random variability in the spatial distribution of these fine-scale feature maps or their supporting
⁎ Corresponding author at: School of Psychology, UNSW Australia, Sydney, NSW 2052, Australia. E-mail address: [email protected] (C.W.G. Clifford).
http://dx.doi.org/10.1016/j.neuroimage.2014.12.055 1053-8119/© 2014 Elsevier Inc. All rights reserved.
vasculature (Boynton, 2005; Kamitani and Tong, 2005, 2006; Haynes and Rees, 2005; Swisher et al., 2010). However, it has also been claimed that only much coarser scale biases in the way orientation is mapped to the cortical surface allow information about oriented image structure to be recovered from patterns of activity in early visual areas (Freeman et al., 2011, 2013). One such coarse-scale bias that has been observed in the response of early visual cortex to orientation is radial bias: a preference for orientations radial to the point of fixation. Radial bias has been observed in the topographic representations of the visual field that characterize the early areas of visual cortex (Sasaki et al., 2006; Clifford et al., 2009; Mannion et al., 2010; Freeman et al., 2011, 2013), although it is likely that a component of it is inherited from earlier in the visual pathway (Levick and Thibos, 1982; Schall et al., 1986; Shou and Leventhal, 1989; Smith et al., 1990). When subjects are presented with a widefield, obliquely oriented grating, for example, the response in the two quadrants of the visual field where the orientation is close to radial is typically larger than the response to the opposite oblique orientation (Sasaki et al., 2006). Thus, simply observing the pattern of activity across the four quadrants of early visual areas would be sufficient to discriminate which of two oblique orientations is being viewed by the subject (Fig. 1). To remove the confounding effect of radial bias on information about spatial image structure, several studies have used spirals as stimuli (Mannion et al., 2009; Seymour et al., 2010; Alink et al., 2013; Freeman et al., 2013). The logic of this manipulation is that, in a pair of
220
C.W.G. Clifford, D.J. Mannion / NeuroImage 110 (2015) 219–222
Fig. 1. Local orientation structure of oblique gratings relative to fixation. (A) For a centrally fixating observer viewing an oblique grating, two diagonally opposite quadrants of the visual field (outlined in white) contain predominantly radial orientations while the orientations in the other two quadrants (outlined in black) are predominantly tangential. (B) This configuration is reversed for an orthogonal grating.
Then, for any meridian orientation θ0:
spirals of opposite sense, the local orientation at corresponding points in the two images is always at equal but opposite angles from radial (Fig. 2A). Formally, let I+ and I− be two similar spirals of opposite sense:
I− ðr; θ−θ0 Þ ¼ expði ðkr r−kθ ðθ−θ0 Þ þ φ0 ÞÞ W ðr Þ
ð3Þ
Iþ ðr; θÞ ¼ expði ðkr r þ kθ θ þ φ0 ÞÞ W ðr Þ
ð1Þ
I− ðr; θ−θ0 Þ ¼ expði ðkr r þ kθ ðθ0 − θÞ þ φ0 ÞÞ W ðr Þ
ð4Þ
I− ðr; θÞ ¼ expði ðkr r−kθ θ þ φ0 ÞÞ W ðr Þ
ð2Þ
I− ðr; θ−θ0 Þ ¼ Iþ ðr; θ0 − θÞ
ð5Þ
where kr and kθ are the radial and angular frequency, respectively, of the spiral, W(r) is any circularly symmetric stimulus window, and φ0 determines the spatial phase.
Thus, spirals of opposite sense are mirror images of one another over any meridian (Fig. 2B) and a bias to response more strongly to radial orientations will provide no information upon which to discriminate which of two opposite spirals is being viewed. It is also evident that a
Fig. 2. Symmetry between spirals of opposite sense. (A) In a pair of spirals of opposite sense, local image orientation at corresponding points in the two images is always at equal but opposite angles from radial. (B) Spirals of opposite sense are mirror images of one another over any meridian.
C.W.G. Clifford, D.J. Mannion / NeuroImage 110 (2015) 219–222
221
response that is completely invariant over orientation will be similarly uninformative with regard to spiral sense. Despite this, Carlson (2014) has recently claimed that spirals of opposite sense can be discriminated from a representation of local image structure that is unbiased with respect to orientation. On this basis, it was argued that “there is no need to posit a bias either at fine grain or coarse-scale representation to account for decoding spiral sense” (Carlson, 2014). Here, we refute this claim. Specifically, we demonstrate that Carlson's (2014) reported decoding ability is consistent with biases introduced by the digital implementation of discrete orientation filters. Such biases stem from the inevitable loss of rotational invariance when sampling a continuous function on a discrete grid of pixels (Perona, 1995) and are unrelated to the brain's processing of orientation.1 In his computational analysis, Carlson (2014) convolved stimulus images with banks of oriented Gabor filters. However, digital implementation of banks of Gabor filters inevitably results in a small but systematic bias in response power as a function of orientation (Fig. 3A). Specifically, although response power is equal at the orientation of each filter, it varies systematically across intermediate orientations (Freeman and Adelson, 1991; Perona, 1995). This is a problem for the analysis of spiral stimuli, as across the image they contain structure at all orientations. Formally, let P(⊖) be the response power of the filter bank in response to orientation ⊖. Then, for the response to spirals of opposite sense to be equal: P ðθ−θ0 Þ ¼ P ðθ0 − θÞ
ð6Þ
For this requirement to hold over the whole stimulus, it must apply for all meridian orientations, θ0: P ðθ−θ0 Þ ¼ P ðθ0 −θÞ; ∀θ0
ð7Þ
i:e:; P ðθÞ ¼ const
ð8Þ
Thus, for the bank of filters to be effectively unbiased in its response to spiral stimuli, it is necessary for its response power to be equal across all orientations, not just at the peak orientations of the individual filters (Fig. 3B, C). For the illustration of orientation bias in response to gratings in Fig. 3A, we used a bank of 8 Gabor filters with orientations of 0°, 22.5°, 45°, 67.5°, 90°, 112.5°, 135°, and 157.5°, a sine-phase carrier with a frequency of 0.025 cycles per pixel and a two-dimensional Gaussian envelope with σ = 20 pixels, implemented in a 121 × 121 pixel array. The spatial frequency of the grating stimuli was 0.025 cycles per pixel, matching the peak tuning of the filters. The response power of the filter bank varies systematically with the orientation of the stimulus grating. Specifically, power is maximal at the orientation of each of the 8 filters and minimal midway between filters (Fig. 3A). Thus, the response profile is invariant to reflection only about the preferred orientation of any one of the filters (Fig. 3B) and not to reflection about any intermediate orientation (Fig. 3C). Since spirals of opposite sense are mirror images of one another over any meridian (Fig. 2B), it follows that the difference in response power to spirals of opposite sense will only be zero along the meridians through the image corresponding to the orientations of the filters. To illustrate the consequences of orientation bias in the filter bank on the response to spirals, we convolved the same set of filters with complex spiral stimuli defined according to Eqs. (1) and (2) with radial frequency, kr = 0.1 cycles per pixel, and angular frequency, kθ = 10, in 1 Our critique applies specifically to the decoding of spiral sense. Carlson (2014) also demonstrated that the response to oriented gratings at the edge of the stimulus aperture could masquerade as radial bias. We believe that Carlson's analysis of the response to oriented gratings is sound in theory, although empirically our (Clifford et al., 2009) and others' retinotopic data (e.g. Sasaki et al., 2006) indicate that radial bias is not restricted to the edge of the aperture.
Fig. 3. Bias arising from digital implementation of filters. (A) The overall power of a bank of oriented Gabor filters convolved with a sinusoidal grating inevitably shows a small but systematic bias with the orientation of the grating. Here, response power is maximal at the orientation of each filter and minimal midway between filters. The response of the filter bank as a function of stimulus orientation is (B) invariant to reflection about the preferred orientation of the any one of the filters but (C) not to reflection about any intermediate orientation. Thus, the difference in response power to spirals of opposite sense will only be zero along the meridians through the image corresponding to the orientations of the filters.
an annular contrast window with raised cosine edges (Fig. 2A). The total response power of the filter bank (summed over all filter orientations) to a complex spiral stimulus shows a spatial distribution that essentially mimics the contrast window in which the stimulus is presented (Fig. 4A). The difference in total response power between spiral pairs of opposite sense (Fig. 4B) follows a distinctive spatial pattern that shows a marked similarity with the corresponding simulated data plots from Figs. 7B and D of Carlson (2014). Several characteristics of the spatial distribution of differential response power are notable: 1. The differential response power is roughly 3 orders of magnitude less than the overall response power (Fig. 4C) and is thus of the same order as the modulation of response power with grating orientation (Fig. 3A). 2. The spatial structure of the differential response power has an angular frequency of 8, characteristic of the filter bank, as opposed to the angular frequency of 10 in the stimuli. 3. The meridians of zero differential response power lie along the orientations of the filters. Each of these observations indicates that the difference between the response to spirals of opposite sense is a consequence of inadvertent orientation bias in the filter bank. These biases are introduced by the digital implementation of discrete orientation filters and are unrelated to the brain's processing of orientation. The analyses that we have presented demonstrate the utility of spirals in controlling for the radial bias and, contrary to the claim of Carlson (2014), the necessity of some other coarse- or fine-scale bias to account for decoding of spiral sense. Spirals remain sensitive to other identified coarse-scale biases, such as differences in the level of response to horizontal and vertical orientations (Mannion et al., 2010), which are likely to
222
C.W.G. Clifford, D.J. Mannion / NeuroImage 110 (2015) 219–222
Fig. 4. Consequences of orientation biases in filter bank for response to spirals. (A) Total response power summed over all filter orientations to a complex spiral stimulus. (B) Difference in total response power between spiral pairs of opposite sense. (C) Horizontal slice through total (magenta) and differential (cyan) response power. Location of slice is indicated by colored lines in (A) and (B). Responses are normalized to the maximum total response power and displayed on a logarithmic scale. Dotted lines show response powers differing by three orders of magnitude.
contribute to or explain the capacity to decode orientation from the multivariate pattern of fMRI activity in early visual cortex (Mannion, 2010; Alink et al., 2013; Freeman et al., 2013; Maloney, 2015).
Acknowledgment Colin Clifford is supported by Australian Research Council Future Fellowship FT110100150.
References Alink, A., Krugliak, A., Walther, A., Kriegeskorte, N., 2013. fMRI orientation decoding in V1 does not require global maps or globally coherent orientation stimuli. Front. Psychol. 4, 493. Boynton, G.M., 2005. Imaging orientation selectivity: decoding conscious perception in V1. Nat. Neurosci. 8, 541–542. Brouwer, G.J., Heeger, D.J., 2009. Decoding and reconstructing color from responses in human visual cortex. J. Neurosci. 29, 13992–14003. Carlson, T.A., 2014. Orientation decoding in human visual cortex: new insights from an unbiased perspective. J. Neurosci. 34 (24), 8373–8383. Clifford, C.W.G., Mannion, D.J., McDonald, J.S., 2009. Radial biases in the processing of motion and motion-defined contours by human visual cortex. J. Neurophysiol. 102, 2974–2981. Clifford, C.W.G., Mannion, D.J., Seymour, K.J., McDonald, J.S., Bartels, A., 2011. Are coarse-scale orientation maps really necessary for orientation decoding? J. Neurosci. (available at: http://www.jneurosci.org/content/31/13/4792/reply). Freeman, W., Adelson, E., 1991. The design and use of steerable filters. IEEE Trans. Pattern. Anal. Mach. Intell. 13, 891–906. Freeman, J., Brouwer, G.J., Heeger, D.J., Merriam, E.P., 2011. Orientation decoding depends on maps, not columns. J. Neurosci. 31, 4792–4804. Freeman, J., Heeger, D.J., Merriam, E.P., 2013. Coarse-scale biases for spirals and orientation in human visual cortex. J. Neurosci. 33, 19695–19703. Goddard, E., Mannion, D.J., McDonald, J.S., Solomon, S.G., Clifford, C.W.G., 2010. Combination of subcortical colour channels in human visual cortex. J. Vis. 10 (5), 1–17 (25). Haynes, J.D., Rees, G., 2005. Predicting the orientation of invisible stimuli from activity in human primary visual cortex. Nat. Neurosci. 8, 686–691. Kamitani, Y., Sawahata, Y., 2010. Spatial smoothing hurts localization but not information: pitfalls for brain mappers. Neuroimage 49, 1949–1952. Kamitani, Y., Tong, F., 2005. Decoding the visual and subjective contents of the human brain. Nat. Neurosci. 8, 679–685. Kamitani, Y., Tong, F., 2006. Decoding seen and attended motion directions from activity in the human visual cortex. Curr. Biol. 16, 1096–1102. Levick, W.R., Thibos, L.N., 1982. Analysis of orientation bias in cat retina. J. Physiol. 329, 243–261.
Logothetis, N.K., 2008. What we can do and what we cannot do with fMRI. Nature 453, 869–878. Maloney, R.T., 2015. The basis of orientation decoding in human primary visual cortex: fine- or coarse-scale biases? J. Neurophysiol. 113, 1–3. Mannion (2010) Orientation anisotropies in human visual cortex. Unpublished PhD Thesis, The University of Sydney. Mannion, D.J., McDonald, J.S., Clifford, C.W.G., 2009. Discrimination of the local orientation structure of spiral Glass patterns early in human visual cortex. Neuroimage 46, 511–515. Mannion, D.J., McDonald, J.S., Clifford, C.W.G., 2010. Orientation anisotropies in human visual cortex. J. Neurophysiol. 103, 3465–3471. Op de Beeck, H.P., 2010. Against hyperacuity in brain reading: spatial smoothing does not hurt multivariate fMRI analyses? Neuroimage 49, 1943–1948. Parkes, L., Marsman, J., Oxley, D., Goulermas, J., Wuerger, S., 2009. Multivoxel fMRI analysis of color tuning in human primary visual cortex. J. Vis. 9 (1), 1–13 (1). Perona, P., 1995. Deformable kernels for early vision. IEEE Trans. Pattern. Anal.Mach.Intell. 17, 488–499. Sasaki, Y., Rajimehr, R., Kim, B.W., Ekstrom, L.B., Vanduffel, W., Tootell, R.B., 2006. The radial bias: a different slant on visual orientation sensitivity in human and nonhuman primates. Neuron 51, 661–670. Schall, J.D., Perry, V.H., Leventhal, A.G., 1986. Retinal ganglion cell dendritic fields in oldworld monkeys are oriented radially. Brain Res. 368, 18–23. Seymour, K.J., Clifford, C.W.G., Logothetis, N.K., Bartels, A., 2009. The coding of color, motion and their conjunction in the human visual cortex. Curr. Biol. 19, 177–183. Seymour, K.J., Clifford, C.W.G., Logothetis, N.K., Bartels, A., 2010. Coding and binding of color and form in visual cortex. Cereb. Cortex 20, 1946–1954. Shou, T.D., Leventhal, A.G., 1989. Organized arrangement of orientation-sensitive relay cells in the cat's dorsal lateral geniculate nucleus. J. Neurosci. 9, 4287–4302. Smith 3rd, E.L., Chino, Y.M., Ridder 3rd, W.H., Kitagawa, K., Langston, A., 1990. Orientation bias of neurons in the lateral geniculate nucleus of macaque monkeys. Vis. Neurosci. 5, 525–545. Sumner, P., Anderson, E.J., Sylvester, R., Haynes, J.D., Rees, G., 2008. Combined orientation and colour information in human V1 for both L–M and S-cone chromatic axes. Neuroimage 39, 814–824. Swisher, J.D., Gatenby, J.G., Gore, J.C., Wolfe, B.A., Moon, C.-H., Kim, S.-G., Tong, F., 2010. Multiscale pattern analysis of orientation-selective activity in the primary visual cortex. J. Neurosci. 30, 325–330. Vanduffel, W., Tootell, R.B., Schoups, A.A., Orban, G.A., 2002. The organization of orientation selectivity throughout macaque visual cortex. Cereb. Cortex 12, 647–662. Xiao, Y., Wang, Y., Felleman, D.J., 2003. A spatially organized representation of colour in macaque cortical area V2. Nature 421, 535–539. Xiao, Y., Casti, A., Xiao, J., Kaplan, E., 2007. Hue maps in primate striate cortex. Neuroimage 35, 771–786. Yacoub, E., Harel, N., Ugurbil, K., 2008. High-field fMRI unveils orientation columns in humans. Proc. Natl. Acad. Sci. U. S. A. 105, 10607–10612. Zhang, X., Qiu, J., Zhang, Y., Han, S., Fang, F., 2014. Misbinding of color and motion in human visual cortex. Curr. Biol. 16, 1354–1360.
