Representation of haptic objects during mental rotation in congenital blindness.

Perceptual & Motor Skills: Physical Development & Measurement 2014, 118, 2, 587-607. © Perceptual & Motor Skills 2014

REPRESENTATION OF HAPTIC OBJECTS DURING MENTAL ROTATION IN CONGENITAL BLINDNESS1, 2 BURAK GÜÇLÜ AND SERKAN ÇELİK

CİVAN ILCİ

Institute of Biomedical Engineering

Center for Visual Disabilities

Boğaziçi University, Turkey Summary.—The representation of haptic objects by three groups of participants (sighted, blindfolded, and congenitally blind) was studied in a mental-rotation task. Three models were tested. The participants explored a standard object continuously with the left hand and tried to find the mirror object among two alternatives explored sequentially with the right hand. Sighted participants were tested in the visual version of the task. The accuracy of judgments was very high (>95%) for all groups, and the blind group had the highest identification times. Correlation analyses were performed between (both single-trial and average) identification times and angular differences. The identification times of the sighted and blindfolded groups increased as linear functions of the angular difference between the mirror and the standard stimuli, supporting the classical model. The identification times of the blind group changed non-monotonically and were consistent with an antiparallel image (180⬚ rotation superimposed) in the mental representation. The dual code model did not fit the data well for any participant group. The performance differences between the blindfolded and blind groups may be attributed to a modified mapping function from the object-properties-processing sub-system to the visual buffer, which was conjectured to be available also to the blind group while processing haptic objects.

Mental rotation is the rotational transformation of an object's mental representation. The identification time measured for this cognitive process is usually a linear function of the angular orientation difference between the final and initial mental representations. Since its discovery by Shepard and Metzler (1971), mental rotation has been studied primarily in vision and formed the basis for the analog-representation theory in visual cognition. More recent theories have elaborated this theory by showing that an object can be stored in multiple views and mental rotation occurs between the observed image and the nearest stored representation (Tarr & Pinker, 1989). According to the lively debate on the neural correlates of visual mental imagery (Kosslyn, Thompson, & Ganis, 2006), visual perception and visual imagery may share similar cortical structures and mechanisms. However, clinical cases which suggest double dissociation showed that Address correspondence to Burak Güçlü, Ph.D., Institute of Biomedical Engineering, Boğaziçi University, Kandilli Campus, Çengelköy 34684 İstanbul, Turkey or e-mail (burak.guclu@boun. edu.tr). 2 This work was supported by the Boğaziçi University Research Fund (13XP8) granted to Dr. Güçlü. We thank Lale Akarun, Hale Saybaşili, Reşit Canbeyli, and Bart Farell for helpful comments. 1

DOI 10.2466/15.22.PMS.118k20w0

20-PMS_Guclu_130143.indd 587

ISSN 0031-5125

25/03/14 8:52 PM

588

B. GÜÇLÜ, ET AL.

these similarities should be interpreted with caution (Bartolomeo, 2008). The conceptual model put forth by Kosslyn and colleagues allows one to formulate new hypotheses regarding information-processing sub-systems to modify, refine, and unify the theories supported by different research groups (Kosslyn, et al., 2006). Perhaps, the most critical tests for models on visual imagery come from studies performed with visually impaired individuals (Dulin, Hatwell, Pylyshyn, & Chokron, 2008). It was shown by numerous researchers that congenitally blind people can form spatial images and mentally manipulate them, albeit with some limitations; e.g., Arditi, Holtzman, & Kosslyn (1988) found that congenitally blind participants do not mentally image larger objects as appreciably farther away as the sighted do, probably due to previous haptic experience which only allows for objects within arms' reach. Additionally, blind people fail to show a decrease in pointing span with increasing mental image distance; therefore, they violate the laws of perspective. Similarly, during the recall of nouns with high “visual” content, they have difficulty in forming composite images (De Beni & Cornoldi, 1988). This is attributed to the fact that blind people are predominantly subject to haptic and auditory inputs that are typically processed in a serial manner. Sighted and blind people can produce and understand haptic pictures (Heller, 2002). Haptic object images can be mentally rotated, with greatest correlated activity appearing in the parietal cortex (Prather & Sathian, 2002). Interestingly, it was found that “visual” areas (e.g., V1, V2) are also recruited by blind participants for performing haptic spatial tasks (Hamilton & Pascual-Leone, 1998; Merabet, Thut, Murray, Andrews, Hsiao, & Pascual-Leone, 2004), and this also seems to occur after de-afferentation in newly blind people (Sadato, Okada, Kubota, & Yonekura, 2004). These results supported an image generation mechanism in which mental images can be spatial and depictive (Kosslyn, et al., 2006). Spatial images are formed based on an object map consisting of locations of objects and their parts. Depictive images, on the other hand, originate by an activation pattern in the visual buffer. Both types of images can be formed based on information from associative memory as well as by the real-time flow of sensory inputs. It is assumed that after an initial somatosensory processing, mental images formed by haptic inputs may also be subserved by the mechanisms and the neural structures involved with visual mental imagery. This may be why people with congenital blindness perform differently in mental imagery tasks. The goal in this study was to explore the representation of haptic objects by sighted, blindfolded, and congenitally blind groups in a mental-rotation task. The main contribution of the study is that three alternative models/hypotheses were tested based on the information-processing sub-systems of Kosslyn, et al. (2006). This approach yielded a novel finding about mental imagery in the blind group.


25/03/14 8:52 PM

REPRESENTATION OF HAPTIC OBJECTS

589

The results of Klein, Dubois, Mangin, Kherif, Flandin, Poline, Denis, et al. (2004) suggested that the activation pattern in the visual buffer is altered with a change of orientation in imagery during a mental-rotation task. This implies that the mapping function from the object-propertiesprocessing sub-system to the visual buffer is continuously modulated, which was hypothesized here to be specifically different in the congenitally blind group because of the visual deficit. Although an object map may be formed based on haptic inputs, it was expected that the blind group would use a different strategy in the mental-rotation task compared to the sighted group, according to the modified mapping function. If the sighted and blind groups were to perform similarly, the results would be consistent with the classical model by Shepard and Metzler (1971) explained in Fig. 1A. The first panel in Fig. 1A is the standard stimulus (Symbol 1) presented at an angle of α in the given reference frame (e.g., 20º in the clockwise direction with the upper vertical). The middle panel is the mental image of the stimulus in the same reference frame. In the third panel, Symbol 2 is the mental image of the comparison stimulus (e.g., at 150º in the given reference frame). The image of the comparison stimulus has to be mentally rotated in the nearest direction for an angle of β to match the image in the middle panel. Therefore, β = |150º – 20º| = 130º. Since the data from the blind group did not conform to this model, an alternative mapping function was verified from the haptic input to the mental image, which includes an antiparallel (180º rotation) part superimposed on the original image (Fig. 1B). Although the first panel in Fig. 1B is identical to that of Fig. 1A, the mental image in the middle panel is different; it now has two short limbs. The third panel in Fig. 1B shows how the image of the comparison stimulus (Symbol 2) is mentally rotated toward the short limb in the nearest direction for an angle of β. Since the additional short limb of the antiparallel modified image is at 200º in the given reference frame (20º + 180º = 200º), the rotation angle becomes β = |150º – 200º| = 50º, very different from the prediction of the first model. Corballis and McLaren (1984) showed that a mental-rotation strategy exists for up-down mirror images as well as left-right mirror images. The modified image enables faster comparison of both types of mirror images by reducing the rotation angle (to the nearest short limb in the given example). Having established support for the antiparallel image hypothesis for the blind group, it was of interest whether the representation of angle had a propositional format for that group (Kosslyn, et al., 2006; Dulin, et al., 2008). This hypothesis implies a dual code such that angles would be first extracted with respect to a template mental image from memory in the given reference frame (i.e., as a depictive representation). The template image was assumed to be upright and to have the antiparallel form


25/03/14 8:52 PM

590

B. GÜÇLÜ, ET AL.

(A)

(B)

(C)

FIG. 1. Mental rotation models tested in this article. (A) The classical model: the standard stimulus (1) is presented at an angle α in the upright reference frame. The gray object is the postulated image of the mental representation of the standard stimulus. The representation of the comparison stimulus (2) is rotated through angle β toward the mental representation of the standard in the nearest direction. (B) The antiparallel model: the representation of the standard stimulus also includes an antiparallel (rotated 180⬚) representation of the input superimposed on the postulated image. The representation of the comparison stimulus (2) also has this form and is rotated through angle β towards the representation of the standard in the nearest direction. (C) The dual code model: first, the angle information for the standard (1) and the comparison (2) stimuli is extracted with respect to a template representation in the antiparallel depictive format. Then, the mental rotation takes effect non-pictorially by rotating the representation of (2) to that of (1) through angle β regardless of a frame of reference. These angles (and their difference) are assumed to be in propositional format, and are therefore shown at random, i.e., unspecified, orientations in the last panel.


25/03/14 8:52 PM


591

(see the first and middle panels in Fig. 1C). Next, the rotation would be performed without a reference frame (see the randomly oriented stimulus representations in the third panel of Fig. 1C) in an abstract, non-pictorial process based on the propositional format for angles and their difference. To explain this hypothesis quantitatively, Symbol 1 in Fig. 1C is the standard stimulus, and Symbol 2 is the comparison stimulus. Following the above models, the standard stimulus is at an angle of α = 20º, and the comparison stimulus is at an angle of 30º (180º –150º = 30º) with the template image in the given reference frame. Then, another result is found: β = |30º – 20º| = 10º. All three models described so far have the same standard and comparison stimuli but require different rotation angles, and thus produce different identification times. It is interesting to note that the classical model presented in Fig. 1A could not be re-tested for the propositional format, because there is no antiparallel image (which generates another angular zero reference) in that model and the same identification times would be obtained if tested. Marmor and Zaback (1976) used a same–different judgment task in which the participants compared a pair of haptic objects that differed by a rotation but were either identical (same) when rotated or mirror-imaged (different). The identification times for both blind and sighted groups increased as a linear function of the angular discrepancy between stimuli. A congenitally blind group performed the task more slowly than a lateblind group. The blindfolded sighted group was the fastest. On the other hand, Dellantonio and Spagnolo (1990) found contradictory results for a sighted sample. In two same–different judgment tasks, one with entirely different haptic objects and one with mirror objects, they found no effect of the size of the angular difference between simultaneously presented objects. They found a rotation effect only for a single-stimulus task in which a stimulus was judged to be the same or the mirror image of a pre-learned stimulus. These results suggested that orientation effects in haptic mental rotation may be dependent on the particular task. An additional novelty of the current study was to use a task that minimized the influence of participants' criterion from haptic mirror-image discrimination. The task presented here had the standard stimulus continuously explored while the mirror image is found among two freely explored alternatives. METHOD Participants There were three participant groups: sighted (n = 7; 3 women, 4 men; age range = 24–28 yr.), blindfolded sighted (n = 7; 3 women, 4 men; age range = 23–27 yr.), and congenitally blind (n = 6; 2 women, 4 men; age range = 18– 36 yr.). All participants were right-handed according to the Edinburgh Inventory (Oldfield, 1971). The causes of blindness were retinopathy for four


25/03/14 8:52 PM

592

B. GÜÇLÜ, ET AL.

participants, glaucoma for one participant, and retinoblastoma for one participant. Four of the participants in the blind group had some light perception, but none could perceive any visual form since birth. The experiment adhered to the tenets of the Declaration of Helsinki for testing human participants, and an informed consent was obtained for each participant. Materials The stimuli were wooden L-shaped haptic objects glued on 125 rectangular cardboard bases (top panel in Fig. 2). The longer limb of the Lshape was 5 cm, and the shorter limb was 3 cm. The thickness of the objects and their depth, i.e., their relief with respect to the base, were 1 cm. The experimenter placed the cardboard bases in a horizontal frame on the testing table to prevent movement. The participant sat at the table with the mid-sagittal plane of his or her head perpendicular to the horizontal frame axis at its center. Therefore, the participant's head always faced the front, because the reference frame for haptic stimuli is important (Prather & Sathian, 2002). Each rectangular base had three objects. The leftmost object was the standard object. The two objects at the right of the standard object were the normal and mirror objects. Each object had an orientation angle randomly selected from five alternatives (0, 45, 90, 135, or 180⬚). These orientations were measured with respect to the axis passing through the object's center and parallel to the short side of the base (see Fig. 1). The normal object was just a rotated version of the standard object. The mirror object was the reflected version of the standard object about the plane perpendicular to the cardboard base and parallel to the object's long axis (see Fig. 2). Procedure The experiment had a factorial design with the identification time as the dependent variable. The factors were the participant group (sighted and eyes open, sighted and blindfolded, and congenitally blind), the rotation of the standard object, and the rotation of the mirror object. The participant groups were tested in three blocks. Each participant was first trained on a subset of stimuli until he or she reached a constant accuracy in order to reduce learning effects (Tarr & Pinker, 1989). The blind participants were taught about the mirror stimuli by explaining that the objects which are mirror images of each other can never match with rotations on the haptic-image plane. The triplets of stimuli were randomly presented by the experimenter one cardboard at a time. The participant's task was to find the mirror stimulus and call out its location as soon as a judgment is made, i.e., the second or the third as in Fig. 2. The Sighted, unblindfolded group performed the task visually to set a baseline. The Blindfolded sighted and Congenitally


25/03/14 8:52 PM


593

FIG. 2. Haptic objects and the exploration procedure. L-shaped objects were presented on a rectangular cardboard (top panel). The standard stimulus was always explored by the left hand. The normal (rotated version of the standard) and mirror (mirror-image version of the standard) stimuli were explored by the right hand. The order of the normal and mirror stimuli was randomized. The stimuli randomly appeared in one of five rotated orientations (0, 45, 90, 135, or 180º in the reference frame of Fig. 1A). During exploration, the left hand was always kept on the standard stimulus (middle and bottom panels). The participants were free to alternate exploring the normal and mirror stimuli with their right hands as many times as they wished, but each stimulus had to be explored at least once. The exploration procedure (and the stopwatch) started when a participant first touched the objects by using both hands as shown. The experimenter did not control whether the right hand initially touched the middle stimulus (middle panel) or the rightmost stimulus (bottom panel), and bimanual exploration was not allowed for a given stimulus object.

blind groups performed the task haptically. The haptically tested participants were required to explore the standard stimulus with their left hands continuously during the task and to explore the normal or mirror stimuli freely but at least once with their right hands. They were allowed to switch between the normal and mirror stimuli as many times as they wished before calling out their judgments. This prevented a confounding


25/03/14 8:52 PM

594

B. GÜÇLÜ, ET AL.

effect due to the participant's judgment criterion (see the Discussion section), but required approximately constant interhemispheric transfer. The placements of the normal and mirror stimuli were randomized and counterbalanced. As such, any bias which could occur depending on where the participants began exploring the items with their right hands was minimized (see Fig. 2 and the Discussion section). Because this exploration was always sequential, the sighted group was initially instructed to perform the visual task similarly in a sequential manner. In other words, they focused on the normal and mirror stimuli separately. However, the experimenter could not guarantee against simultaneous visual inspection. The judgments and the identification times were recorded by the experimenter. The identification times were measured by using a digital stopwatch with 0.01 sec. resolution. The stopwatch started as soon as a participant first touched the objects with both hands as shown in Fig. 2 (or as the stimuli were presented visually to the sighted participants) and stopped with the voice of the participant. The participants were free to start with the right hand on the middle stimulus (middle panel in Fig. 2) or the rightmost stimulus (bottom panel in Fig. 2). However, bimanual exploration was not allowed for a given object. The participants were debriefed after the experiment. Analysis For each orientation angle condition, accuracy percentage was found among the participants in a given group, and the average identification time was calculated from only the correct judgments. Pearson correlations between the accuracies and the identification times (both average and single-trial) were reported. Three-way analyses of variance (ANOVAs) were performed to assess the effect of the experimental factors on accuracy and identification time. After ANOVA, multiple comparisons of the means were performed using the Tukey–Kramer criterion to test relationships between the factors. The 95% confidence intervals (CI) were reported in these post hoc analyses for differences between means. Best fit lines were calculated to study the time course of the mental-rotation process. All analyses were performed in MATLAB Version 7.6 (The MathWorks, Inc., Natick, MA). The analyses were repeated as appropriate for the hypothetical models presented in Fig. 1. RESULTS Summary of Factor Effects The accuracy of judgments was very high for every participant group. The percentages of accuracy averaged across all conditions are plotted in Fig. 3A. The blindfolded participants had the highest accuracy (98.9%,


25/03/14 8:52 PM


595

FIG. 3. General summary of results: (A) Average accuracies of the participant groups given as percentages. (B) Average identification times of the participant groups: the error bars are the standard errors.

SE = 0.4). The blind participants had the lowest accuracy (95.7%, SE = 0.9). Sighted participants were in-between (97.3%, SE = 0.6). Three-way ANOVA showed that the main effect due to the participant group was significant (F2, 300 = 5.7, p = .004, partial η2 = 0.04). The main effects due to the rotation of the standard stimulus, rotation of the mirror stimulus, and all interaction effects were insignificant. Post hoc tests showed that the difference in accuracy between the blindfolded and blind participants was indeed significant (95%CI = 0.96, 5.29).


25/03/14 8:52 PM

596

B. GÜÇLÜ, ET AL.

Average identification times are plotted in Fig. 3B for each participant group. They were lowest for the sighted participants (M = 1.34 sec., SE = 0.02), higher for the blindfolded (M = 4.29 sec., SE = 0.06), and highest for the blind participants (M = 8.14 sec., SE = 0.26). There was a significant main effect due to the participant group (F2, 300 = 530.6, p < .001, partial η2 = 0.78) and a significant interaction effect due to the rotation of the standard stimulus × the rotation of the mirror stimulus (F16, 300 = 2.4, p = .002, partial η2 = 0.12). No other significant effects could be found. According to the post hoc tests, visual identification times of the sighted participants were significantly lower than the haptic identification times of the blindfolded participants (95%CI = 2.5, 3.4). The haptic identification times of the blindfolded participants were in turn lower than those of the blind participants (95%CI = 3.6, 4.3). The correlation between accuracy and identification time was calculated for each participant group. It is important to note that because the sample sizes were low, the accuracy for a given condition only took one of a few percentage values, while the identification times could vary continuously. For all stimulus conditions included (n = 125), there was a low but significant negative correlation in the sighted group (r = –.30, p < .001) and a slightly higher correlation in the blind group (r = –.37, p < .001). The correlation was not statistically significant in the blindfolded group (r = –.15, p = .09). These results showed that the average identification time of the correct judgments was expected to be higher if some participants could not find the mirror stimulus. In other words, the participants who could correctly find the mirror stimulus in those trials spent somewhat more time before giving a response. However, since the accuracies were heavily represented by a limited number of high values and the correlation coefficients were low, we did not investigate these correlations further in detail. Mental-rotation Effect in the Classical Model The hypothetical models presented in the Introduction predict alternative mental-rotation angles for some of the identical stimulus conditions. Therefore, the accuracies and identification times were tabulated separately for all possible angular differences in a given model (Table 1). Due to relatively high standard deviations, no interesting pattern was observed in the accuracy values. For the classical model interpretation (Fig. 1A), the effect of mental rotation was investigated by plotting the average identification times (Table 1) as a function of the angular difference between the mirror and the standard stimulus (Fig. 4A–C). Note that the y-axes in these plots have different scales. A linear trend on the plots would show the classical mental rotation effect. Additionally, it would support the model presented in Figure 1A. The data given in Figure 4A–C showed that there was indeed an angular orientation effect for each participant group. For sighted participants (Fig. 4A), there


25/03/14 8:52 PM

597


TABLE 1 MEAN ACCURACIES AND IDENTIFICATION TIMES FOR ANGULAR DIFFERENCES TESTED IN THE HYPOTHETICAL MODELS, WITH STANDARD DEVIATIONS GIVEN IN PARENTHESES. THE LAST ROW INCLUDES THE GROSS AVERAGES REGARDLESS OF ANY ANGULAR DIFFERENCE AND MODEL

Model

Angular Difference (°) 0

Classical model

Blindfolded Group Accuracy, %

Accuracy, %

Identification Time, sec.

4.07

98.0

6.83

(0.62)

(5.5)

(2.18)

1.30 (0.16)

(0.0)

97.5

1.28

99.3

4.16

94.6

7.36

(6.4)

(0.19)

(4.5)

(0.70)

(11.6)

(2.42)

95.7

1.40

98.6

4.41

95.6

9.40

(7.6)

(0.23)

(4.3)

(0.70)

(10.7)

(3.09)

99.3

1.37

96.4

4.49

95.8

9.25

(3.2)

(0.20)

(6.3)

(0.69)

(7.4)

(3.33)

180

95.7

1.42

4.55

95.0

8.47

(6.9)

(0.25)

(0.78)

(11.2)

(3.06)

0

97.1

1.33

4.21

97.1

7.30

(5.8)

(0.20)

(0.0)

(0.69)

(7.5)

(2.53)

45

98.1

1.31

98.3

4.27

95.0

7.99

(5.6)

(0.20)

(5.3)

(0.71)

(10.3)

(2.87)

90

95.7

1.40

98.6

4.41

95.4

9.40

(7.6)

(0.23)

(4.4)

(0.70)

(10.8)

(3.09)

96.5

1.36

99.7

4.31

95.9

7.86

(6.9)

(0.22)

(2.1)

(0.71)

(10.1)

(3.10)

98.1

1.31

98.3

4.27

95.0

7.99

(5.6)

(0.20)

(5.3)

(0.71)

(10.3)

(2.87)

96.4

1.36

98.6

4.29

97.5

9.20

(1.4)

(0.21)

(4.4)

(0.69)

(6.1)

(2.49)

97.3

1.34

98.9

4.29

95.7

8.14

(6.2)

(0.21)

(4.3)

(0.70)

(9.7)

(2.92)

135

0 45 90 Overall

100.0

Blind Group


(5.3)

90

Dual code model

Accuracy, %


97.7

45

Antiparallel model

Sighted Group

100.0 (0.0) 100.0

was a high and marginally significant correlation between the identification time and the angular difference between the mirror and standard stimuli (r = 0.88, p = .05). Similarly, blindfolded participants show a significant and high correlation between identification time and angular difference (r = .97, p = .007) as seen in Fig. 4B. The correlation for blind participants was not significant (r = .72, p = .17), and the data showed a non-monotonic trend (Fig. 4C). The equations for the best-fit lines are also given in Fig. 4. In addition to average identification times described above, the present authors also tested


25/03/14 8:52 PM

598

B. GÜÇLÜ, ET AL.

FIG. 4. Orientation effects as explained by the hypothetical model in Fig. 1A. Average identification times are plotted as a function of angular difference. The error bars are the standard errors. The dotted lines are the best-fit lines. Their equations and goodness-of-fit values (R2) are given on the plots. (A) Classical mental rotation effect in the sighted group who performed the task visually. (B) Classical mental rotation effect in the blindfolded group who performed the task haptically. (C) Non-monotonic orientation effect in the blind group who performed the task haptically.


25/03/14 8:52 PM


599

the orientation effect based on data from single trials (n = 125 for each participant group). The correlation between single-trial identification time and the angular difference was significant (due to high sample size) but very low for each group (sighted: r = .22, p = .01; blindfolded: r = .24, p = .007; blind: r = .29, p = .001). According to both types of correlation analyses, it was concluded that the visual task performed by the sighted group and the haptic task performed by the blindfolded group conformed to the classical mental rotation model well. Although the haptic task performed by the blind group clearly showed an orientation effect, on average, it yielded a non-monotonic effect if the angular differences were interpreted based on Fig. 1A. The linear equations in Fig. 4A and B indicate that the mental rotation rate according to the classical model is approximately 1.25º/msec. for the visual stimuli and 0.34º/msec. for the haptic stimuli (in the blindfolded group). These values corresponded to inverse slopes of the best-fit lines. Therefore, it was found that haptic mental rotation was about four times slower than visual mental rotation for the sighted group. Antiparallel and Dual Code Models The data were re-interpreted according to the antiparallel model depicted in Fig. 1B. The average identification times are given in Table 1 for each participant group. Correlations based on average identification times yielded high but non-significant (due to low sample size) coefficients (sighted: r = .73, p = .48; blindfolded: r = .97, p = .15; blind: r = .98, p = .12). To obtain a new plot, first the abscissa value for each single-trial identification-time measurement was modified with respect to the antiparallel model. Then, the identification times were averaged for each angular difference value. The resultant graph for blind participants is given in Fig. 5A. Note that the non-monotonic effect no longer exists, mostly due to the fact that large angles ( > 90º) were transformed into smaller angles. The correlation analyses were repeated based on single-trial identification times. Only the blind group had a significant correlation (sighted: r = .11, p = .24; blindfolded: r = .10, p = .26; blind: r = .26, p = .004). It seems that the antiparallel model fits the data for the blind group better than the other groups. According to this model, the haptic mental rotation rate was 0.04º/msec. for the blind group (see the equation in Fig. 5A). In the dual code model depicted in Fig. 1C, the angle information is first extracted according to the antiparallel template. Then, the angles are represented in a propositional format and the angular difference is calculated based on that. When the data were re-interpreted based on this model (Table 1), the average identification times yielded a high, but nonsignificant, correlation within the blind group (sighted: r = –.10, p = .933; blindfolded: r = –.56, p = .624; blind: r = .91, p = .28). There was a monotonic orientation effect for that group (Fig. 5B). However, analyses based on


25/03/14 8:52 PM

600

B. GÜÇLÜ, ET AL.

FIG. 5. Orientation effects in the blind group according to new hypothetical models. (A) Antiparallel model as explained in Fig. 1B. (B) Dual code model as explained in Fig. 1C. Average identification times are plotted as a function of angular difference. The error bars are the standard errors. The dotted lines are the best-fit lines. Their equations and goodness-offit values (R2) are given on the plots.

single-trial identification times gave almost zero correlation for all participant groups (sighted: r = –.04, p = .64; blindfolded: r = –.02, p = .85; blind: r = .13, p = .14). Therefore, it was concluded that the described dual code model was not a good fit for any of the participant groups’ data. DISCUSSION The results for the sighted and blindfolded groups presented in this article are consistent with the classical model of mental rotation based on the depictive representation of mental images. The blind group also showed an orientation effect, but unlike the blindfolded group, appeared


25/03/14 8:52 PM


601

to utilize antiparallel (180º rotation superimposed) depictive representations for both standard and comparison stimuli. This new finding can be explained by the idea that the mapping function from the object-properties-processing sub-system to the visual buffer is altered with blindness (Kosslyn, et al., 2006). Dulin and Hatwell (2006) showed that congenitally blind people with experience perform better (higher accuracy) than those with early and late blindness without experience in using raisedline materials. In this experiment, the blindfolded group's accuracy was slightly but significantly higher (~ 3.2%) than the blind group's accuracy. Although this may seem contradictory to the effects of haptic experience, the two groups probably have different mapping functions as mentioned above. Since early blindness impairs cognitive maps that cause small perceptual fields and difficulties with the third dimension, significantly higher identification times were obtained in the blind group compared to blindfolded group. Nevertheless, Dulin and Hatwell's (2006) particular study and the high accuracy in this experiment imply that the haptic performance of people with blindness can improve with training. This can be utilized for developing educational resources and methods. Methodological Issues A novel task was used which required discriminating the same- and mirror-image versions of the standard object within a set of three simultaneously presented stimuli. Haptic tasks used in previous studies presented the standard object and only one test object in either a simultaneous or a sequential same–different judgment task. The main reason for not using a simple same vs different/mirror judgment is related to signal detection theory in psychophysics. Same/mirror judgment is a no/yes task that depends on the participant's criterion (liberal or conservative) for reporting the mirror (or “yes”). Even in the same group, this task may result in largely different correct detection and false positive rates among participants. To prevent this criterion effect, the procedure described in this article was similar to a forced-choice method in psychophysics. The results from previous studies in the literature are somewhat contradictory. Marmor and Zaback (1976) found a linear orientation effect with simultaneously presented same-different judgment pairs in blind participants. Similarly, Carpenter and Eisenberg (1978) obtained a linear effect with the single-stimulus paradigm, in which the blind group decided whether a letter was normal or the mirror version without a comparison stimulus. On the other hand, Dellantonio and Spagnolo (1990) could not find a rotation effect in a sighted group with simultaneously presented same–different/mirror pairs and only found a linear trend


25/03/14 8:52 PM

602

B. GÜÇLÜ, ET AL.

with the single-stimulus paradigm. This discrepancy may be due to unknown criterion effects (as can be explained based on signal detection theory) previously not controlled. The current method was limited such that it required sequential scanning of the normal and mirror stimuli with the right hand, and therefore, yielded higher identification times than those in the studies cited above. There was also some variability in the identification times because of that. The systematic effect of beginning at a particular object location was reduced by randomizing the position of the mirror stimulus. After the experiment, the authors compared the average accuracies and identification times for the two positions of the mirror stimulus in a given participant group. The accuracies were not significantly affected by the position of the mirror stimulus (all p’s > .05). The position also did not change the identification times significantly except for the blindfolded group. The blindfolded participants had slightly higher identification times when the mirror stimulus was placed in the middle position (4.46 sec. vs. 4.12 sec.; one-tailed t test, p = .003). This may be because some of these participants explored the mirror stimulus twice, probably by checking it again after they explored the middle and the rightmost stimulus once consecutively. In order not to increase identification times further, we did not allow bimanual exploration of the stimuli. Given three objects for every stimulus set, bimanual exploration of each object sequentially would require a considerable amount of time and additional memory load. However, people with blindness generally use bimanual exploration with an unfamiliar object; e.g., Ballesteros and Reales (2004) reported that bimanual exploration yielded better accuracy than unimanual exploration and faster response times in haptic discrimination of symmetry. This may be because hand positions are related to the body reference frame better during bimanual exploration. It was previously shown that a participant's hand rotation can modulate the reference frame (Carpenter and Eisenberg, 1978). Additionally, Millar and Al-Attar (2004) showed that disrupting bodycentered reference frame increased errors substantially. When debriefed after the experiment, the blind group participants described a strategy similar to mental rotation, but they also stated that they rotated the comparison object to a “straight” location. This “straight” location may correspond to the representation in the antiparallel model as supported by the results. Such an extended “straight” reference may arise in people with blindness because of their essential daily tasks. Perhaps the most important ability is to maintain postural stability perpendicular to ground (Heller, 2002). Therefore, “vertical” and “horizontal” directions may have differing roles in their perception of haptic objects. Therefore, the blind participants may have used the antiparallel representation of the standard


25/03/14 8:52 PM


603

object as their “vertical” reference. An alternative explanation is related to bimanual exploration. The “straight” reference described as such may refer to adjusting the mental representation according to the body axis. Blind people develop many strategies that allow them to outperform sighted individuals in certain tasks (Ballesteros, Bardisa, Millar, & Reales, 2005; Rovira, Deschamps, & Baena-Gomez, 2011). The reason why the blind group had greater identification times than the blindfolded group may partially be because the blind group was not allowed to perform with their “natural,” i.e., bimanual, strategy. Identification Times Unlike Tarr and Pinker (1989), no main effect due to the absolute rotation angle of the standard or mirror stimuli was found. Nevertheless, the data are consistent with their hypothesis regarding multiple views, i.e., representations of inputs. The following steps of sub-processing are proposed for the task used in the current study. As the standard is explored, a mental representation at the given angle is immediately formed as one of the multiple views suggested by Tarr and Pinker (1989). This representation becomes the standard representation for the rotation task. For the blind group, the antiparallel representation is formed. As the comparison stimulus is explored, its mental representation at the presented angle is formed as one of the multiple views. All the steps listed so far would take constant processing times regardless of orientation and can be estimated by the intercepts in Figs. 4A, 4B, and 5A. Then, the representation of the comparison stimulus is mentally rotated to match the standard representation in the nearest direction. Processing time for this step would be dependent on orientation difference and can be estimated by the slopes in Figs. 4A, 4B, and 5A. Tarr and Pinker (1989) also suggested that comparison of a stimulus with its mirror image may require a step which involves flipping in depth (see also Hamm, Johnson, & Corballis, 2004). This phenomenon was not explicitly studied here, but the participants did not describe this step in their debriefing comments after the experiment. The mental rotation rates and intercepts presented in the current article are correlated with each other. Sighted participants who judged visually performed the fastest as measured by both rotation rates and intercepts. The blind group, who judged haptically, performed the slowest in both measures. The rate found for visual mental rotation (1.25º/msec.) is very similar to 1.00º/msec. reported by Tarr and Pinker (1989). However, their intercept (identification time at 0º: 0.75 sec.) was smaller than ours (1.29 sec.). This discrepancy may be because our task required scanning three objects and therefore could require more processing time. Marmor and Zaback (1976) reported 0.25º/msec. haptic mental-rotation rate for the blindfolded


25/03/14 8:52 PM

604

B. GÜÇLÜ, ET AL.

group and 0.06º/msec. for the early blind group. These rates are similar to those found in the current study (0.34º/msec. for the blindfolded group and 0.04º/msec. for the blind group). On the other hand, the intercepts are still higher than those reported by Marmor and Zaback (1976), which is probably again due to additional sub-processing in our task which had sequential exploration with the right hand. Hypothetical Models The model put forth by Kosslyn, et al. (2006) for visual perception has several sub-systems. It has been argued that some neural structures and mechanisms within that model may also be used during visual imagery. It is a reasonable hypothesis that some mechanisms may even be shared by any mental imagery including that originating from haptic inputs. Although the scope of this article was not to verify this claim directly, three hypothetical models were presented for mental representation of haptic objects. During somatosensory processing, object properties and spatial properties are encoded separately (Reed, Klatzky, & Halgren, 2005). We conjectured that haptic information is also routed to the pathways for visual mental imagery, which helps us to “visualize” objects recognized through haptic interaction. Weatherly, Ball, and Stacks (1997) showed that “visualizers” had greater accuracy in mental-rotation tasks. Specifically, we propose that the mental representation used by the blind group in a haptic rotation task is different than that used by the blindfolded group. We attribute this to a modified mapping function from object information to the visual buffer, and hence a different, i.e., antiparallel, mental representation. The antiparallel model implies the compression of angles larger than 90º into a range of 0º to 90º. Although one may expect lower identification times because of that, such a result was not obtained. This suggests that the mapping function from the object-properties-processing sub-system to the visual buffer may also be slower in people with blindness, next to the methodological factors mentioned above. It is sometimes argued that the mental-rotation effect is observed because of tacit knowledge. In other words, the difference between the angles would be calculated, and the identification time would be correlated with this difference simply because the participants would expect that result themselves (see Kosslyn, et al., 2006, for a detailed account of this claim). There was no attempt to address this question here, but a hypothetical dual code model was tested, which included both a depictive and a propositional mental representation for angles. In brief, after the angle information is extracted according to a depictive format, the “rotation” is performed without a reference frame based on the propositional representation of angles and their difference. Although mean identification times fit only slightly better to the antiparallel model than the dual code model for the blind group, single-trial identification times were uncorrelated ac-


25/03/14 8:52 PM


605

cording to the dual code model. The word “rotation” was not used in the experimental instructions, but one may argue that a rotation effect originating from a propositional representation would be indicative of previous knowledge about angle differences, which is not necessarily so in a depictive format for naïve participants. Haptic Imagery and Sensory Substitution Experiments clearly showed that haptic imagery exists in participants with congenital blindness, who are better than blindfolded sighted participants in tactile picture perception (Heller, 2002). Moreover, visual imagery is helpful for haptic mental rotation (Prather & Sathian, 2002), and visual training was found to enhance haptic recognition, but haptic knowledge was not much useful for visual recognition (Behrmann & Ewell, 2003). This further supports the idea that certain visual areas are used in haptic processing. Taylor-Clarke, Kennett, and Haggard (2004) reported that even a noninformative visual image of a finger can enhance tactile acuity. Future studies on haptic rotation can benefit from this cross-modal interaction. Haptic manipulation of visual objects or visually enhanced haptic objects may be tested in computer-generated virtual environments. We would predict that the latter situation would improve a mental-rotation task performance. Sampaio, Maris, and Bach-y-Rita (2001) have been partially successful in mapping visual information to electro-tactile stimulation on the tongue. Their device may be useful for studying imagery from a radical point of view. Can the mental representation of passive tactile inputs be sufficient for conveying visual information? Lenay, Gapenne, Hanneton, Marque, and Genouëlle (2003) presented a detailed critique of sensory substitution devices. These devices create a new “mode” of perception very limited in terms of “substituting” for a sensory modality. Moreover, they pointed out the absolute necessity of active exploration by sensorimotor cycles, which is currently difficult to integrate into the devices. Cattaneo, Vecchi, Monegato, Pece, and Cornoldi (2007) found that visual deficits occurring later in life are more disruptive on spatial imagery abilities. Compensatory mechanisms adopted by people with congenital blindness may be convenient for developing new educational devices. Unfortunately, there is little evidence-based research on the benefits of assistive technology for individuals with visual impairment (Kelly & Smith, 2011). We hope for a change in the near future. REFERENCES

ARDITI, A., HOLTZMAN, J. D., & KOSSLYN, S. M. (1988) Mental imagery and sensory experience in congenital blindness. Neuropsychologia, 26, 1-12. BALLESTEROS, S., BARDISA, D., MILLAR, S., & REALES, J. M. (2005) The haptic test battery: a new instrument to test tactual abilities in blind and visually impaired and sighted children. British Journal of Visual Impairment, 23, 11-24.


25/03/14 8:52 PM

606

B. GÜÇLÜ, ET AL.

BALLESTEROS, S., & REALES, J. M. (2004) Visual and haptic discrimination of symmetry in unfamiliar displays extended in the z-axis. Perception, 33, 315-327. BARTOLOMEO, P. (2008) The neural correlates of visual mental imagery: an ongoing debate. Cortex, 44, 107-108. BEHRMANN, M., & EWELL, C. (2003) Expertise in tactile pattern recognition. Psychological Science, 14, 480-486. CARPENTER, P. A., & EISENBERG, P. (1978) Mental rotation and the frame of reference in blind and sighted individuals. Perception & Psychophysics, 23, 117-124. CATTANEO, Z., VECCHI, T., MONEGATO, M., PECE, A., & CORNOLDI, C. (2007) Effects of late visual impairment on mental representations activated by visual and tactile stimuli. Brain Research, 1148, 170-176. CORBALLIS, M. C., & MCLAREN, R. (1984) Winding one's Ps and Qs: mental rotation and mirror-image discrimination. Journal of Experimental Psychology: Human Perception and Performance, 10, 318-327. DE BENI, R., & CORNOLDI, C. (1988) Imagery limitations in totally congenitally blind subjects. Journal of Experimental Psychology: Learning, Memory, and Cognition, 14, 650655. DELLANTONIO, A., & SPAGNOLO, F. (1990) Mental rotation of tactual stimuli. Acta Psychologica, 73, 245-257. DULIN, D., & HATWELL, Y. (2006) The effects of visual experience and training in raisedline materials on the mental spatial imagery of blind persons. Journal of Visual Impairment & Blindness, 100, 414-424. DULIN, D., HATWELL, Y., PYLYSHYN, Z., & CHOKRON, S. (2008) Effects of peripheral and central visual impairment on mental imagery capacity. Neuroscience and Biobehavioral Reviews, 32, 1396-1408. HAMILTON, R. H., & PASCUAL-LEONE, A. (1998) Cortical plasticity associated with Braille learning. Trends in Cognitive Sciences, 2, 168-174. HAMM, J. P., JOHNSON, B. W., & CORBALLIS, M. C. (2004) One good turn deserves another: an event-related brain potential study of rotated mirror-normal letter discriminations. Neuropsychologia, 42, 810-820. HELLER, M. A. (2002) Tactile picture perception in sighted and blind people. Behavioural Brain Research, 135, 65-68. KELLY, S. M., & SMITH, D. W. (2011) The impact of assistive technology on the educational performance of students with visual impairments: a synthesis of the research. Journal of Visual Impairment & Blindness, 105, 73-83. KLEIN, I., DUBOIS, J., MANGIN, J-F., KHERIF, F., FLANDIN, G., POLINE, J-B., DENIS, M., KOSSLYN, S. M., & LE BIHAN, D. (2004) Retinotopic organization of visual mental images as revealed by functional magnetic resonance imaging. Cognitive Brain Research, 22, 26-31. KOSSLYN, S. M., THOMPSON, W. L., & GANIS, G. (2006) The case for mental imagery. New York: Oxford Univer. Press. LENAY, C., GAPENNE, O., HANNETON, S., MARQUE, C., & GENOUËLLE, C. (2003) Sensory substitution: limits and perspectives. In Y. Hatwell, A. Streri, & E. Gentaz (Eds.), Touching for knowing: cognitive psychology of haptic manual perception. Philadelphia, PA: John Benjamins. Pp. 275-292. MARMOR, G. S., & ZABACK, L. A. (1976) Mental rotation by the blind: does mental rotation depend on visual imagery? Journal of Experimental Psychology: Human Perception and Performance, 2, 515-521.


25/03/14 8:52 PM


607

MERABET, L., THUT, G., MURRAY, B., ANDREWS, J., HSIAO, S., & PASCUAL-LEONE, A. (2004) Feeling by sight or seeing by touch. Neuron, 42, 173-179. MILLAR, S., & AL-ATTAR, Z. (2004) External and body-centered frames of reference in spatial memory: evidence from touch. Perception & Psychophysics, 66, 51-59. OLDFIELD, R. C. (1971) The assessment and analysis of handedness: the Edinburgh inventory. Neuropsychologia, 9, 97-113. PRATHER, S. C., & SATHIAN, K. (2002) Mental rotation of tactile stimuli. Cognitive Brain Research, 14, 91-98. REED, C. L., KLATZKY, R. L., & HALGREN, E. (2005) What vs. where in touch: an fMRI study. NeuroImage, 25, 718-726. ROVIRA, K., DESCHAMPS, L., & BAENA-GOMEZ, D. (2011) Mental rotation in blind and sighted adolescents: the effects of haptic strategies. Revue Européenne de Psychologie Appliqué, 61, 153-160. SADATO, N., OKADA, T., KUBOTA, K., & YONEKURA, Y. (2004) Tactile discrimination activates the visual cortex of the recently blind naive to Braille: a functional magnetic resonance imaging study in humans. Neuroscience Letters, 359, 49-52. SAMPAIO, E., MARIS, S., & BACH-Y-RITA, P. (2001) Brain plasticity: ‘visual’ acuity of blind persons via the tongue. Brain Research, 908, 204-207. SHEPARD, R. N., & METZLER, J. (1971) Mental rotation of three-dimensional objects. Science, 171(972), 701-703. TARR, M. J., & PINKER, S. (1989) Mental rotation and orientation-dependence in shape recognition. Cognitive Psychology, 21, 233-282. TAYLOR-CLARKE, M., KENNETT, S., & HAGGARD, P. (2004) Persistence of visual-tactile enhancement in humans. Neuroscience Letters, 354, 22-25. WEATHERLY, D. C., BALL, S. E., & STACKS, J. R. (1997) Reliance on visual imagery and its relation to mental rotation. Perceptual & Motor Skills, 85, 431-434. Accepted February 21, 2014.


25/03/14 8:52 PM

Copyright of Perceptual & Motor Skills is the property of Ammons Scientific, Ltd. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.

Mental rotation of tactile stimuli: using directional haptic cues in mobile devices.

An investigation of sex differences in spatial ability: mental rotation of three-dimensional objects.

Effects of congenital blindness on the subcortical representation of speech cues.

Mental Rotation as an Indicator of Motor Representation in Patients with Mild Cognitive Impairment.

Haptic recognition of familiar objects: examining lateralization of specificity effects.

Congenital Word-Blindness.

Age-related differences of neural connectivity during mental rotation.

Characteristics of Haptic Peripersonal Spatial Representation of Object Relations.

Spatial representation of objects in the young blind child.

Age-related deficits in conjunctive representation of complex objects.

Invisibility of moving objects: a core symptom of motion blindness.

Visuo-haptic multisensory object recognition, categorization, and representation.

Developmental differences in mental rotation.

Deconstructing mental rotation.

Perceptual illusion of rotation of three-dimensional objects.

A Case of Congenital Word-Blindness.

Visual and haptic integration in the estimation of softness of deformable objects.

Are developments in mental scanning and mental rotation related?

What does physical rotation reveal about mental rotation?

Capacity for Visual Features in Mental Rotation.

Evaluation of Pseudo-Haptic Interactions with Soft Objects in Virtual Environments.

Tactile mental body parts representation in obesity.

Enhanced chemosensory detection of negative emotions in congenital blindness.

Morphometric changes of the corpus callosum in congenital blindness.