Accepted Manuscript Title: A Systematic Review of Multisensory Cognitive-Affective Integration in Schizophrenia Author: Huai-Hsuan Tseng Matthijs G. Bossong Gemma Modinos Kuan-Ming Chen Philip McGuire Paul Allen PII: DOI: Reference:

S0149-7634(15)00123-2 http://dx.doi.org/doi:10.1016/j.neubiorev.2015.04.019 NBR 2181

To appear in: Received date: Revised date: Accepted date:

17-7-2014 25-3-2015 26-4-2015

Please cite this article as: Tseng, H.-H., Bossong, M.G., Modinos, G., Chen, K.-M., McGuire, P., Allen, P.,A Systematic Review of Multisensory CognitiveAffective Integration in Schizophrenia, Neuroscience and Biobehavioral Reviews (2015), http://dx.doi.org/10.1016/j.neubiorev.2015.04.019 This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

*Highlights (for review)

Highlights 1. Multisensory integration (MSI) is a spontaneous perceptual-cognitive process. 2. Deficits in MSI may hinder prompt and appropriate behavioural responses

4. MSI for emotional stimuli in schizophrenia may also be altered.

ip t

3. MSI for non-emotional linguistic stimuli is impaired in schizophrenia.

Ac ce pt e

d

M

an

us

cr

5. Functional alterations in temporal and frontal brain areas underlie MSI deficits.

1 Page 1 of 38

*Manuscript

A Systematic Review of Multisensory Cognitive-Affective Integration in Schizophrenia

Department of Psychosis Studies, Institute of Psychiatry, King’s College London, United

cr

1

ip t

Huai-Hsuan Tseng1,2,3,*, Matthijs G. Bossong1, Gemma Modinos1, Kuan-Ming Chen4, Philip McGuire1, Paul Allen1

Kingdom

Research Imaging Centre, Centre of Addiction and Mental Health, University of Toronto,

us

2

Toronto, Canada Ju Shan Hospital, Taoyuan, Taiwan

4

Science Education Center, National Taiwan Normal University, Taipei, Taiwan

M

an

3

references

d

Word Count: Abstract: 159 words; Main text: around 4243 words, 2 tables, 2 figures, 63

Ac ce pt e

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

1 Page 2 of 38

Abstract: The etymology of schizophrenia implies poor functional integration of sensory, cognitive and affective processes. Multisensory integration (MSI) is a spontaneous perceptual-cognitive process by which relevant information from multiple sensory modalities is extracted to generate

ip t

a holistic experience. Deficits in MSI may hinder prompt and appropriate behavioural responses in a complex and transient environment. Despite extensive investigation of sensory, cognitive

cr

and affective processing in patients with schizophrenia, little is known about how MSI is

us

affected in the illness. We systemically searched the PubMed electronic database and reviewed twenty-nine behavioural and neuroimaging studies examining MSI in patients with schizophrenia.

an

The available evidence indicates impaired MSI for non-emotional stimuli in schizophrenia, especially for linguistic information. There is also evidence for altered MSI for emotional stimuli,

M

although findings are inconsistent and may be modality-specific. Brain functional alterations in

d

the superior temporal cortex and inferior frontal cortex appear to underlie the deficits in both non-emotional and emotional MSI. The limitations of the experimental paradigms used and

Ac ce pt e

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

directions for future research are also discussed.

2 Page 3 of 38

1. Introduction The etymology of schizophrenia means "splitting of mental functions”; implying defective functional integration of information. In addition to widespread cognitive impairment, patients

ip t

with schizophrenia have prominent deficits in emotional processing (Edwards et al., 2002), and this is closely related to problems with social cognition and interpersonal functioning (e.g.,

cr

Brittain et al., 2010; Penn et al., 1997; Pinkham et al., 2003) and psychotic symptoms (Laroi et

us

al., 2010; Tseng et al., 2013). Studies of social cognition in schizophrenia have used emotionladen stimuli usually presented in a single sensory modality, such as visual face (Gur et al., 2002;

an

Kohler et al., 2003; Pinkham et al., 2005) or auditory prosodic voice stimuli (Leentjens et al., 1998; Leitman et al., 2005) (for a comprehensive reivew, see Edwards et al., 2002). The

M

integration of emotionally laden information across sensory modalities is required to provide a holistic and complete representation of emotional meaning and its social relevance.

d

Real-time non-emotional and emotional signals are simultaneously transmitted from the environment to the observer through multiple sensory modalities. By combining information

Ac ce pt e

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

from different sensory modalities, relevant signals are easier to filter and distinguish from background perceptual noise. The integration of multiple signals into meaningful information can initiate adaptive behaviours (Ethofer et al., 2006; Koelewijn et al., 2010). The interplay among different sensory inputs usually generates a holistic experience, different from the input via any single sensory modality (McGurk and MacDonald, 1976). The underlying neurocognitive process that allows for the integration of separate streams of information from different sensory modalities into a holistic experience is usually referred to as multisensory integration (MSI) (de Jong et al., 2009). The current article provides a systematic review of studies examining multisensory integration in schizophrenia, with an emphasis on the integration of emotional information.

3 Page 4 of 38

2. Multi-sensory integration (MSI) MSI is a spontaneous perceptual and cognitive process that occurs early when incoming signals

ip t

are adjacent temporally and/or spatially (de Jong et al., 2009). MSI can occur either along a

temporal or a spatial dimension, or both, and experimental paradigms using speech, emotional or

cr

social information have been developed to study its mechanisms. Experimentally, when

us

multisensory information is congruent, better perceptual sensitivity is observed resulting in greater recognition accuracy and shorter response latencies. This is referred to as a congruent

an

facilitation effect (Miller, 1982, 1986; Schröger and Widmann, 1998). Contrarily, when the multisensory information is incongruent, an incongruent interference effect, can reduce

M

perceptual sensitivity and accuracy, and increase response latencies. In addition, when

d

information across sensory modalities is incongruent, the perceivers’ judgment tends to be dominated or influenced by the most reliable source of sensory input (Collignon et al., 2008;

Ac ce pt e

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

Welch and Warren, 1980). Sometimes, a perceptual illusion may occur. For example, the McGurk-MacDonald effect (McGurk and MacDonald, 1976) describes the phenomenon by which incongruent visual information exerts a bias on auditory speech perception, for instance a simultaneous presentation of a heard syllable /ba/ and a seen syllable /ga/ is perceived as /da/. Another example is the ventriloquist effect (Bertelson and Radeau, 1981; Choe et al., 1975), describing the phenomenon by which the puppeteer’s voice seems to originate from the mouth of the puppet, suggesting that misaligned visual cues disrupt auditory localization.

In social contexts, emotional information is often conveyed across multiple sensory modalities, for example, simultaneously through speech content and prosody, as well as facial and bodily signals. Experimental studies report that, compared to neutral stimuli, emotion-laden stimuli 4 Page 5 of 38

capture attention automatically (Vuilleumier and Huang, 2009). This effect has been demonstrated in a MSI study using emotional speech stimuli, in which the MSI of emotional information took place before the semantic content was processed (de Gelder and Vroomen, 2000). When subjects perceive emotional multisensory information, similar to non-emotional

ip t

stimuli, an emotional congruent facilitation effect leading to increased accuracy and/or reduced reaction times (RTs) is observed. In contrast, an emotional incongruent interference effect is

us

cr

associated with decreased accuracy and/or increased RTs.

Empirical findings in healthy volunteers suggest that multisensory integrative processes are

an

obligatory in nature (Muller et al., 2011), irrespective of the congruency of information. However, several studies report defective MSI in patients with schizophrenia (de Gelder et al.,

M

2003; de Gelder et al., 2005; de Jong et al., 2009; Ross et al., 2007; Williams et al., 2010).

d

Impaired MSI in patients with schizophrenia is likely to be a major disadvantage in complex, information-abundant environments, with a potential negative impact on social functioning.

Ac ce pt e

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

However, the nature and the extent of MSI impairments in patients with schizophrenia are unclear, particularly for emotion-laden stimuli.

3. Search strategy and results

We used the Preferred Reporting Items for Systematic Review and Meta-Analyses (PRISMA) for guidance regarding reporting of search, extraction and synthesis of results in the current review (Liberati et al., 2009; Moher et al., 2009). An electronic search was performed using the following criteria in PubMed database: ("crossmodal" OR "audiovisual" OR "audio-visual" OR "multisensory" OR "multi-sensory") AND ("integration" OR "interference" OR "conflict") AND ("schizophrenia" OR "psychosis"). Reports in English using human subjects, published in paperback or accessed online before 5 Page 6 of 38

August, 2013 were included. In addition, the references cited by the included studies were scrutinized to ensure the inclusion of other relevant studies.

A total of forty-eight studies were identified. Nineteen studies that did not originally investigate

ip t

MSI in schizophrenic spectrum disorders were excluded (two review articles, nine articles in healthy populations, and eight articles that did not use MSI as one of the main research

cr

paradigm). In patients with schizophrenia, fourteen studies investigated non-emotional MSI and

us

seven studies examined emotional MSI (see Table 1). We also include one multimodal case study investigating multisensory hallucinations, two post-mortem studies that assessed brain

an

areas related to MSI, one study examined unusual proneness to visuo-tactile integration in schizophrenia, and four studies using a MSI task as part of a neurological examination battery in

M

schizophrenia patients and their relatives (See Table 2). Overall, a total of twenty-nine studies

d

were included in the current review. The protocol for the selection of research adheres to the PRISMA guidelines for a systematic review and is illustrated in Figure 1.

Ac ce pt e

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

4. Non-emotional multisensory integration in schizophrenia This section focuses on empirical studies of MSI in patients with schizophrenia, using stimuli without emotion-laden features. Fourteen studies that used non-emotional multisensory integration tasks to compare MSI between patients with schizophrenia and healthy controls have been identified. Study characteristics, sample sizes, task paradigm, and results are summarized in Table 1.

Six of fourteen studies that are described used cognitive-behavioural paradigms. Of these, one study used a target-detection paradigm, and three used lip-reading paradigm (congruent or incongruent in the information between heard and seen syllables); the others used spatial or 6 Page 7 of 38

temporal resolution, or visual temporal order paradigms. Three of the fourteen studies measured neural responses to different experimental paradigms using electrophysiological techniques, including electroencephalography (EEG), magnetic electroencephalography (MEG) and eventrelated potentials (ERP). Of these, two used far-near judgment paradigm and one used mixed lip-

ip t

reading and non-lip reading paradigm. For clarity, these results are summarised based on the

congruency of the stimuli used. Five of fourteen studies used functional Magnetic Resonance

cr

Imaging (fMRI) examining the neural correlates of MSI in patients with schizophrenia; three

us

used a lip-reading paradigm, and other two fMRI studies used a functional connectivity approach

an

during MSI in patients with schizophrenia.

During the simultaneous presentation of congruent auditory and visual stimuli, healthy controls

M

show shorter reaction times (RT) relative to unimodal conditions, demonstrating a congruent

d

facilitation effect. However, this facilitation effect on RT (Williams et al., 2010) and accuracy is reduced in patients with schizophrenia. In line with this study, Ross et al. (2007) demonstrated

Ac ce pt e

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

this reduced effect using a linguistic lip reading paradigm in which congruent auditory and visual linguistic signals were presented simultaneously. Although healthy volunteers and patients with schizophrenia both demonstrate increased recognition accuracy during congruent trials, this facilitation effect was significantly reduced relative to healthy controls.

When audiovisual linguistic signals are incongruent, as would be expected, there is a decrease in accuracy and compound products of visual and auditory phonemes (i.e., McGurk-MacDonald effect (McGurk and MacDonald, 1976)). However, patients with schizophrenia show a reduced McGurk-MacDonald effect, relative to controls, indicating reduced interference (de Gelder et al., 2003; Pearl et al., 2009). MSI deficits in schizophrenia have also been investigated at the neural level using evoked brain responses (Stekelenburg et al., 2013). When incongruent audio-visual 7 Page 8 of 38

stimuli were presented (either linguistically or non-linguistically), a suppressed amplitude and a decreased latency of the auditory evoked brain response were observed in healthy participants mainly in the right posterior superior temporal gyrus, right middle temporal gyrus and left inferior frontal gyrus. However, the suppression in amplitude and decreased response latency

ip t

were not seen in patients with schizophrenia during incongruent audiovisual trials.

cr

Other studies report contradictory findings. For example, a comparable ventriloquist effect (MSI

us

of spatially incongruent stimuli) has been observed both in healthy participants and in patients with schizophrenia (de Gelder et al., 2003). While examining multisensory processing in

an

schizophrenia using unisensory versus multisensory temporal tasks, two studies found that patients with schizophrenia were less able to discriminate stimuli in rapid serial presentation

M

relative to healthy controls, but were not disadvantaged when integrating simultaneous non-

d

linguistic multimodal information (de Boer-Schellekens et al., 2013; Foucher et al., 2007).

Ac ce pt e

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

However, among these studies using linguistic lip-reading paradigms, only one study failed to demonstrate a reduced congruent facilitation effect in patients with schizophrenia (Surguladze et al., 2001), suggesting that attenuated MSI is more prominent in a linguistic context. This interpretation is supported by findings from two non-linguistic MSI studies that report comparable facilitation effects in controls and patients (Stephen et al., 2013; Stone et al., 2011).

Five studies examined the neural correlates of MSI in patients with schizophrenia using fMRI. During an audiovisual lip-reading paradigm, healthy controls showed greater activation in auditory, visual, and attentional networks during incongruent relative to congruent trials. In contrast, patients with schizophrenia showed the reverse pattern of activation (i.e greater activation for congruent relative to incongruent stimuli) in the right inferior frontal gyrus, 8 Page 9 of 38

bilateral superior and middle temporal gyri, cingulate gyrus and precuneus. (Szycik et al., 2009a). A study using a semantic priming paradigm reports greater activation during multisensory relative to unisensory trials in superior frontal gyrus, middle temporal gyrus, anterior cingulate and angular gyrus in healthy controls. However, the magnitude of increase was reduced in

ip t

patients with schizophrenia (Sass et al., 2013). These fMRI studies suggest that, during MSI, brain activation is altered in patients with schizophrenia in networks responsible for sensory and

cr

linguistic processing, including the superior and middle temporal cortex, superior and inferior

us

frontal gyrus (Sass et al., 2013; Szycik et al., 2009a). The compromised neural substrates also involve cingulate cortex and precuneus (Szycik et al., 2009a), which suggests the potential

an

modulatory role of dysfunctional attentional networks on MSI deficits in patients with

M

schizophrenia.

d

Functional connectivity during MSI has also been investigated in patients with schizophrenia in two out of five fMRI studies. Areas responsible for higher order integrative function in superior

Ac ce pt e

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

temporal sulcus (STS) and inferior frontal gyrus (IFG) were investigated in two fMRI studies using audiovisual integration paradigms. During a metaphoric gestures processing task, participants were asked to look at ‘concrete’ gestures that were mentioned in spoken words, or ‘abstract’ gestures that were associated with spoken contents. Whilst performing the task, STS is functionally connected to the middle temporal gyrus, ventral IFG and other disparate regions of the brain. However, decreased functional connectivity between STS, middle temporal gyrus and ventral IFG was observed in patients with schizophrenia during the abstract relative to the concrete gestures condition (Straube et al., 2013). Similarly, during a lip-reading task, altered connectivity patterns between right posterior STS and left IFG (Broca’s area) for congruent relative to incongruent trials were seen in patients with schizophrenia relative to controls (Szycik et al., 2013). These results suggest altered or aberrant connectivity in STS and IFG networks 9 Page 10 of 38

during MSI in schizophrenia, particularly while integrating linguistically abstract or incongruent stimuli.

These neurofunctional alterations suggest a plausible neural basis for the interpersonal

ip t

communication problems that are common in patients with schizophrenia. The role of the

bilateral STS in MSI is further highlighted by the finding that the area is an important substrate

cr

for multisensory hallucinations (Jardri et al., 2009). In concert with the post-mortem finding of

us

thinner cortical layers of posterior temporal gyrus and angular gyrus in schizophrenia (Smiley et al., 2012; Smiley et al., 2009), neurobiological changes in the STS and the temporal-parietal

an

junction appear to underlie MSI impairments in schizophrenia.

M

Overall, patients with schizophrenia show both reduced congruent facilitation and incongruent

d

interference effects as a consequence of impaired integration of concurrent audiovisual information. In particular, patients consistently show the reduced facilitation effect during

Ac ce pt e

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

linguistic paradigms (de Gelder et al., 2003; Pearl et al., 2009; Ross et al., 2007; Sass et al., 2013; Stekelenburg et al., 2013; Straube et al., 2013; Szycik et al., 2013), this effect is less consistently reported in studies using non-linguistic stimuli (de Boer-Schellekens et al., 2013; de Gelder et al., 2003; Foucher et al., 2007; Stone et al., 2011; Williams et al., 2010). Given these findings, de Gelder and colleagues (de Gelder et al., 2003) argued for a specific impairment in the audiovisual integration of cues with communicative value rather than a general impairment across perceptual domains. Findings from evoked brain response and fMRI studies using linguistic paradigms also report deficits of multisensory processing in schizophrenia at the neural level (Sass et al., 2013; Stekelenburg et al., 2013; Straube et al., 2013; Szycik et al., 2009a; Szycik et al., 2013). The compromised neural substrates mainly include posterior superior temporal gyrus, STS, middle temporal gyrus and IFG, with less adaptive activation patterns and 10 Page 11 of 38

abnormal functional connectivity between these regions, particularly during incongruent and

ip t

abstract conditions.

cr

5. Emotional MSI in schizophrenia

us

Seven studies have examined multisensory processing using emotion-laden stimuli in patients with schizophrenia. Study characteristics, sample sizes, task designs and a summary of the

an

results are shown in Table 1.

M

Five of seven studies used behavioural paradigms. Of these, one study investigated olfactory-

d

visual emotional integration (Seubert et al., 2010), three investigated audiovisual integration using facial and prosodic stimuli (de Gelder et al., 2005; de Jong et al., 2010; de Jong et al., 2009)

Ac ce pt e

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

and one using whole-body expressions with affective prosody (Van den Stock et al., 2011). Four studies report that patients with schizophrenia were less able to integrate emotional information across different sensory modalities (de Gelder et al., 2005; de Jong et al., 2010; de Jong et al., 2009; Seubert et al., 2010), resulting in reduced facilitation effects.

de Jong and colleagues (2010) further examined the role of modality-specific attention that modulates emotional MSI in schizophrenia. In healthy controls, a cross-modal facilitating influence of congruent facial stimuli on the processing of vocal emotion information was attenuated in the presence of concurrent unisensory non-emotional distractors. The attenuating effect of distractors was absent or even reversed in patients with schizophrenia (de Jong et al., 2010), suggesting a reduced modality-specific attentional modulation effect. However, two 11 Page 12 of 38

studies reported an increased cross-modal influence in schizophrenia (de Gelder et al., 2005; Van den Stock et al., 2011). Relative to healthy controls, patients showed an increased facilitation effect brought about by congruent facial expressions during categorization of emotional voices

(visual) emotional body language (Van den Stock et al., 2011).

ip t

(de Gelder et al., 2005), and an increased cross-modal influence of vocal emotion on unisensory

cr

Two studies have further examined the neurofunctional basis of emotional MSI in schizophrenia

us

(Muller et al., 2013b; Muller et al., 2012). Muller and colleagues asked participants to rate emotional and neutral faces whilst being distracted by emotional or neutral sounds (Muller et al.,

an

2012). Patients with schizophrenia showed a significant attenuation in amplitudes associated with visual processing during emotionally incongruent stimulus pairs, but not during congruent

M

pairs. An attenuated brain response in patients was also observed for single-modality face stimuli,

d

indicating the presence of early visual processing deficits in schizophrenia during emotion processing, which can be ameliorated while providing congruent multisensory emotional

Ac ce pt e

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

information.

Using meta-analytic connectivity modelling analyses, Muller et al. (2013b) examined the inferior parietal cortex connectivity in healthy controls, patients with depression, and patients with schizophrenia. Inferior parietal cortex is deactivated during audiovisual congruent stimuli in healthy controls, whilst being interconnected with the memory network (Muller et al., 2013a). However, patients with schizophrenia failed to deactivate this area during the congruent condition, indicating a dysregulation in inferior parietal cortex in schizophrenia, which implies an impaired memory function required during emotional MSI. Behavioural findings are inconsistent. Three studies (de Gelder et al., 2005; de Jong et al., 2009; Seubert et al., 2010) reported decreased integration of emotional information in patients 12 Page 13 of 38

with schizophrenia, whilst three (de Gelder et al., 2005; de Jong et al., 2010; Van den Stock et al., 2011) whilst three reported an increased congruent facilitation effect (with one study reported opposite results in different modalities). Changes in associated neural substrate are also observed, mainly involving inferior parietal cortex (Muller et al., 2013b). It is unclear why patients show

ip t

both the reduced and increased bi-modal facilitation effects during emotional MSI and a greater number of studies are required before firm conclusions can be drawn. It is possible that the

us

cr

sensory modality and the experimental context have influenced the finding of existing studies.

an

6. Discussion

Experimental paradigms used for examining MSI in patients with schizophrenia differ in task

M

complexities and dimensions of integration (i.e., along spatial or temporal dimension) making it

d

difficult to draw firm conclusions. In general, patients with schizophrenia show deficits in integrating multisensory information that result in reduced congruent facilitation and/or reduced

Ac ce pt e

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

incongruent interference effect (de Gelder et al., 2003; de Gelder et al., 2005; de Jong et al., 2010; de Jong et al., 2009; Muller et al., 2013b; Muller et al., 2012; Pearl et al., 2009; Ross et al., 2007; Sass et al., 2013; Seubert et al., 2010; Stekelenburg et al., 2013; Straube et al., 2013; Szycik et al., 2009b; Szycik et al., 2013; Williams et al., 2010). In addition to a generalised reduction of temporal processing resolution, patients with schizophrenia are also less able to discriminate consecutive stimuli presented across sensory modalities (de Boer-Schellekens et al., 2013; Foucher et al., 2007). However, they seemingly perform well to integrate simple audiovisual stimuli (e.g. light, shapes, and sounds) along spatial and temporal dimensions (de BoerSchellekens et al., 2013; de Gelder et al., 2003; Foucher et al., 2007).

13 Page 14 of 38

Most studies using linguistic paradigms report impaired MSI in patients with schizophrenia. However, findings from studies using non-linguistic paradigms are less consistent (de BoerSchellekens et al., 2013; de Gelder et al., 2003; Foucher et al., 2007; Muller et al., 2013b; Muller et al., 2012; Seubert et al., 2010; Stone et al., 2011; Van den Stock et al., 2011; Williams et al.,

ip t

2010). This suggests that audiovisual integration of linguistic information in schizophrenia may be selectively impaired, implicating a specific deficit in integrating multisensory stimuli with a

cr

communicative value rather than a general impairment across perceptual domains (de Gelder et

us

al., 2003).

an

The underlying mechanistic alterations associated with MSI deficits in schizophrenia remain unclear although it is likely that greater attentional demands are required for patients with

M

schizophrenia when performing MSI tasks relative to healthy controls (de Jong et al., 2010).

d

Thus, pre-existing attentional deficits in patients with schizophrenia (Egan et al., 2000; Mesholam-Gately et al., 2009) may decrease facilitation effects typically brought about by

Ac ce pt e

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

utilising additional information from a second sensory modality.

In particular, modality-specific attention is required to modulate the MSI of emotional information, and this modulatory role is deficient in patients with schizophrenia (de Jong et al., 2010). However, despite the existence of attentional deficits, an enhanced facilitation effect during MSI has also been reported in patients during non-linguistic paradigms (Stephen et al., 2013; Stone et al., 2011). Whilst difficult to explain, this enhanced facilitation effect could arise through the attention-capturing nature of multisensory stimuli which compensates for the preexisting attention deficits in schizophrenia, and ameliorate MSI impairments.

14 Page 15 of 38

Alternatively, lateral temporal lobe regions involved in linguistic processing (Hickok and Poeppel, 2007) are widely affected by the neuropathological changes seen in patients with schizophrenia (Shenton et al., 2001). This may potentially explain the more prominent MSI deficits in schizophrenia using linguistic MSI paradigm and implicate that underlie deficits may

ip t

be specific to linguistic processing.

cr

The evidence for the existence of an emotional MSI deficit in schizophrenia is inconsistent. Both

us

decreased and increased integration of emotion-laden multisensory stimuli has been reported. There are several possibilities for the inconsistent findings. Firstly, unisensory emotional

an

processing difficulties in schizophrenia have been robustly demonstrated in visual modality and auditory modality separately (Edwards et al., 2002). Unisensory deficits in schizophrenia are

M

likely to impact on the integration of emotional information arriving across modalities. However,

d

emotion-laden multisensory stimuli automatically are known to capture stimulus-driven attention (Carretie, 2014; Carretie et al., 2004), which may ameliorate MSI impairments that are seen

Ac ce pt e

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

when non-emotional stimuli are used.

The inconsistency among findings may also result from experimental paradigms. Findings from studies contrasting congruent to incongruent emotional trials (de Gelder et al., 2005; de Jong et al., 2010; de Jong et al., 2009) represent the additive effect of congruent facilitation and incongruent interference effects. Using performance for emotionally incongruent stimuli as a baseline to demonstrate facilitation effects may not be optimal, as conflict detection during the processing of incongruent stimuli demands more attentional resources (Driver and Noesselt, 2008), and uniquely activates the cingulate-fronto-parietal network involved in conflict monitoring and resolution (Muller et al., 2011). Interference effects brought about by emotionally incongruent stimuli may involve less emotional processing and be compounded by 15 Page 16 of 38

an attentional deficit in patients with schizophrenia, thus the emotional congruent-incongruent contrast may not unequivocally represent the deficits of emotional MSI.

Finally, an auditory-dominance effect during MSI in patients with schizophrenia may also

ip t

contribute to inconsistencies between studies. Different from a visual-dominance effect observed in healthy controls, patients might rely more on auditory information (de Gelder et al., 2005;

cr

Ross et al., 2007; Van den Stock et al., 2011) while perceiving concurrent audiovisual

us

information. Such an effect may explain increased cross-modal influence of voices on emotional bodily expression compared to healthy controls (Van den Stock et al., 2011), and decreased

an

influence of emotional faces on voices (de Jong et al., 2009) reported in patients with

M

schizophrenia.

d

However, an auditory dominance effect in schizophrenia has not been conclusively established. The exaggerated cross-modal facilitation effect in schizophrenia is bi-directional, not only found

Ac ce pt e

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

in vocal on facial emotion, but also in facial on vocal emotion (de Gelder et al., 2005). In addition, patients with schizophrenia rely less on the visual modality than healthy controls only for specific types of visual stimuli used in a novel paradigm (Stone et al., 2011). A critical evaluation of the auditory predominance hypothesis on MSI is warranted. Comparison of the multisensory performance during incongruent stimuli to the performance during unisensory visual stimuli could demonstrate a modality-specific interference effect.

Evidence from neuroimaging studies indicates that MSI deficits are associated with neurofunctional and anatomical alterations in the STS and superior temporal gyrus, areas important for higher-order speech integration (Calvert et al., 2001; Driver and Noesselt, 2008; Ethofer et al., 2006). MSI deficits in schizophrenia are associated with alterations in the inferior 16 Page 17 of 38

frontal gyrus, an area crucial for semantic and linguistic function (Sass et al., 2013; Stekelenburg et al., 2013; Straube et al., 2013; Szycik et al., 2009b; Szycik et al., 2013), and may account for behavioural findings suggesting a selective MSI deficit for linguistic information. Altered engagement of default mode and attentional networks in patients with schizophrenia may also

ip t

affected MSI (de Jong et al., 2010; Sass et al., 2013; Szycik et al., 2009b; Szycik et al., 2013),

cr

given the attentional demands of MSI tasks.

us

Research into MSI deficits in schizophrenia may also be confounded by clinical factors, such as illness chronicity and antipsychotic medications. In this regard, studies in patients with a first

an

episode of psychosis, or in people with prodromal symptoms, may circumvent some of these limitations. Similar to patients with schizophrenia (Gur et al., 2002; Hempel et al., 2003;

M

Williams et al., 2004), patients with first episode psychosis and people with prodromal signs and

d

symptoms show anatomical and functional abnormalities in the neural circuitry associated with emotional processing, such as in the right lingual and fusiform gyrus and prefronto-limbic

Ac ce pt e

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

functional connectivity (Modinos et al., 2010; Seiferth et al., 2008; Smieskova et al., 2010). These alterations may affect patients’ ability to effectively integrate multi-modal emotional information (Pauly et al., 2010; Seiferth et al., 2008). However, to date, no studies have examined the neural substrates of MSI in first episode or ultra high-risk cohorts.

A multi-stage emotional information-processing model (Adolphs, 2002; Wildgruber et al., 2009, see Figure 2) proposes an initial decoding stage in each sensory modality (facial, FFA and prosodic; prosodic, middle section of superior temporal gyrus). Multimodal emotion-laden sensory information are integrated in posterior STG/STS before being processed by corticolimbic regions (amygdala, parahippocampal area, inferior frontal gyrus) where the emotional significance of the input signal is evaluated and interpreted. Base on this model, 17 Page 18 of 38

neurofunctional and neuroantomical changes in STG/STS reported in ultra-high risk cohorts (Broome et al., 2009; Fusar-Poli et al., 2011; Fusar-Poli et al., 2007; Mechelli et al., 2011; Seiferth et al., 2008; Smieskova et al., 2010; Tognin et al., 2013) may underlie emotional

emotion-laden information are often multimodal in daily life.

ip t

processing difficulties that are characteristic of the prodromal phase of the illness, whilst

cr

MSI deficits at the initial perceptual stage may have a cascading impact on higher order

us

emotional processing; it is conceivable that impairment at an initial sensory integration stage will impact on the output to the later evaluation and interpretation stage. Functional MRI studies

an

explicitly designed to examine and isolate emotional processing deficits at both the early and late processing stages will help to clarify the nature of emotional processing deficits, and how these

M

deficits develop during the early and prodromal stage of schizophrenia. Of note, though not

d

discussed in the current review, evidence from an animal study suggests that the neurotransmitter glutamate may be important for MSI (Jacklin et al., 2012), based on its effect on attentional and

Ac ce pt e

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

default mode network (Hu et al., 2013). Glutamate alterations have been robustly established in schizophrenia, first episode schizophrenia, and ultra high-risk states (Marsman et al., 2013; Merritt et al., 2013).

7. Limitations

A limiting factor for the current review, making it difficult to reach more definite conclusions regarding cognitive and affective MSI impairments in schizophrenia, is the inconsistency between experimental paradigms across behavioural, electrophysiological and neurofunctional studies. These experimental paradigms measure various components of MSI; such as the perceptual domains, the cognitive domain, the spatial domain, the temporal domain, as well as varying contents of stimuli to be integrated. Contents of stimuli used in these paradigms are also 18 Page 19 of 38

of varying level of complexity (simple stimuli, such as target-detection paradigm, and complex stimuli with linguistic or emotional component, such as lip-reading paradigm, semantic priming paradigm and emotion recognition paradigm; see Table 1). Simple stimuli involve mainly perceptual processing, whereas the complex stimuli require additional cognitive, semantic or

ip t

affective processing. Furthermore, there are methodological concerns regarding the baseline

conditions used across different MSI studies. Some studies have chosen to contrast performance

cr

with an opposing congruency condition rather than a single modality baseline, thus might have

us

examined different components of MSI. It is essential for future researchers to identify the pertinent components of MSI to be observed before choosing adequate paradigms, and to

an

minimise the potential confounding factors; for example, acuity to signals in different sensory modalities potentially differs between patients in schizophrenia and healthy subjects. Ensuring

M

consistency between experimental paradigms used will enable better isolation of the

d

neuropsychological, neurophysiological and neurofunctional mechanisms underlying the cognitive-affective MSI impairments in schizophrenia. Another limitation is that the literature

Ac ce pt e

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

search strategy of the current review used one fixed set of search parameters and terms in PubMed. Although the reference sections of the cited studies were thoroughly scrutinised, it is possible that we did not include all the existing MSI studies in schizophrenia, particularly if some studies did not use integration/interference/conflict as one of their MESH terms.

8. Conclusions

MSI is an important perceptual and cognitive process. It is crucial in everyday life to integrate relevant information before responding promptly and appropriately in a complex and transient environment. The available literature indicates that patients with schizophrenia demonstrate impairments in the integration of non-emotional audiovisual stimuli, especially for complex linguistic stimuli. Altered MSI for emotional stimuli has also been reported in patients, although 19 Page 20 of 38

the effects may vary across target modalities. The neurofunctional mechanisms that are involved in MSI, particularly in the superior temporal and parietal cortices, appear to be affected in schizophrenia. Dysfunction of attentional networks may also compromise MSI. Further neuroimaging research is needed to understand the development and chronology of MSI deficits

ip t

in schizophrenia, first episode psychosis and prodromal populations. Such deficits which may be associated with early, but subtle, emotional and social cognition deficits are seen in psychosis

cr

risk cohorts (Addington et al., 2012; Amminger et al., 2012a; Amminger et al., 2012b;

us

Thompson et al., 2012). Novel therapies aimed at ameliorating MSI deficits in high-risk cohorts

d

M

an

may be effective in delaying or preventing the onset of full-blown psychosis.

Ac ce pt e

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

20 Page 21 of 38

Acknowledgement Dr Paul Allen is supported by a NARSAD Independent Investigator Award. Dr Matthijs Bossong

d

M

an

us

cr

ip t

is supported by a Rubicon grant from the Netherlands Organisation for Scientific Research.

Ac ce pt e

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

21 Page 22 of 38

Conflict of interest

d

M

an

us

cr

ip t

The authors have declared that no competing interests exist.

Ac ce pt e

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

22 Page 23 of 38

Role of the Funding Source The funders had no role in study design, data collection and analysis, decision to publish, or

d

M

an

us

cr

ip t

preparation of the manuscript.

Ac ce pt e

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

23 Page 24 of 38

References Addington, J., Piskulic, D., Perkins, D., Woods, S.W., Liu, L., Penn, D.L., 2012. Affect recognition in people at clinical high risk of psychosis. Schizophr Res 140, 87-92. Adolphs, R., 2002. Neural systems for recognizing emotion. Current opinion in neurobiology 12, 169-

ip t

177. Amminger, G.P., Schafer, M.R., Klier, C.M., Schlogelhofer, M., Mossaheb, N., Thompson, A., Bechdolf,

cr

A., Allott, K., McGorry, P.D., Nelson, B., 2012a. Facial and vocal affect perception in people at ultrahigh risk of psychosis, first-episode schizophrenia and healthy controls. Early intervention in psychiatry

us

6, 450-454.

Amminger, G.P., Schafer, M.R., Papageorgiou, K., Klier, C.M., Schlogelhofer, M., Mossaheb, N.,

an

Werneck-Rohrer, S., Nelson, B., McGorry, P.D., 2012b. Emotion recognition in individuals at clinical high-risk for schizophrenia. Schizophr Bull 38, 1030-1039.

discordance. Percept Psychophys 29, 578-584.

M

Bertelson, P., Radeau, M., 1981. Cross-modal bias and perceptual fusion with auditory-visual spatial

d

Brittain, P., Ffytche, D.H., McKendrick, A., Surguladze, S., 2010. Visual processing, social cognition and functional outcome in schizophrenia. Psychiatry Res 178, 270-275.

Ac ce pt e

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

Broome, M.R., Matthiasson, P., Fusar-Poli, P., Woolley, J.B., Johns, L.C., Tabraham, P., Bramon, E., Valmaggia, L., Williams, S.C., Brammer, M.J., Chitnis, X., McGuire, P.K., 2009. Neural correlates of executive function and working memory in the 'at-risk mental state'. The British journal of psychiatry : the journal of mental science 194, 25-33.

Calvert, G.A., Hansen, P.C., Iversen, S.D., Brammer, M.J., 2001. Detection of audio-visual integration sites in humans by application of electrophysiological criteria to the BOLD effect. Neuroimage 14, 427438. Carretie, L., 2014. Exogenous (automatic) attention to emotional stimuli: a review. Cognitive, affective & behavioral neuroscience. Carretie, L., Hinojosa, J.A., Martin-Loeches, M., Mercado, F., Tapia, M., 2004. Automatic attention to emotional stimuli: neural correlates. Human brain mapping 22, 290-299.

24 Page 25 of 38

Choe, C.S., Welch, R.B., Gilford, R.M., Juola, J.F., 1975. Ventriloquist Effect - Visual Dominance or Response Bias. Perception & Psychophysics 18, 55-60. Collignon, O., Girard, S., Gosselin, F., Roy, S., Saint-Amour, D., Lassonde, M., Lepore, F., 2008. Audiovisual integration of emotion expression. Brain Res 1242, 126-135.

ip t

de Boer-Schellekens, L., Stekelenburg, J.J., Maes, J.P., Van Gool, A.R., Vroomen, J., 2013. Sound improves diminished visual temporal sensitivity in schizophrenia. Acta Psychol (Amst).

cr

de Gelder, B., Vroomen, J., 2000. Bimodal emotion perception: integration across separate modalities, cross-modal perceptual grouping or perception of multimodal events? Cognition Emotion 14, 321-324.

us

de Gelder, B., Vroomen, J., Annen, L., Masthof, E., Hodiamont, P., 2003. Audio-visual integration in schizophrenia. Schizophrenia research 59, 211-218.

an

de Gelder, B., Vroomen, J., de Jong, S.J., Masthoff, E.D., Trompenaars, F.J., Hodiamont, P., 2005. Multisensory integration of emotional faces and voices in schizophrenics. Schizophr Res 72, 195-203.

M

de Jong, J.J., Hodiamont, P.P., de Gelder, B., 2010. Modality-specific attention and multisensory integration of emotions in schizophrenia: reduced regulatory effects. Schizophr Res 122, 136-143.

d

de Jong, J.J., Hodiamont, P.P., Van den Stock, J., de Gelder, B., 2009. Audiovisual emotion recognition in schizophrenia: Reduced integration of facial and vocal affect. Schizophr Res 107, 286-293.

Ac ce pt e

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

Driver, J., Noesselt, T., 2008. Multisensory interplay reveals crossmodal influences on 'sensory-specific' brain regions, neural responses, and judgments. Neuron 57, 11-23. Edwards, J., Jackson, H.J., Pattison, P.E., 2002. Emotion recognition via facial expression and affective prosody in schizophrenia: a methodological review. Clin Psychol Rev 22, 789-832. Egan, M.F., Goldberg, T.E., Gscheidle, T., Weirich, M., Bigelow, L.B., Weinberger, D.R., 2000. Relative risk of attention deficits in siblings of patients with schizophrenia. Am J Psychiatry 157, 1309-1316. Ethofer, T., Pourtois, G., Wildgruber, D., 2006. Investigating audiovisual integration of emotional signals in the human brain, in: S. Anders, G.E.M.J.J.K., Wildgruber, D. (Eds.), Progress in Brain Research. Elsevier, pp. 345-361.

25 Page 26 of 38

Foucher, J.R., Lacambre, M., Pham, B.T., Giersch, A., Elliott, M.A., 2007. Low time resolution in schizophrenia Lengthened windows of simultaneity for visual, auditory and bimodal stimuli. Schizophrenia research 97, 118-127. Fusar-Poli, P., Broome, M.R., Matthiasson, P., Woolley, J.B., Mechelli, A., Johns, L.C., Tabraham, P.,

ip t

Bramon, E., Valmaggia, L., Williams, S.C., McGuire, P., 2011. Prefrontal function at presentation directly related to clinical outcome in people at ultrahigh risk of psychosis. Schizophr Bull 37, 189-198.

cr

Fusar-Poli, P., Perez, J., Broome, M., Borgwardt, S., Placentino, A., Caverzasi, E., Cortesi, M., Veggiotti, P., Politi, P., Barale, F., McGuire, P., 2007. Neurofunctional correlates of vulnerability to psychosis: a

us

systematic review and meta-analysis. Neurosci Biobehav Rev 31, 465-484.

Gur, R.E., McGrath, C., Chan, R.M., Schroeder, L., Turner, T., Turetsky, B.I., Kohler, C., Alsop, D.,

schizophrenia. Am J Psychiatry 159, 1992-1999.

an

Maldjian, J., Ragland, J.D., Gur, R.C., 2002. An fMRI study of facial emotion processing in patients with

M

Hempel, A., Hempel, E., Schonknecht, P., Stippich, C., Schroder, J., 2003. Impairment in basal limbic function in schizophrenia during affect recognition. Psychiatry Res 122, 115-124.

Neuroscience 8, 393-402.

d

Hickok, G., Poeppel, D., 2007. The cortical organization of speech processing. Nature reviews.

Ac ce pt e

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

Hu, Y., Chen, X., Gu, H., Yang, Y., 2013. Resting-state glutamate and GABA concentrations predict task-induced deactivation in the default mode network. The Journal of neuroscience : the official journal of the Society for Neuroscience 33, 18566-18573. Jacklin, D.L., Goel, A., Clementino, K.J., Hall, A.W., Talpos, J.C., Winters, B.D., 2012. Severe crossmodal object recognition deficits in rats treated sub-chronically with NMDA receptor antagonists are reversed by systemic nicotine: implications for abnormal multisensory integration in schizophrenia. Neuropsychopharmacology 37, 2322-2331. Jardri, R., Pins, D., Bubrovszky, M., Lucas, B., Lethuc, V., Delmaire, C., Vantyghem, V., Despretz, P., Thomas, P., 2009. Neural functional organization of hallucinations in schizophrenia: multisensory dissolution of pathological emergence in consciousness. Conscious Cogn 18, 449-457.

26 Page 27 of 38

Koelewijn, T., Bronkhorst, A., Theeuwes, J., 2010. Attention and the multiple stages of multisensory integration: A review of audiovisual studies. Acta Psychol (Amst) 134, 372-384. Kohler, C.G., Turner, T.H., Bilker, W.B., Brensinger, C.M., Siegel, S.J., Kanes, S.J., Gur, R.E., Gur, R.C., 2003. Facial emotion recognition in schizophrenia: intensity effects and error pattern. Am J

ip t

Psychiatry 160, 1768-1774. Laroi, F., Fonteneau, B., Mourad, H., Raballo, A., 2010. Basic emotion recognition and psychopathology

cr

in schizophrenia. The Journal of nervous and mental disease 198, 79-81.

Leentjens, A.F., Wielaert, S.M., van Harskamp, F., Wilmink, F.W., 1998. Disturbances of affective

us

prosody in patients with schizophrenia; a cross sectional study. Journal of Neurology, Neurosurgery & Psychiatry 64, 375-378.

an

Leitman, D.I., Foxe, J.J., Butler, P.D., Saperstein, A., Revheim, N., Javitt, D.C., 2005. Sensory contributions to impaired prosodic processing in schizophrenia. Biol Psychiatry 58, 56-61.

M

Liberati, A., Altman, D.G., Tetzlaff, J., Mulrow, C., Gotzsche, P.C., Ioannidis, J.P., Clarke, M., Devereaux, P.J., Kleijnen, J., Moher, D., 2009. The PRISMA statement for reporting systematic reviews

b2700.

d

and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. Bmj 339,

Ac ce pt e

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

Marsman, A., van den Heuvel, M.P., Klomp, D.W., Kahn, R.S., Luijten, P.R., Hulshoff Pol, H.E., 2013. Glutamate in schizophrenia: a focused review and meta-analysis of (1)H-MRS studies. Schizophr Bull 39, 120-129.

McGurk, H., MacDonald, J., 1976. Hearing lips and seeing voices. Nature 264, 746-748. Mechelli, A., Riecher-Rossler, A., Meisenzahl, E.M., Tognin, S., Wood, S.J., Borgwardt, S.J., Koutsouleris, N., Yung, A.R., Stone, J.M., Phillips, L.J., McGorry, P.D., Valli, I., Velakoulis, D., Woolley, J., Pantelis, C., McGuire, P., 2011. Neuroanatomical abnormalities that predate the onset of psychosis: a multicenter study. Archives of general psychiatry 68, 489-495. Merritt, K., McGuire, P., Egerton, A., 2013. Relationship between Glutamate Dysfunction and Symptoms and Cognitive Function in Psychosis. Frontiers in psychiatry 4, 151.

27 Page 28 of 38

Mesholam-Gately, R.I., Giuliano, A.J., Goff, K.P., Faraone, S.V., Seidman, L.J., 2009. Neurocognition in first-episode schizophrenia: a meta-analytic review. Neuropsychology 23, 315-336. Miller, J., 1982. Divided attention: Evidence for coactivation with redundant signals. Cognitive Psychology 14, 247-279.

ip t

Miller, J., 1986. Timecourse of coactivation in bimodal divided attention. Attention, Perception, & Psychophysics 40, 331-343.

cr

Modinos, G., Ormel, J., Aleman, A., 2010. Altered activation and functional connectivity of neural

systems supporting cognitive control of emotion in psychosis proneness. Schizophr Res 118, 88-97.

us

Moher, D., Liberati, A., Tetzlaff, J., Altman, D.G., Group, P., 2009. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Bmj 339, b2535.

an

Muller, V.I., Cieslik, E.C., Kellermann, T.S., Eickhoff, S.B., 2013a. Crossmodal emotional integration in major depression. Social cognitive and affective neuroscience.

M

Muller, V.I., Cieslik, E.C., Laird, A.R., Fox, P.T., Eickhoff, S.B., 2013b. Dysregulated left inferior

Neurosci 7, 268.

d

parietal activity in schizophrenia and depression: functional connectivity and characterization. Front Hum

Muller, V.I., Habel, U., Derntl, B., Schneider, F., Zilles, K., Turetsky, B.I., Eickhoff, S.B., 2011.

Ac ce pt e

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

Incongruence effects in crossmodal emotional integration. Neuroimage 54, 2257-2266. Muller, V.I., Kellermann, T.S., Seligman, S.C., Turetsky, B.I., Eickhoff, S.B., 2012. Modulation of Affective Face Processing Deficits in Schizophrenia by Congruent Emotional Sounds. Social cognitive and affective neuroscience.

Pauly, K., Seiferth, N.Y., Kellermann, T., Ruhrmann, S., Daumann, B., Backes, V., Klosterkotter, J., Shah, N.J., Schneider, F., Kircher, T.T., Habel, U., 2010. The interaction of working memory and emotion in persons clinically at risk for psychosis: An fMRI pilot study. Schizophr Res 120, 167-176. Pearl, D., Yodashkin-Porat, D., Katz, N., Valevski, A., Aizenberg, D., Sigler, M., Weizman, A., Kikinzon, L., 2009. Differences in audiovisual integration, as measured by McGurk phenomenon, among adult and adolescent patients with schizophrenia and age-matched healthy control groups. Comprehensive psychiatry 50, 186-192.

28 Page 29 of 38

Penn, D.L., Corrigan, P.W., Bentall, R.P., Racenstein, J.M., Newman, L., 1997. Social cognition in schizophrenia. Psychological Bulletin 121, 114-132. Pinkham, A., Penn, D., Wangelin, B., Perkins, D., Gerig, G., Gu, H., Lieberman, J., 2005. Facial emotion perception and fusiform gyrus volume in first episode schizophrenia. Schizophr Res 79, 341-343.

ip t

Pinkham, A.E., Penn, D.L., Perkins, D.O., Lieberman, J., 2003. Implications for the neural basis of social cognition for the study of schizophrenia. Am J Psychiatry 160, 815-824.

cr

Ross, L.A., Saint-Amour, D., Leavitt, V.M., Molholm, S., Javitt, D.C., Foxe, J.J., 2007. Impaired

multisensory processing in schizophrenia: deficits in the visual enhancement of speech comprehension

us

under noisy environmental conditions. Schizophrenia research 97, 173-183.

Sass, K., Heim, S., Sachs, O., Straube, B., Schneider, F., Habel, U., Kircher, T., 2013. Neural correlates

an

of semantic associations in patients with schizophrenia. Eur Arch Psychiatry Clin Neurosci.

integration. Psychophysiology 35, 755-759.

M

Schröger, E., Widmann, A., 1998. Speeded responses to audiovisual signal changes result from bimodal

Seiferth, N.Y., Pauly, K., Habel, U., Kellermann, T., Shah, N.J., Ruhrmann, S., Klosterkotter, J.,

for psychosis. Neuroimage 40, 289-297.

d

Schneider, F., Kircher, T., 2008. Increased neural response related to neutral faces in individuals at risk

Ac ce pt e

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

Seubert, J., Loughead, J., Kellermann, T., Boers, F., Brensinger, C.M., Habel, U., 2010. Multisensory integration of emotionally valenced olfactory-visual information in patients with schizophrenia and healthy controls. Journal of psychiatry & neuroscience : JPN 35, 185-194. Shenton, M.E., Dickey, C.C., Frumin, M., McCarley, R.W., 2001. A review of MRI findings in schizophrenia. Schizophr Res 49, 1-52.

Smieskova, R., Fusar-Poli, P., Allen, P., Bendfeldt, K., Stieglitz, R.D., Drewe, J., Radue, E.W., McGuire, P.K., Riecher-Rossler, A., Borgwardt, S.J., 2010. Neuroimaging predictors of transition to psychosis--a systematic review and meta-analysis. Neurosci Biobehav Rev 34, 1207-1222. Smiley, J.F., Konnova, K., Bleiwas, C., 2012. Cortical thickness, neuron density and size in the inferior parietal lobe in schizophrenia. Schizophr Res 136, 43-50.

29 Page 30 of 38

Smiley, J.F., Rosoklija, G., Mancevski, B., Mann, J.J., Dwork, A.J., Javitt, D.C., 2009. Altered volume and hemispheric asymmetry of the superficial cortical layers in the schizophrenia planum temporale. Eur J Neurosci 30, 449-463. Stekelenburg, J.J., Maes, J.P., Van Gool, A.R., Sitskoorn, M., Vroomen, J., 2013. Deficient multisensory

ip t

integration in schizophrenia: an event-related potential study. Schizophr Res 147, 253-261. Stephen, J.M., Coffman, B.A., Jung, R.E., Bustillo, J.R., Aine, C.J., Calhoun, V.D., 2013. Using joint

cr

ICA to link function and structure using MEG and DTI in schizophrenia. Neuroimage 83, 418-430. Stone, D.B., Urrea, L.J., Aine, C.J., Bustillo, J.R., Clark, V.P., Stephen, J.M., 2011. Unisensory

us

processing and multisensory integration in schizophrenia: A high-density electrical mapping study. Neuropsychologia 49, 3178-3187.

an

Straube, B., Green, A., Sass, K., Kircher, T., 2013. Superior Temporal Sulcus Disconnectivity During Processing of Metaphoric Gestures in Schizophrenia. Schizophr Bull.

M

Surguladze, S.A., Calvert, G.A., Brammer, M.J., Campbell, R., Bullmore, E.T., Giampietro, V., David, A.S., 2001. Audio-visual speech perception in schizophrenia: an fMRI study. Psychiatry research 106, 1-

d

14.

Szycik, G.R., Munte, T.F., Dillo, W., Mohammadi, B., Samii, A., Emrich, H.M., Dietrich, D.E., 2009a.

Ac ce pt e

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

Audiovisual integration of speech is disturbed in schizophrenia: an fMRI study. Schizophr Res 110, 111118.

Szycik, G.R., Munte, T.F., Dillo, W., Mohammadi, B., Samii, A., Emrich, H.M., Dietrich, D.E., 2009b. Audiovisual integration of speech is disturbed in schizophrenia: an fMRI study. Schizophrenia research 110, 111-118.

Szycik, G.R., Ye, Z., Mohammadi, B., Dillo, W., Te Wildt, B.T., Samii, A., Frieling, H., Bleich, S., Munte, T.F., 2013. Maladaptive connectivity of Broca's area in schizophrenia during audiovisual speech perception: An fMRI study. Neuroscience. Thompson, A., Papas, A., Bartholomeusz, C., Allott, K., Amminger, G.P., Nelson, B., Wood, S., Yung, A., 2012. Social cognition in clinical "at risk" for psychosis and first episode psychosis populations. Schizophr Res 141, 204-209.

30 Page 31 of 38

Tognin, S., Riecher-Rossler, A., Meisenzahl, E.M., Wood, S.J., Hutton, C., Borgwardt, S.J., Koutsouleris, N., Yung, A.R., Allen, P., Phillips, L.J., McGorry, P.D., Valli, I., Velakoulis, D., Nelson, B., Woolley, J., Pantelis, C., McGuire, P., Mechelli, A., 2013. Reduced parahippocampal cortical thickness in subjects at ultra-high risk for psychosis. Psychological medicine, 1-10.

ip t

Tseng, H.H., Chen, S.H., Liu, C.M., Howes, O., Huang, Y.L., Hsieh, M.H., Liu, C.C., Shan, J.C., Lin, Y.T., Hwu, H.G., 2013. Facial and prosodic emotion recognition deficits associate with specific clusters

cr

of psychotic symptoms in schizophrenia. PloS one 8, e66571.

Van den Stock, J., de Jong, S.J., Hodiamont, P.P., de Gelder, B., 2011. Perceiving emotions from bodily

us

expressions and multisensory integration of emotion cues in schizophrenia. Social neuroscience 6, 537547.

an

Vuilleumier, P., Huang, Y.M., 2009. Emotional Attention: Uncovering the Mechanisms of Affective Biases in Perception. Curr Dir Psychol Sci 18, 148-152.

Psychological Bulletin 88, 638-667.

M

Welch, R.B., Warren, D.H., 1980. Immediate Perceptual Response to Intersensory Discrepancy.

d

Wildgruber, D., Ethofer, T., Grandjean, D., Kreifelts, B., 2009. A cerebral network model of speech prosody comprehension. International Journal of Speech-Language Pathology 11, 277-281.

Ac ce pt e

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65

Williams, L.E., Light, G.A., Braff, D.L., Ramachandran, V.S., 2010. Reduced multisensory integration in patients with schizophrenia on a target detection task. Neuropsychologia 48, 3128-3136. Williams, L.M., Das, P., Harris, A.W., Liddell, B.B., Brammer, M.J., Olivieri, G., Skerrett, D., Phillips, M.L., David, A.S., Peduto, A., Gordon, E., 2004. Dysregulation of arousal and amygdala-prefrontal systems in paranoid schizophrenia. Am J Psychiatry 161, 480-489.

31 Page 32 of 38

cr

ip t

Table(s)

Tables

Author

Method and Task paradigm

Non-emotional

us

Table 1. Summary of Non-emotional and Emotional Studies of Multisensory Integration in Schizophrenia

Sample size

Cognitive –Behavioural ventriloquist effect Lip-reading paradigm, incongruent

18 SCH, 12 NC

Pearl et al. (2009)

Cognitive –Behavioural Lip-reading paradigm, incongruent Cognitive –Behavioural Lip-reading paradigm, congruent Cognitive –Behavioural Target-detection paradigm Cognitive –Behavioural visual temporal order judgment task

30 SCH, 20 HC

No group differences in spatial multisensory integration  lip-reading on audiovisual speech perception  audiovisual integrative process in SCH

18 SCH, 18 HC

 benefit from congruent information in SCH

20SCH, 20 HC

Foucher et al. (2007)

Cognitive –Behavioural simultaneity judgement paradigm

30 SCH, 33 HC

 benefit on RT for congruent information in SCH No deficits in the integration of low-level auditory and visual information  visual temporal order judgement, No specific deficits in multisensory integration  resolution of time perception

Stone et al. (2011)

ERP Far-near judgment

Williams et al. (2010)

Ac c

M

ep te

de Boer-Schellekens et al. (2013)

d

Ross et al. (2007)

an

de Gelder et al. (2003)

Stekelenburg et al. (2013)

Stephen et al. (2013)

Sass et al. (2013)

16 SCH, 16 HC

EEG/ERP mixed lip-reading non lip-reading paradigm Joint independent component analysis of MEG and DTI Far-near judgment

18 SCH, 18 HC

fMRI semantic priming paradigm

14 SCH, 14 HC,

29 SCH, 29 HC

 benefit from congruent information in schizophrenia  absolute magnitude of evoked brain response from spatially congruent information in SCH  suppression of auditory evoked brain response  RT facilitation in schizophrenia  RT facilitation benefit from congruent information in schizophrenia  activation in superior frontal gyrus, middle temporal gyrus, anterior cingulate and angular gyrus SCH

Page 33 of 38

14 SCH, 7 HC

fMRI Lip-reading paradigm, congruent vs incongruent fMRI Lip-reading paradigm, congruent vs incongruent

15 SCH, 15 HC

Cognitive –Behavioural happy-sad face continuum face on voices voice on faces Cognitive –Behavioural happy-sad and happy-fearful face-voice pairs, congruent vs incongruent face on voices Cognitive –Behavioural happy-sad and happy-fearful face-voice pairs with or without visual or auditory distractors, congruent vs incongruent face on voices

13 SCH, 13 HC

Cognitive –Behavioural pleasant and unpleasant olfactory priming on judging facial expression, congruent or incongruent Cognitive –Behavioural whole-body expressions with human or animal vocalizations, congruent or incongruent ERP happy, neutral, fear audiovisual emotional information, congruent or incongruent

24 SCH, 24 HC

15 SCH, 15HC

M

Szycik et al. (2013)

cr

ip t Szycik et al. (2009a)

16 SCH, 16HC

us

Surguladze et al. (2001)

fMRI Videos of an actor performing gestures in a concrete and abstract sentence context fMRI and Behavioural Lip-reading paradigm, incongruent

an

Straube et al. (2013)

Emotional

ep te

d

de Gelder et al. (2005)

de Jong et al. (2009)

Ac c

de Jong et al. (2010a)

Seubert et al. (2010)

Van den Stock et al. (2011)

Muller et al. (2012)

50 SCH, 46 NonSCH, HC

50 SCH, 46 NonSCH psychotic

 connectivity from STS to middle temporal gyrus and ventral IFG during the abstract relative to concrete gestures No group differences on audiovisual speech perception Group × congruency: less activation difference or opposite activation pattern for incongruent than for congruent stimuli in SCH  adaptive connectivity in right posterior STS and left inferior frontal gyrus (Broca’s area)

 influence of emotional voice on face continuum in SCH  influence of emotional face on voice continuum in SCH  benefit from congruent relative to incongruent information in SCH Comparing to conditions without distractors:  benefit when with distractors in HC similar benefit when with visual distractors in SCH  benefit when with auditory distractors in SCH  olfactory priming facilitation on disgust recognition in SCH, mainly unpleasant odour priming

SCH, non-SCH psychotic

 cross-modality influence from human voices

18 SCH, 18 HC

 amplitudes associated with visual processing during emotionally incongruent stimulus pairs

Page 34 of 38

ip t cr

fMRI happy, neutral, fear audiovisual information, congruent or incongruent

15 SCH, 15 HC

 deactivation of the left IPC during congruent audiovisual information

us

Muller et al. (2013b)

Ac c

ep te

d

M

an

HC, healthy control; SCH, patients with schizophrenia; RT, reaction times; STS, superior temporal sulcus; IFG, inferior frontal gyrus; IPC, inferior parietal cortex.

Page 35 of 38

ip t Method and Task paradigm rubber hand illusion paradigm

Jardri et al. (2009b)

Case report, multimodal

an

Ferri et al. (2013)

Sample size

Post-mortem study

Smiley et al. (2009b)

Post-mortem study

Compton et al. (2006)

Neurological examination includes audiovisual integration Neurological examination includes audiovisual integration Neurological examination includes audiovisual integration Neurological examination includes audiovisual integration

d

ep te

Sanders et al. (2006)

M

Smiley et al. (2012b)

Sanders et al. (2000)

Sense of ownership over the rubber hand

1 SCH

activation in bilateral STS, occipital temporal sulcus and anterior cingulate cortex during multisensory hallucination No differences in gray volumes of planum temporal;  upper layer thickness of the caudal of planum temporal in SCH No differences in gray matter volume in inferior parietal lobule;  thickness in angular gyrus relative to supramarginal gyrus in SCH audiovisual integration task was included in a ‘sensory integration factor’ audiovisual integration task was included in a ‘cognitive-perceptual factor’ audiovisual integration task showed statistically significant heritability performance of audiovisual integration task: SCH+OCD > OCD .

24 SCH, 24 Control 19 SCH, 18 Control 110 SCH 103SCH 96 relatives of SCH 30 SCH, 30 OCD, 16 SCH+OCD, 13 OCD with poor insight

Ac c

Tumkaya et al. (2012)

21 SCH, 17 HC

us

Author

cr

Table 2. Summary of other studies related to multisensory integration in schizophrenia

HC, healthy control; SCH, patients with schizophrenia; OCD, patients with obsessive compulsive disorder; STS, superior temporal sulcus.

Page 36 of 38

Figure

Figures

48 studies found in

cr

ip t

literature search

excluded

7 studies

d

using emotional experimental paradigm

8 remaining studies were included due to relevance

Ac ce pt e

14 studies using nonemotional experimental paradigms

M

an

29 studies, meeting the inclusion criteria, included in this review

us

19 studies

Figure 1 Flowchart of systematically reviewing protocol

1 Page 37 of 38

Ac ce pt e

d

M

an

us

cr

ip t

Figure 2 Multi-stage model of multisensory processing in schizophrenia (Adapted from Wildgruber et al., 2009)

SCH, patients with schizophrenia; A1, primary auditory cortex; V1, primary visual cortex; STC, superior temporal cortex; IFC, inferior frontal cortex; OFC, orbital frontal cortex; MFC, medial frontal cortex

2 Page 38 of 38