SCHRES-06385; No of Pages 6 Schizophrenia Research xxx (2015) xxx–xxx
Contents lists available at ScienceDirect
Schizophrenia Research journal homepage: www.elsevier.com/locate/schres
Virtual reality job interview training and 6-month employment outcomes for individuals with schizophrenia seeking employment Matthew J. Smith a,b,⁎, Michael F. Fleming a,c, Michael A. Wright a, Andrea G. Roberts a,b, Laura Boteler Humm d,1, Dale Olsen d,1, Morris D. Bell e a
Northwestern University Feinberg School of Medicine, Department of Psychiatry and Behavioral Sciences, Chicago, IL, USA Northwestern University Feinberg School of Medicine, Warren Wright Adolescent Center, Chicago, IL, USA Northwestern University Feinberg School of Medicine, Department of Family Medicine, Chicago, IL, USA d SIMmersion LLC, Columbia, MD, USA e Yale School of Medicine, Department of Psychiatry, Department of Veteran Affairs, West Haven, CT, USA b c
a r t i c l e
i n f o
Article history: Received 5 March 2015 Received in revised form 12 May 2015 Accepted 12 May 2015 Available online xxxx Keywords: Schizophrenia Virtual reality training Job interview skills Vocational outcomes
a b s t r a c t Background: Individuals with schizophrenia have low employment rates and the job interview presents a critical barrier for them to obtain employment. Virtual reality training has demonstrated efficacy at improving interview skills and employment outcomes among multiple clinical populations. However, the effects of this training on individuals with schizophrenia are unknown. This study evaluated the efficacy of virtual reality job interview training (VR-JIT) at improving job interview skills and employment outcomes among individuals with schizophrenia in a small randomized controlled trial (n = 21 VR-JIT trainees, n = 11 waitlist controls). Methods: Trainees completed up to 10 h of virtual interviews using VR-JIT, while controls received services as usual. Primary outcome measures included two pre-test and two post-test video-recorded role-play interviews scored by blinded raters with expertise in human resources and self-reported interviewing selfconfidence. Six-month follow-up data on employment outcomes were collected. Results: Trainees reported that the intervention was easy-to-use, helpful, and prepared them for future interviews. Trainees demonstrated increased role-play scores between pre-test and post-test while controls did not (p = 0.001). After accounting for neurocognition and months since prior employment, trainees had greater odds of receiving a job offer by 6 month follow-up compared to controls (OR: 8.73, p = 0.04) and more training was associated with fewer weeks until receiving a job offer (r = −0.63, p b 0.001). Conclusions: Results suggest that VR-JIT is acceptable to trainees and may be efficacious for improving job interview skills in individuals with schizophrenia. Moreover, trainees had greater odds of receiving a job offer by 6-month follow-up. Future studies could evaluate the effectiveness of VR-JIT within community-based services. © 2015 Elsevier B.V. All rights reserved.
1. Introduction Less than 20% of individuals with schizophrenia attain employment (Rosenheck et al., 2006; Salkever et al., 2007), while this rate climbs to 30–40% for individuals in Individual Placement and Support (IPS) services (Drake and Bond, 2011). Successfully navigating the job interview is critical to attaining employment and is the most proximal step to attaining employment for IPS clients (Corbiere et al., 2011). Moreover, social cognitive and neurocognitive deficits that characterize schizophrenia have been associated with poorer vocational outcomes (Eack ⁎ Corresponding author at: Northwestern University Feinberg School of Medicine, Department of Psychiatry and Behavioral Sciences, 710 N. Lake Shore Dr, Abbott Hall 13th floor, Chicago, IL 60611. Tel.: 1 312 503 2542, fax: 1 312 503 0527. E-mail address: [email protected] (M.J. Smith). 1 Dr. Olsen and Ms. Humm contributed to the design of the study and manuscript preparation, but were not involved in the collection or analysis of data.
et al., 2011; Vargas et al., 2014; Martinez-Dominguez et al., 2015), and likely increase the difficulty of interviewing. Although individuals with schizophrenia self-identify having poor interview skills and want services to enhance these skills (Solinski et al., 1992; Marwaha and Johnson, 2006), there is a paucity of evidence-based interventions targeting interview skills (Bell and Weinstein, 2011). Recently, two randomized controlled trials (RCTs) evaluated virtual reality job interview training (VR-JIT), which demonstrated acceptability and efficacy at improving interview skills and self-confidence for individuals primarily diagnosed with mood disorders and veterans with posttraumatic stress disorder (PTSD) (Smith et al., 2014b, 2015a). Moreover, VR-JIT trainees sustained their self-confidence and were more likely to receive job offers than controls when evaluated six months later. Additionally, completing more VR-JIT trials was associated with greater odds of receiving a job offer and spending fewer weeks searching for employment (Smith et al., 2015b).
http://dx.doi.org/10.1016/j.schres.2015.05.022 0920-9964/© 2015 Elsevier B.V. All rights reserved.
Please cite this article as: Smith, M.J., et al., Virtual reality job interview training and 6-month employment outcomes for individuals with schizophrenia seeking employment, Schizophr. Res. (2015), http://dx.doi.org/10.1016/j.schres.2015.05.022
2
M.J. Smith et al. / Schizophrenia Research xxx (2015) xxx–xxx
Thus, we hypothesized that 1) individuals with schizophrenia randomized to training would find VR-JIT acceptable and enhance their interviewing skills and interviewing self-confidence compared to waitlist controls; 2) trainees with schizophrenia would sustain their enhanced self-confidence and have greater odds of receiving a job offer compared to controls at 6-month follow-up; and 3) a greater number of completed VR-JIT trials, improved VR-JIT performance across trials, and self-confidence at the conclusion of training would be correlated with receiving a job offer and with fewer weeks until a job was offered. We generated these directional hypotheses based on our prior research (Smith et al., 2015b). 2. Methods 2.1. Participants Participants included 32 individuals with schizophrenia or schizoaffective disorder recruited through Northwestern University's Schizophrenia Research Group. Bachelor's or Ph.D.-level research staff determined diagnoses (and antipsychotic treatment) using the Structured Clinical Interview for the DSM-IV-TR (SCID-IV) (First et al., 2002). Inclusion criteria included: 18–55 years old, minimum of a 6th grade reading level using the Wide Range Achievement Test-IV (WRAT-IV) (Wilkinson and Robertson, 2006), willingness to be videorecorded, unemployed or underemployed, and actively seeking employment. Exclusion criteria included: having a medical illness that significantly comprised cognition (e.g., traumatic brain injury), uncorrected vision or hearing problem, or current substance abuse. Northwestern University's Institutional Review Board approved the study protocol and all participants provided informed consent. Once enrolled, participants were randomized using a random number generator into training (n = 17) or treatment-as-usual waitlist control (n = 8) groups at an estimated ratio of 2 to 1 to optimize VR-JIT evaluation. Four trainees and three controls with schizophrenia completed prior RCTs of VR-JIT (Smith et al., 2014b, 2015a) and their data were included in all current analyses to optimize statistical power with final sample sizes of n = 21 trainees and n = 11 controls. Participants were re-contacted after 6 months and asked to complete a followup survey. Of the original 32 participants, 30 (94%) completed the follow-up survey and 2 (6%) were lost to contact. 2.2. Intervention Virtual Reality Job Interview Training (VR-JIT) is a computer-based intervention developed by SIMmersion LLC (http://www.simmersion. com) to improve interviewing skills for adults with a range of disabilities. See Supplemental Material and Smith et al. (2014b) for details on VR-JIT design, use, and delivery (e.g., fidelity training). 2.3. Study procedures Pre-test measures included demographic, clinical, and vocational interviews; clinical, neurocognitive, and social cognitive assessments; and self-reported self-confidence and two standardized video-recorded roleplays. Participants were randomized following pre-test assessments. Trainees completed up to 10 h of VR-JIT (approximately 20 trials) over the course of 5 visits (across 5–10 business days). Controls received services-as-usual for 5–10 business days. Both groups returned after ten business days to complete post-test measures of self-confidence and two standardized video-recorded role-plays, while trainees also completed the Treatment Experience Questionnaire (TEQ). After the post-test visit, all controls were invited from the waitlist to use VR-JIT. Two controls used VR-JIT and crossed-over to the training group (Fig. 1). Research staff contacted participants to complete a follow-up survey over the phone or via email approximately 6 months after completing the efficacy trial. Two controls were unreachable by phone, mail, and
Fig. 1. Consort diagram of study participation.
email, and were lost to contact. Overall, 23 VR-JIT and 7 controls completed follow-up (Fig. 1). 2.4. Study measures 2.4.1. Participant characteristics We assessed demographic characteristics and vocational history via self-report. Symptoms were assessed using global ratings from the Scale for the Assessment of Positive Symptoms and the Scale for the assessment of Negative Symptoms (Andreasen, 1983a, 1983b). For the 7 participants who completed prior studies (Smith et al., 2014b, 2015a), we used ratings and notes from the Mini-International Neuropsychiatric Interview (Sheehan et al., 1998) to inform symptom ratings and SCID diagnosis. 2.4.2. Cognition We assessed global cognitive ability with the total score from the Repeatable Battery for the Assessment of Neuropsychological Status (Randolph et al., 1998). We assessed basic social cognition using the Bell-Lysaker Emotion Recognition Task (Bell et al., 1997), and advanced social cognition using an emotional perspective-taking task (Smith et al., 2014c). Accuracy ratings for each task were generated using the number of correct responses. 2.4.3. VR-JIT acceptability We recorded trainee attendance across five sessions and the number of minutes (600 min possible) that they engaged in virtual interviews. Trainees completed the TEQ to evaluate whether they felt that VR-JIT was easy to use, enjoyable, helpful, instilled confidence, and prepared them for interviews (Bell and Weinstein, 2011). The TEQ used a 7-point Likert scale with higher scores reflecting more positive views (α = 0.86).
Please cite this article as: Smith, M.J., et al., Virtual reality job interview training and 6-month employment outcomes for individuals with schizophrenia seeking employment, Schizophr. Res. (2015), http://dx.doi.org/10.1016/j.schres.2015.05.022
M.J. Smith et al. / Schizophrenia Research xxx (2015) xxx–xxx
3
Table 1 Baseline characteristics.
Demographics Age, mean years (SD) Gender (% male) Parental education, mean years (SD) Race % Caucasian % African-American % Latino Vocational history Months since prior employment, mean (SD)a Prior full-time employment (%) Prior paid employment (any type) (%) Prior participation in vocational training program (%) Cognitive function Neurocognition, mean (SD) Basic social cognition, mean (SD) Advanced social cognition, mean (SD) Clinical history Duration of illness, mean years (SD) Clinical symptoms Hallucinations Delusions Bizarre behavior Thought disorder Affective flattening Alogia Avolitionb Anhedoniac Attentionc Chlorpromazine equivalent Treated with atypical antipsychotic medication (%) Treated with typical antipsychotic medication (%) a b c
Control group (n = 11)
VR-JIT group (n = 21)
p-Value
39.1 (10.6) 54.5% 11.7 (1.8)
40.8 (12.2) 52.4% 12.9 (3.4)
0.76 0.91 0.46
27.3% 54.5% 18.2%
28.6% 66.7% 4.8%
0.46
26.9 (41.0) 63.6% 100% 27.3%
65.2 (92.4) 71.4% 95.2% 38.1%
0.27 0.65 0.46 0.54
72.5 (12.6) 0.71 (0.2) 0.77 (0.1)
74.8 (16.4) 0.69 (0.1) 0.74 (0.1)
0.76 0.67 0.88
16.0 (11.4)
19.5 (11.2)
0.35
2.36 (2.1) 2.36 (2.1) 0.36 (0.8) 1.18 (1.3) 1.64 (1.4) 1.27 (1.3) 2.90 (0.8) 2.38 (1.6) 1.75 (0.9) 245.25 (164.9) 81.8% 18.2%
1.71 (1.8) 2.50 (1.9) 0.95 (1.2) 0.90 (1.3) 1.86 (1.6) 1.14 (1.5) 2.90 (1.1) 3.73 (1.4) 2.00 (1.5) 340.81 (197.9) 90.0% 10.0%
0.39 0.88 0.24 0.56 0.73 0.79 0.86 0.05 0.73 0.18 0.52
Two outliers excluded for VR-JIT trainees: 324 months and 276 months. One VR-JIT trainee was not rated on this domain. Three controls and six VR-JIT trainees were not rated on this domain.
2.4.4. VR-JIT efficacy 2.4.4.1. Role-play performance. Job interview role-plays (approximately 20 min each) were scored across nine domains: 1) conveying oneself as a hard worker, 2) sounding easy to work with, 3) conveying that one behaves professionally, 4) negotiation skills (requesting Thursdays off), 5) sharing things in a positive way, 6) sounding honest, 7) sounding interested in the position, 8) comfort level, and 9) establishing rapport; which were adapted from prior work (Huffcutt, 2011). Role-play videos were randomly assigned to two blinded raters with more than 15 years of experience in human resources. Total scores for two pre-test and two post-test role-plays were computed across the nine domains (range of 1–5, higher scores reflecting better performance), and averaged to reflect a single score.
2.4.5. Six-month follow-up measures The follow-up survey asked participants to reflect on the past 6 months and report the 1) total number of weeks they searched for employment, 2) number of job interviews completed, and 3) number of job offers received and accepted. The survey reassessed interviewing self-confidence (α = 0.89), and assessed whether trainees felt that VRJIT prepared them for real interviews, helped them attain employment, and if revisiting VR-JIT would prepare them for future interviews. These items were rated on a 5-point Likert scale (1 = strongly disagree to 5 = strongly agree).
3. Data analysis 3.1. Efficacy study
2.4.4.2. Interviewing self-confidence. Participants rated their selfconfidence at interviewing using a 7-point Likert scale to answer nine questions, with higher scores reflecting more positive views (e.g., “How comfortable are you going on a job interview?”). Total scores at pre-test and post-test had strong internal consistencies (α = 0.95 and α = 0.92, respectively).
2.4.4.3. VR-JIT process measures. We recorded trainees' VR-JIT performance scores, number of completed trials, and time spent performing virtual interviews. Each virtual interview was scored 0–100 using an algorithm that assessed the appropriateness of responses based on eight domains: negotiation skills, conveying that you're a hard worker, sounding easy to work with, sharing things in a positive way, sounding honest, sounding interested in the position, behaving professionally, and establishing interviewer rapport.
Between-group differences for demographics, vocational history, cognition, and clinical history were assessed with a Mann–Whitney independent samples test or chi-square analysis. We used descriptive statistics to assess VR-JIT acceptability via session attendance, VR-JIT usage Table 2 Mean characteristics of VR-JIT acceptability (SD). Attendance measures % session attendance Elapsed simulation time (min) TEQ items Ease of use Enjoyable Helpful Instilled confidence Prepared for interviews
90.0% 537.9 (102.9) 6.0 (1.0) 6.7 (0.7) 6.6 (0.7) 5.9 (1.2) 6.0 (1.0)
Note. Scale for TEQ, 1 = extremely unhelpful to 7 = extremely helpful.
Please cite this article as: Smith, M.J., et al., Virtual reality job interview training and 6-month employment outcomes for individuals with schizophrenia seeking employment, Schizophr. Res. (2015), http://dx.doi.org/10.1016/j.schres.2015.05.022
4
M.J. Smith et al. / Schizophrenia Research xxx (2015) xxx–xxx
4. Results 4.1. Pre-test between-group characteristics Trainees and controls did not differ with respect to demographics as well as clinical, cognitive, and vocational history (all p N 0.10). Despite random assignment, anhedonia ratings differed between groups (but were missing for nine participants) (p b 0.05) (Table 1). 4.2. VR-JIT acceptability
Fig. 2. VR-JIT learning curve in individuals with schizophrenia. This figure plots the average score for each successive VR-JIT virtual interview trial. Trials 1–3 at easy, trials 4–6 at medium, and trials 7–17 at hard. Model fit, R2 = 0.86.
(in minutes), and mean responses to the TEQ. A repeated measures analysis of variance (RM-ANOVA), using group as a fixed factor and assessment scores as repeated measures, evaluated whether primary outcomes (role-play performance and interviewing self-confidence) significantly improved between pre-test and post-test for trainees as compared to controls. Cohen's d effect sizes characterized the withinparticipant differences. We evaluated VR-JIT performance score improvement across trials as a process measure by computing linear regression slopes for each participant based on the regression of their performance scores on the log of trial number. The group-level performance average for each successive VR-JIT trial was plotted with a report of the R-square from the regression of average performance on the log of trial number.
VR-JIT sessions were well attended, trainees completed mean = 15.7 (SD = 4.3) trials, and participants reported that VR-JIT was easy to use, enjoyable, helpful, and increased their self-confidence in interviewing (Table 2). 4.3. VR-JIT process measures VR-JIT performance scores significantly improved across the number of completed trials (Fig. 2). The slope (mean = 4.8, SD = 3.7) suggests that performance improved 4.8 points for every 1 point increase in the natural log of trial number (R-squared = 0.86). 4.4. Primary outcomes for efficacy study RM-ANOVA revealed a significant group-by-time interaction for role-play performance (F1,30 = 13.9, p = 0.001). Trainees improved their role-play performance between pre-test and post-test, which was characterized by a large effect size (d = 0.92). Controls appeared to regress to the mean (d = −0.42) (Table 3). RM-ANOVA revealed a non-significant group-by-time interaction for self-confidence scores (F1,30 = 1.6, p = 0.11), while observed improvement by trainees was characterized by a medium effect size (d = 0.58) (Table 3).
3.2. Six month follow-up
4.5. 6-month vocational outcomes
Among trainees, we conducted a pairwise t-test to evaluate whether post-test self-confidence was sustained at 6-month follow-up. We conducted a logistic regression with job offer (1 = yes, 0 = no) as the dependent variable to evaluate whether or not trainees had higher odds of receiving a job offer than controls. Neurocognition and number of months since prior employment were included as covariates given their relationship to vocational outcomes in this population (Burke-Miller et al., 2006; Catty et al., 2008). Odds ratios (OR) were generated with 95% confidence intervals. Nagelkerke R2 provided the model's proportion of explained variance. We conducted point serial correlations to evaluate whether receiving a job offer was associated with VR-JIT process measures (i.e., total number of completed VR-JIT trials, the VR-JIT performance slope, total amount of time spent with VRJIT) and self-confidence at 6-month follow-up. We conducted Pearson correlations to evaluate whether the number of weeks searching for employment was associated with VR-JIT process measures and selfconfidence at 6-month follow-up.
Trainees sustained interviewing self-confidence between the posttest (m = 50.2, SD = 8.8) and 6-month follow-up (m = 49.0, SD = 9.5) (T20 = 0.92, p N 0.10, d = −0.13). This analysis excluded the two controls who crossed-over to the VR-JIT group. More trainees received job offers (47.8%) than controls (14.3%), which was a trend-level difference (p = 0.055). The groups did not differ with respect to the proportion of participants who completed interviews, accepted job offers, total number of interviews completed, or number of weeks they searched for employment (all p N 0.10) (Table 4). The logistic regression revealed that the odds of attaining a job offer was 8.73 times higher for trainees compared to controls (OR = 8.73, p = 0.04; 95% CI = 1.17, 65.00) after accounting for neurocognition and months since prior employment (non-significant predictors). Overall, the model explained 20.3% of the variance in job offers (Table 5). Among trainees, completing more VR-JIT trials correlated with fewer weeks searching for employment (r = − 0.63, p b 0.001) and greater improvement in self-confidence (r = 0.46, p = 0.03). A larger VR-JIT
Table 3 Unadjusted means for efficacy study primary outcome measures. Control group
Role-play performancea Interviewing self-confidencea a
VR–JIT group
Pretest mean (SD)
Posttest mean (SD)
Cohen's d
Pretest mean (SD)
Posttest mean (SD)
Cohen's d
34.9 (3.6) 41.9 (14.0)
33.6 (3.3) 44.2 (11.5)
−0.42 0.30
33.8 (5.9) 42.5 (13.7)
36.5 (4.4) 50.2 (8.8)
0.92 0.58
One training group participant scored 3 standard deviations below the mean on the pretest and posttest means and was removed from this particular analysis.
Please cite this article as: Smith, M.J., et al., Virtual reality job interview training and 6-month employment outcomes for individuals with schizophrenia seeking employment, Schizophr. Res. (2015), http://dx.doi.org/10.1016/j.schres.2015.05.022
M.J. Smith et al. / Schizophrenia Research xxx (2015) xxx–xxx
5
Table 4 6-month follow-up between-group differences. N Mean total weeks looking for a job (SD) Mean total job interviews completed (SD) % of subjects who completed job interviews % of subjects who received job offer % of subjects who accepted job offer
6 1 1
Control group (n = 7) 17.3 (8.5) 1.3 (0.8) 85.7% 14.3% 100.0%
performance slope correlated with greater improvement in role-play performance (r = 0.46, p = 0.03). Remaining correlations were non significant (Supplemental Table 1). More than 80% of trainees strongly agreed or agreed that VR-JIT prepared them for real interviews and they would use VR-JIT again to enhance their skills. Fifty-six percent of trainees agreed or strongly agreed that VR-JIT helped them get a job.
5. Discussion We examined the acceptability and efficacy of VR-JIT in employmentseeking individuals with schizophrenia. Trainees reported that VR-JIT was easy to use, enjoyable and helpful, and improved their confidence for future interviews. Also, trainees improved their virtual interview scores across increasing levels of difficulty and their role-play performance scores. At 6-month follow-up, trainees had greater odds of receiving a job offer. Moreover, the regression was conducted while covarying for known predictors of employment (Burke-Miller et al., 2006; Catty et al., 2008). Also, more training was associated with fewer weeks searching for employment and trainees felt VR-JIT prepared them for interviews they encountered in real-life. The observed improvement in interviewing skills in this cohort is consistent with the evaluation of VR-JIT in other clinical populations (Smith et al., 2014a, 2014b, 2015a). Also, our findings that trainees with schizophrenia had greater odds of receiving job offers and higher doses of training were related to fewer weeks of job searching replicated the results from a recent 6-month follow-up study of a cohort primarily diagnosed with mood disorders or PTSD (Smith et al., 2015b). There are several directions for future research. First, one could evaluate whether VR-JIT improves vocational outcomes in larger samples enrolled in evidence-based vocational services such as IPS, which has better vocational outcomes than conventional practice (Drake and Bond, 2011). Since the vast majority of individuals with schizophrenia spectrum disorders do not have access to IPS, VR-JIT could be evaluated independently or as a complement to available vocational services. Due to the web-based interface, VR-JIT can be widely disseminated to treatment centers with little access to vocational services. Several limitations must be considered. There was limited statistical power due to a small sample. Data on the types of jobs attained or pay received were not collected, and 6-month data was self-reported.
Table 5 VR-JIT as a predictor of receiving a job offer.
Step 1a Neurocognition Months since prior employment Step 2b VR-JIT (yes or no) Nagelkerke R2 a
Step 1 OR (C.I. 95%)
Step 2 OR (C.I. 95%)
0.97 (0.92–1.01) 1.00 (0.99–1.00)
0.97 (0.92–1.01) 1.00 (0.99–1.00)
– 0.033
8.73 (1.17–65.00)⁎ 0.203⁎
Step 1 Omnibus Test of Model Coefficients, chi-square = 0.73, df = 2, p = 0.69. Step 2 Omnibus Test of Model Coefficients, chi-square = 4.15, df = 1, p = 0.02. ⁎ p b 0.05. b
N
19 11 9
VR-JIT group (n = 23)
p-Value
11.3 (10.0) 2.3 (2.3) 82.6% 47.8% 81.8%
0.11 0.31 0.85 0.055 0.22
The findings only generalize to individuals actively seeking jobs. Also, two controls crossed-over to the training group, which enhances our understanding of VR-JIT but could limit the generalizability of controls. We did not assess Parkinsonian symptoms or anticholinergic treatment, which could impact interviewing performance. All participants were paid for their study efforts (including the intervention phase), which may create bias. Thus, future research needs to evaluate outcomes by unpaid trainees. Both groups had similar demographic, clinical, and cognitive characteristics, and job-seeking behaviors (i.e., proportion of completed interviews, length of job search), which are notable strengths of the sample. 6. Conclusions In conclusion, VR-JIT is a promising intervention given that trainees improve their interviewing skills and have greater odds of receiving a job offer. Moreover, the amount of training was associated with receiving a job offer more quickly. Future studies should evaluate whether VRJIT can enhance vocational outcomes for individuals with schizophrenia with and without access to evidence-based vocational services. Supplementary data to this article can be found online at http://dx. doi.org/10.1016/j.schres.2015.05.022. Role of funding source This study was primarily funded by the Department of Psychiatry and Behavioral Sciences at Northwestern University Feinberg School of Medicine. Additional funding for the study was provided by an NIMH grant awarded to Dr. Dale Olsen (R44 MH080496).
Contributors All authors have made significant scientific contributions to this manuscript. Dr. Matthew J. Smith contributed to the conceptualization of the study, conducted the statistical analyses, and wrote the first draft of the manuscript. Drs. Fleming, Olsen, and Bell contributed to the conceptualization of the study and assisted with manuscript editing. Mr. Wright and Ms. Boteler Humm assisted with study conceptualization and manuscript editing. Ms. Roberts assisted with writing the introduction and overall manuscript editing. All authors approved the final manuscript. Conflict of interest Dr. Olsen and Laura Boteler Humm are employed by and own shares in SIMmersion LLC. They contributed to the manuscript, but were not involved in analyzing the data. Dr. Bell was a paid consultant by SIMmersion LLC to assist with the development of VRJIT. Dr. Bell and his family do not have a financial stake in the company. The remaining authors report no conflicts of interest.
Acknowledgments Support for this work was provided by the Department of Psychiatry and Behavioral Sciences. The authors would like to acknowledge Dr. Zoran Martinovich for advising on statistical issues, the research staff at Northwestern University's Schizophrenia Research Group and Clinical Research Program for study coordination and data collection, and our participants for volunteering their time.
References Andreasen, N.C., 1983a. The Scale for the Assessment of Negative Symptoms. The University of Iowa, Iowa City, IA. Andreasen, N.C., 1983b. The Scale for the Assessment of Positive Symptoms. The University of Iowa, Iowa City, IA. Bell, M.D., Weinstein, A., 2011. Simulated job interview skill training for people with psychiatric disability: feasibility and tolerability of virtual reality training. Schizophr. Bull. 37 (Suppl. 2), S91–S97.
Please cite this article as: Smith, M.J., et al., Virtual reality job interview training and 6-month employment outcomes for individuals with schizophrenia seeking employment, Schizophr. Res. (2015), http://dx.doi.org/10.1016/j.schres.2015.05.022
6
M.J. Smith et al. / Schizophrenia Research xxx (2015) xxx–xxx
Bell, M., Bryson, G., Lysaker, P., 1997. Positive and negative affect recognition in schizophrenia: a comparison with substance abuse and normal control subjects. Psychiatry Res. 73 (1-2), 73–82. Burke-Miller, J.K., Cook, J.A., Grey, D.D., Razzano, L.A., Blyler, C.R., Leff, H.S., Gold, P.B., Goldberg, R.W., Mueser, K.T., Cook, W.L., Hoppe, S.K., Stewart, M., Blankertz, L., Dudek, K., Taylor, A.L., Carey, M.A., 2006. Demographic characteristics and employment among people with severe mental illness in a multisite study. Community Ment. Health J. 42 (2), 143–159. Catty, J., Lissouba, P., White, S., Becker, T., Drake, R.E., Fioritti, A., Knapp, M., Lauber, C., Rossler, W., Tomov, T., van Busschbach, J., Wiersma, D., Burns, T., 2008. Predictors of employment for people with severe mental illness: results of an international six-centre randomised controlled trial. Br. J. Psychiatry J. Ment. Sci. 192 (3), 224–231. Corbiere, M., Zaniboni, S., Lecomte, T., Bond, G., Gilles, P.Y., Lesage, A., Goldner, E., 2011. Job acquisition for people with severe mental illness enrolled in supported employment programs: a theoretically grounded empirical study. J. Occup. Rehabil. 21 (3), 342–354. Drake, R.E., Bond, G.R., 2011. IPS supported employment: a 20 year update. Am. J. Psychiatr. Rehabil. 14 (3), 155–164. Eack, S.M., Hogarty, G.E., Greenwald, D.P., Hogarty, S.S., Keshavan, M.S., 2011. Effects of cognitive enhancement therapy on employment outcomes in early schizophrenia: results from a two-year randomized trial. Res. Soc. Work. Pract. 21 (1), 32–42. First, M.B., Spitzer, R.L., Miriam, G., Williams, J.B.W., 2002. Structured Clinical Interview for DSM-IV-TR Axis I Disorders, Research Version, Non-patient Edition. Biometrics Research, New York State Psychiatric Institute, New York, NY. Huffcutt, A.I., 2011. An empirical review of the employment interview construct literature. Int. J. Sel. Assess. 19 (1), 62–81. Martinez-Dominguez, S., Penades, R., Segura, B., Gonzalez-Rodriguez, A., Catalan, R., 2015. Influence of social cognition on daily functioning in schizophrenia: study of incremental validity and mediational effects. Psychiatry Res. 225 (3), 374–380. Marwaha, S., Johnson, S., 2006. View and experiences of employment among people with psychosis: a qualitative descriptive study. Int. J. Soc. Psychiatry 51 (4), 302–316. Myers, W.R., 2000. Handling missing data in clinical trials: an overview. Drug Inf. J. 34, 525–533. Randolph, C., Tierney, M.C., Mohr, E., Chase, T.N., 1998. The Repeatable Battery for the Assessment of Neuropsychological Status (RBANS): preliminary clinical validity. J. Clin. Exp. Neuropsychol. 20 (3), 310–319.
Rosenheck, R., Leslie, D., Keefe, R., McEvoy, J., Swartz, M., Perkins, D., Stroup, S., Hsiao, J.K., Lieberman, J., 2006. Barriers to employment for people with schizophrenia. Am. J. Psychiatry 163 (3), 411–417. Salkever, D.S., Karakus, M.C., Slade, E.P., Harding, C.M., Hough, R.L., Rosenheck, R.A., Swartz, M.S., Barrio, C., Yamada, A.M., 2007. Measures and predictors of community-based employment and earnings of persons with schizophrenia in a multisite study. Psychiatr. Serv. 58 (3), 315–324. Sheehan, D.V., Lecrubier, Y., Sheehan, K.H., Amorim, P., Janavs, J., Weiller, E., Hergueta, T., Baker, R., Dunbar, G.C., 1998. The Mini-International Neuropsychiatric Interview (M.I.N.I.): the development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. J. Clin. Psychiatry 59 (Suppl 20), 22–33 quiz 34–57. Smith, M.J., Ginger, E.J., Wright, K., Wright, M.A., Taylor, J.L., Humm, L.B., Olsen, D.E., Bell, M.D., Fleming, M.F., 2014a. Virtual reality job interview training in adults with autism spectrum disorder. J. Autism Dev. Disord. 44 (10), 2450–2463. Smith, M.J., Ginger, E.J., Wright, M., Wright, K., Boteler Humm, L., Olsen, D., Bell, M.D., Fleming, M.F., 2014b. Virtual reality job interview training for individuals with psychiatric disabilities. J. Nerv. Ment. Dis. 202 (9), 659–667. Smith, M.J., Horan, W.P., Cobia, D.J., Karpouzian, T.M., Fox, J.M., Reilly, J.L., Breiter, H.C., 2014c. Performance-based empathy mediates the influence of working memory on social competence in schizophrenia. Schizophr. Bull. 40 (4), 824–834. Smith, M.J., Boteler Humm, L., Fleming, M.F., Jordan, N., Wright, M.A., Ginger, E.J., Wright, K., Olsen, D., Bell, M.D., 2015a. Virtual reality job interview training for veterans with posttraumatic stress disorder. J. Vocat. Rehabil 42 (3), 271–280. Smith, M.J., Fleming, M.F., Wright, M.A., Jordan, N., Boteler Humm, L., Olsen, D., Bell, M.D., 2015b. Job offers among individuals with severe mental illness after virtual reality job interview training in press. Psychiatr. Serv. Solinski, S., Jackson, H.J., Bell, R.C., 1992. Prediction of employability in schizophrenic patients. Schizophr. Res. 7 (2), 141–148. Vargas, G., Strassnig, M., Sabbag, S., Gould, F., Durand, D., Stone, L., Patterson, T.L., Harvey, P.D., 2014. The course of vocational functioning in patients with schizophrenia: reexamining social drift. Schizophrenia research. Cognition 1 (1), e41–e46. Wilkinson, G.S., Robertson, G.J., 2006. Wide Range Achievement Test 4 Professional Manual. Psychological Assessment Resources, Lutz, FL.
Please cite this article as: Smith, M.J., et al., Virtual reality job interview training and 6-month employment outcomes for individuals with schizophrenia seeking employment, Schizophr. Res. (2015), http://dx.doi.org/10.1016/j.schres.2015.05.022
Contents lists available at ScienceDirect
Schizophrenia Research journal homepage: www.elsevier.com/locate/schres
Virtual reality job interview training and 6-month employment outcomes for individuals with schizophrenia seeking employment Matthew J. Smith a,b,⁎, Michael F. Fleming a,c, Michael A. Wright a, Andrea G. Roberts a,b, Laura Boteler Humm d,1, Dale Olsen d,1, Morris D. Bell e a
Northwestern University Feinberg School of Medicine, Department of Psychiatry and Behavioral Sciences, Chicago, IL, USA Northwestern University Feinberg School of Medicine, Warren Wright Adolescent Center, Chicago, IL, USA Northwestern University Feinberg School of Medicine, Department of Family Medicine, Chicago, IL, USA d SIMmersion LLC, Columbia, MD, USA e Yale School of Medicine, Department of Psychiatry, Department of Veteran Affairs, West Haven, CT, USA b c
a r t i c l e
i n f o
Article history: Received 5 March 2015 Received in revised form 12 May 2015 Accepted 12 May 2015 Available online xxxx Keywords: Schizophrenia Virtual reality training Job interview skills Vocational outcomes
a b s t r a c t Background: Individuals with schizophrenia have low employment rates and the job interview presents a critical barrier for them to obtain employment. Virtual reality training has demonstrated efficacy at improving interview skills and employment outcomes among multiple clinical populations. However, the effects of this training on individuals with schizophrenia are unknown. This study evaluated the efficacy of virtual reality job interview training (VR-JIT) at improving job interview skills and employment outcomes among individuals with schizophrenia in a small randomized controlled trial (n = 21 VR-JIT trainees, n = 11 waitlist controls). Methods: Trainees completed up to 10 h of virtual interviews using VR-JIT, while controls received services as usual. Primary outcome measures included two pre-test and two post-test video-recorded role-play interviews scored by blinded raters with expertise in human resources and self-reported interviewing selfconfidence. Six-month follow-up data on employment outcomes were collected. Results: Trainees reported that the intervention was easy-to-use, helpful, and prepared them for future interviews. Trainees demonstrated increased role-play scores between pre-test and post-test while controls did not (p = 0.001). After accounting for neurocognition and months since prior employment, trainees had greater odds of receiving a job offer by 6 month follow-up compared to controls (OR: 8.73, p = 0.04) and more training was associated with fewer weeks until receiving a job offer (r = −0.63, p b 0.001). Conclusions: Results suggest that VR-JIT is acceptable to trainees and may be efficacious for improving job interview skills in individuals with schizophrenia. Moreover, trainees had greater odds of receiving a job offer by 6-month follow-up. Future studies could evaluate the effectiveness of VR-JIT within community-based services. © 2015 Elsevier B.V. All rights reserved.
1. Introduction Less than 20% of individuals with schizophrenia attain employment (Rosenheck et al., 2006; Salkever et al., 2007), while this rate climbs to 30–40% for individuals in Individual Placement and Support (IPS) services (Drake and Bond, 2011). Successfully navigating the job interview is critical to attaining employment and is the most proximal step to attaining employment for IPS clients (Corbiere et al., 2011). Moreover, social cognitive and neurocognitive deficits that characterize schizophrenia have been associated with poorer vocational outcomes (Eack ⁎ Corresponding author at: Northwestern University Feinberg School of Medicine, Department of Psychiatry and Behavioral Sciences, 710 N. Lake Shore Dr, Abbott Hall 13th floor, Chicago, IL 60611. Tel.: 1 312 503 2542, fax: 1 312 503 0527. E-mail address: [email protected] (M.J. Smith). 1 Dr. Olsen and Ms. Humm contributed to the design of the study and manuscript preparation, but were not involved in the collection or analysis of data.
et al., 2011; Vargas et al., 2014; Martinez-Dominguez et al., 2015), and likely increase the difficulty of interviewing. Although individuals with schizophrenia self-identify having poor interview skills and want services to enhance these skills (Solinski et al., 1992; Marwaha and Johnson, 2006), there is a paucity of evidence-based interventions targeting interview skills (Bell and Weinstein, 2011). Recently, two randomized controlled trials (RCTs) evaluated virtual reality job interview training (VR-JIT), which demonstrated acceptability and efficacy at improving interview skills and self-confidence for individuals primarily diagnosed with mood disorders and veterans with posttraumatic stress disorder (PTSD) (Smith et al., 2014b, 2015a). Moreover, VR-JIT trainees sustained their self-confidence and were more likely to receive job offers than controls when evaluated six months later. Additionally, completing more VR-JIT trials was associated with greater odds of receiving a job offer and spending fewer weeks searching for employment (Smith et al., 2015b).
http://dx.doi.org/10.1016/j.schres.2015.05.022 0920-9964/© 2015 Elsevier B.V. All rights reserved.
Please cite this article as: Smith, M.J., et al., Virtual reality job interview training and 6-month employment outcomes for individuals with schizophrenia seeking employment, Schizophr. Res. (2015), http://dx.doi.org/10.1016/j.schres.2015.05.022
2
M.J. Smith et al. / Schizophrenia Research xxx (2015) xxx–xxx
Thus, we hypothesized that 1) individuals with schizophrenia randomized to training would find VR-JIT acceptable and enhance their interviewing skills and interviewing self-confidence compared to waitlist controls; 2) trainees with schizophrenia would sustain their enhanced self-confidence and have greater odds of receiving a job offer compared to controls at 6-month follow-up; and 3) a greater number of completed VR-JIT trials, improved VR-JIT performance across trials, and self-confidence at the conclusion of training would be correlated with receiving a job offer and with fewer weeks until a job was offered. We generated these directional hypotheses based on our prior research (Smith et al., 2015b). 2. Methods 2.1. Participants Participants included 32 individuals with schizophrenia or schizoaffective disorder recruited through Northwestern University's Schizophrenia Research Group. Bachelor's or Ph.D.-level research staff determined diagnoses (and antipsychotic treatment) using the Structured Clinical Interview for the DSM-IV-TR (SCID-IV) (First et al., 2002). Inclusion criteria included: 18–55 years old, minimum of a 6th grade reading level using the Wide Range Achievement Test-IV (WRAT-IV) (Wilkinson and Robertson, 2006), willingness to be videorecorded, unemployed or underemployed, and actively seeking employment. Exclusion criteria included: having a medical illness that significantly comprised cognition (e.g., traumatic brain injury), uncorrected vision or hearing problem, or current substance abuse. Northwestern University's Institutional Review Board approved the study protocol and all participants provided informed consent. Once enrolled, participants were randomized using a random number generator into training (n = 17) or treatment-as-usual waitlist control (n = 8) groups at an estimated ratio of 2 to 1 to optimize VR-JIT evaluation. Four trainees and three controls with schizophrenia completed prior RCTs of VR-JIT (Smith et al., 2014b, 2015a) and their data were included in all current analyses to optimize statistical power with final sample sizes of n = 21 trainees and n = 11 controls. Participants were re-contacted after 6 months and asked to complete a followup survey. Of the original 32 participants, 30 (94%) completed the follow-up survey and 2 (6%) were lost to contact. 2.2. Intervention Virtual Reality Job Interview Training (VR-JIT) is a computer-based intervention developed by SIMmersion LLC (http://www.simmersion. com) to improve interviewing skills for adults with a range of disabilities. See Supplemental Material and Smith et al. (2014b) for details on VR-JIT design, use, and delivery (e.g., fidelity training). 2.3. Study procedures Pre-test measures included demographic, clinical, and vocational interviews; clinical, neurocognitive, and social cognitive assessments; and self-reported self-confidence and two standardized video-recorded roleplays. Participants were randomized following pre-test assessments. Trainees completed up to 10 h of VR-JIT (approximately 20 trials) over the course of 5 visits (across 5–10 business days). Controls received services-as-usual for 5–10 business days. Both groups returned after ten business days to complete post-test measures of self-confidence and two standardized video-recorded role-plays, while trainees also completed the Treatment Experience Questionnaire (TEQ). After the post-test visit, all controls were invited from the waitlist to use VR-JIT. Two controls used VR-JIT and crossed-over to the training group (Fig. 1). Research staff contacted participants to complete a follow-up survey over the phone or via email approximately 6 months after completing the efficacy trial. Two controls were unreachable by phone, mail, and
Fig. 1. Consort diagram of study participation.
email, and were lost to contact. Overall, 23 VR-JIT and 7 controls completed follow-up (Fig. 1). 2.4. Study measures 2.4.1. Participant characteristics We assessed demographic characteristics and vocational history via self-report. Symptoms were assessed using global ratings from the Scale for the Assessment of Positive Symptoms and the Scale for the assessment of Negative Symptoms (Andreasen, 1983a, 1983b). For the 7 participants who completed prior studies (Smith et al., 2014b, 2015a), we used ratings and notes from the Mini-International Neuropsychiatric Interview (Sheehan et al., 1998) to inform symptom ratings and SCID diagnosis. 2.4.2. Cognition We assessed global cognitive ability with the total score from the Repeatable Battery for the Assessment of Neuropsychological Status (Randolph et al., 1998). We assessed basic social cognition using the Bell-Lysaker Emotion Recognition Task (Bell et al., 1997), and advanced social cognition using an emotional perspective-taking task (Smith et al., 2014c). Accuracy ratings for each task were generated using the number of correct responses. 2.4.3. VR-JIT acceptability We recorded trainee attendance across five sessions and the number of minutes (600 min possible) that they engaged in virtual interviews. Trainees completed the TEQ to evaluate whether they felt that VR-JIT was easy to use, enjoyable, helpful, instilled confidence, and prepared them for interviews (Bell and Weinstein, 2011). The TEQ used a 7-point Likert scale with higher scores reflecting more positive views (α = 0.86).
Please cite this article as: Smith, M.J., et al., Virtual reality job interview training and 6-month employment outcomes for individuals with schizophrenia seeking employment, Schizophr. Res. (2015), http://dx.doi.org/10.1016/j.schres.2015.05.022
M.J. Smith et al. / Schizophrenia Research xxx (2015) xxx–xxx
3
Table 1 Baseline characteristics.
Demographics Age, mean years (SD) Gender (% male) Parental education, mean years (SD) Race % Caucasian % African-American % Latino Vocational history Months since prior employment, mean (SD)a Prior full-time employment (%) Prior paid employment (any type) (%) Prior participation in vocational training program (%) Cognitive function Neurocognition, mean (SD) Basic social cognition, mean (SD) Advanced social cognition, mean (SD) Clinical history Duration of illness, mean years (SD) Clinical symptoms Hallucinations Delusions Bizarre behavior Thought disorder Affective flattening Alogia Avolitionb Anhedoniac Attentionc Chlorpromazine equivalent Treated with atypical antipsychotic medication (%) Treated with typical antipsychotic medication (%) a b c
Control group (n = 11)
VR-JIT group (n = 21)
p-Value
39.1 (10.6) 54.5% 11.7 (1.8)
40.8 (12.2) 52.4% 12.9 (3.4)
0.76 0.91 0.46
27.3% 54.5% 18.2%
28.6% 66.7% 4.8%
0.46
26.9 (41.0) 63.6% 100% 27.3%
65.2 (92.4) 71.4% 95.2% 38.1%
0.27 0.65 0.46 0.54
72.5 (12.6) 0.71 (0.2) 0.77 (0.1)
74.8 (16.4) 0.69 (0.1) 0.74 (0.1)
0.76 0.67 0.88
16.0 (11.4)
19.5 (11.2)
0.35
2.36 (2.1) 2.36 (2.1) 0.36 (0.8) 1.18 (1.3) 1.64 (1.4) 1.27 (1.3) 2.90 (0.8) 2.38 (1.6) 1.75 (0.9) 245.25 (164.9) 81.8% 18.2%
1.71 (1.8) 2.50 (1.9) 0.95 (1.2) 0.90 (1.3) 1.86 (1.6) 1.14 (1.5) 2.90 (1.1) 3.73 (1.4) 2.00 (1.5) 340.81 (197.9) 90.0% 10.0%
0.39 0.88 0.24 0.56 0.73 0.79 0.86 0.05 0.73 0.18 0.52
Two outliers excluded for VR-JIT trainees: 324 months and 276 months. One VR-JIT trainee was not rated on this domain. Three controls and six VR-JIT trainees were not rated on this domain.
2.4.4. VR-JIT efficacy 2.4.4.1. Role-play performance. Job interview role-plays (approximately 20 min each) were scored across nine domains: 1) conveying oneself as a hard worker, 2) sounding easy to work with, 3) conveying that one behaves professionally, 4) negotiation skills (requesting Thursdays off), 5) sharing things in a positive way, 6) sounding honest, 7) sounding interested in the position, 8) comfort level, and 9) establishing rapport; which were adapted from prior work (Huffcutt, 2011). Role-play videos were randomly assigned to two blinded raters with more than 15 years of experience in human resources. Total scores for two pre-test and two post-test role-plays were computed across the nine domains (range of 1–5, higher scores reflecting better performance), and averaged to reflect a single score.
2.4.5. Six-month follow-up measures The follow-up survey asked participants to reflect on the past 6 months and report the 1) total number of weeks they searched for employment, 2) number of job interviews completed, and 3) number of job offers received and accepted. The survey reassessed interviewing self-confidence (α = 0.89), and assessed whether trainees felt that VRJIT prepared them for real interviews, helped them attain employment, and if revisiting VR-JIT would prepare them for future interviews. These items were rated on a 5-point Likert scale (1 = strongly disagree to 5 = strongly agree).
3. Data analysis 3.1. Efficacy study
2.4.4.2. Interviewing self-confidence. Participants rated their selfconfidence at interviewing using a 7-point Likert scale to answer nine questions, with higher scores reflecting more positive views (e.g., “How comfortable are you going on a job interview?”). Total scores at pre-test and post-test had strong internal consistencies (α = 0.95 and α = 0.92, respectively).
2.4.4.3. VR-JIT process measures. We recorded trainees' VR-JIT performance scores, number of completed trials, and time spent performing virtual interviews. Each virtual interview was scored 0–100 using an algorithm that assessed the appropriateness of responses based on eight domains: negotiation skills, conveying that you're a hard worker, sounding easy to work with, sharing things in a positive way, sounding honest, sounding interested in the position, behaving professionally, and establishing interviewer rapport.
Between-group differences for demographics, vocational history, cognition, and clinical history were assessed with a Mann–Whitney independent samples test or chi-square analysis. We used descriptive statistics to assess VR-JIT acceptability via session attendance, VR-JIT usage Table 2 Mean characteristics of VR-JIT acceptability (SD). Attendance measures % session attendance Elapsed simulation time (min) TEQ items Ease of use Enjoyable Helpful Instilled confidence Prepared for interviews
90.0% 537.9 (102.9) 6.0 (1.0) 6.7 (0.7) 6.6 (0.7) 5.9 (1.2) 6.0 (1.0)
Note. Scale for TEQ, 1 = extremely unhelpful to 7 = extremely helpful.
Please cite this article as: Smith, M.J., et al., Virtual reality job interview training and 6-month employment outcomes for individuals with schizophrenia seeking employment, Schizophr. Res. (2015), http://dx.doi.org/10.1016/j.schres.2015.05.022
4
M.J. Smith et al. / Schizophrenia Research xxx (2015) xxx–xxx
4. Results 4.1. Pre-test between-group characteristics Trainees and controls did not differ with respect to demographics as well as clinical, cognitive, and vocational history (all p N 0.10). Despite random assignment, anhedonia ratings differed between groups (but were missing for nine participants) (p b 0.05) (Table 1). 4.2. VR-JIT acceptability
Fig. 2. VR-JIT learning curve in individuals with schizophrenia. This figure plots the average score for each successive VR-JIT virtual interview trial. Trials 1–3 at easy, trials 4–6 at medium, and trials 7–17 at hard. Model fit, R2 = 0.86.
(in minutes), and mean responses to the TEQ. A repeated measures analysis of variance (RM-ANOVA), using group as a fixed factor and assessment scores as repeated measures, evaluated whether primary outcomes (role-play performance and interviewing self-confidence) significantly improved between pre-test and post-test for trainees as compared to controls. Cohen's d effect sizes characterized the withinparticipant differences. We evaluated VR-JIT performance score improvement across trials as a process measure by computing linear regression slopes for each participant based on the regression of their performance scores on the log of trial number. The group-level performance average for each successive VR-JIT trial was plotted with a report of the R-square from the regression of average performance on the log of trial number.
VR-JIT sessions were well attended, trainees completed mean = 15.7 (SD = 4.3) trials, and participants reported that VR-JIT was easy to use, enjoyable, helpful, and increased their self-confidence in interviewing (Table 2). 4.3. VR-JIT process measures VR-JIT performance scores significantly improved across the number of completed trials (Fig. 2). The slope (mean = 4.8, SD = 3.7) suggests that performance improved 4.8 points for every 1 point increase in the natural log of trial number (R-squared = 0.86). 4.4. Primary outcomes for efficacy study RM-ANOVA revealed a significant group-by-time interaction for role-play performance (F1,30 = 13.9, p = 0.001). Trainees improved their role-play performance between pre-test and post-test, which was characterized by a large effect size (d = 0.92). Controls appeared to regress to the mean (d = −0.42) (Table 3). RM-ANOVA revealed a non-significant group-by-time interaction for self-confidence scores (F1,30 = 1.6, p = 0.11), while observed improvement by trainees was characterized by a medium effect size (d = 0.58) (Table 3).
3.2. Six month follow-up
4.5. 6-month vocational outcomes
Among trainees, we conducted a pairwise t-test to evaluate whether post-test self-confidence was sustained at 6-month follow-up. We conducted a logistic regression with job offer (1 = yes, 0 = no) as the dependent variable to evaluate whether or not trainees had higher odds of receiving a job offer than controls. Neurocognition and number of months since prior employment were included as covariates given their relationship to vocational outcomes in this population (Burke-Miller et al., 2006; Catty et al., 2008). Odds ratios (OR) were generated with 95% confidence intervals. Nagelkerke R2 provided the model's proportion of explained variance. We conducted point serial correlations to evaluate whether receiving a job offer was associated with VR-JIT process measures (i.e., total number of completed VR-JIT trials, the VR-JIT performance slope, total amount of time spent with VRJIT) and self-confidence at 6-month follow-up. We conducted Pearson correlations to evaluate whether the number of weeks searching for employment was associated with VR-JIT process measures and selfconfidence at 6-month follow-up.
Trainees sustained interviewing self-confidence between the posttest (m = 50.2, SD = 8.8) and 6-month follow-up (m = 49.0, SD = 9.5) (T20 = 0.92, p N 0.10, d = −0.13). This analysis excluded the two controls who crossed-over to the VR-JIT group. More trainees received job offers (47.8%) than controls (14.3%), which was a trend-level difference (p = 0.055). The groups did not differ with respect to the proportion of participants who completed interviews, accepted job offers, total number of interviews completed, or number of weeks they searched for employment (all p N 0.10) (Table 4). The logistic regression revealed that the odds of attaining a job offer was 8.73 times higher for trainees compared to controls (OR = 8.73, p = 0.04; 95% CI = 1.17, 65.00) after accounting for neurocognition and months since prior employment (non-significant predictors). Overall, the model explained 20.3% of the variance in job offers (Table 5). Among trainees, completing more VR-JIT trials correlated with fewer weeks searching for employment (r = − 0.63, p b 0.001) and greater improvement in self-confidence (r = 0.46, p = 0.03). A larger VR-JIT
Table 3 Unadjusted means for efficacy study primary outcome measures. Control group
Role-play performancea Interviewing self-confidencea a
VR–JIT group
Pretest mean (SD)
Posttest mean (SD)
Cohen's d
Pretest mean (SD)
Posttest mean (SD)
Cohen's d
34.9 (3.6) 41.9 (14.0)
33.6 (3.3) 44.2 (11.5)
−0.42 0.30
33.8 (5.9) 42.5 (13.7)
36.5 (4.4) 50.2 (8.8)
0.92 0.58
One training group participant scored 3 standard deviations below the mean on the pretest and posttest means and was removed from this particular analysis.
Please cite this article as: Smith, M.J., et al., Virtual reality job interview training and 6-month employment outcomes for individuals with schizophrenia seeking employment, Schizophr. Res. (2015), http://dx.doi.org/10.1016/j.schres.2015.05.022
M.J. Smith et al. / Schizophrenia Research xxx (2015) xxx–xxx
5
Table 4 6-month follow-up between-group differences. N Mean total weeks looking for a job (SD) Mean total job interviews completed (SD) % of subjects who completed job interviews % of subjects who received job offer % of subjects who accepted job offer
6 1 1
Control group (n = 7) 17.3 (8.5) 1.3 (0.8) 85.7% 14.3% 100.0%
performance slope correlated with greater improvement in role-play performance (r = 0.46, p = 0.03). Remaining correlations were non significant (Supplemental Table 1). More than 80% of trainees strongly agreed or agreed that VR-JIT prepared them for real interviews and they would use VR-JIT again to enhance their skills. Fifty-six percent of trainees agreed or strongly agreed that VR-JIT helped them get a job.
5. Discussion We examined the acceptability and efficacy of VR-JIT in employmentseeking individuals with schizophrenia. Trainees reported that VR-JIT was easy to use, enjoyable and helpful, and improved their confidence for future interviews. Also, trainees improved their virtual interview scores across increasing levels of difficulty and their role-play performance scores. At 6-month follow-up, trainees had greater odds of receiving a job offer. Moreover, the regression was conducted while covarying for known predictors of employment (Burke-Miller et al., 2006; Catty et al., 2008). Also, more training was associated with fewer weeks searching for employment and trainees felt VR-JIT prepared them for interviews they encountered in real-life. The observed improvement in interviewing skills in this cohort is consistent with the evaluation of VR-JIT in other clinical populations (Smith et al., 2014a, 2014b, 2015a). Also, our findings that trainees with schizophrenia had greater odds of receiving job offers and higher doses of training were related to fewer weeks of job searching replicated the results from a recent 6-month follow-up study of a cohort primarily diagnosed with mood disorders or PTSD (Smith et al., 2015b). There are several directions for future research. First, one could evaluate whether VR-JIT improves vocational outcomes in larger samples enrolled in evidence-based vocational services such as IPS, which has better vocational outcomes than conventional practice (Drake and Bond, 2011). Since the vast majority of individuals with schizophrenia spectrum disorders do not have access to IPS, VR-JIT could be evaluated independently or as a complement to available vocational services. Due to the web-based interface, VR-JIT can be widely disseminated to treatment centers with little access to vocational services. Several limitations must be considered. There was limited statistical power due to a small sample. Data on the types of jobs attained or pay received were not collected, and 6-month data was self-reported.
Table 5 VR-JIT as a predictor of receiving a job offer.
Step 1a Neurocognition Months since prior employment Step 2b VR-JIT (yes or no) Nagelkerke R2 a
Step 1 OR (C.I. 95%)
Step 2 OR (C.I. 95%)
0.97 (0.92–1.01) 1.00 (0.99–1.00)
0.97 (0.92–1.01) 1.00 (0.99–1.00)
– 0.033
8.73 (1.17–65.00)⁎ 0.203⁎
Step 1 Omnibus Test of Model Coefficients, chi-square = 0.73, df = 2, p = 0.69. Step 2 Omnibus Test of Model Coefficients, chi-square = 4.15, df = 1, p = 0.02. ⁎ p b 0.05. b
N
19 11 9
VR-JIT group (n = 23)
p-Value
11.3 (10.0) 2.3 (2.3) 82.6% 47.8% 81.8%
0.11 0.31 0.85 0.055 0.22
The findings only generalize to individuals actively seeking jobs. Also, two controls crossed-over to the training group, which enhances our understanding of VR-JIT but could limit the generalizability of controls. We did not assess Parkinsonian symptoms or anticholinergic treatment, which could impact interviewing performance. All participants were paid for their study efforts (including the intervention phase), which may create bias. Thus, future research needs to evaluate outcomes by unpaid trainees. Both groups had similar demographic, clinical, and cognitive characteristics, and job-seeking behaviors (i.e., proportion of completed interviews, length of job search), which are notable strengths of the sample. 6. Conclusions In conclusion, VR-JIT is a promising intervention given that trainees improve their interviewing skills and have greater odds of receiving a job offer. Moreover, the amount of training was associated with receiving a job offer more quickly. Future studies should evaluate whether VRJIT can enhance vocational outcomes for individuals with schizophrenia with and without access to evidence-based vocational services. Supplementary data to this article can be found online at http://dx. doi.org/10.1016/j.schres.2015.05.022. Role of funding source This study was primarily funded by the Department of Psychiatry and Behavioral Sciences at Northwestern University Feinberg School of Medicine. Additional funding for the study was provided by an NIMH grant awarded to Dr. Dale Olsen (R44 MH080496).
Contributors All authors have made significant scientific contributions to this manuscript. Dr. Matthew J. Smith contributed to the conceptualization of the study, conducted the statistical analyses, and wrote the first draft of the manuscript. Drs. Fleming, Olsen, and Bell contributed to the conceptualization of the study and assisted with manuscript editing. Mr. Wright and Ms. Boteler Humm assisted with study conceptualization and manuscript editing. Ms. Roberts assisted with writing the introduction and overall manuscript editing. All authors approved the final manuscript. Conflict of interest Dr. Olsen and Laura Boteler Humm are employed by and own shares in SIMmersion LLC. They contributed to the manuscript, but were not involved in analyzing the data. Dr. Bell was a paid consultant by SIMmersion LLC to assist with the development of VRJIT. Dr. Bell and his family do not have a financial stake in the company. The remaining authors report no conflicts of interest.
Acknowledgments Support for this work was provided by the Department of Psychiatry and Behavioral Sciences. The authors would like to acknowledge Dr. Zoran Martinovich for advising on statistical issues, the research staff at Northwestern University's Schizophrenia Research Group and Clinical Research Program for study coordination and data collection, and our participants for volunteering their time.
References Andreasen, N.C., 1983a. The Scale for the Assessment of Negative Symptoms. The University of Iowa, Iowa City, IA. Andreasen, N.C., 1983b. The Scale for the Assessment of Positive Symptoms. The University of Iowa, Iowa City, IA. Bell, M.D., Weinstein, A., 2011. Simulated job interview skill training for people with psychiatric disability: feasibility and tolerability of virtual reality training. Schizophr. Bull. 37 (Suppl. 2), S91–S97.
Please cite this article as: Smith, M.J., et al., Virtual reality job interview training and 6-month employment outcomes for individuals with schizophrenia seeking employment, Schizophr. Res. (2015), http://dx.doi.org/10.1016/j.schres.2015.05.022
6
M.J. Smith et al. / Schizophrenia Research xxx (2015) xxx–xxx
Bell, M., Bryson, G., Lysaker, P., 1997. Positive and negative affect recognition in schizophrenia: a comparison with substance abuse and normal control subjects. Psychiatry Res. 73 (1-2), 73–82. Burke-Miller, J.K., Cook, J.A., Grey, D.D., Razzano, L.A., Blyler, C.R., Leff, H.S., Gold, P.B., Goldberg, R.W., Mueser, K.T., Cook, W.L., Hoppe, S.K., Stewart, M., Blankertz, L., Dudek, K., Taylor, A.L., Carey, M.A., 2006. Demographic characteristics and employment among people with severe mental illness in a multisite study. Community Ment. Health J. 42 (2), 143–159. Catty, J., Lissouba, P., White, S., Becker, T., Drake, R.E., Fioritti, A., Knapp, M., Lauber, C., Rossler, W., Tomov, T., van Busschbach, J., Wiersma, D., Burns, T., 2008. Predictors of employment for people with severe mental illness: results of an international six-centre randomised controlled trial. Br. J. Psychiatry J. Ment. Sci. 192 (3), 224–231. Corbiere, M., Zaniboni, S., Lecomte, T., Bond, G., Gilles, P.Y., Lesage, A., Goldner, E., 2011. Job acquisition for people with severe mental illness enrolled in supported employment programs: a theoretically grounded empirical study. J. Occup. Rehabil. 21 (3), 342–354. Drake, R.E., Bond, G.R., 2011. IPS supported employment: a 20 year update. Am. J. Psychiatr. Rehabil. 14 (3), 155–164. Eack, S.M., Hogarty, G.E., Greenwald, D.P., Hogarty, S.S., Keshavan, M.S., 2011. Effects of cognitive enhancement therapy on employment outcomes in early schizophrenia: results from a two-year randomized trial. Res. Soc. Work. Pract. 21 (1), 32–42. First, M.B., Spitzer, R.L., Miriam, G., Williams, J.B.W., 2002. Structured Clinical Interview for DSM-IV-TR Axis I Disorders, Research Version, Non-patient Edition. Biometrics Research, New York State Psychiatric Institute, New York, NY. Huffcutt, A.I., 2011. An empirical review of the employment interview construct literature. Int. J. Sel. Assess. 19 (1), 62–81. Martinez-Dominguez, S., Penades, R., Segura, B., Gonzalez-Rodriguez, A., Catalan, R., 2015. Influence of social cognition on daily functioning in schizophrenia: study of incremental validity and mediational effects. Psychiatry Res. 225 (3), 374–380. Marwaha, S., Johnson, S., 2006. View and experiences of employment among people with psychosis: a qualitative descriptive study. Int. J. Soc. Psychiatry 51 (4), 302–316. Myers, W.R., 2000. Handling missing data in clinical trials: an overview. Drug Inf. J. 34, 525–533. Randolph, C., Tierney, M.C., Mohr, E., Chase, T.N., 1998. The Repeatable Battery for the Assessment of Neuropsychological Status (RBANS): preliminary clinical validity. J. Clin. Exp. Neuropsychol. 20 (3), 310–319.
Rosenheck, R., Leslie, D., Keefe, R., McEvoy, J., Swartz, M., Perkins, D., Stroup, S., Hsiao, J.K., Lieberman, J., 2006. Barriers to employment for people with schizophrenia. Am. J. Psychiatry 163 (3), 411–417. Salkever, D.S., Karakus, M.C., Slade, E.P., Harding, C.M., Hough, R.L., Rosenheck, R.A., Swartz, M.S., Barrio, C., Yamada, A.M., 2007. Measures and predictors of community-based employment and earnings of persons with schizophrenia in a multisite study. Psychiatr. Serv. 58 (3), 315–324. Sheehan, D.V., Lecrubier, Y., Sheehan, K.H., Amorim, P., Janavs, J., Weiller, E., Hergueta, T., Baker, R., Dunbar, G.C., 1998. The Mini-International Neuropsychiatric Interview (M.I.N.I.): the development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. J. Clin. Psychiatry 59 (Suppl 20), 22–33 quiz 34–57. Smith, M.J., Ginger, E.J., Wright, K., Wright, M.A., Taylor, J.L., Humm, L.B., Olsen, D.E., Bell, M.D., Fleming, M.F., 2014a. Virtual reality job interview training in adults with autism spectrum disorder. J. Autism Dev. Disord. 44 (10), 2450–2463. Smith, M.J., Ginger, E.J., Wright, M., Wright, K., Boteler Humm, L., Olsen, D., Bell, M.D., Fleming, M.F., 2014b. Virtual reality job interview training for individuals with psychiatric disabilities. J. Nerv. Ment. Dis. 202 (9), 659–667. Smith, M.J., Horan, W.P., Cobia, D.J., Karpouzian, T.M., Fox, J.M., Reilly, J.L., Breiter, H.C., 2014c. Performance-based empathy mediates the influence of working memory on social competence in schizophrenia. Schizophr. Bull. 40 (4), 824–834. Smith, M.J., Boteler Humm, L., Fleming, M.F., Jordan, N., Wright, M.A., Ginger, E.J., Wright, K., Olsen, D., Bell, M.D., 2015a. Virtual reality job interview training for veterans with posttraumatic stress disorder. J. Vocat. Rehabil 42 (3), 271–280. Smith, M.J., Fleming, M.F., Wright, M.A., Jordan, N., Boteler Humm, L., Olsen, D., Bell, M.D., 2015b. Job offers among individuals with severe mental illness after virtual reality job interview training in press. Psychiatr. Serv. Solinski, S., Jackson, H.J., Bell, R.C., 1992. Prediction of employability in schizophrenic patients. Schizophr. Res. 7 (2), 141–148. Vargas, G., Strassnig, M., Sabbag, S., Gould, F., Durand, D., Stone, L., Patterson, T.L., Harvey, P.D., 2014. The course of vocational functioning in patients with schizophrenia: reexamining social drift. Schizophrenia research. Cognition 1 (1), e41–e46. Wilkinson, G.S., Robertson, G.J., 2006. Wide Range Achievement Test 4 Professional Manual. Psychological Assessment Resources, Lutz, FL.
Please cite this article as: Smith, M.J., et al., Virtual reality job interview training and 6-month employment outcomes for individuals with schizophrenia seeking employment, Schizophr. Res. (2015), http://dx.doi.org/10.1016/j.schres.2015.05.022
