The Clinical Neuropsychologist
ISSN: 1385-4046 (Print) 1744-4144 (Online) Journal homepage: http://www.tandfonline.com/loi/ntcn20
Factors Associated with Word Memory Test Performance in Persons with Medically Documented Traumatic Brain Injury Mark Sherer, Lynne C. Davis, Angelle M. Sander, Todd G. Nick, Chunqiao Luo, Nicholas Pastorek & Robin Hanks To cite this article: Mark Sherer, Lynne C. Davis, Angelle M. Sander, Todd G. Nick, Chunqiao Luo, Nicholas Pastorek & Robin Hanks (2015) Factors Associated with Word Memory Test Performance in Persons with Medically Documented Traumatic Brain Injury, The Clinical Neuropsychologist, 29:4, 522-541, DOI: 10.1080/13854046.2015.1052763 To link to this article: http://dx.doi.org/10.1080/13854046.2015.1052763
Published online: 11 Jun 2015.
Submit your article to this journal
Article views: 188
View related articles
View Crossmark data
Full Terms & Conditions of access and use can be found at http://www.tandfonline.com/action/journalInformation?journalCode=ntcn20 Download by: [University of Manitoba Libraries]
Date: 13 November 2015, At: 16:08
The Clinical Neuropsychologist, 2015 Vol. 29, No. 4, 522–541, http://dx.doi.org/10.1080/13854046.2015.1052763
Factors Associated with Word Memory Test Performance in Persons with Medically Documented Traumatic Brain Injury Mark Sherer1, Lynne C. Davis1, Angelle M. Sander2, Todd G. Nick3, Chunqiao Luo3, Nicholas Pastorek4, and Robin Hanks5 1
Brain Injury Research Center, TIRR Memorial Hermann, Houston, TX, USA Department of Physical Medicine and Rehabilitation, Baylor College of Medicine and Harris Health System, Brain Injury Research Center, TIRR Memorial Hermann, Houston, TX, USA 3 Biostatistics Program, Department of Pediatrics, College of Medicine, University of Arkansas for Medical Sciences, Little Rock, AR, USA 4 Rehabilitation and Extended Care Line, Department of Physical Medicine and Rehabilitation, Michael E. DeBakey VA Medical Center, Baylor College of Medicine, Houston, TX, USA 5 Department of Physical Medicine and Rehabilitation, Rehabilitation Institute of Michigan, Wayne State University School of Medicine, Detroit, MI, USA
The Clinical Neuropsychologist 2015.29:522-541.
2
Objectives: (1) To examine the rate of poor performance validity in a large, multicenter, prospectively accrued cohort of community dwelling persons with medically documented traumatic brain injury (TBI), (2) to identify factors associated with Word Memory Test (WMT) performance in persons with TBI. Method: This was a prospective cohort, observational study of 491 persons with medically documented TBI. Participants were administered a battery of cognitive tests, questionnaires on emotional distress and post-concussive symptoms, and a performance validity test (WMT). Additional data were collected by interview and review of medical records. Results: One hundred and seventeen participants showed poor performance validity using the standard cutoff. Variable cluster analysis was conducted as a data reduction strategy. Findings revealed that the 10 cognitive tests and questionnaires could be summarized as 4 indices of emotional distress, speed of cognitive processing, verbal memory, and verbal fluency. Regression models revealed that verbal memory, emotional distress, age, and injury severity (time to follow commands) made unique contribution to prediction of poor performance validity. Conclusions: Poor performance validity was common in a research sample of persons with medically documented TBI who were not evaluated in conjunction with litigation, compensation claims, or current report of symptoms. Poor performance validity was associated with poor performance on cognitive tests, greater emotional distress, lower injury severity, and greater age. Many participants expected to have residual deficits based on initial injury severity showed poor performance validity. Keywords: Traumatic brain injury; Performance validity; Effort; Word Memory Test; Neuropsychological assessment.
INTRODUCTION Performance validity tests (PVTs) are designed to aid in the detection of suboptimal effort on cognitive testing. The importance of incorporating PVTs into
Address correspondence to: Mark Sherer, Ph.D., Brain Injury Research Center, TIRR Memorial Hermann, 1333 Moursund Street, Houston, TX 77030, USA. E-mail: [email protected] (Received 10 December 2014; accepted 14 May 2015)
© 2015 Taylor & Francis
The Clinical Neuropsychologist 2015.29:522-541.
TBI AND PERFORMANCE VALIDITY
523
neuropsychological evaluation procedures is well established (Bush et al., 2005; Heilbronner, Sweet, Morgan, Larrabee, & Millis, 2009; Larrabee, 2012) as failure to examine test performance validity can yield misinterpretation of test data, inaccurate diagnoses, misguided recommendations, and inappropriate allocation of resources. PVT failure is associated with poor neuropsychological test performance (Green, Rohling, Lees-Haley, & Allen, 2001; Lange, Iverson, Brooks, & Rennison, 2010). The presence of external incentives, such as financial compensation, renders PVT administration particularly important when evaluating persons with traumatic brain injury (TBI) (Larrabee, 2000). Estimates suggest that PVT failure rates approach 40% in forensic settings among individuals claiming neuropsychological dysfunction (Larrabee, 2003; Mittenberg, Patton, Canyock, & Condit, 2002). The Word Memory Test (WMT; Green, 2003) is a well-researched and commonly used PVT that is generally found to have high sensitivity and specificity (Donders & Strong, 2013). Reports indicate 99–100% accuracy of the WMT in discriminating between persons asked to exert full effort and those instructed to simulate memory dysfunction (Brockhaus & Merten, 2004; Tan, Slick, Strauss, & Hultsch, 2002). The computerized version involves presentation of a list of semantically related pairs of common words across two learning trials. Forced-choice recognition trials (immediate and delayed) are then administered, which permits evaluation of consistency of responding in addition to examination of performance accuracy. Four additional subtests are administered following the delayed recognition trial, including multiple choice cued recognition, paired associates recall, short-delay free recall, and optional long-delay free recall. Cutoff scores indicated in the test manual are used to classify performance as pass/fail (Green, 2003). The relationship between PVTs and the measurement of cognitive abilities has been well studied. WMT failure is associated with poorer performance on neuropsychological measures, particularly those assessing memory, motor skills, executive functioning, and perceptual organization (Gervais, Rohling, Green, & Ford, 2004; Green et al., 2001). WMT recognition scores are relatively insensitive to brain damage, as persons with moderate to severe TBI have obtained average scores of 95% correct for Immediate Recognition, Delayed Recognition, and Consistency trials (Green, 2003; Green, Lees-Haley, & Allen, 2002), which compare to 97.8–99.5% correct in healthy controls (Green, 2003; Tan et al., 2002). An inverse relationship between TBI severity level and WMT performance has been reported; individuals with mild TBI have shown significantly poorer WMT performance than those with severe TBI (Green, Iverson, & Allen, 1999). In a large series of persons evaluated in conjunction with disability or personal injuries claims, Flaro, Green, and Robertson (2007) reported a WMT failure rate of 40% in persons with mild TBI, contrasting sharply with a 21% failure rate in the moderate–severe TBI group. Similarly, in this investigation, persons with abnormal CT/MRI brain scans performed better on the WMT as compared to those with normal scans (Flaro et al., 2007). Previous studies of WMT performance among persons with TBI have yielded variable rates of WMT failure, ranging from 0 (Konrad et al., 2011) to 31% (Krishnan & Donders, 2011) in civilian samples, from 21 to 40% in predominantly compensationseeking samples (Flaro et al., 2007), and from 27.8 (Lange, Pancholi, Bhagwat, Anderson-Barnes, & French, 2012) to 68.4% (Russo, 2012) in veteran samples. Importantly, the vast majority of investigations have involved TBI diagnosis based on
The Clinical Neuropsychologist 2015.29:522-541.
524
MARK SHERER ET AL.
retrospective self-report (Armistead-Jehle & Buican, 2013; Konrad et al., 2011; Lange et al., 2012; Lippa et al., 2014; McCormick, Yoash-Gantz, McDonald, Campbell, & Tupler, 2013; Russo, 2012; Wisdom et al., 2014), persons who were referred for neuropsychological evaluation because of cognitive complaints (Bowden, Shores, & Mathias, 2006; Donders & Boonstra, 2007; Krishnan & Donders, 2011; Lange et al., 2012; Lippa et al., 2014; Meyers, Volbrecht, Axelrod, & Reinsch-Boothby, 2011; Russo, 2012; Stevens, Friedel, Mehren, & Merten, 2008), and those seeking financial compensation related to their injuries (Bowden et al., 2006; Flaro et al., 2007; Green, 2007; Green et al., 1999, 2001; Stevens et al., 2008). Samples of individuals with self-reported difficulties thought to be suggestive of TBI are particularly common among investigations of veterans. The reliability of selfreported symptoms and injury characteristics among veterans has been questioned (Stein & McAllister, 2009; Van Dyke, Axelrod & Schutte, 2010). Self-reported alteration of consciousness sustained in combat can result from causes other than TBI, including stress, inadequate sleep, and syncope (Hoge, Goldberg, & Castro, 2009). Additional weaknesses of self-report methods of TBI diagnosis include inaccuracy of memory (Carlson et al., 2011), particularly since these reports often occur months or years after injury. Possible symptom exaggeration or response bias, secondary gain (financial and/or psychological), inaccurate evaluation of pre-injury status, and personal biases in openness to acknowledging difficulties are also problematic features of self-reported information (Gunstad & Suhr, 2001; Vanderploeg & Belanger, 2013). Self-reported cognitive complaints have shown poor correlation with objective neuropsychological test scores in various clinical groups, including TBI (Drag, Spencer, Walker, Pangilinan, & Bieliauskas, 2012; Spencer, Drag, Walker, & Bieliauskas, 2010). Study samples consisting of persons with TBI who are referred for neuropsychological evaluation may show poor generalizability to the broader population of persons with TBI, as they constitute subsets of individuals with cognitive complaints that are not reported by many individuals in non-referred research settings. Meta-analytic evidence suggests that cognitive impairments among persons with mild TBI in referred clinical samples are associated with effect sizes of d = .74 versus effect sizes of d = .04 standard deviations among prospectively recruited samples (Belanger, Curtiss, Demery, Lebowitz, & Vanderploeg, 2005). Study samples consisting exclusively or predominantly of compensation-seeking persons are also unlikely to accurately represent the broader population of those with TBI due to the potentially biasing effect of secondary gain. Estimates suggest that PVT performance contributes more to the variance in neuropsychological test scores than TBI severity level in compensation-seeking persons (Green et al., 2001; Stevens et al., 2008). Effort level appears to contribute four to five times more to the variance in neuropsychological scores as compared to TBI severity (Green et al., 2001; Rohling & Demakis, 2010). To our knowledge, investigations of PVT performance in persons with medically documented TBI in samples with broad generalizability to the general population of individuals with TBI are lacking. Our literature search revealed no studies examining WMT performance in persons with medically documented TBI across a broad range of TBI severity using samples that were not assessed in connection with ongoing complaints or compensation claims. Konrad et al. (2011) did report on a convenience sample of 33 persons with medically diagnosed mild TBI drawn from a potential sample of 233 persons with mild TBI. In this study, no participants with mild TBI showed poor
The Clinical Neuropsychologist 2015.29:522-541.
TBI AND PERFORMANCE VALIDITY
525
performance validity on a German adaptation of the WMT. The purposes of this study were (1) to examine the rate of poor performance validity in a large, multicenter, prospectively accrued cohort of community dwelling persons with medically documented TBI most of whom required inpatient rehabilitation in the acute post-injury period, (2) to identify factors associated with WMT performance in a large sample of persons with medically documented TBI who were not referred due to cognitive complaints or compensation-seeking. After initial review of this manuscript, the reviewers and editor recommended that we add content regarding the Genuine Memory Impairment Profile (GMIP) for the WMT. While scores below the recommended cutoffs for the WMT “easy” tests (Immediate Recognition, Delayed Recognition, Consistency) are generally interpreted as indicating poor performance validity, it is possible that a person could be so cognitively impaired that he/she scores below cutoffs for the easy tests even when giving a full effort. The GMIP is an attempt to detect this occurrence. Green and others have recommended a cutoff for the discrepancy between the average score on the easy tests and the average score on the “hard” tests (Multiple Choice, Paired Associates, Free Recall) based on the notion that persons with very severe memory impairment who are giving a valid effort should still perform substantially better on the easy tests than on the hard tests while persons not giving valid effort may just perform very poorly on both easy and hard tests. So in applying the GMIP, persons showing a discrepancy above the recommended cutoff may have genuine memory impairment, while those scoring a discrepancy below the cutoff are thought to have poor performance validity. The GMIP was initially developed for assessments of persons with possible dementia. Indeed, Green, Flaro, and Courtney (2009) indicate that even when the GMIP pattern is found poor effort may still be the explanation unless other factors indicate memory impairment as severe as seen in dementia may be present. Unfortunately, the GMIP has not been shown to be specific to severe memory impairment as over 25% of non-patients asked to feign memory impairment produce the GMIP on the WMT (Armistead-Jehle & Denney, 2015). Several investigators have raised concerns regarding use of the GMIP methodology unless patients are likely to have severe to profound memory/cognitive impairment (Axelrod & Schutte, 2010; Donders & Strong, 2013; Meyers et al., 2014). In an effort to contribute some new knowledge regarding the possible use of the GMIP in persons with TBI, we compared persons who met criteria for poor performance validity and produced the GMIP pattern to those who met criteria for poor performance validity and did not produce the GMIP pattern.
METHOD Study participants Participants for this study were enrolled in a study of symptoms, perceived strengths, impairments, and environmental supports for persons with TBI in the post-acute period of recovery (Sherer et al., in press). The present investigation is a secondary analysis of a subset of the data collected for the initial study. Additional description of the study sample can be found in Sherer et al. (in press). Participants for this investigation were recruited and enrolled at three study sites. Institutional review board approval was obtained at each site. Inclusion criteria were as follows:
526
MARK SHERER ET AL.
(1) contemporaneous definitive medical documentation of TBI, (2) aged 18–64, (3) capacity to give informed consent, (4) ability to complete all study measures in English. Exclusion criteria were (1) presence of a medical or psychiatric condition that affected functioning to a greater extent than TBI, (2) failure to complete the WMT. Participant recruitment and data collection
The Clinical Neuropsychologist 2015.29:522-541.
Participants were a convenience sample of persons who had previously participated in TBI related research at the three study sites. Potential participants were contacted to determine interest in participating in this investigation. If they indicated interest, they were consented and administered the study assessment battery. Study enrollment extended from October 2011 through September 2013. All participants were oriented. Assessment sessions generally lasted 3 h or longer. Study measures Data were collected by medical record review, interview, and administration of questionnaires and cognitive tests. Data collected by medical record review included date of birth, date of injury, injury severity (Glasgow Coma Scale score [GCS], length of coma, duration of post-traumatic amnesia [PTA]), and length of stay in days (acute care hospital, rehabilitation hospital). Data collected by interview included race/ ethnicity, sex, marital status, years of education, employment at time of injury, annual earnings, and alcohol and drug use history. Earnings for the year prior to injury were captured in 10 categories indicating earnings in tens of thousands of dollars so category 0 = no income, category 1 = less than $10,000, category 2 = $10,000–$19,999, and so forth up to category 9 which indicated $80,000 or more. Problem alcohol use was defined as heavy use (>14 drinks per week for men, >7 drinks per week for women) or binge use (consumption of 5 or more drinks on one day in the past month) and problem drug use was defined as any use of illicit drugs (Corrigan, Bogner, Lamb-Hart, & Sivak-Sears, 2003). Cognitive tests and questionnaires were selected based on the National Institute on Neurological Disorders and Stroke TBI Common Data Elements recommendations (Wilde et al., 2010). Cognitive tests were the Rey Auditory Verbal Learning Test (RAVLT), the Trail Making Test, verbal fluency (FAS), and the Symbol Search and Coding subtests from the Wechsler Adult Intelligence Scale-IV (WAIS-IV). The RAVLT (Rey, 1958) is a list learning task. The respondent is read a list of 15 words on 5 occasions and after each reading, repeats back as many of the words as he/she can recall. After an interference trial, the respondent again is asked to recall the original list and then, again, after a 30 min delay. RAVLT scores used for this investigation were the total words recalled for the 5 learning trials (RAVLT Trials 1-5) and the words recalled on the 30 min delayed recall (RAVLT Long-Delay). The Trail Making Test (Reitan & Wolfson, 1985) consists of two subtests, Trails A and Trails B. For Trails A, the respondent connects circles containing numbers in ascending order as quickly as possible and, for Trails B, the respondent connects circles containing numbers and letters in ascending and alphabetical order as quickly as possible alternating between numbers and letters. Scores for the two Trails measures were number of seconds needed to complete the tasks.
The Clinical Neuropsychologist 2015.29:522-541.
TBI AND PERFORMANCE VALIDITY
527
For the verbal fluency test (Gladsjo et al., 1999), the respondent is given 1 min each to say as many words as possible that begin with the letters, “F,” “A,” and “S.” The score is the total number of words given across the three trials. Coding and Symbol Search are subtests of the Wechsler Adult Intelligence Scale-Fourth Edition (WAIS-IV) and define the Processing Speed Index (Wechsler, 2008). On the Coding subtest, the respondent write symbols under the numbers they are matched with on a key grid. This task is timed. For Symbol Search, the respondent must find shapes that match samples from an array of shapes including some foils. This task is timed. As appropriate, raw scores on cognitive tests were converted to normed scores using the applicable norms sets. Performance validity was measured with the WMT. Poor validity could represent an attempt to perform poorly, inattention, or other response sets that result in noncredible patterns of responses. As recommended by Green (2007), both dichotomous (valid versus invalid) and continuous indices of performance validity were employed. Participants were also administered the Patient Health Questionnaire—9 (PHQ-9), Generalized Anxiety Disorder—7 (GAD-7), and Rivermead Post-Concussion Symptom Questionnaire (RPQ). The PHQ-9 (Kroenke, Spitzer, & Williams, 2001) was derived from the Patient Health Questionnaire (Spitzer, Kroenke, & Williams, 1999), a selfreport survey used to diagnose mental disorder in primary care settings. The PHQ-9 has been validated as a measure of depression in persons with TBI (Cook et al., 2011). The GAD-7 (Spitzer, Kroenke, Williams, & Löwe, 2006) was developed as a complementary measure to the PHQ-9 for assessment in anxiety of persons with TBI. While not studied as extensively as the PHQ-9, the GAD-7 has been shown to be useful in TBI research (Harch et al., 2012). Scores for the PHQ-9 and GAD-7 were total scores for each measure. The RPQ (King, Crawford, Wenden, Moss, & Wade, 1995) is a measure of symptoms reported by persons who have sustained TBI. For this questionnaire, respondents rate the severity of symptoms experienced in the past 24 h as compared to their intensity prior to the TBI. Data analysis Descriptive data were presented as means and standard deviations (SD) for continuous-type variables (age, years of education, time since injury, PTA, length of stay rehabilitation, time to follow commands, cognitive tests, questionnaires) and frequencies and percentages for categorical-type variables (sex, race/ethnicity, marital status, problem drinker, drug use, income, GCS). The amount of missing data was small with less than 2% missing on all candidate predictors listed above except for time to follow commands, which had 18% missing. To include all persons in the regression models discussed below, imputation was used to predict the missing data. The imputations were based on computing regression models for each candidate predictor that predicted the missing value, while adding a random component. This random component avoids biased results. Since the amount of missing data was small, only one random draw of the data was used instead of analyzing multiple imputed data sets. In addition to including all of the variables mentioned below to predict the missing data for each variable, the variables WMT group, race/ethnicity, marital status, PTA, LOS rehab, and GCS group were included to improve the predictions of the missing data.
The Clinical Neuropsychologist 2015.29:522-541.
528
MARK SHERER ET AL.
To reduce the number of cognitive test and questionnaire measures to avoid fitting too many candidate predictor variables for regression analyses, a variable cluster analysis was performed to group the cognitive tests and questionnaires into sets and then a principal component analysis was conducted to generate a composite score for each set. The variable cluster analysis was performed on the cognitive tests (Symbol Search, Coding, Trails A, Trails B, RAVLT Trials 1-5, RAVLT Long-Delay, FAS) and questionnaires (RPQ, GAD-7, PHQ-9) using the VARCLUS procedure in SAS to group the measures (SAS Institute, Inc., 2009). This procedure groups the measures into clusters so that measures within a cluster are more strongly related to each other than to measures in other clusters. Next, a principal component (PC) analysis was computed within each proposed set (cluster) and the first PC score of the set was used as the measure or composite function of the set. Two outcomes derived from the WMT were analyzed using logistic and linear regression models. First, a dichotomous outcome of valid versus invalid (WMT group) was derived by classifying any participant who scored below the recommended cutoff on any of the Immediate Recognition, Delayed Recognition, or Consistency scores as invalid and all others as valid (Green, 2003). Next, as recommended by Green (2007), we also created a continuous outcome indicating the degree of validity by averaging the Immediate Recognition, Delayed Recognition, and Consistency scores (Avg WMT). Higher scores on this index indicated better performance validity. The dichotomous WMT outcome was submitted to logistic regression, while the continuous WMT outcome was submitted to ordinary linear regression. In both cases, unadjusted (simple) and adjusted (multivariable) models were computed to examine the unadjusted and adjusted effects, respectively, of each candidate predictor. The pre-specified candidate predictors included age (at time of assessment), sex, education, income, problem drinker, drug use, time since injury, time to follow commands, and the cluster scores derived from the cluster analysis that represented the 10 cognitive tests and questionnaires. For the multivariable models instead of using stepwise regression models which often produce overly optimistic estimates, full model fits were computed by including all pre-specified candidate predictors for the adjusted analysis. For all continuous-type candidate predictors, we used natural splines to relax the linearity assumption for both the adjusted and unadjusted linear regression models. Splines are mathematical representations of curved shapes. In linear regression models, splines are used to fit more flexible curves to the data than the simplest form of linear regression allows. Without splines, the regression models would only test for linear relationships between independent variables and the outcome. The non-linear term for each independent variable was retained in all models, adjusted and unadjusted, if the non-linear p-value was less than .20 in either the multivariable logistic or linear regression model. Effects for the continuous-type independent variable were shown as interquartile coefficients which indicate the change in the value of the outcome as the value of the independent variable changes from the 25th to the 75th percentile. To measure the overall strength of association of the models, R-square and C index are reported for the linear and logistic models, respectively. The C index is equivalent to the receiver operating characteristic curve. Because these measures tend to be overly optimistic, the validated R-square and C index were computed based on a bootstrap (200 bootstrap samples) resampling plan. For all analyses, the RMS package in R (version 3.1.1) programming language was used to account for non-linear terms and to
TBI AND PERFORMANCE VALIDITY
529
compute validated R-square measures (Harrell, 2009). For persons with poor performance validity, those meeting criteria for the GMIP were compared those who did not meet criteria for the GMIP.
RESULTS
The Clinical Neuropsychologist 2015.29:522-541.
Description of the study sample Five hundred four persons were enrolled in the study and all of these completed at least partial assessments. Thirteen (2.6%) failed to complete the WMT which was the key study measure for these analyses, and all 13 were excluded from study, yielding a study sample of 491 participants. In all cases, participants did not complete the WMT due to terminating the assessment session before this measure was obtained. Descriptive data describing demographic characteristics, pre-injury functioning, injury severity, and performance on study measures are provided in Table 1. Scores for the RPQ, PHQ-9, and GAD-7 are raw scores. Scores for the RAVLT Trials 1-5, RAVLT Long-Delay, Trails A, Trails B, and FAS are T-scores. Scores for the WAIS-IV Symbol Search and Coding subtests are scaled scores. As is common in studies of persons with TBI, most were males of young to middle age. Since most participants received inpatient rehabilitation following their injuries, it is expected that most sustained moderate or severe injuries. As indicated by data in Table 2, if one assumes that persons requiring intubation or medical sedation shortly after injury likely sustained moderate or severe TBI, 343 (70%) participants sustained moderate or severe TBI. Four hundred twenty (86%) participants underwent inpatient rehabilitation for problems related to their TBI. Performance validity for GCS severity categories Table 2 shows the proportions of persons with good performance validity for each GCS category. Persons with mild, moderate, or severe TBI showed essentially the same proportions of good performance validity, while persons who required intubation or medication had higher rates of valid scores and those missing GCS scores had lower rates of valid scores. Data reduction Data reduction analyses performed on study measures (Symbol Search, Trails A, Trails B, Coding, RAVLT Trials 1-5, RAVLT Long-Delay, FAS, RPQ, GAD-7, PHQ-9) identified four clusters. Cluster 1 was defined by the RPQ, GAD-7, and PHQ-9 and represents the degree of emotional distress experienced by participants. Cluster 2 was defined by Symbol Search, Coding, Trails A, and Trails B and represents speed of cognitive processing. Cluster 3 was defined by RAVLT Trials 1-5 and RAVLT Long-Delay and represents verbal memory. Finally, Cluster 4 was defined by FAS and indicates participants’ verbal fluency. The first principal component score for each cluster was used as the summary score for each of the four clusters. These scores explained 85, 70, 90, and 100% of the variability of the four clusters, respectively.
530
MARK SHERER ET AL.
The Clinical Neuropsychologist 2015.29:522-541.
Table 1. Description of the study sample on outcome, demographics, injury severity, and predictors (N = 491) Categorical descriptors
n (%) Missing
n (%)
Male Race/ethnicity White Black Hispanic Other Marital statusa Never married Married/common law Separated/divorced Widowed Drugs (yes) Problem drinker (yes) Incomeb Q1:$9,999 or less Median:$10,000–$19,999 Q3:$30,000–$39,999 GCS groupa Severe (3–8) Moderate (9–12) Mild (13–15) Intubated Medicated WMT good effort
0 (0%) 0 (0%)
369 (75%)
0 (0%)
150 (33%) 47 (10%) 118 (26%) 93 (20%) 53 (11%) 374 (76%)
Continuous descriptors
n (%) Missing
Mean (SD)
Age (years) Education (years) LOS rehab (days) Time since injury (years) Time to follow (days) PTA duration (days) Rivermead post-concussion GAD-7 PHQ-9 Symbol search Trails A seconds Trails B seconds Coding RAVLT Trials 1-5 total RAVLT long-delay free recall FAS WMT average
0 (0%) 0 (0%) 38 (7.7%) 3 (.6%) 88 (17.9%) 154 (31.4%) 1 (.2%) 0 (0%) 2 (.4%) 4 (.8%) 2 (.4%) 8 (1.6%) 4 (.8%) 0 (0%) 2 (.4%) 0 (0%) 0 (%)
38.0 (13.2) 12.7 (2.3) 20.8 (18.3) 6.2 (6.8) 6.7 (9.4) 21.4 (19.7) 23.8 (13.4) 5.4 (5.5) 6.9 (5.9) 7.5 (3.0) 42.3 (13.5) 44.3 (13.0) 7.1 (3.0) 34.9 (14.8) 35.7 (15.9) 42.8 (12.0) 91.5 (11.5)
a
220 (45%) 181 (37%) 72 (15%) 18 (4%) 2 (.4%)
0 (0%) 0 (0%) 0 (0%)
86 (18%) 80 (16%) 42 (9%) 30 (6.1%)
n (%) are computed based on non-missing data. Q1 and Q3 represent the first and third quartiles.
b
269 (55%) 124 (25%) 94 (19%) 12 (2%) 187 (38%) 267 (54%)
TBI AND PERFORMANCE VALIDITY
531
Table 2. WMT invalid/valid effort distributions % (n) for Glasgow Coma Scale severity groups (N = 491) GCS group Severe (3–8) Moderate (9–12) Mild (13–15) Intubated Medicated Missing
N 150 47 118 93 53 30
WMT invalid 38 13 29 19 7 11
(25%) (28%) (25%) (20%) (13%) (37%)
WMT valid 112 (75%) 34 (72%) 89 (75%) 74 (80%) 46 (87%) 19 (63%)
The Clinical Neuropsychologist 2015.29:522-541.
Regression analyses As shown in Table 1, for the dichotomous WMT outcome, 374 (76%) of participants showed valid scores and 117 (24%) showed invalid scores. For the continuous index of performance validity the mean (SD) value was 91.5 (11.5). In addition to the summary scores for each cluster described above, independent variables included in regression models were gender, age in years at time of examination, years of education, income category (0–9), problem alcohol use (yes/no), drug use (yes/no), time to follow commands in days, and time from injury to assessment in years. To incorporate nonlinear trends, four additional degrees of freedom were included for age and the first three clusters given that the non-linear p-values were less than .20 for either model (all p < .20). For both multivariable models, there are 16 total degrees of freedom including 12 independent variables with four non-linear terms. Note that because there were only 117 persons in the least frequent outcome category (limiting sample size) for the full model logistic regression, this model was at risk for over-fitting based on a standard 10 persons per one degree of freedom rule. However, the full model linear regression is not likely to be over-fitted given up to 49 degrees of freedom could be fitted based on a standard rule requiring 10 persons per every degree of freedom fitted. For this reason, the full model linear regression should be considered the primary analysis. Results of the logistic regression models are presented in Table 3. Unadjusted (univariable) effects are provided for all factors, but we focus on the adjusted effects. Age at time of injury, time to follow commands, Cluster 1 (emotional distress), and Cluster 3 (memory) were associated with of performance validity. Those with younger age at assessment, longer time to follow commands (more severe injury), less emotional distress, and more intact memory were more likely to show valid scores. The greatest effect was for memory. Those scoring at the 75th percentile on memory (better memory) were 7.3 times as likely to have valid scores as those scoring at the 25th percentile on memory (poorer memory). Stated another way, those obtaining scores indicating poorer memory had only .14 times the odds of showing valid scores as those obtaining scores indicating better memory. Those reporting less emotional distress at the 25th percentile were about twice as likely to have valid scores as those reporting greater emotional distress at the 75th percentile. Those with younger age (26 years) were about twice as likely to have valid scores as those with older age (50 years). Those with greater injury severity (9.5 days to follow commands) were about 30% more likely to have valid scores than those with milder injury severity (.5 days to follow commands). The C index was .84, and the validated index was .81 using bootstrap resampling.
532
MARK SHERER ET AL. Table 3. Unadjusted and adjusted results of logistic regression analysis predicting probability of poor performance validity (N = 491)
The Clinical Neuropsychologist 2015.29:522-541.
Unadjusted (simple) model
Multivariable model
Predictor
Comparison
Effect (OR)
95% CI
p
Effect (OR)
95% CI
p
Gender Age Education Income category Problem drinker Drug use Time to follow Time since injury Cluster 1 Cluster 2 Cluster 3 Cluster 4
M, F 26, 50 years 11, 14 years 1, 4 No, yes No, yes .5, 9.5 days 1.0, 9.9 years −.8, .6 −.6, .7 −.8, .8 −.6, .7
1.21 .52 1.76 1.10 .71 .85 1.05 .78 .48 2.39 5.82 1.31
.74, 1.98 .33, .81 1.31, 2.36 .86, 1.39 .47, 1.09 .56, 1.30 .86, 1.28 .60, 1.00 .31, .74 1.62, 3.52 3.67, 9.24 1.00, 1.73
.45
ISSN: 1385-4046 (Print) 1744-4144 (Online) Journal homepage: http://www.tandfonline.com/loi/ntcn20
Factors Associated with Word Memory Test Performance in Persons with Medically Documented Traumatic Brain Injury Mark Sherer, Lynne C. Davis, Angelle M. Sander, Todd G. Nick, Chunqiao Luo, Nicholas Pastorek & Robin Hanks To cite this article: Mark Sherer, Lynne C. Davis, Angelle M. Sander, Todd G. Nick, Chunqiao Luo, Nicholas Pastorek & Robin Hanks (2015) Factors Associated with Word Memory Test Performance in Persons with Medically Documented Traumatic Brain Injury, The Clinical Neuropsychologist, 29:4, 522-541, DOI: 10.1080/13854046.2015.1052763 To link to this article: http://dx.doi.org/10.1080/13854046.2015.1052763
Published online: 11 Jun 2015.
Submit your article to this journal
Article views: 188
View related articles
View Crossmark data
Full Terms & Conditions of access and use can be found at http://www.tandfonline.com/action/journalInformation?journalCode=ntcn20 Download by: [University of Manitoba Libraries]
Date: 13 November 2015, At: 16:08
The Clinical Neuropsychologist, 2015 Vol. 29, No. 4, 522–541, http://dx.doi.org/10.1080/13854046.2015.1052763
Factors Associated with Word Memory Test Performance in Persons with Medically Documented Traumatic Brain Injury Mark Sherer1, Lynne C. Davis1, Angelle M. Sander2, Todd G. Nick3, Chunqiao Luo3, Nicholas Pastorek4, and Robin Hanks5 1
Brain Injury Research Center, TIRR Memorial Hermann, Houston, TX, USA Department of Physical Medicine and Rehabilitation, Baylor College of Medicine and Harris Health System, Brain Injury Research Center, TIRR Memorial Hermann, Houston, TX, USA 3 Biostatistics Program, Department of Pediatrics, College of Medicine, University of Arkansas for Medical Sciences, Little Rock, AR, USA 4 Rehabilitation and Extended Care Line, Department of Physical Medicine and Rehabilitation, Michael E. DeBakey VA Medical Center, Baylor College of Medicine, Houston, TX, USA 5 Department of Physical Medicine and Rehabilitation, Rehabilitation Institute of Michigan, Wayne State University School of Medicine, Detroit, MI, USA
The Clinical Neuropsychologist 2015.29:522-541.
2
Objectives: (1) To examine the rate of poor performance validity in a large, multicenter, prospectively accrued cohort of community dwelling persons with medically documented traumatic brain injury (TBI), (2) to identify factors associated with Word Memory Test (WMT) performance in persons with TBI. Method: This was a prospective cohort, observational study of 491 persons with medically documented TBI. Participants were administered a battery of cognitive tests, questionnaires on emotional distress and post-concussive symptoms, and a performance validity test (WMT). Additional data were collected by interview and review of medical records. Results: One hundred and seventeen participants showed poor performance validity using the standard cutoff. Variable cluster analysis was conducted as a data reduction strategy. Findings revealed that the 10 cognitive tests and questionnaires could be summarized as 4 indices of emotional distress, speed of cognitive processing, verbal memory, and verbal fluency. Regression models revealed that verbal memory, emotional distress, age, and injury severity (time to follow commands) made unique contribution to prediction of poor performance validity. Conclusions: Poor performance validity was common in a research sample of persons with medically documented TBI who were not evaluated in conjunction with litigation, compensation claims, or current report of symptoms. Poor performance validity was associated with poor performance on cognitive tests, greater emotional distress, lower injury severity, and greater age. Many participants expected to have residual deficits based on initial injury severity showed poor performance validity. Keywords: Traumatic brain injury; Performance validity; Effort; Word Memory Test; Neuropsychological assessment.
INTRODUCTION Performance validity tests (PVTs) are designed to aid in the detection of suboptimal effort on cognitive testing. The importance of incorporating PVTs into
Address correspondence to: Mark Sherer, Ph.D., Brain Injury Research Center, TIRR Memorial Hermann, 1333 Moursund Street, Houston, TX 77030, USA. E-mail: [email protected] (Received 10 December 2014; accepted 14 May 2015)
© 2015 Taylor & Francis
The Clinical Neuropsychologist 2015.29:522-541.
TBI AND PERFORMANCE VALIDITY
523
neuropsychological evaluation procedures is well established (Bush et al., 2005; Heilbronner, Sweet, Morgan, Larrabee, & Millis, 2009; Larrabee, 2012) as failure to examine test performance validity can yield misinterpretation of test data, inaccurate diagnoses, misguided recommendations, and inappropriate allocation of resources. PVT failure is associated with poor neuropsychological test performance (Green, Rohling, Lees-Haley, & Allen, 2001; Lange, Iverson, Brooks, & Rennison, 2010). The presence of external incentives, such as financial compensation, renders PVT administration particularly important when evaluating persons with traumatic brain injury (TBI) (Larrabee, 2000). Estimates suggest that PVT failure rates approach 40% in forensic settings among individuals claiming neuropsychological dysfunction (Larrabee, 2003; Mittenberg, Patton, Canyock, & Condit, 2002). The Word Memory Test (WMT; Green, 2003) is a well-researched and commonly used PVT that is generally found to have high sensitivity and specificity (Donders & Strong, 2013). Reports indicate 99–100% accuracy of the WMT in discriminating between persons asked to exert full effort and those instructed to simulate memory dysfunction (Brockhaus & Merten, 2004; Tan, Slick, Strauss, & Hultsch, 2002). The computerized version involves presentation of a list of semantically related pairs of common words across two learning trials. Forced-choice recognition trials (immediate and delayed) are then administered, which permits evaluation of consistency of responding in addition to examination of performance accuracy. Four additional subtests are administered following the delayed recognition trial, including multiple choice cued recognition, paired associates recall, short-delay free recall, and optional long-delay free recall. Cutoff scores indicated in the test manual are used to classify performance as pass/fail (Green, 2003). The relationship between PVTs and the measurement of cognitive abilities has been well studied. WMT failure is associated with poorer performance on neuropsychological measures, particularly those assessing memory, motor skills, executive functioning, and perceptual organization (Gervais, Rohling, Green, & Ford, 2004; Green et al., 2001). WMT recognition scores are relatively insensitive to brain damage, as persons with moderate to severe TBI have obtained average scores of 95% correct for Immediate Recognition, Delayed Recognition, and Consistency trials (Green, 2003; Green, Lees-Haley, & Allen, 2002), which compare to 97.8–99.5% correct in healthy controls (Green, 2003; Tan et al., 2002). An inverse relationship between TBI severity level and WMT performance has been reported; individuals with mild TBI have shown significantly poorer WMT performance than those with severe TBI (Green, Iverson, & Allen, 1999). In a large series of persons evaluated in conjunction with disability or personal injuries claims, Flaro, Green, and Robertson (2007) reported a WMT failure rate of 40% in persons with mild TBI, contrasting sharply with a 21% failure rate in the moderate–severe TBI group. Similarly, in this investigation, persons with abnormal CT/MRI brain scans performed better on the WMT as compared to those with normal scans (Flaro et al., 2007). Previous studies of WMT performance among persons with TBI have yielded variable rates of WMT failure, ranging from 0 (Konrad et al., 2011) to 31% (Krishnan & Donders, 2011) in civilian samples, from 21 to 40% in predominantly compensationseeking samples (Flaro et al., 2007), and from 27.8 (Lange, Pancholi, Bhagwat, Anderson-Barnes, & French, 2012) to 68.4% (Russo, 2012) in veteran samples. Importantly, the vast majority of investigations have involved TBI diagnosis based on
The Clinical Neuropsychologist 2015.29:522-541.
524
MARK SHERER ET AL.
retrospective self-report (Armistead-Jehle & Buican, 2013; Konrad et al., 2011; Lange et al., 2012; Lippa et al., 2014; McCormick, Yoash-Gantz, McDonald, Campbell, & Tupler, 2013; Russo, 2012; Wisdom et al., 2014), persons who were referred for neuropsychological evaluation because of cognitive complaints (Bowden, Shores, & Mathias, 2006; Donders & Boonstra, 2007; Krishnan & Donders, 2011; Lange et al., 2012; Lippa et al., 2014; Meyers, Volbrecht, Axelrod, & Reinsch-Boothby, 2011; Russo, 2012; Stevens, Friedel, Mehren, & Merten, 2008), and those seeking financial compensation related to their injuries (Bowden et al., 2006; Flaro et al., 2007; Green, 2007; Green et al., 1999, 2001; Stevens et al., 2008). Samples of individuals with self-reported difficulties thought to be suggestive of TBI are particularly common among investigations of veterans. The reliability of selfreported symptoms and injury characteristics among veterans has been questioned (Stein & McAllister, 2009; Van Dyke, Axelrod & Schutte, 2010). Self-reported alteration of consciousness sustained in combat can result from causes other than TBI, including stress, inadequate sleep, and syncope (Hoge, Goldberg, & Castro, 2009). Additional weaknesses of self-report methods of TBI diagnosis include inaccuracy of memory (Carlson et al., 2011), particularly since these reports often occur months or years after injury. Possible symptom exaggeration or response bias, secondary gain (financial and/or psychological), inaccurate evaluation of pre-injury status, and personal biases in openness to acknowledging difficulties are also problematic features of self-reported information (Gunstad & Suhr, 2001; Vanderploeg & Belanger, 2013). Self-reported cognitive complaints have shown poor correlation with objective neuropsychological test scores in various clinical groups, including TBI (Drag, Spencer, Walker, Pangilinan, & Bieliauskas, 2012; Spencer, Drag, Walker, & Bieliauskas, 2010). Study samples consisting of persons with TBI who are referred for neuropsychological evaluation may show poor generalizability to the broader population of persons with TBI, as they constitute subsets of individuals with cognitive complaints that are not reported by many individuals in non-referred research settings. Meta-analytic evidence suggests that cognitive impairments among persons with mild TBI in referred clinical samples are associated with effect sizes of d = .74 versus effect sizes of d = .04 standard deviations among prospectively recruited samples (Belanger, Curtiss, Demery, Lebowitz, & Vanderploeg, 2005). Study samples consisting exclusively or predominantly of compensation-seeking persons are also unlikely to accurately represent the broader population of those with TBI due to the potentially biasing effect of secondary gain. Estimates suggest that PVT performance contributes more to the variance in neuropsychological test scores than TBI severity level in compensation-seeking persons (Green et al., 2001; Stevens et al., 2008). Effort level appears to contribute four to five times more to the variance in neuropsychological scores as compared to TBI severity (Green et al., 2001; Rohling & Demakis, 2010). To our knowledge, investigations of PVT performance in persons with medically documented TBI in samples with broad generalizability to the general population of individuals with TBI are lacking. Our literature search revealed no studies examining WMT performance in persons with medically documented TBI across a broad range of TBI severity using samples that were not assessed in connection with ongoing complaints or compensation claims. Konrad et al. (2011) did report on a convenience sample of 33 persons with medically diagnosed mild TBI drawn from a potential sample of 233 persons with mild TBI. In this study, no participants with mild TBI showed poor
The Clinical Neuropsychologist 2015.29:522-541.
TBI AND PERFORMANCE VALIDITY
525
performance validity on a German adaptation of the WMT. The purposes of this study were (1) to examine the rate of poor performance validity in a large, multicenter, prospectively accrued cohort of community dwelling persons with medically documented TBI most of whom required inpatient rehabilitation in the acute post-injury period, (2) to identify factors associated with WMT performance in a large sample of persons with medically documented TBI who were not referred due to cognitive complaints or compensation-seeking. After initial review of this manuscript, the reviewers and editor recommended that we add content regarding the Genuine Memory Impairment Profile (GMIP) for the WMT. While scores below the recommended cutoffs for the WMT “easy” tests (Immediate Recognition, Delayed Recognition, Consistency) are generally interpreted as indicating poor performance validity, it is possible that a person could be so cognitively impaired that he/she scores below cutoffs for the easy tests even when giving a full effort. The GMIP is an attempt to detect this occurrence. Green and others have recommended a cutoff for the discrepancy between the average score on the easy tests and the average score on the “hard” tests (Multiple Choice, Paired Associates, Free Recall) based on the notion that persons with very severe memory impairment who are giving a valid effort should still perform substantially better on the easy tests than on the hard tests while persons not giving valid effort may just perform very poorly on both easy and hard tests. So in applying the GMIP, persons showing a discrepancy above the recommended cutoff may have genuine memory impairment, while those scoring a discrepancy below the cutoff are thought to have poor performance validity. The GMIP was initially developed for assessments of persons with possible dementia. Indeed, Green, Flaro, and Courtney (2009) indicate that even when the GMIP pattern is found poor effort may still be the explanation unless other factors indicate memory impairment as severe as seen in dementia may be present. Unfortunately, the GMIP has not been shown to be specific to severe memory impairment as over 25% of non-patients asked to feign memory impairment produce the GMIP on the WMT (Armistead-Jehle & Denney, 2015). Several investigators have raised concerns regarding use of the GMIP methodology unless patients are likely to have severe to profound memory/cognitive impairment (Axelrod & Schutte, 2010; Donders & Strong, 2013; Meyers et al., 2014). In an effort to contribute some new knowledge regarding the possible use of the GMIP in persons with TBI, we compared persons who met criteria for poor performance validity and produced the GMIP pattern to those who met criteria for poor performance validity and did not produce the GMIP pattern.
METHOD Study participants Participants for this study were enrolled in a study of symptoms, perceived strengths, impairments, and environmental supports for persons with TBI in the post-acute period of recovery (Sherer et al., in press). The present investigation is a secondary analysis of a subset of the data collected for the initial study. Additional description of the study sample can be found in Sherer et al. (in press). Participants for this investigation were recruited and enrolled at three study sites. Institutional review board approval was obtained at each site. Inclusion criteria were as follows:
526
MARK SHERER ET AL.
(1) contemporaneous definitive medical documentation of TBI, (2) aged 18–64, (3) capacity to give informed consent, (4) ability to complete all study measures in English. Exclusion criteria were (1) presence of a medical or psychiatric condition that affected functioning to a greater extent than TBI, (2) failure to complete the WMT. Participant recruitment and data collection
The Clinical Neuropsychologist 2015.29:522-541.
Participants were a convenience sample of persons who had previously participated in TBI related research at the three study sites. Potential participants were contacted to determine interest in participating in this investigation. If they indicated interest, they were consented and administered the study assessment battery. Study enrollment extended from October 2011 through September 2013. All participants were oriented. Assessment sessions generally lasted 3 h or longer. Study measures Data were collected by medical record review, interview, and administration of questionnaires and cognitive tests. Data collected by medical record review included date of birth, date of injury, injury severity (Glasgow Coma Scale score [GCS], length of coma, duration of post-traumatic amnesia [PTA]), and length of stay in days (acute care hospital, rehabilitation hospital). Data collected by interview included race/ ethnicity, sex, marital status, years of education, employment at time of injury, annual earnings, and alcohol and drug use history. Earnings for the year prior to injury were captured in 10 categories indicating earnings in tens of thousands of dollars so category 0 = no income, category 1 = less than $10,000, category 2 = $10,000–$19,999, and so forth up to category 9 which indicated $80,000 or more. Problem alcohol use was defined as heavy use (>14 drinks per week for men, >7 drinks per week for women) or binge use (consumption of 5 or more drinks on one day in the past month) and problem drug use was defined as any use of illicit drugs (Corrigan, Bogner, Lamb-Hart, & Sivak-Sears, 2003). Cognitive tests and questionnaires were selected based on the National Institute on Neurological Disorders and Stroke TBI Common Data Elements recommendations (Wilde et al., 2010). Cognitive tests were the Rey Auditory Verbal Learning Test (RAVLT), the Trail Making Test, verbal fluency (FAS), and the Symbol Search and Coding subtests from the Wechsler Adult Intelligence Scale-IV (WAIS-IV). The RAVLT (Rey, 1958) is a list learning task. The respondent is read a list of 15 words on 5 occasions and after each reading, repeats back as many of the words as he/she can recall. After an interference trial, the respondent again is asked to recall the original list and then, again, after a 30 min delay. RAVLT scores used for this investigation were the total words recalled for the 5 learning trials (RAVLT Trials 1-5) and the words recalled on the 30 min delayed recall (RAVLT Long-Delay). The Trail Making Test (Reitan & Wolfson, 1985) consists of two subtests, Trails A and Trails B. For Trails A, the respondent connects circles containing numbers in ascending order as quickly as possible and, for Trails B, the respondent connects circles containing numbers and letters in ascending and alphabetical order as quickly as possible alternating between numbers and letters. Scores for the two Trails measures were number of seconds needed to complete the tasks.
The Clinical Neuropsychologist 2015.29:522-541.
TBI AND PERFORMANCE VALIDITY
527
For the verbal fluency test (Gladsjo et al., 1999), the respondent is given 1 min each to say as many words as possible that begin with the letters, “F,” “A,” and “S.” The score is the total number of words given across the three trials. Coding and Symbol Search are subtests of the Wechsler Adult Intelligence Scale-Fourth Edition (WAIS-IV) and define the Processing Speed Index (Wechsler, 2008). On the Coding subtest, the respondent write symbols under the numbers they are matched with on a key grid. This task is timed. For Symbol Search, the respondent must find shapes that match samples from an array of shapes including some foils. This task is timed. As appropriate, raw scores on cognitive tests were converted to normed scores using the applicable norms sets. Performance validity was measured with the WMT. Poor validity could represent an attempt to perform poorly, inattention, or other response sets that result in noncredible patterns of responses. As recommended by Green (2007), both dichotomous (valid versus invalid) and continuous indices of performance validity were employed. Participants were also administered the Patient Health Questionnaire—9 (PHQ-9), Generalized Anxiety Disorder—7 (GAD-7), and Rivermead Post-Concussion Symptom Questionnaire (RPQ). The PHQ-9 (Kroenke, Spitzer, & Williams, 2001) was derived from the Patient Health Questionnaire (Spitzer, Kroenke, & Williams, 1999), a selfreport survey used to diagnose mental disorder in primary care settings. The PHQ-9 has been validated as a measure of depression in persons with TBI (Cook et al., 2011). The GAD-7 (Spitzer, Kroenke, Williams, & Löwe, 2006) was developed as a complementary measure to the PHQ-9 for assessment in anxiety of persons with TBI. While not studied as extensively as the PHQ-9, the GAD-7 has been shown to be useful in TBI research (Harch et al., 2012). Scores for the PHQ-9 and GAD-7 were total scores for each measure. The RPQ (King, Crawford, Wenden, Moss, & Wade, 1995) is a measure of symptoms reported by persons who have sustained TBI. For this questionnaire, respondents rate the severity of symptoms experienced in the past 24 h as compared to their intensity prior to the TBI. Data analysis Descriptive data were presented as means and standard deviations (SD) for continuous-type variables (age, years of education, time since injury, PTA, length of stay rehabilitation, time to follow commands, cognitive tests, questionnaires) and frequencies and percentages for categorical-type variables (sex, race/ethnicity, marital status, problem drinker, drug use, income, GCS). The amount of missing data was small with less than 2% missing on all candidate predictors listed above except for time to follow commands, which had 18% missing. To include all persons in the regression models discussed below, imputation was used to predict the missing data. The imputations were based on computing regression models for each candidate predictor that predicted the missing value, while adding a random component. This random component avoids biased results. Since the amount of missing data was small, only one random draw of the data was used instead of analyzing multiple imputed data sets. In addition to including all of the variables mentioned below to predict the missing data for each variable, the variables WMT group, race/ethnicity, marital status, PTA, LOS rehab, and GCS group were included to improve the predictions of the missing data.
The Clinical Neuropsychologist 2015.29:522-541.
528
MARK SHERER ET AL.
To reduce the number of cognitive test and questionnaire measures to avoid fitting too many candidate predictor variables for regression analyses, a variable cluster analysis was performed to group the cognitive tests and questionnaires into sets and then a principal component analysis was conducted to generate a composite score for each set. The variable cluster analysis was performed on the cognitive tests (Symbol Search, Coding, Trails A, Trails B, RAVLT Trials 1-5, RAVLT Long-Delay, FAS) and questionnaires (RPQ, GAD-7, PHQ-9) using the VARCLUS procedure in SAS to group the measures (SAS Institute, Inc., 2009). This procedure groups the measures into clusters so that measures within a cluster are more strongly related to each other than to measures in other clusters. Next, a principal component (PC) analysis was computed within each proposed set (cluster) and the first PC score of the set was used as the measure or composite function of the set. Two outcomes derived from the WMT were analyzed using logistic and linear regression models. First, a dichotomous outcome of valid versus invalid (WMT group) was derived by classifying any participant who scored below the recommended cutoff on any of the Immediate Recognition, Delayed Recognition, or Consistency scores as invalid and all others as valid (Green, 2003). Next, as recommended by Green (2007), we also created a continuous outcome indicating the degree of validity by averaging the Immediate Recognition, Delayed Recognition, and Consistency scores (Avg WMT). Higher scores on this index indicated better performance validity. The dichotomous WMT outcome was submitted to logistic regression, while the continuous WMT outcome was submitted to ordinary linear regression. In both cases, unadjusted (simple) and adjusted (multivariable) models were computed to examine the unadjusted and adjusted effects, respectively, of each candidate predictor. The pre-specified candidate predictors included age (at time of assessment), sex, education, income, problem drinker, drug use, time since injury, time to follow commands, and the cluster scores derived from the cluster analysis that represented the 10 cognitive tests and questionnaires. For the multivariable models instead of using stepwise regression models which often produce overly optimistic estimates, full model fits were computed by including all pre-specified candidate predictors for the adjusted analysis. For all continuous-type candidate predictors, we used natural splines to relax the linearity assumption for both the adjusted and unadjusted linear regression models. Splines are mathematical representations of curved shapes. In linear regression models, splines are used to fit more flexible curves to the data than the simplest form of linear regression allows. Without splines, the regression models would only test for linear relationships between independent variables and the outcome. The non-linear term for each independent variable was retained in all models, adjusted and unadjusted, if the non-linear p-value was less than .20 in either the multivariable logistic or linear regression model. Effects for the continuous-type independent variable were shown as interquartile coefficients which indicate the change in the value of the outcome as the value of the independent variable changes from the 25th to the 75th percentile. To measure the overall strength of association of the models, R-square and C index are reported for the linear and logistic models, respectively. The C index is equivalent to the receiver operating characteristic curve. Because these measures tend to be overly optimistic, the validated R-square and C index were computed based on a bootstrap (200 bootstrap samples) resampling plan. For all analyses, the RMS package in R (version 3.1.1) programming language was used to account for non-linear terms and to
TBI AND PERFORMANCE VALIDITY
529
compute validated R-square measures (Harrell, 2009). For persons with poor performance validity, those meeting criteria for the GMIP were compared those who did not meet criteria for the GMIP.
RESULTS
The Clinical Neuropsychologist 2015.29:522-541.
Description of the study sample Five hundred four persons were enrolled in the study and all of these completed at least partial assessments. Thirteen (2.6%) failed to complete the WMT which was the key study measure for these analyses, and all 13 were excluded from study, yielding a study sample of 491 participants. In all cases, participants did not complete the WMT due to terminating the assessment session before this measure was obtained. Descriptive data describing demographic characteristics, pre-injury functioning, injury severity, and performance on study measures are provided in Table 1. Scores for the RPQ, PHQ-9, and GAD-7 are raw scores. Scores for the RAVLT Trials 1-5, RAVLT Long-Delay, Trails A, Trails B, and FAS are T-scores. Scores for the WAIS-IV Symbol Search and Coding subtests are scaled scores. As is common in studies of persons with TBI, most were males of young to middle age. Since most participants received inpatient rehabilitation following their injuries, it is expected that most sustained moderate or severe injuries. As indicated by data in Table 2, if one assumes that persons requiring intubation or medical sedation shortly after injury likely sustained moderate or severe TBI, 343 (70%) participants sustained moderate or severe TBI. Four hundred twenty (86%) participants underwent inpatient rehabilitation for problems related to their TBI. Performance validity for GCS severity categories Table 2 shows the proportions of persons with good performance validity for each GCS category. Persons with mild, moderate, or severe TBI showed essentially the same proportions of good performance validity, while persons who required intubation or medication had higher rates of valid scores and those missing GCS scores had lower rates of valid scores. Data reduction Data reduction analyses performed on study measures (Symbol Search, Trails A, Trails B, Coding, RAVLT Trials 1-5, RAVLT Long-Delay, FAS, RPQ, GAD-7, PHQ-9) identified four clusters. Cluster 1 was defined by the RPQ, GAD-7, and PHQ-9 and represents the degree of emotional distress experienced by participants. Cluster 2 was defined by Symbol Search, Coding, Trails A, and Trails B and represents speed of cognitive processing. Cluster 3 was defined by RAVLT Trials 1-5 and RAVLT Long-Delay and represents verbal memory. Finally, Cluster 4 was defined by FAS and indicates participants’ verbal fluency. The first principal component score for each cluster was used as the summary score for each of the four clusters. These scores explained 85, 70, 90, and 100% of the variability of the four clusters, respectively.
530
MARK SHERER ET AL.
The Clinical Neuropsychologist 2015.29:522-541.
Table 1. Description of the study sample on outcome, demographics, injury severity, and predictors (N = 491) Categorical descriptors
n (%) Missing
n (%)
Male Race/ethnicity White Black Hispanic Other Marital statusa Never married Married/common law Separated/divorced Widowed Drugs (yes) Problem drinker (yes) Incomeb Q1:$9,999 or less Median:$10,000–$19,999 Q3:$30,000–$39,999 GCS groupa Severe (3–8) Moderate (9–12) Mild (13–15) Intubated Medicated WMT good effort
0 (0%) 0 (0%)
369 (75%)
0 (0%)
150 (33%) 47 (10%) 118 (26%) 93 (20%) 53 (11%) 374 (76%)
Continuous descriptors
n (%) Missing
Mean (SD)
Age (years) Education (years) LOS rehab (days) Time since injury (years) Time to follow (days) PTA duration (days) Rivermead post-concussion GAD-7 PHQ-9 Symbol search Trails A seconds Trails B seconds Coding RAVLT Trials 1-5 total RAVLT long-delay free recall FAS WMT average
0 (0%) 0 (0%) 38 (7.7%) 3 (.6%) 88 (17.9%) 154 (31.4%) 1 (.2%) 0 (0%) 2 (.4%) 4 (.8%) 2 (.4%) 8 (1.6%) 4 (.8%) 0 (0%) 2 (.4%) 0 (0%) 0 (%)
38.0 (13.2) 12.7 (2.3) 20.8 (18.3) 6.2 (6.8) 6.7 (9.4) 21.4 (19.7) 23.8 (13.4) 5.4 (5.5) 6.9 (5.9) 7.5 (3.0) 42.3 (13.5) 44.3 (13.0) 7.1 (3.0) 34.9 (14.8) 35.7 (15.9) 42.8 (12.0) 91.5 (11.5)
a
220 (45%) 181 (37%) 72 (15%) 18 (4%) 2 (.4%)
0 (0%) 0 (0%) 0 (0%)
86 (18%) 80 (16%) 42 (9%) 30 (6.1%)
n (%) are computed based on non-missing data. Q1 and Q3 represent the first and third quartiles.
b
269 (55%) 124 (25%) 94 (19%) 12 (2%) 187 (38%) 267 (54%)
TBI AND PERFORMANCE VALIDITY
531
Table 2. WMT invalid/valid effort distributions % (n) for Glasgow Coma Scale severity groups (N = 491) GCS group Severe (3–8) Moderate (9–12) Mild (13–15) Intubated Medicated Missing
N 150 47 118 93 53 30
WMT invalid 38 13 29 19 7 11
(25%) (28%) (25%) (20%) (13%) (37%)
WMT valid 112 (75%) 34 (72%) 89 (75%) 74 (80%) 46 (87%) 19 (63%)
The Clinical Neuropsychologist 2015.29:522-541.
Regression analyses As shown in Table 1, for the dichotomous WMT outcome, 374 (76%) of participants showed valid scores and 117 (24%) showed invalid scores. For the continuous index of performance validity the mean (SD) value was 91.5 (11.5). In addition to the summary scores for each cluster described above, independent variables included in regression models were gender, age in years at time of examination, years of education, income category (0–9), problem alcohol use (yes/no), drug use (yes/no), time to follow commands in days, and time from injury to assessment in years. To incorporate nonlinear trends, four additional degrees of freedom were included for age and the first three clusters given that the non-linear p-values were less than .20 for either model (all p < .20). For both multivariable models, there are 16 total degrees of freedom including 12 independent variables with four non-linear terms. Note that because there were only 117 persons in the least frequent outcome category (limiting sample size) for the full model logistic regression, this model was at risk for over-fitting based on a standard 10 persons per one degree of freedom rule. However, the full model linear regression is not likely to be over-fitted given up to 49 degrees of freedom could be fitted based on a standard rule requiring 10 persons per every degree of freedom fitted. For this reason, the full model linear regression should be considered the primary analysis. Results of the logistic regression models are presented in Table 3. Unadjusted (univariable) effects are provided for all factors, but we focus on the adjusted effects. Age at time of injury, time to follow commands, Cluster 1 (emotional distress), and Cluster 3 (memory) were associated with of performance validity. Those with younger age at assessment, longer time to follow commands (more severe injury), less emotional distress, and more intact memory were more likely to show valid scores. The greatest effect was for memory. Those scoring at the 75th percentile on memory (better memory) were 7.3 times as likely to have valid scores as those scoring at the 25th percentile on memory (poorer memory). Stated another way, those obtaining scores indicating poorer memory had only .14 times the odds of showing valid scores as those obtaining scores indicating better memory. Those reporting less emotional distress at the 25th percentile were about twice as likely to have valid scores as those reporting greater emotional distress at the 75th percentile. Those with younger age (26 years) were about twice as likely to have valid scores as those with older age (50 years). Those with greater injury severity (9.5 days to follow commands) were about 30% more likely to have valid scores than those with milder injury severity (.5 days to follow commands). The C index was .84, and the validated index was .81 using bootstrap resampling.
532
MARK SHERER ET AL. Table 3. Unadjusted and adjusted results of logistic regression analysis predicting probability of poor performance validity (N = 491)
The Clinical Neuropsychologist 2015.29:522-541.
Unadjusted (simple) model
Multivariable model
Predictor
Comparison
Effect (OR)
95% CI
p
Effect (OR)
95% CI
p
Gender Age Education Income category Problem drinker Drug use Time to follow Time since injury Cluster 1 Cluster 2 Cluster 3 Cluster 4
M, F 26, 50 years 11, 14 years 1, 4 No, yes No, yes .5, 9.5 days 1.0, 9.9 years −.8, .6 −.6, .7 −.8, .8 −.6, .7
1.21 .52 1.76 1.10 .71 .85 1.05 .78 .48 2.39 5.82 1.31
.74, 1.98 .33, .81 1.31, 2.36 .86, 1.39 .47, 1.09 .56, 1.30 .86, 1.28 .60, 1.00 .31, .74 1.62, 3.52 3.67, 9.24 1.00, 1.73
.45
