T O W A R D S P A T I E N T C O L L A B O R A T I O N IN C O G N I T I V E A S S E S S M E N T : S P E C I F I C I T Y , SENSITIVITY, A N D I N C R E M E N T A L V A L I D I T Y OF S E L F - R E P O R T L2
Carolyn E. Schwartz, Sc.D. Frontier Science & Technology Research Foundation, Inc., Department of Psychiatry Deaconess Hospital, Harvard Medical School
Elizabeth Kozora, Ph.D. National Jewish Center for Immunology & Respiratory Medicine, University of Colorado School of Medicine
Qi Zeng, M.S. Harvard School of Public Health
ABSTRACT
of congruence between patient-reported and objective assessment of symptoms has become more salient. If patient-reported assessments are reliable and valid, then clinicians and researchers could rely on self-reports for primary outcome measures as well as for initial screening tools for preventive intervention. One domain of particular relevance for a broad range of chronic diseases is neuropsychological functioning. Neuropsychological assessment investigates cognitive abilities using standardized tests given by a trained health professional and can vary from very brief (i.e. five minutes) screening measures to comprehensive evaluations (i.e. eight hours). Neuropsychological testing is sometimes expensive and complicated and may not correspond to how patients perceive themselves to be getting along on a day-to-day basis (1, p. 114). Patient self:reports are reputed to suffer from limited reliability and validity (2), while costly assessments collected by experts are deemed of high value (3). If patients' assessments of neuropsychological symptoms are specific and sensitive, then querying patients represents a potentially valuable collaboration between patients and clinicians, which could result in less expensive screening procedures for neuropsychological impairment. The purpose of the present work is to evaluate the specificity and sensitivity of patientreported neuropsychological performance and to evaluate the incremental validity of patient self-report beyond what is described through neuropsychological testing. The congruence between subjective and objective indices is relevant to a broad range of outcomes research. In the domain of health-related quality-of-life, research on the relationship between clinical, laboratory, and self-reported measures suggests that they reflect rather different aspects of physical well-being and as such are not interchangeable (4). Bijlsma et al. (4) found that rheumatoid arthritis patients' reports of physical function accounted for only 7-16% of the variance in clinical measures of functional status (e.g. grip strength, walking time) and that laboratory measures were not associated with any of the selfreported indices. The concordance between clinicians' and patients' assessments of functioning may be higher for physical functioning than for mental functioning and seems to be dependent on the patient's disability level (5). Whereas some investigators conclude that formal assessments of outcome are needed (5), others cite high concordance rates (72% to 86%)
The present work addressed the specificity and sensitivity of patient-reported cognitive ability using both cross-sectional and longitudinal data, and the incremental validity of patient self-report in addition to knowledge gained through neuropsychological tests. We examined a sample of individuals with multiple sclerosis (N = 130) as a model of chronic illness where neuropsychological deficits are relatively common. Results revealed that 64% of the sample reported noticing some problems with memory or confusion. Very high levels o f reported problems were not consistent with objective testing, whereas moderate levels of noticing problems were congruent with test resuits. This pattern suggests a curvilinear relationship between self-reported and objective assessment. The moderate reporters seem to be attending to subtle increases in deficits over time. Results also supported the incremental validity of combining subjective and objective indices, but only when the high reporters were excluded. We conclude that patients can provide important complementary data which may promote preventive care. (Ann Behav Med 1996, 18(3):177-184)
INTRODUCTION As changes in the health care system force an emphasis on inexpensive means of evaluation and treatment, the question Preparation of this manuscript was supported in part by grants from the National Multiple Sclerosis Society (FG 880-A1 and RG 2577-A-2) and the Fetzer Institute (#563) to Dr. Schwartz. 2 We would like to thank for their helpful input Drs. Thomas Kay, Elsie Gulick, Andrea Kronman, Stephen Rao, Frederick Foley, Arthur Barsky, and Patricia Rieker; Pam Sorensen, Harry Costin, and anonymous reviewers. We would also like to thank all of the patients who participated in this study. Reprint Address: C. E. Schwartz, Sc.D., Frontier Science, 303 Boylston Street, Brookline, MA 02146. 9 1996 by The Society of Behavioral Medicine. 177
178
ANNALS OF BEHAVIORAL MEDICINE
and conclude that the patient's perspective can provide useful information to guide care (6). The neuropsychological domain presents a particular methodological challenge because the very process of awareness may be impaired by the disease process. For example, lesions in the frontal lobe can result in impaired executive function and insight (7). Lack of awareness or "anosognosia" (8) is a pervasive phenomenon, having been described in a variety of neurological, psychiatric, and toxic disorders (9) and resulting from multiple different brain deficits. Research on deficits in neuropsychologicai awareness (3,10) has revealed that the direction of error may not be consistent, with some brain-damaged patients showing unawareness in the face of profound deficits (10) and others rating themselves lower than normals regardless of the severity of the injury (11). Thus, patients with severe cognitive deficits may report no problems (i.e. unawareness) or many cognitive problems (i.e. an accurate assessment of difficulties). In the present work, we hypothesize that the relationship between self-reported and objective assessment is curvilinear, such that patients reporting very many problems perform better on objective testing than those reporting moderate numbers of problems and similarly to those reporting no problem. Although many investigators conclude that patient self-report of neuropsychological functioning is unreliable (2) and invalid (12) and should not be used for clinical purposes (3), others suggest that patients are reporting meaningful difficulties in areas that are not easily measured by neuropsychological measures (1,13). In fact, Goldstein and McCue (14) found that the discrepancy in functional ratings among patients, informants, and test results diminished when the tests being considered more closely resembled complex everyday activities. Thus, unawareness may be a multidimensional construct (3), and measures of perceived cognitive problems can serve an important function in increasing our understanding of the life activities that may be disrupted by cognitive deficits (13). Mediators of Accuracy Three factors may influence the accuracy of self-report and consequently should be considered in a meaningful examination of congruence of objective and subjective assessment. The first factor is depression, a salient problem for chronically ill patients whose depression may be a reaction to the losses caused by the disease (15), structural brain damage (16), genetic factors (17), or to combinations of the above. When people are depressed, they may be less likely to think clearly (18) and may complain of more cognitive problems (19), although their memory does not seem to be affected (19,20). Furthermore, patients may r,eport problems with recall, but the deficits may actually be in attention or learning. Depressed people may be likely to rate their functional status worse (9,12) and may be more realistic in their self-perceptions of competence than non-depressed people (21). Thus, subjective-objective congruence may be attenuated by depression. A second factor which may influence the measurable agreement between subjective and objective assessment is related to the fact that cognitive impairment may have an insidious onset. Among patient populations with high premorbid functioning, patients may recognize subtle deficits in their cognitive performance which may not be labeled as deficient based on neuropsychological tests (22). Individuals with high premorbid intellectual functioning may notice early problems that
S c h w a r t z et al. are undetectable by neuropsychological screening tests because they are likely to live within highly demanding contexts (23), to be employed in areas which emphasize intellectual function, and to socialize with other highly functioning individuals (24). Despite high levels of premorbid functioning, we hypothesize that patients who report noticing cognitive problems are responding to real rather than imagined problems. Consequently, we expect that those who report noticing cognitive problems will be more likely to perform worse on objective neuropsychological tests than patients who report not noticing such problems and will exhibit a progressive decline in performance over one year of follow-up. Finally, sociodemographic factors can mediate the relationship between self- and objective assessment. Among cancer patients, age moderated the relationship between self-report and objective neuropsychological performance, such that older patients (i.e. over 36 years) but not younger patients showed greater congruence (25). Among human immunodeficiency virus (HIV)-positive patients, education did not moderate the relationship between self-report and objective neuropsychological functioning, but more educated subjects were more accurate self-observers in regard to motor performance (19). These cofactors are usually controlled by defining impairment using normative criteria which correct for relevant sociodemographic factors, such as age, education, and gender (26).
Hypotheses The present work addressed the specificity and sensitivity of patient-reported neuropsychological performance using both cross-sectional and longitudinal analyses, and the incremental validity of patient self-report beyond information gained through neuropsychological testing to predict psychosocial role limitations. We examined a sample of individuals with multiple sclerosis (MS), a chronic illness where neuropsychological deficits are relatively common (i.e. 43% prevalence rate) (27), are associated with clinical indicators of disease progression (28), and have a salient impact on role performance and employment status (29). We hypothesized that: (a) the relationship between self-reported and objective neuropsychological performance is not linear, and that patients who report a large number of problems are less consistent with objective testing than those who report none or moderate numbers of problems (i.e. specificity); (b) once inconsistent (i.e. overly-vigilant) reporters are excluded from the analysis, the relationship between self-reported and objective neuropsychological performance will appear to be linear [in other words, noticing problems will be associated with worse performance on neuropsychological tests at baseline and over time (i.e. sensitivity)]; and (c) combining patient self-report and objective performance measures will yield increased accuracy in predicting psychosocial role limitations than either assessment method alone (i.e. incremental validity). We sought to evaluate the incremental validity of self-report and objective assessment on psychosocial, rather than physical, role limitations because of the documented effect of cognitive problems on psychosocial functioning (29) and the documented lack of association with physical disability (27). Whereas the first hypothesis suggests a curvilinear relationship between subjective and objective assessment, the second suggests a linear relationship for a restricted range of the patient population. Using different cohorts (i.e. full versus restricted subsamples) is necessary to elucidate the specificity and sensitivity of patient
Cognitive Self-Assessment
V O L U M E 18, N U M B E R 3, 1996
TABLE 1 Demographic Data Variables
Mean
SD
Range
Age Expanded Disability Status Score (EDSS) Duration
43.2
8.93
21-64
4.74 8.26
1.74 6.71
1-8.5 1-37
Categories Sex Marital Status
Course of Disease Employment
Occupation
Education
Income
Female Male Married Single Divorced/Widowed Relapsing/Remitting Chronic Progressive Part/Full Time Disabled Other Professional/Managerial Clerical Skilled Labor Other High School Graduate or Less 1-4 years of College 1 or more years of Professional or Graduate School Less than $30,000 $30,000 to 59,000 $60,000 and Higher
Percentage 72.6% 27.4% 62.7% 20.1% 17.2% 41.5% 58.5% 43.3% 33.7% 22.3% 68.2% 18.1% 6.5% 7.2% 18.8% 51.6% 29.6% 34.5% 35.7% 29.8%
self-report and to model how reported difficulties and objective assessments can complement each other. METHOD
Subjects Subjects were 130 people with a neurologist-confirmed diagnosis of MS who were recruited from the MS Clinic registry of a large Boston teaching hospital (41%), newspaper advertisements (25%), MS Society mailings (18%), word of mouth (12%), and other hospital-based physician referrals (5%). These participants were interested in participating in a randomized controlled trial of two psychosocial interventions (30) and were excluded if they demonstrated difficulty understanding the nature of the study or were actively psychotic, as determined in a clinical interview by one of us (CES). Study participants had a mean age of 43 years (SD = 9.0); tended to be Caucasian, married, middle-aged people with some college education; and approximately one-third had an income of less than $30,000 (see Table 1). Nearly half of the study participants were employed, most reported having had a professional or managerial occupation, and 34% were disabled from work. Fifty-eight percent of the participants had progressive disease, and 42% had relapsing-remitting disease. The average disease duration was eight years (range 1-37), and the mean level of neurologic disability was an Expanded Disability Status Score (EDSS) of 4.71 (range = 1 to 8.5), which means that most participants had some ambulation disability. The gender ratio approximated the general MS population (31), as 73% of the participants were
179
female. Study participants were on a variety of medications, including drugs for fatigue (19%), spasticity (27%), anxiety (15%) or depression (22%) (27% were on either anti-anxiety or anti-depressant medications), and incontinence (15%). None of the participants were on anti-psychotic drugs, and 2% of the patients were on mood-stabilizers. The sample participants were largely free (i.e. less than 2% of sample) of serious comorbidities such as cancer or heart disease, and a minority reported comorbidities including emotional problems (24%), other (23%), arthritis (7%), high blood pressure (6%), ulcer (5%), and anemia (5%).
Procedure Participants were interviewed to determine their eligibility for the study and subsequently entered into the study. After the interview, a brief neuropsychological battery of standardized tests was given to assess verbal and spatial memory, information processing, cognitive flexibility, and complex attention (see below). At the close of this intake procedure, interested participants were given a questionnaire packet to complete at home and to return by mail. One year later, participants were interviewed and given the neuropsychological battery and were asked to complete the questionnaire by mail. One hundred and seventy-two people were interviewed at baseline, of which 136 (79%) were randomized to one of two social support interventions, and 130 (76%) completed the full one-year of follow-up [see Schwartz and Fox (32) for a careful examination of the selection bias in the present sample].
Measures Self-reported cognitive complaints were assessed using responses to the semi-structured intake interview. The interview question used in the present analyses asked whether the person had "noticed any problems with memory and confusion," and responses were coded according to the type(s) of problems mentioned (e.g. short-term memory, information processing, word recall, etc.). A four-level score of noted problems categorized responses into zero, one, two, or three and greater cognitive problems mentioned. Self-reported affective complaints were also assessed using responses to the semi-structured intake interview, creating an indicator variable score of " 1 " if the patient mentioned mood swings or depression. Neurologic disability was assessed using the Expanded Disability Status Score (33), an observer-reported scale which ranges from zero to ten, where a score of six refers to requiring a cane to ambulate, and a score of ten refers to death due to MS. This scale is the most standard measure of neurologic disability used in the MS population. The Sickness Impact Profile (SIP) (34) was used to assess role performance limitations for the incremental validity analysis. This self-report measure of health status contains 136 items in twelve categories and includes a psychosocial summary score which is a composite of the following subscal~s: social interaction, alertness behavior, emotional behavior, and communication. The SIP has documented test-retest and alpha reliability (35) and convergent and discriminant validity (36). Objective neuropsychological performance was assessed using the Rao et al. cognitive battery (27-29), the Wisconsin Card Sorting Task (37), and the Trail Making Test (38). The Rao battery includes standardized cognitive tests to assess verbal and spatial memory, complex attention, and verbal fluency,
180
ANNALS OF BEHAVIORAL
MEDICINE
including the Selective Reminding Test (SRT) (39), the Controlled Oral Word Association Task (40-42), the Paced Auditory Serial Addition Test (PASAT) (43,44), the Symbol-Digit Task (45), and the 10/36 Spatial Recall Test (46). These tests were chosen from a larger battery because of their documented sensitivity of 0.71 and specificity of 0.94 in discriminating cognitively intact from impaired MS patients. The Wisconsin Card Sorting Task is a measure of cognitive flexibility with documented sensitivity to focal frontal lobe problems (i.e. executive function) (47,48). A recent study has also shown that the Wisconsin Card Sorting Task is sensitive to white matter lesions among people with MS (49). Executive function refers to higher cognitive abilities involving the conceptualization and initiation of activity (50). The Trail Making Test is a measure of complex attention and sequencing with documented test-retest reliability and known sensitivity to the effects of brain injury (38,51,52). All neuropsychological test scores included in this study were adjusted for age, educational level, and gender using normative standards. Normative data were also available to adjust for the practice effect of readministering the tests after one year. Normative scores used published standards and data from healthy controls provided by Dr. Rao and colleagues. The neuropsychological composite, the dependent variable, was the mean of converted T-scores using normative standards as described below. Raw scores were adjusted for age, gender, and education level; were converted to Z scores using normative data; and were converted to T-scores by the transformation T = 50 + 10Z.
Statistical Analysis Specificity is defined to be the probability of accurately not endorsing a problem when one does not have it (53). We began by performing a threshold analysis to determine what "failure" threshold maximized the pattern of congruence between subjective and objective assessment. We examined mean number of tests failed using the following failure thresholds on neuropsychological tests: T-scores of 60, 50, 40, 30, and 20. We then selected the threshold that maximized the apparent consistency of self-report and objective indices and created the following three groupings of patients: Group One included those who noticed no problems; Group Two included those who noticed one or two problems; and Group Three included those patients who noticed three or more problems. These groupings were used in subsequent analyses which addressed sensitivity and incremental validity. Thus, the clinical significance of the number of tests failed at a given threshold will not be addressed psychometrically, but rather the incremental validity analyses will address the issue substantively. Sensitivity is defined to be the probability of accurately identifying a problem when one has it (53). Sensitivity was evaluated by separating the patients into the above groups and computing multinomial logistic regression models to examine the ability of the objective assessment (independent variable) to p~'edict patient self-report (dependent variable). We tested for the potential confounding effect of reported affective complaints on reporting other cognitive problems. Analysis of variance revealed a significant difference among the groups on reported affective complaints (F = 6.97, p < 0.001). Consequently, the logistic regression models included reported affective complaints as a covariate, so that the estimated relationship of subjective and objective assessment would be statistically independent of reported affective complaints.
S c h w a r t z et al. TABLE 2 Descriptive Statistics of Neuropsychological Tests Mean T-Score (Standard Deviation) Test Selective Reminding Test Long-Term Storage Continuous Long-Term Retrieval Controlled Oral Word Association Number correct PASAT Symbol Digit Wisconsin Card Sort Number Categories Achieved Trail-Making Test Part A Part B Spatial (10/36) Neuropsychological Composite (mean T-score)
Baseline
One-Year
38.6 (7.1) 39.7 (6.9)
34.0 (4.9) 38.9 (7.2)
36.1 (5.1) 44.9 (12.1) 31.5 (14.2)
34.4 (4.7) 43.7 (13.4) 31.9 (15.1)
44.5 (15.5)
41.8 (8.1)
35.2 (13.6) 38.8 (14.6) 39.3 (6.9)
36.2 (13.4) 39.1 (12.3) 33.8 (4.9)
41.1 (5.8)
38.1 (5.7)
Two summaries of objective assessment were used for the above logistic analyses: (a) the baseline neuropsychological composite, and (b) a neuropsychological slope score which summarizes change over time. The latter score was derived using growth curve analysis (54-57), which computes individual slopes to assess change over time on the dependent variable of interest. This technique is superior to simply using difference scores because it considers both initial status and variability, whereas difference scores consider only increment from initial status (54). In interpreting the slope scores, we corrected the slope using the mean slope, so that ratios of greater than one reflected improved performance over time, and ratios of less than one reflected declining performance over time. It was unnecessary to adjust for age, education, or gender, as normative criteria were used to derive T-scores. Incremental validity was evaluated using general linear modeling, where the dependent variable was psychosocial role limitations, a documented social cost of cognitive deficits (29). The independent variables were patient self-reported cognitive complaints and either the baseline neuropsychological composite or the neuropsychological slope score. The covariates were patient-reported affective complaints and neurologic disability [i.e. the EDSS (33)]. The independent contribution of each predictor was evaluated separately and together to determine changes in explained variance due to patient report, objective assessment, and both (i.e. incremental validity). Diagnostic statistics were examined for all reported models to test for heteroscedasticity and multicollinearity. Data were analyzed using SAS PC (58) and S Plus (59) software packages. RESULTS Neuropsychologieal Performance As shown in Table 2, the mean performance on all of the individual neuropsychological tests was below normal with an appreciable proportion of the sample performing in the impaired range. Frequency distributions revealed that no one in the sample performed below the 10th percentile or above the 70th percentile on any of the neuropsychological tests. As these patients had participated in a randomized trial of two psycho-
C o g n i t i v e Self-Assessment
V O L U M E 18, N U M B E R 3, 1996
Specificity of Self-Assessment Neuropsychological Performance (Number of tests failed, i.e.,T-score 0.85). Thus, there was no differential effect of the interventions on neuropsychological function over time. Specificity of Self-Assessment Sixty-four percent of the sample reported noticing some problems with memory or confusion of whom 25% reported noticing one cognitive problem, 24% reported noticing two problems, and 15% reported noticing three or more problems. The threshold analysis examining the congruence of patient self-report with increasingly conservative impairment criteria revealed that a T-score impairment criterion of 40 or below distinguished these groups of patients. This analysis revealed a pattern: there was a linear trend indicating increasingly poor performance as the number of patient-reported cognitive complaints increased from none to two, but patients who reported three or more cognitive complaints performed better than those who complained of moderate (i.e. one or two) numbers of complaints and similarly to those who reported no cognitive complaints. This pattern supports our first hypothesis that the relationship between selfreported and objective neuropsychological performance is curvilinear and that patients who report a large number of problems are less consistent with objective testing than those who report none or moderate numbers of problems (Figure 1). Subsequent analyses consequently separated the patients into three cohorts: Group One included patients who noticed no problems; Group Two included those who noticed one or two problems; and Group Three included those patients who noticed three or more problems. This last group will henceforth be referred to as the "overly vigilant," because those patients reported more cognitive complaints but their objective performance was actually better than patients reporting fewer complaints. Sensitivity of Self-Assessment Multinomial logistic regression analyses were done to evaluate the differential sensitivity of self-report after adjusting
181
for affective complaints. We found that slope estimates of objective assessment were better predictors of patient self-report than were baseline estimates (c statistic = 0.62 and 0.51, respectively, where a value of 0.50 would be expected by chance). These findings partially support the hypothesis that noticing problems will be associated with worse performance on neuropsychological tests at baseline and over time (i.e. sensitivity). In fact, people who reported noticing cognitive difficulties performed worse than those who did not on neuropsychological tests over time but not at baseline (i.e. sensitivity). Patients seemed to be attending to relatively subtle deficits (i.e. decline over time) which may not be considered impaired according to baseline performance. Incremental Validity To understand the incremental validity of combining subjective and objective assessment, we built hierarchical models, all of which contained the covariates of EDSS and reported affective complaints. These hierarchical models tested the predictive significance of: (a) the baseline neuropsychological composite; (b) the slope neuropsychological composite; (c) reported cognitive complaints; and (d) the full baseline model [i.e. the combined predictive significance of objective (baseline or slope) performance and subjective assessment]. The parameter estimates for the full models are listed in the first five rows of Table 3 to give the reader a sense of the magnitude of the association between subjective and objective assessment in predicting psychosocial role limitations. The next four rows of Table 3 provide the model R E statistics for each hierarchical model as well as for the full model. Psychosocial role limitations were predicted at two points in time: at baseline and at one year. The incremental validity of these predictors were tested among two cohorts of patients: the whole sample and a restricted sample where the overly-vigilant patients were excluded. We found that patient-reported complaints and baseline objective assessment were complementary and statistically significant predictors of psychosocial role limitations at baseline and one year among the whole sample, but that slope objective assessment was not a complementary and significant predictor among the whole sample (Table 3). Among the subsample which excluded the overly-vigilant patients, we found that patient-reported complaints and objective assessment (both at baseline and over time) were complementary and statistically significant predictors of psychosocial role limitations at baseline and one year. Thus, patient-reported complaints and objective assessment are incrementally valid indicators of psychosocial role limitations, and this complementarity is pronounced among the subsample of patients whose self-reports are linearly associated with objective assessment. Further, the models using the slope estimate of performance change accounted for more variance in psychosocial role limitations than the models which included baseline performance, suggesting that the additional information provided by patients may be due to subtle deficits accrued over time. DISCUSSION Our results suggest that MS patients who report cognitive complaints demonstrate greater progression of such deficits over time, and a combination of objective and subjective measures of cognitive deficit accounted for greater variance in role limitations than either measure alone. We found that the people whose self-assessment was most consistent with objective test-
182
ANNALS OF BEHAVIORAL
MEDICINE
S c h w a r t z et al.
TABLE 3 Incremental Validity of Objective and Subjective Assessment for Psychosocial Role Limitations Outcome
Psychosocial Role Limitations Whole Sample
Predictor
Outcome@ Baseline
EDSS Reported Affective Problems Baseline Neuropsychological Composite Slope of Neuropsychological Composite Reported Cognitive Problems Model R2 Baseline Composite Alone Slope Composite Alone Reported Cognitive Problems Full Model tp *p **p ***p ****p
Outcome@ 1 year
Overly-Vigilant Patients Excluded Outcome@ 1 year
0.11 0.18*
0.11 0. t 1
0.22* 0.11
-0.15+ -0.35***
-0.35*** -0.32***
--0.09 0.35***
0.16 -0.18 0.25****
0.19 -0.20 0.28****
0.12 0.17 -0.18"***
Outcome@ Baseline 0.13 0.09 -0.18+ -0.37*** 0.17 -0.20 0.21"***
Outcome@ 1 year 0.11 0.02 -0.43*** -0.32*** 0.32 -0.30 0.37****
Outcome@ 1 year 0.11 -0.003 --0.41"** 0.33*** 0.31 0.32 -0.35****
Carolyn E. Schwartz, Sc.D. Frontier Science & Technology Research Foundation, Inc., Department of Psychiatry Deaconess Hospital, Harvard Medical School
Elizabeth Kozora, Ph.D. National Jewish Center for Immunology & Respiratory Medicine, University of Colorado School of Medicine
Qi Zeng, M.S. Harvard School of Public Health
ABSTRACT
of congruence between patient-reported and objective assessment of symptoms has become more salient. If patient-reported assessments are reliable and valid, then clinicians and researchers could rely on self-reports for primary outcome measures as well as for initial screening tools for preventive intervention. One domain of particular relevance for a broad range of chronic diseases is neuropsychological functioning. Neuropsychological assessment investigates cognitive abilities using standardized tests given by a trained health professional and can vary from very brief (i.e. five minutes) screening measures to comprehensive evaluations (i.e. eight hours). Neuropsychological testing is sometimes expensive and complicated and may not correspond to how patients perceive themselves to be getting along on a day-to-day basis (1, p. 114). Patient self:reports are reputed to suffer from limited reliability and validity (2), while costly assessments collected by experts are deemed of high value (3). If patients' assessments of neuropsychological symptoms are specific and sensitive, then querying patients represents a potentially valuable collaboration between patients and clinicians, which could result in less expensive screening procedures for neuropsychological impairment. The purpose of the present work is to evaluate the specificity and sensitivity of patientreported neuropsychological performance and to evaluate the incremental validity of patient self-report beyond what is described through neuropsychological testing. The congruence between subjective and objective indices is relevant to a broad range of outcomes research. In the domain of health-related quality-of-life, research on the relationship between clinical, laboratory, and self-reported measures suggests that they reflect rather different aspects of physical well-being and as such are not interchangeable (4). Bijlsma et al. (4) found that rheumatoid arthritis patients' reports of physical function accounted for only 7-16% of the variance in clinical measures of functional status (e.g. grip strength, walking time) and that laboratory measures were not associated with any of the selfreported indices. The concordance between clinicians' and patients' assessments of functioning may be higher for physical functioning than for mental functioning and seems to be dependent on the patient's disability level (5). Whereas some investigators conclude that formal assessments of outcome are needed (5), others cite high concordance rates (72% to 86%)
The present work addressed the specificity and sensitivity of patient-reported cognitive ability using both cross-sectional and longitudinal data, and the incremental validity of patient self-report in addition to knowledge gained through neuropsychological tests. We examined a sample of individuals with multiple sclerosis (N = 130) as a model of chronic illness where neuropsychological deficits are relatively common. Results revealed that 64% of the sample reported noticing some problems with memory or confusion. Very high levels o f reported problems were not consistent with objective testing, whereas moderate levels of noticing problems were congruent with test resuits. This pattern suggests a curvilinear relationship between self-reported and objective assessment. The moderate reporters seem to be attending to subtle increases in deficits over time. Results also supported the incremental validity of combining subjective and objective indices, but only when the high reporters were excluded. We conclude that patients can provide important complementary data which may promote preventive care. (Ann Behav Med 1996, 18(3):177-184)
INTRODUCTION As changes in the health care system force an emphasis on inexpensive means of evaluation and treatment, the question Preparation of this manuscript was supported in part by grants from the National Multiple Sclerosis Society (FG 880-A1 and RG 2577-A-2) and the Fetzer Institute (#563) to Dr. Schwartz. 2 We would like to thank for their helpful input Drs. Thomas Kay, Elsie Gulick, Andrea Kronman, Stephen Rao, Frederick Foley, Arthur Barsky, and Patricia Rieker; Pam Sorensen, Harry Costin, and anonymous reviewers. We would also like to thank all of the patients who participated in this study. Reprint Address: C. E. Schwartz, Sc.D., Frontier Science, 303 Boylston Street, Brookline, MA 02146. 9 1996 by The Society of Behavioral Medicine. 177
178
ANNALS OF BEHAVIORAL MEDICINE
and conclude that the patient's perspective can provide useful information to guide care (6). The neuropsychological domain presents a particular methodological challenge because the very process of awareness may be impaired by the disease process. For example, lesions in the frontal lobe can result in impaired executive function and insight (7). Lack of awareness or "anosognosia" (8) is a pervasive phenomenon, having been described in a variety of neurological, psychiatric, and toxic disorders (9) and resulting from multiple different brain deficits. Research on deficits in neuropsychologicai awareness (3,10) has revealed that the direction of error may not be consistent, with some brain-damaged patients showing unawareness in the face of profound deficits (10) and others rating themselves lower than normals regardless of the severity of the injury (11). Thus, patients with severe cognitive deficits may report no problems (i.e. unawareness) or many cognitive problems (i.e. an accurate assessment of difficulties). In the present work, we hypothesize that the relationship between self-reported and objective assessment is curvilinear, such that patients reporting very many problems perform better on objective testing than those reporting moderate numbers of problems and similarly to those reporting no problem. Although many investigators conclude that patient self-report of neuropsychological functioning is unreliable (2) and invalid (12) and should not be used for clinical purposes (3), others suggest that patients are reporting meaningful difficulties in areas that are not easily measured by neuropsychological measures (1,13). In fact, Goldstein and McCue (14) found that the discrepancy in functional ratings among patients, informants, and test results diminished when the tests being considered more closely resembled complex everyday activities. Thus, unawareness may be a multidimensional construct (3), and measures of perceived cognitive problems can serve an important function in increasing our understanding of the life activities that may be disrupted by cognitive deficits (13). Mediators of Accuracy Three factors may influence the accuracy of self-report and consequently should be considered in a meaningful examination of congruence of objective and subjective assessment. The first factor is depression, a salient problem for chronically ill patients whose depression may be a reaction to the losses caused by the disease (15), structural brain damage (16), genetic factors (17), or to combinations of the above. When people are depressed, they may be less likely to think clearly (18) and may complain of more cognitive problems (19), although their memory does not seem to be affected (19,20). Furthermore, patients may r,eport problems with recall, but the deficits may actually be in attention or learning. Depressed people may be likely to rate their functional status worse (9,12) and may be more realistic in their self-perceptions of competence than non-depressed people (21). Thus, subjective-objective congruence may be attenuated by depression. A second factor which may influence the measurable agreement between subjective and objective assessment is related to the fact that cognitive impairment may have an insidious onset. Among patient populations with high premorbid functioning, patients may recognize subtle deficits in their cognitive performance which may not be labeled as deficient based on neuropsychological tests (22). Individuals with high premorbid intellectual functioning may notice early problems that
S c h w a r t z et al. are undetectable by neuropsychological screening tests because they are likely to live within highly demanding contexts (23), to be employed in areas which emphasize intellectual function, and to socialize with other highly functioning individuals (24). Despite high levels of premorbid functioning, we hypothesize that patients who report noticing cognitive problems are responding to real rather than imagined problems. Consequently, we expect that those who report noticing cognitive problems will be more likely to perform worse on objective neuropsychological tests than patients who report not noticing such problems and will exhibit a progressive decline in performance over one year of follow-up. Finally, sociodemographic factors can mediate the relationship between self- and objective assessment. Among cancer patients, age moderated the relationship between self-report and objective neuropsychological performance, such that older patients (i.e. over 36 years) but not younger patients showed greater congruence (25). Among human immunodeficiency virus (HIV)-positive patients, education did not moderate the relationship between self-report and objective neuropsychological functioning, but more educated subjects were more accurate self-observers in regard to motor performance (19). These cofactors are usually controlled by defining impairment using normative criteria which correct for relevant sociodemographic factors, such as age, education, and gender (26).
Hypotheses The present work addressed the specificity and sensitivity of patient-reported neuropsychological performance using both cross-sectional and longitudinal analyses, and the incremental validity of patient self-report beyond information gained through neuropsychological testing to predict psychosocial role limitations. We examined a sample of individuals with multiple sclerosis (MS), a chronic illness where neuropsychological deficits are relatively common (i.e. 43% prevalence rate) (27), are associated with clinical indicators of disease progression (28), and have a salient impact on role performance and employment status (29). We hypothesized that: (a) the relationship between self-reported and objective neuropsychological performance is not linear, and that patients who report a large number of problems are less consistent with objective testing than those who report none or moderate numbers of problems (i.e. specificity); (b) once inconsistent (i.e. overly-vigilant) reporters are excluded from the analysis, the relationship between self-reported and objective neuropsychological performance will appear to be linear [in other words, noticing problems will be associated with worse performance on neuropsychological tests at baseline and over time (i.e. sensitivity)]; and (c) combining patient self-report and objective performance measures will yield increased accuracy in predicting psychosocial role limitations than either assessment method alone (i.e. incremental validity). We sought to evaluate the incremental validity of self-report and objective assessment on psychosocial, rather than physical, role limitations because of the documented effect of cognitive problems on psychosocial functioning (29) and the documented lack of association with physical disability (27). Whereas the first hypothesis suggests a curvilinear relationship between subjective and objective assessment, the second suggests a linear relationship for a restricted range of the patient population. Using different cohorts (i.e. full versus restricted subsamples) is necessary to elucidate the specificity and sensitivity of patient
Cognitive Self-Assessment
V O L U M E 18, N U M B E R 3, 1996
TABLE 1 Demographic Data Variables
Mean
SD
Range
Age Expanded Disability Status Score (EDSS) Duration
43.2
8.93
21-64
4.74 8.26
1.74 6.71
1-8.5 1-37
Categories Sex Marital Status
Course of Disease Employment
Occupation
Education
Income
Female Male Married Single Divorced/Widowed Relapsing/Remitting Chronic Progressive Part/Full Time Disabled Other Professional/Managerial Clerical Skilled Labor Other High School Graduate or Less 1-4 years of College 1 or more years of Professional or Graduate School Less than $30,000 $30,000 to 59,000 $60,000 and Higher
Percentage 72.6% 27.4% 62.7% 20.1% 17.2% 41.5% 58.5% 43.3% 33.7% 22.3% 68.2% 18.1% 6.5% 7.2% 18.8% 51.6% 29.6% 34.5% 35.7% 29.8%
self-report and to model how reported difficulties and objective assessments can complement each other. METHOD
Subjects Subjects were 130 people with a neurologist-confirmed diagnosis of MS who were recruited from the MS Clinic registry of a large Boston teaching hospital (41%), newspaper advertisements (25%), MS Society mailings (18%), word of mouth (12%), and other hospital-based physician referrals (5%). These participants were interested in participating in a randomized controlled trial of two psychosocial interventions (30) and were excluded if they demonstrated difficulty understanding the nature of the study or were actively psychotic, as determined in a clinical interview by one of us (CES). Study participants had a mean age of 43 years (SD = 9.0); tended to be Caucasian, married, middle-aged people with some college education; and approximately one-third had an income of less than $30,000 (see Table 1). Nearly half of the study participants were employed, most reported having had a professional or managerial occupation, and 34% were disabled from work. Fifty-eight percent of the participants had progressive disease, and 42% had relapsing-remitting disease. The average disease duration was eight years (range 1-37), and the mean level of neurologic disability was an Expanded Disability Status Score (EDSS) of 4.71 (range = 1 to 8.5), which means that most participants had some ambulation disability. The gender ratio approximated the general MS population (31), as 73% of the participants were
179
female. Study participants were on a variety of medications, including drugs for fatigue (19%), spasticity (27%), anxiety (15%) or depression (22%) (27% were on either anti-anxiety or anti-depressant medications), and incontinence (15%). None of the participants were on anti-psychotic drugs, and 2% of the patients were on mood-stabilizers. The sample participants were largely free (i.e. less than 2% of sample) of serious comorbidities such as cancer or heart disease, and a minority reported comorbidities including emotional problems (24%), other (23%), arthritis (7%), high blood pressure (6%), ulcer (5%), and anemia (5%).
Procedure Participants were interviewed to determine their eligibility for the study and subsequently entered into the study. After the interview, a brief neuropsychological battery of standardized tests was given to assess verbal and spatial memory, information processing, cognitive flexibility, and complex attention (see below). At the close of this intake procedure, interested participants were given a questionnaire packet to complete at home and to return by mail. One year later, participants were interviewed and given the neuropsychological battery and were asked to complete the questionnaire by mail. One hundred and seventy-two people were interviewed at baseline, of which 136 (79%) were randomized to one of two social support interventions, and 130 (76%) completed the full one-year of follow-up [see Schwartz and Fox (32) for a careful examination of the selection bias in the present sample].
Measures Self-reported cognitive complaints were assessed using responses to the semi-structured intake interview. The interview question used in the present analyses asked whether the person had "noticed any problems with memory and confusion," and responses were coded according to the type(s) of problems mentioned (e.g. short-term memory, information processing, word recall, etc.). A four-level score of noted problems categorized responses into zero, one, two, or three and greater cognitive problems mentioned. Self-reported affective complaints were also assessed using responses to the semi-structured intake interview, creating an indicator variable score of " 1 " if the patient mentioned mood swings or depression. Neurologic disability was assessed using the Expanded Disability Status Score (33), an observer-reported scale which ranges from zero to ten, where a score of six refers to requiring a cane to ambulate, and a score of ten refers to death due to MS. This scale is the most standard measure of neurologic disability used in the MS population. The Sickness Impact Profile (SIP) (34) was used to assess role performance limitations for the incremental validity analysis. This self-report measure of health status contains 136 items in twelve categories and includes a psychosocial summary score which is a composite of the following subscal~s: social interaction, alertness behavior, emotional behavior, and communication. The SIP has documented test-retest and alpha reliability (35) and convergent and discriminant validity (36). Objective neuropsychological performance was assessed using the Rao et al. cognitive battery (27-29), the Wisconsin Card Sorting Task (37), and the Trail Making Test (38). The Rao battery includes standardized cognitive tests to assess verbal and spatial memory, complex attention, and verbal fluency,
180
ANNALS OF BEHAVIORAL
MEDICINE
including the Selective Reminding Test (SRT) (39), the Controlled Oral Word Association Task (40-42), the Paced Auditory Serial Addition Test (PASAT) (43,44), the Symbol-Digit Task (45), and the 10/36 Spatial Recall Test (46). These tests were chosen from a larger battery because of their documented sensitivity of 0.71 and specificity of 0.94 in discriminating cognitively intact from impaired MS patients. The Wisconsin Card Sorting Task is a measure of cognitive flexibility with documented sensitivity to focal frontal lobe problems (i.e. executive function) (47,48). A recent study has also shown that the Wisconsin Card Sorting Task is sensitive to white matter lesions among people with MS (49). Executive function refers to higher cognitive abilities involving the conceptualization and initiation of activity (50). The Trail Making Test is a measure of complex attention and sequencing with documented test-retest reliability and known sensitivity to the effects of brain injury (38,51,52). All neuropsychological test scores included in this study were adjusted for age, educational level, and gender using normative standards. Normative data were also available to adjust for the practice effect of readministering the tests after one year. Normative scores used published standards and data from healthy controls provided by Dr. Rao and colleagues. The neuropsychological composite, the dependent variable, was the mean of converted T-scores using normative standards as described below. Raw scores were adjusted for age, gender, and education level; were converted to Z scores using normative data; and were converted to T-scores by the transformation T = 50 + 10Z.
Statistical Analysis Specificity is defined to be the probability of accurately not endorsing a problem when one does not have it (53). We began by performing a threshold analysis to determine what "failure" threshold maximized the pattern of congruence between subjective and objective assessment. We examined mean number of tests failed using the following failure thresholds on neuropsychological tests: T-scores of 60, 50, 40, 30, and 20. We then selected the threshold that maximized the apparent consistency of self-report and objective indices and created the following three groupings of patients: Group One included those who noticed no problems; Group Two included those who noticed one or two problems; and Group Three included those patients who noticed three or more problems. These groupings were used in subsequent analyses which addressed sensitivity and incremental validity. Thus, the clinical significance of the number of tests failed at a given threshold will not be addressed psychometrically, but rather the incremental validity analyses will address the issue substantively. Sensitivity is defined to be the probability of accurately identifying a problem when one has it (53). Sensitivity was evaluated by separating the patients into the above groups and computing multinomial logistic regression models to examine the ability of the objective assessment (independent variable) to p~'edict patient self-report (dependent variable). We tested for the potential confounding effect of reported affective complaints on reporting other cognitive problems. Analysis of variance revealed a significant difference among the groups on reported affective complaints (F = 6.97, p < 0.001). Consequently, the logistic regression models included reported affective complaints as a covariate, so that the estimated relationship of subjective and objective assessment would be statistically independent of reported affective complaints.
S c h w a r t z et al. TABLE 2 Descriptive Statistics of Neuropsychological Tests Mean T-Score (Standard Deviation) Test Selective Reminding Test Long-Term Storage Continuous Long-Term Retrieval Controlled Oral Word Association Number correct PASAT Symbol Digit Wisconsin Card Sort Number Categories Achieved Trail-Making Test Part A Part B Spatial (10/36) Neuropsychological Composite (mean T-score)
Baseline
One-Year
38.6 (7.1) 39.7 (6.9)
34.0 (4.9) 38.9 (7.2)
36.1 (5.1) 44.9 (12.1) 31.5 (14.2)
34.4 (4.7) 43.7 (13.4) 31.9 (15.1)
44.5 (15.5)
41.8 (8.1)
35.2 (13.6) 38.8 (14.6) 39.3 (6.9)
36.2 (13.4) 39.1 (12.3) 33.8 (4.9)
41.1 (5.8)
38.1 (5.7)
Two summaries of objective assessment were used for the above logistic analyses: (a) the baseline neuropsychological composite, and (b) a neuropsychological slope score which summarizes change over time. The latter score was derived using growth curve analysis (54-57), which computes individual slopes to assess change over time on the dependent variable of interest. This technique is superior to simply using difference scores because it considers both initial status and variability, whereas difference scores consider only increment from initial status (54). In interpreting the slope scores, we corrected the slope using the mean slope, so that ratios of greater than one reflected improved performance over time, and ratios of less than one reflected declining performance over time. It was unnecessary to adjust for age, education, or gender, as normative criteria were used to derive T-scores. Incremental validity was evaluated using general linear modeling, where the dependent variable was psychosocial role limitations, a documented social cost of cognitive deficits (29). The independent variables were patient self-reported cognitive complaints and either the baseline neuropsychological composite or the neuropsychological slope score. The covariates were patient-reported affective complaints and neurologic disability [i.e. the EDSS (33)]. The independent contribution of each predictor was evaluated separately and together to determine changes in explained variance due to patient report, objective assessment, and both (i.e. incremental validity). Diagnostic statistics were examined for all reported models to test for heteroscedasticity and multicollinearity. Data were analyzed using SAS PC (58) and S Plus (59) software packages. RESULTS Neuropsychologieal Performance As shown in Table 2, the mean performance on all of the individual neuropsychological tests was below normal with an appreciable proportion of the sample performing in the impaired range. Frequency distributions revealed that no one in the sample performed below the 10th percentile or above the 70th percentile on any of the neuropsychological tests. As these patients had participated in a randomized trial of two psycho-
C o g n i t i v e Self-Assessment
V O L U M E 18, N U M B E R 3, 1996
Specificity of Self-Assessment Neuropsychological Performance (Number of tests failed, i.e.,T-score 0.85). Thus, there was no differential effect of the interventions on neuropsychological function over time. Specificity of Self-Assessment Sixty-four percent of the sample reported noticing some problems with memory or confusion of whom 25% reported noticing one cognitive problem, 24% reported noticing two problems, and 15% reported noticing three or more problems. The threshold analysis examining the congruence of patient self-report with increasingly conservative impairment criteria revealed that a T-score impairment criterion of 40 or below distinguished these groups of patients. This analysis revealed a pattern: there was a linear trend indicating increasingly poor performance as the number of patient-reported cognitive complaints increased from none to two, but patients who reported three or more cognitive complaints performed better than those who complained of moderate (i.e. one or two) numbers of complaints and similarly to those who reported no cognitive complaints. This pattern supports our first hypothesis that the relationship between selfreported and objective neuropsychological performance is curvilinear and that patients who report a large number of problems are less consistent with objective testing than those who report none or moderate numbers of problems (Figure 1). Subsequent analyses consequently separated the patients into three cohorts: Group One included patients who noticed no problems; Group Two included those who noticed one or two problems; and Group Three included those patients who noticed three or more problems. This last group will henceforth be referred to as the "overly vigilant," because those patients reported more cognitive complaints but their objective performance was actually better than patients reporting fewer complaints. Sensitivity of Self-Assessment Multinomial logistic regression analyses were done to evaluate the differential sensitivity of self-report after adjusting
181
for affective complaints. We found that slope estimates of objective assessment were better predictors of patient self-report than were baseline estimates (c statistic = 0.62 and 0.51, respectively, where a value of 0.50 would be expected by chance). These findings partially support the hypothesis that noticing problems will be associated with worse performance on neuropsychological tests at baseline and over time (i.e. sensitivity). In fact, people who reported noticing cognitive difficulties performed worse than those who did not on neuropsychological tests over time but not at baseline (i.e. sensitivity). Patients seemed to be attending to relatively subtle deficits (i.e. decline over time) which may not be considered impaired according to baseline performance. Incremental Validity To understand the incremental validity of combining subjective and objective assessment, we built hierarchical models, all of which contained the covariates of EDSS and reported affective complaints. These hierarchical models tested the predictive significance of: (a) the baseline neuropsychological composite; (b) the slope neuropsychological composite; (c) reported cognitive complaints; and (d) the full baseline model [i.e. the combined predictive significance of objective (baseline or slope) performance and subjective assessment]. The parameter estimates for the full models are listed in the first five rows of Table 3 to give the reader a sense of the magnitude of the association between subjective and objective assessment in predicting psychosocial role limitations. The next four rows of Table 3 provide the model R E statistics for each hierarchical model as well as for the full model. Psychosocial role limitations were predicted at two points in time: at baseline and at one year. The incremental validity of these predictors were tested among two cohorts of patients: the whole sample and a restricted sample where the overly-vigilant patients were excluded. We found that patient-reported complaints and baseline objective assessment were complementary and statistically significant predictors of psychosocial role limitations at baseline and one year among the whole sample, but that slope objective assessment was not a complementary and significant predictor among the whole sample (Table 3). Among the subsample which excluded the overly-vigilant patients, we found that patient-reported complaints and objective assessment (both at baseline and over time) were complementary and statistically significant predictors of psychosocial role limitations at baseline and one year. Thus, patient-reported complaints and objective assessment are incrementally valid indicators of psychosocial role limitations, and this complementarity is pronounced among the subsample of patients whose self-reports are linearly associated with objective assessment. Further, the models using the slope estimate of performance change accounted for more variance in psychosocial role limitations than the models which included baseline performance, suggesting that the additional information provided by patients may be due to subtle deficits accrued over time. DISCUSSION Our results suggest that MS patients who report cognitive complaints demonstrate greater progression of such deficits over time, and a combination of objective and subjective measures of cognitive deficit accounted for greater variance in role limitations than either measure alone. We found that the people whose self-assessment was most consistent with objective test-
182
ANNALS OF BEHAVIORAL
MEDICINE
S c h w a r t z et al.
TABLE 3 Incremental Validity of Objective and Subjective Assessment for Psychosocial Role Limitations Outcome
Psychosocial Role Limitations Whole Sample
Predictor
Outcome@ Baseline
EDSS Reported Affective Problems Baseline Neuropsychological Composite Slope of Neuropsychological Composite Reported Cognitive Problems Model R2 Baseline Composite Alone Slope Composite Alone Reported Cognitive Problems Full Model tp *p **p ***p ****p
Outcome@ 1 year
Overly-Vigilant Patients Excluded Outcome@ 1 year
0.11 0.18*
0.11 0. t 1
0.22* 0.11
-0.15+ -0.35***
-0.35*** -0.32***
--0.09 0.35***
0.16 -0.18 0.25****
0.19 -0.20 0.28****
0.12 0.17 -0.18"***
Outcome@ Baseline 0.13 0.09 -0.18+ -0.37*** 0.17 -0.20 0.21"***
Outcome@ 1 year 0.11 0.02 -0.43*** -0.32*** 0.32 -0.30 0.37****
Outcome@ 1 year 0.11 -0.003 --0.41"** 0.33*** 0.31 0.32 -0.35****
