Journal of of Personality 84:1, February Journal Personality ••:••, •• 2014 2016

On Being the Same in Different Places: Evaluating Frame-of-Reference Effects Across Two Social Contexts

C © V 2014 Wiley Periodicals, Inc. 2014 Wiley Periodicals, Inc.

DOI: 10.1111/jopy.12144 DOI: 10.1111/jopy.12144

John E. Kurtz1 and Amy Pastva Palfrey2 1 2

Villanova University St. Johns University

Abstract Frame-of-reference (FOR) effects in personality assessment are demonstrated when self-rated items oriented to specific contexts (e.g., workplace) show better predictive validity than noncontextualized items. Empirical support of FOR effects typically relies on job performance ratings or academic grades for criteria.The current study evaluates FOR effects using ratings of personality provided by informants from the home or school context. Items from the NEO Five-Factor Inventory (NEO-FFI; Costa & McCrae, 1992) were contextualized to the home and school contexts to create NEO-Home and NEO-School versions. One hundred fifty-eight college students completed the NEO-Home and NEO-School questionnaires, and 161 college students completed the standard, noncontextualized NEO-FFI. All participants recruited one peer from college and at least one parent to complete standard rater versions of the NEO-FFI. Contextualized self-ratings did not show FOR effects. NEO-Home self-ratings did not correlate higher with parent ratings than with peer ratings, and NEO-School self-ratings did not correlate higher with peer ratings than with parent ratings. Standard NEO-FFI self-ratings generally showed higher self-informant agreement with both types of informants than contextualized self-ratings.The pattern of correlations suggests that validity is enhanced more by specific trait-informant combinations than by the contextualization of items to social contexts.

It is important for trait theories of personality to examine stability and change of trait expression across time and situations. The trait-situation debate, highlighted by the seminal challenge of Mischel (1968), has called attention to traitsituation interactions in personality science (Kenrick & Funder, 1988). For example, researchers have examined contextual variability in personality expression across social roles, such as friend, coworker, or romantic partner (Baird & Lucas, 2011; Donahue, Robins, Roberts, & John, 1993; Heller, Watson, Komar, Min, & Perunovic, 2007; Roberts, 2007). To examine the effects of different social contexts on selfdescriptions, many studies of contextual factors present questionnaire items that explicitly orient the respondent to particular social roles or settings. Contextualized personality questionnaires have been applied by industrial-organizational psychologists interested in the use of personality assessment for personnel selection. Taking inspiration from Wright and Mischel (1987), the concept of frame-of-reference (FOR) effects proposes that rewriting test items to orient them to the workplace setting will increase their validity for predicting behavior at work. The original demonstration of FOR effects was reported by Schmit, Ryan, Stierwalt, and Powell (1995). They found that adding the phrase “at work” to items from the NEO Five-Factor Inventory (NEO-FFI; Costa & McCrae, 1992) yielded lower Neuroticism

scores and higher Agreeableness and Conscientiousness scores relative to standard, noncontextualized items. However, the score differences do not necessarily reflect changes in actual behavior in the workplace. Work-contextualized self-ratings may only reflect what individuals know about the types of behavior that are important to maintain a job, such as being highly dependable, getting along with coworkers, and maintaining a stable mood. In a second study, Schmit et al. (1995) used undergraduate students’ current grade point average (GPA) to appraise the validity of NEO-FFI items contextualized with the phrase “at school.” Schoolcontextualized Conscientiousness scores showed higher correlations than noncontextualized scores in predicting current

The authors thank Brian Connelly and two anonymous reviewers for their helpful comments and suggestions. We thank Evan Good and Joy Lee for their assistance with data analysis and Amanda Sizemore, Julia Hernandez, Elizabeth McKernan, Sarah Tarquini, Emily Iobst, Lindsay Schauble, and Nicole Cross for their assistance with protocol administration and data management. We are also grateful to Psychological Assessment Resources, Inc., for granting permission to alter the test materials for the purpose of this study. Correspondence concerning this article should be addressed to John E. Kurtz, Department of Psychology, Villanova University, 800 Lancaster Avenue, Villanova, PA 19085-1699. Email: [email protected].

1214

GPA; the “at school” and standard versions of NEO-FFI Conscientiousness correlated .41 and .27, respectively, with GPA. A series of subsequent studies have replicated the findings from Schmit et al. (1995), suggesting that personality test items with a school or workplace FOR predict performance better than standard items. Specifically, three studies (Bing, Whanger, Davison, & Van Hook, 2004; Lievens, De Corte, & Schollaert, 2008; Reddock, Biderman, & Nguyen, 2011) have demonstrated that adding the phrase “at school” to personality measures increased the correlations between students’ self-ratings of Conscientiousness and GPA. Shaffer and Postlethwaite (2012) performed a meta-analysis of 15 studies that examined whether adding an “at work” FOR to personality measures enhances the prediction of supervisor ratings of job performance. This body of work included five unpublished dissertations to account for possible file drawer effects. Shaffer and Postlethwaite showed higher mean validity coefficients for “at work” FOR scales versus noncontextualized scales, with rho values ranging from .30 for Conscientiousness to .19 for Openness. Moreover, several studies (e.g., Pace & Brannick, 2010) demonstrate that contextualized scales obtain significant incremental validity over standard, noncontextualized scales to predict performance criteria. It is noteworthy that the majority of studies supporting FOR effects have utilized concurrent performance criteria, such as supervisor ratings of incumbent employees or GPA of established students. Although performance criteria are a logical choice for the practical purposes of prediction and selection, it is likely that this method introduces some degree of criterion contamination. In most studies of academic performance, the student participants had knowledge of the criterion, as most students know their current GPA.1 Self-ratings of competence, order, and self-discipline oriented to the school setting may be an indirect way to assess self-perceptions of academic performance. All of the published studies of workplace performance identified by Shaffer and Postlethwaite (2012) used incumbent employees, but asking employees to rate how conscientious or agreeable they are “at work” may be rather similar to asking them what their current supervisor would think of their performance on the job. A study showing that FOR trait measures can better predict independent observations of behavior, or other criteria unknown to the respondent at the time of testing, would present a more convincing demonstration of contextual effects on personality expression and the validity of FOR assessment practices. Informant ratings of personality offer an appealing alternative for evaluating the validity of FOR effects. Research on meta-perception suggests that people are not very accurate in their judgments about how others view their personality traits (Kenny & DePaulo, 1993; Vazire & Carlson, 2010), so these criteria may be less contaminated by the target’s expectations. Moreover, informant ratings demonstrate comparable and sometimes better validity to predict behavior and outcomes relative to self-ratings (e.g., Kolar, Funder, & Colvin, 1996; Kurtz, Puher, & Cross, 2012; Spain, Eaton, & Funder, 2000).

Kurtz Kurtz & & Palfrey

If there are appreciable differences in personality trait expression across social contexts, then observers from different contexts should provide divergent ratings of a target’s personality traits. Prior research has shown parents and peers to be generally comparable in their judgmental accuracy of personality traits, despite observing their targets in distinctively different social roles and life contexts (e.g., Bem & Allen, 1974; Funder, Kolar, & Blackman, 1995; Kenrick & Stringfield, 1980; Kurtz, Tarquini, & Iobst, 2008). Thus, informant ratings of personality represent potentially informative but as yet underutilized criteria for the evaluation of FOR effects and the study of differences in trait expression across social contexts. Two studies to date have investigated the relationships between FOR trait ratings by the self and generalized trait ratings by informants from those FOR settings. Small and Diefendorff (2006) gathered trait ratings from supervisors and coworkers for a sample of 143 adults employed full-time in a wide variety of occupations. These employees provided selfratings under standard and work-specific instructions. Trait ratings by workplace informants did not correlate higher with the “at work” self-ratings of traits than with standard selfratings. Bongard, Martin, Seip, and al’Absi (2011) utilized informant ratings to examine the validity of self-reported anger expression using generalized, work-specific, and homespecific measures. Each employee recruited a cohabitating “partner” and a “colleague” from the workplace to complete generalized and home- or work-contextualized measures of anger expression. Only the ratings of outwardly expressed anger (AO) showed the predicted FOR pattern. Self-partner correlations were higher than self-colleague correlations for measures of inwardly directed anger and anger control, regardless of the type of rating instructions (general, “at home,” or “at work”). Thus, existing support for FOR effects using informant ratings is mixed at best. The current study aims to advance the study of FOR effects by asking college students to complete self-ratings of personality traits contextualized to the home and school domains. This study also employs informant ratings as the criterion variables in order to reduce criterion contamination. Each participant recruited at least one parent, who served as the informant for the home domain, and one college peer, who served as the informant for the school domain. Based on the theory of trait-situation interactionism and the previous studies of FOR reviewed by Shaffer and Postlethwaite (2012), the following five hypotheses were tendered: 1. Self-ratings of Five-Factor Model (FFM) traits differ significantly between scales referencing the home or school contexts versus scales using standard rating instructions. 2. Home self-ratings of FFM traits are more highly correlated with parent ratings of the corresponding traits than school self-ratings. 3. Home self-ratings provide incremental validity over school ratings in the prediction of parent ratings.

Frame-of-Reference Across Across Contexts Contexts

3 115

4. School self-ratings of FFM traits are more highly correlated with peer ratings of the corresponding traits than home self-ratings. 5. School self-ratings provide incremental validity over home ratings in the prediction of peer ratings.

METHOD Participants Four hundred seventy-four same-sex dyads of undergraduate students who resided on or near the campus of a private university in the northeastern United States initiated participation in the study. One member of each dyad was assigned the role of target, and the other member was assigned the role of peer. This initial sample of targets was 62% female, and 81% were freshmen. Dyads were further divided into one of two conditions; 231 dyads were assigned to the contextual condition and 243 dyads were assigned to the standard condition. There were no significant differences in targets assigned to the two conditions with respect to gender, class year, or type of relationship with peer. Parents of consenting targets were also invited to participate in the study, and complete data were received from at least one parent for 326 targets. Peers were required to be students from the same university and introduced to the target after starting university. Seven dyads were excluded from the study because the target and peer had been acquainted prior to coming to the university as freshmen, leaving 319 eligible dyads. The rate of parent participation was equivalent between targets assigned to the contextual condition (n = 158) and to the standard condition (n = 161).

gender-specific informant-rating versions, each using a 5-point item response format labeled Strongly Disagree, Disagree, Neutral, Agree, and Strongly Agree. The NEO-FFI was modified for the current study to create two contextualized self-rating versions designed to assess perceptions of FFM traits as they are experienced or expressed when at home and when at school. These versions will be referred to as the NEO-Home and NEO-School questionnaires. The general instructions for the NEO-Home questionnaire asked the target to “Answer these questions as they apply to your thoughts, feelings, and behaviors when you are at home with your family.” The phrase “at home” was added to the beginning, middle, or end of each item contained in the standard NEO-FFI, depending on what made for the smoothest style of expression. The instructions on the NEO-School questionnaire asked the target to “Answer these questions as they apply to your thoughts, feelings, and behaviors when you are at school with your peers.” The phrase “at school” was added to each item in the same location as in the NEO-Home questionnaire. Descriptive statistics for the NEO-Home and NEOSchool scales are compared with those of the standard version in Table 1. Alphas for the parent ratings ranged from .70 for Openness to .89 for Conscientiousness, with a median alpha of .84. Alphas for the peer ratings ranged from .69 for Openness to .90 for Conscientiousness, with a median alpha of .84. The parent-peer agreement correlations were .23 for Agreeableness, .28 for Neuroticism, .35 for Extraversion, .36 for Openness, and .48 for Conscientiousness, which lends support to their potential to represent distinct, contextualized accounts of target personality.2

Procedure Measures NEO Five-Factor Inventory. Several variations of the NEO Five-Factor Inventory (NEO-FFI; Costa & McCrae, 1992) were used in the study. The NEO-FFI is a 60-item structured questionnaire designed to measure the FFM traits of Neuroticism, Extraversion, Openness to Experience, Agreeableness, and Conscientiousness. This measure has both self-rating and

Students were recruited into the study using a departmental human subjects pool, and they received credit toward introductory psychology course requirements in exchange for their participation. Students were asked to bring a same-sex peer to the study who was also a student at the university. If one member of the dyad was enrolled in the introductory psychology course, then that student was assigned the role of target and the other student was assigned the role of peer. Those not

Table 1 Descriptive Statistics for the NEO-Home, NEO-School, and Standard NEO Scales NEO-Home

Neuroticism Extraversion Openness Agreeableness Conscientiousness

NEO-School

Standard NEO

M

SD

α

M

SD

α

M

SD

α

38.9* 54.0 46.8* 55.9 48.9

10.4 12.2 11.1 12.3 11.1

.88 .84 .73 .82 .86

43.1 54.7 46.8* 58.2* 50.3

10.7 11.1 10.6 9.8 10.7

.87 .83 .74 .78 .87

45.0 54.0 51.6 53.8 50.9

11.3 10.4 10.0 11.3 10.3

.88 .81 .74 .79 .87

Note. n = 158 for NEO-Home and NEO-School; n = 161 for standard NEO. All scores are converted to T-scores using NEO-FFI normative data for self-ratings by college students provided in Costa and McCrae (1992). *Difference between contextual NEO mean and standard NEO mean is significant at p < .05.

1416

enrolled in the course participated on a volunteer basis. If both members of the dyad were introductory students, then the roles of target and peer were determined by random assignment. Participants completed the study materials in groups of one to four dyads. Assignment to one of the two instructional conditions, contextual or standard, was conducted in alternating order by session. Targets in the contextual condition were asked to complete the NEO-Home and NEO-School questionnaires; 87 participants completed the home version first and 71 participants completed the school version first. Peers in both instructional conditions rated the personality of the target using the appropriate gender-specific informant version of the NEO-FFI. Two hundred eleven dyads were roommates and 108 were noncohabitating friends. The average length of acquaintanceship between targets and peers was 6.9 months (SD = 5.1), ranging from 2 to 45 months.3 Each target was also asked to recruit his or her parents into the study. Consenting targets addressed one envelope that was used for the initial mailing, and another envelope was used for a reminder letter. The initial mailing included the study materials and a postage-paid return envelope. The reminder letter was sent 2 weeks after the first mailing to all parents; no record was kept of the names and addresses of parents. For targets with ratings returned from both parents (n = 229), the samesex parent was selected, arbitrarily, in order to match the gender of the peer informant. All targets had lived with the rating parent since birth and reported at least once-weekly contact with that parent by telephone, electronic mail, or text message. The distance between the parent’s home and campus ranged from 5 miles to 9,580 miles; the median distance from home to school was 125 miles.

RESULTS As stated in Hypothesis 1, the most basic prediction regarding FOR effects is that contextualization of items will alter the total scores relative to standard instructions. This hypothesis was tested by standardizing the NEO-Home and NEO-School scores for all five trait domains using the college student norms for NEO-FFI self-ratings provided in the test manual (Costa & McCrae, 1992). The same norms were also applied to the noncontextual self-ratings provided by the standard instruction group. Table 1 compares the mean T-scores profiles for all three types of ratings. Openness to Experience self-ratings were significantly lower when contextualized to the home, t(317) = 4.06, p < .001, d = .47, and school settings, t(317) = 4.14, p < .001, d = .47, relative to noncontextualized selfratings of Openness provided by respondents in the standard condition. Ratings of Neuroticism-at-home by contextual respondents were significantly lower than noncontextualized ratings of Neuroticism by standard respondents, t(317) = −4.98, p < .001, d = −.58, and ratings of Agreeableness-atschool were significantly higher than standard Agreeableness ratings, t(317) = 3.71, p < .001, d = .42. All other betweengroups contrasts were nonsignificant.

Kurtz Kurtz & & Palfrey

Table 2 Correlations Between Home- and School-Contextualized SelfRatings, Standard Self-Ratings, and Ratings by Parents and Peers

FFM Trait Neuroticism Home School Standard Extraversion Home School Standard Openness Home School Standard Agreeableness Home School Standard Conscientiousness Home School Standard

Parent Ratings r

Peer Ratings r

.48a .46 .39

.28 .27b .34

.35 .42 .58

.38 .50c .47

.44 .44 .45

.52 .52 .58

.38d .17 .44

.28 .28 .45

.52a .48 .54

.39 .47 .51

Note. n = 158 for NEO-Home and NEO-School ratings; n = 161 for standard NEO ratings. FFM = Five-Factor Model; r = zero-order correlations; values of r in bold indicate convergent correlations predicted by frame-of-reference effects. a Home-parent r is significantly higher (p < .05, one-tailed) than home-peer r. b School-peer r is significantly lower (p < .05. one-tailed) than school-parent r. c School-peer r is significantly higher (p < .05, one-tailed) than home-peer r, dHomeparent r is significantly higher (p < .05, one-tailed) than school-parent r. Significance of differences in r based on tests described in Meng, Rosenthal, and Rubin (1992). All correlations are significant at p < .05.

Hypothesis 2 predicts FOR effects when parent ratings serve as the criterion. That is, NEO-Home ratings of the FFM traits will correlate more highly with parent ratings than NEOSchool ratings of those traits. The relevant correlations are presented in the first column of Table 2, and differences in the magnitude of the NEO-Home and NEO-School correlations were tested for significance (Meng, Rosenthal, & Rubin, 1992). Strong support for FOR effects was observed only for Agreeableness (z = 3.07). The other four traits showed nonsignificant differences in correlation with parent ratings, and the findings for Extraversion were opposite to the FOR pattern. To test Hypothesis 3, hierarchical regression analyses examined the incremental validity of NEO-Home scores over NEO-School scores to predict parent ratings of corresponding traits. A series of hierarchical multiple regression analyses was performed for each of the five traits, entering the NEO-School score in the first step and the NEO-Home score in the second step. The results of these analyses are presented in the top panel of Table 3. NEO-Home scores showed incremental validity over NEO-School scores for four of the five traits; only Openness-at-home failed to increment Openness-at-school in predicting parent ratings of Openness. However, a series of reverse models showed incremental validity patterns contrary

Frame-of-Reference Across Across Contexts Contexts

5 117

Table 3 Incremental Validity of NEO-Home Over NEO-School Scores to Predict Informant Ratings Criterion Parent ratings (hypothesized models) Neuroticism Extraversion Openness Agreeableness Conscientiousness Peer ratings (reverse models) Neuroticism Extraversion Openness Agreeableness Conscientiousness

Part r

ΔR2

F-Change

p

.29 .18 .13 .35 .26

8.2% 3.4% 1.6% 12.5% 6.9%

18.02 6.76 3.17 22.83 15.25

< .001 .010 .077 < .001 < .001

.18 .17 .15 .10 .10

3.2% 3.0% 2.2% 1.0% 1.0%

5.53 6.37 4.90 2.54 2.02

.020 .013 .028 .113 .158

Note. n = 158. Part r = semipartial correlations for NEO-Home scores controlling for NEO-School scores.

Table 4 Incremental Validity of NEO-School Over NEO-Home Scores to Predict Informant Ratings Criterion Peer ratings (hypothesized models) Neuroticism Extraversion Openness Agreeableness Conscientiousness Parent ratings (reverse models) Neuroticism Extraversion Openness Agreeableness Conscientiousness

Part r

ΔR2

F-Change

p

.14 .36 .17 .15 .27

1.9% 13.0% 3.0% 2.2% 7.2%

3.35 27.88 6.55 3.81 14.47

.069 < .001 .011 .053 < .001

.26 .30 .15 –.04 .16

6.5% 9.3% 2.2% 0.2% 2.7%

14.36 18.50 4.31 0.31 6.06

< .001 < .001 .040 .577 .015

Note. n = 158. Part r = semipartial correlations for NEO-School scores controlling for NEO-Home scores.

to FOR effects; the results of these analyses are presented in the bottom panel of Table 3. NEO-School scores significantly incremented NEO-Home scores in predicting parent ratings for three of the five traits. Hypothesis 4 evaluates FOR effects when NEO-School scores are correlated with peer ratings of the corresponding traits. These correlations are contrasted with the NEO-Home correlations in the second column of Table 2. Support for an FOR effect was observed only for Extraversion, assuming a one-tailed significance test (z = 1.67). The other four traits showed nonsignificant differences in correlation with peer ratings, and the findings for Neuroticism were opposite to the FOR pattern. To test Hypothesis 5, another series of hierarchical regression analyses was performed, and these results are presented in the top panel of Table 4. Peer ratings of each trait served as the criterion, with NEO-Home scores of the corresponding trait entered in the first step and NEO-School scores entered in the

second step. NEO-School scores significantly incremented prediction of peer ratings beyond NEO-Home scores for three of the five traits; the increments for Neuroticism and Agreeableness fell short of significance. Again, the reverse models presented in the bottom panel of Table 4 also showed incremental gains contrary to FOR effects. The incremental validity of NEO-Home scores over NEO-School to predict peer ratings was statistically significant for four of the five traits, with a lack of significant increment in prediction only for Agreeableness.4 Table 2 also presents the correlations between noncontextualized NEO-FFI scores and parent and peer ratings of targets in the standard condition. When the convergent correlations predicted by FOR effects (i.e., NEO-Home with parent ratings, NEO-School with peer ratings) are compared with the standard self-informant correlations, higher correlations related to FOR effects are seen in just two of 10 comparisons. The home-parent correlation for Neuroticism and the school-peer correlation for Extraversion are higher than the standard self-informant correlations for these traits. However, when all the relevant correlations are considered together, it is evident that even these two apparent FOR effects can be explained by specific combinations of trait, context, and informant. Neuroticism ratings by parents correlate higher with self-ratings than with peer ratings, regardless of the context to which those self-ratings are oriented. Ratings of Extraversion contextualized to the school setting produced higher correlations than ratings contextualized to the home setting, regardless of which informant rating serves as the criterion. Peers are more accurate informants for Openness, and this difference holds for contextualized and noncontextualized self-ratings alike. The picture is least clear for Agreeableness, but the larger advantage seen with the standard self-ratings for this trait suggests that contextualizing Agreeableness items substantially reduces rather than enhances their validity for predicting informant ratings.

DISCUSSION The present study evaluated FOR effects using trait ratings by informants from two separate social contexts as criteria. Consistent with previous research (Schmit et al., 1995), modifications to the NEO-FFI to create an “at home” and an “at school” FOR altered the scores for three of five trait domains relative to the standard version. However, the finding that modifications to the NEO-FFI changed the scores obtained is a rather predictable one that leaves open to question whether validity for predicting informant ratings was enhanced. Accordingly, informant ratings were used to test whether self-ratings of personality traits with a “home” FOR would correlate more highly with parent ratings than with peer ratings and whether ratings with a “school” FOR would correlate more highly with peer ratings than with parent ratings. The pattern of correlations between the NEO-Home and NEO-School scores and ratings of the same traits by parents and peers generally did not

1618

support FOR effects. Peers were better judges than parents of self-rated Openness to Experience, regardless of context. Parents were better judges than peers of self-rated Neuroticism, regardless of context. Moreover, the inclusion of selfinformant correlations with the standard NEO-FFI revealed a pattern of findings rather different from that predicted by FOR effects. In 17 of 20 comparisons, the standard self-ratings produced higher correlations with informant ratings than either of the contextualized self-ratings. These findings suggest that personality is generally stable across contexts and that matching traits and informants may improve validity more in some cases than matching selfratings to social contexts. Well-acquainted informants may possess knowledge about personality that transcends differences in behavior across contexts (Vazire, 2010). Certain informants may be better judges of certain traits by virtue of their history of acquaintanceship and of the particular settings and contexts in which they make their observations. FOR effects are principally concerned with the accuracy of personality judgments. Funder’s (1995) realistic accuracy model provides a theoretical account of the process by which the self or others produce valid trait ratings. Funder’s concept of “relevance” can be invoked to account for much of the previous FOR literature; judgments of a trait will be more accurate when they are made in settings or situations that afford more opportunities to observe that trait enacted. Most studies of FOR focus on the trait of Conscientiousness and the settings of work or school. The work context is highly relevant to the trait of Conscientiousness, but that does not necessarily mean that a person’s Conscientiousness is different outside of work. Rather, it is easier to access relevant information to make judgments about Conscientiousness when thinking about the work setting than when thinking about relaxing with friends. Nonetheless, that work-situated judgment of Conscientiousness will probably predict whether we are on time when meeting a friend at the movies. As another example, Extraversion is more easily observed and accurately rated in a public setting with the larger peer group than in a dorm room with the roommate (Kurtz & Sherker, 2003). Ultimately, it is premature to conclude that FOR manipulations will increase validity for all traits and all settings. Contextual variability in trait expression may differ across the FFM constructs, and certain combinations of traits and settings, such as Conscientiousness and work, are more closely linked by relevance than other combinations. Confidence in these findings is tempered by several limitations in the current study methodology. First, the samples used raise questions of generalizability, as college students are not representative of the population at large. On the other hand, college students present an ideal situation for the study of contextualized personality, given the clear separation of the two social worlds they inhabit and the amount of time they spend in each of them. Second, only one informant is employed in the home and school settings, and the type of informant is confounded with the context. It is likely that any

Kurtz Kurtz & & Palfrey

second informant rating would increment prediction beyond a single rater; thus, using multiple informants per context would provide a clearer appraisal of incremental prediction due to FOR modifications. The third limitation is the use of a between-subjects design to compare the FOR and standard testing formats, whereas the majority of previous FOR studies have used within-subjects designs for these comparisons. However, within-group designs in FOR research present other methodological problems. When more than one context is evoked in a single assessment session, it may create a demand effect in which respondents feel compelled to rate the same items differently as the instructional context is altered (Baird & Lucas, 2011). The contextualization of personality expression and the prospect of FOR effects might vary for different types of personality variables. FFM constructs represent personality at the broadest and most decontextualized level. McAdams and Pals (2006) propose that there exists a different class of personality constructs called characteristic adaptations that vary across time, place, and social role. Examples of characteristic adaptations include attachment styles, coping mechanisms, and personal goals. Future research may reveal stronger FOR effects by focusing on this class of more putatively malleable personality variables. Finally, only two types of social contexts were investigated here. Although the use of two contexts advances research on FOR effects, it is likely that more can be learned about the stability of personality expression and the possibilities of FOR methods for assessment by going beyond the workplace and school settings. Studies that examine the consistency of trait expression across contexts and over time (e.g., Nave, Sherman, Funder, Hampson, & Goldberg, 2010) will be most informative about the consistency, coherence, and flexibility of the person in different places. Notes 1. The one exception is the study by Bing et al. (2004), who collected personality ratings from students during freshman orientation and then subsequently obtained their freshman year GPA from the university registrar. Interestingly, of all the studies reviewed, the design of Bing and colleagues’ study produced the smallest differences in validity correlations between the school FOR and standard format items. 2. Additional descriptive statistics and intercorrelation matrices of the study variables can be obtained by contacting the first author. 3. Consistent with previous research (e.g., Kurtz & Sherker, 2003; Watson, Hubbard, & Wiese, 2000), length of acquaintanceship between targets and peers significantly moderated self-peer correlations for Neuroticism-at-school. Cohabitation status significantly moderated self-peer correlations for Conscientiousness-at-school, with higher correlations observed among roommates (r = .55) than among noncohabitating friends (r = .41). 4. A reviewer suggested that confirmatory factor analysis would provide a more comprehensive test of FOR effects. For each of the five traits, a model was constructed in which the Self-Home, Self-

Frame-of-Reference Across Across Contexts Contexts

School, parent, and peer ratings were loaded on a general trait factor. By correlating the residual variance estimates for the contextmatched indicators (i.e., NEO-Home with parent, NEO-school with peer), direct estimates of FOR effects were obtained. Both paths were significant for Agreeableness (r = .15 and .13), and the home-parent path for Conscientiousness was significant (r = .20). However, the four general factor loadings were at least twice the magnitude of the FOR residual correlations for all five traits. Consistent with the findings reported here, this approach shows FOR effects to be inconsistent across trait domains and smaller in magnitude than generalized trait effects.

Declaration of Conflicting Interests The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding The authors received no financial support for the research, authorship, and/or publication of this article.

References Baird, B. M., & Lucas, R. E. (2011). “. . . And how about now?”: Effects of item redundancy on contextualized self-reports of personality. Journal of Personality, 79, 1081–1112. Bem, D. J., & Allen, A. A. (1974). On predicting some of the people some of the time: The search for cross-situational consistencies in behavior. Psychological Review, 81, 506–520. Bing, M. N., Whanger, J. C., Davison, H. K., & Van Hook, J. B. (2004). Incremental validity of the frame-of-reference effect in personality scale scores: A replication and extension. Journal of Applied Psychology, 89, 150–154. Bongard, S., Martin, N. M., Seip, M., & al’Absi, M. (2011). Evaluation of a domain-specific anger expression assessment strategy. Journal of Personality Assessment, 93, 56–61. Costa, P. T., & McCrae, R. R. (1992). Revised NEO Personality Inventory (NEO-PI-R) and NEO Five-Factor Inventory (NEOFFI) professional manual. Lutz, FL: Psychological Assessment Resources. Donahue, E. M., Robins, R. W., Roberts, B. W., & John, O. P. (1993). The divided self: Concurrent and longitudinal effects of psychological adjustment and social roles on self-concept differentiation. Journal of Personality and Social Psychology, 64, 834–846. Funder, D. C. (1995). On the accuracy of personality judgment: A realistic approach. Psychological Review, 102, 652–670. Funder, D. C., Kolar, D. C., & Blackman, M. C. (1995). Agreement among judges of personality: Interpersonal relations, similarity, and acquaintanceship. Journal of Personality and Social Psychology, 69, 656–672.

7 119

Heller, D., Watson, D., Komar, J., Min, J. A., & Perunovic, W. Q. E. (2007). Contextualized personality: Traditional and new assessment procedures. Journal of Personality, 75, 1229–1254. Kenny, D. A., & DePaulo, B. M. (1993). Do people know how others view them? An empirical and theoretical account. Psychological Bulletin, 114, 145–161. Kenrick, D. T., & Funder, D. C. (1988). Profiting from controversy: Lessons from the person-situation debate. American Psychologist, 43, 23–34. Kenrick, D. T., & Stringfield, D. O. (1980). Personality traits and the eye of the beholder: Crossing some traditional philosophical boundaries in the search for consistency in all people. Psychological Review, 87, 88–104. Kolar, D. W., Funder, D. C., & Colvin, C. R. (1996). Comparing the accuracy of personality judgements by the self and knowledgeable others. Journal of Personality, 64, 311–337. Kurtz, J. E., Puher, M. A., & Cross, N. A. (2012). Prospective prediction of college adjustment using self- and informant-rated personality traits. Journal of Personality Assessment, 94, 630– 637. Kurtz, J. E., & Sherker, J. L. (2003). Relationship quality, trait similarity, and self-other agreement on personality ratings in college roommates. Journal of Personality, 71, 21–48. Kurtz, J. E., Tarquini, S. J., & Iobst, E. A. (2008). Socially desirable responding in personality assessment: Still more substance than style. Personality and Individual Differences, 45, 22–27. Lievens, F., De Corte, W., & Schollaert, E. (2008). A closer look at the frame-of-reference effect in personality scale scores and validity. Journal of Applied Psychology, 93, 268–278. McAdams, D. P., & Pals, J. L. (2006): A new Big Five: Fundamental principles for an integrative science of personality. American Psychologist, 61, 204–217. Meng, X. L., Rosenthal, R., & Rubin, D. B. (1992). Comparing correlated correlation coefficients. Quantitative Methods in Psychology, 111, 172–175. Mischel, W. (1968). Personality and assessment. New York: Wiley. Nave, C. S., Sherman, R. A., Funder, D. C., Hampson, S. E., & Goldberg, L. R. (2010). On the contextual independence of personality: Teachers’ assessments predict directly observed behavior after four decades. Social Psychological and Personality Science, 1, 327–334. Pace, V. L., & Brannick, M. T. (2010). Improving prediction of work performance through frame-of-reference consistency: Empirical evidence using openness to experience. International Journal of Selection and Assessment, 18, 230–235. Reddock, C. M., Biderman, M. D., & Nguyen, N. T. (2011). The relationship of reliability and validity of personality tests to frame-of-reference instructions and within-person inconsistency. International Journal of Selection and Assessment, 19, 119– 131. Roberts, B. W. (2007). Contextualizing personality psychology. Journal of Personality, 75, 1071–1082. Schmit, M. J., Ryan, A. M., Stierwalt, S. L., & Powell, A. B. (1995). Frame-of-reference effects on personality scale scores and

1820

criterion-related validity. Journal of Applied Psychology, 80, 607– 620. Shaffer, J. A., & Postlethwaite, B. E. (2012). A matter of context: A meta-analytic investigation of the relative validity of contextualized and noncontextualized personality measures. Personnel Psychology, 65, 445–494. Small, E. E., & Diefendorff, J. M. (2006). The impact of contextual self-ratings and observer ratings of personality on the personality– performance relationship. Journal of Applied Social Psychology, 36, 297–320. Spain, J. S., Eaton, L. G., & Funder, D. C. (2000). Perspectives on personality: The relative accuracy of self versus others for the prediction of emotion and behavior. Journal of Personality, 68, 837–867.

Kurtz Kurtz & & Palfrey

Vazire, S. (2010). Who knows what about a person? The self-other knowledge asymmetry (SOKA) model. Journal of Personality and Social Psychology, 98, 281–300. Vazire, S., & Carlson, E. N. (2010). Self-knowledge of personality: Do people know themselves? Social and Personality Psychology Compass, 4/8, 605–620. Watson, D., Hubbard, B., & Wiese, D. (2000). Self-other agreement in personality and affectivity: The role of acquaintanceship, trait visibility, and assumed similarity. Journal of Personality and Social Psychology, 78, 546–558. Wright, J. C., & Mischel, W. (1987). A conditional approach to dispositional constructs: The local predictability of social behavior. Journal of Personality and Social Psychology, 53, 1159– 1177.