when i say When I say … context specificity Cees P M van der Vleuten
People are inclined to think that behaviour, that of others in particular, is governed by stable dispositions. Traits are stable latent entities and if growth happens as a result of experience, this happens continuously over time. Think of intelligence or neuroticism. However, after researchers in personality psychology had been trying to measure traits for over half a century, the realisation dawned that it was not traits but the situation that was the dominant influence on behaviour. This sparked a radical shift to a paradigm of person-bysituation interaction, with traits re-cast as contextdependent states.1 In medical education, the new centrality of context caused quite a stir. Context specificity was discovered more or less concurrently in research of expertise in medical problem solving and in psychometrics.2 Problem-solving ability was no longer conceptualised as a stable entity. Quite the opposite in fact, it was context specific in the extreme, depending crucially on experts’ and learners’ idiosyncratic experiences. The seminal discovery in assessment was that performance on one problem-solving exercise – the Patient Management Problem of the 1960s – was hardly predictive of performance on another exercise, a persistent finding that turned out to be shared across assessment methods. In teasing out variance components, research typically identified person-by-item interaction as the most dominant source of the greatest variance, much greater than the variance associated with the purpose of the measurement, i.e. to discriminate between learners. This implied that to make generalisable inferences, samples had to be large, often unfeasibly so. This Maastricht, the Netherlands
Medical Education 2014: 48: 234–235 doi: 10.1111/medu.12263
234
insight heralded major changes in certification assessment programmes all around the world. The findings from expertise research and psychometrics were replicated in virtually all areas of medical education. Even generic skills, such as professionalism, communication, team performance and leadership proved context specific. Eva2 aptly concluded: ‘context specificity is a profoundly general phenomenon’. Today, context specificity is almost a platitude, but have we really grasped the complexity and magnitude of its theoretical consequences? In psychometric theory, context specificity is dismissed as ‘noise’. But is it? Can we be sure it is ‘noise’ when a doctor fits his actions to the patient or the environment (a hectic emergency department, being on call at night)? What if this is a sign of the doctor’s professional expertise? Durning et al.3 have called for serious study of the multifaceted interactions between the doctor, the patient and the context, based on the assumption that observations may be hard to predict, but nevertheless consistent within certain margins. For example, if we empty a bag of sand on to the ground, we cannot predict where each grain of sand will end up, but we can roughly predict the shape of the heap. The question is do we translate our new understandings back to theory? Has the time come for a new psychometric theory, for probabilistic modelling of outcomes of patient-by-doctor-by-context interactions? Should we explore the nature of contexts? Should we look for a signal in the noise? The assessment community is beginning to endorse the notion that it is safe to rely on professional judgement for capturing complex performances. Correspondence: Cees PM van der Vleuten, Department of Educational Development and Research, Maastricht University, P.O. Box 616, 6200 MD Maastricht, the Netherlands. Tel: 00 31 43 388 5725; E-mail: [email protected]
ª 2014 John Wiley & Sons Ltd. MEDICAL EDUCATION 2014; 48: 234–235
C P M van der Vleuten The new understanding that we are currently documenting is that assessors are not passive, perfectly calibrated measurement instruments, but active agents constructing judgements using information from situational factors and personal experiences.4 From this perspective, different assessors are not expected to make similar judgements, and this may actually be desirable! Indeed, from a learning orientation, different perspectives on performance may be beneficial. Also, we are moving away from scoring by numbers. Complex skills and performances are elusive and need the wealth of information that only narrative can provide. Perhaps we should invoke qualitative methodologies to give meaning to our interpretations. Purposeful sampling, member checking, triangulation and saturation may be sound strategies for making predictions based on complex performances. Should we replace scores by words? So, should we replace psychometric theories with an interpretative theory? Context specificity may be a general phenomenon, but its implications remain to be fully grasped. We will have to study person-by-context interaction and observe its (in)consistencies. We should also re-orientate our theoretical stances to accommodate
context specificity as an integral component. Are we up for the challenge? If it were tomorrow, what would I have written? I wonder…
REFERENCES 1 Ross L, Nisbett RE. The Person and the Situation: Perspectives in Social Psychology. New York, NY: McGraw-Hill 1991. 2 Eva KW. On the generality of specificity. Med Educ 2003;37:587–8. 3 Durning S, Artino AR Jr, Pangaro L, van der Vleuten CPM, Schuwirth L. Context and clinical reasoning: understanding the perspective of the expert’s voice. Med Educ 2011;45:927–38. 4 Govaerts MJ, van der Vleuten CP, Schuwirth LW, Muijtjens AM. Broadening perspectives on clinical performance assessment: rethinking the nature of in-training assessment. Adv Health Sci Educ Theory Pract 2007;12(2):239–60. Received 5 April 2013; editorial comments to author 24 April 2013, accepted for publication 29 April 2013
ª 2014 John Wiley & Sons Ltd. MEDICAL EDUCATION 2014; 48: 234–235
235
People are inclined to think that behaviour, that of others in particular, is governed by stable dispositions. Traits are stable latent entities and if growth happens as a result of experience, this happens continuously over time. Think of intelligence or neuroticism. However, after researchers in personality psychology had been trying to measure traits for over half a century, the realisation dawned that it was not traits but the situation that was the dominant influence on behaviour. This sparked a radical shift to a paradigm of person-bysituation interaction, with traits re-cast as contextdependent states.1 In medical education, the new centrality of context caused quite a stir. Context specificity was discovered more or less concurrently in research of expertise in medical problem solving and in psychometrics.2 Problem-solving ability was no longer conceptualised as a stable entity. Quite the opposite in fact, it was context specific in the extreme, depending crucially on experts’ and learners’ idiosyncratic experiences. The seminal discovery in assessment was that performance on one problem-solving exercise – the Patient Management Problem of the 1960s – was hardly predictive of performance on another exercise, a persistent finding that turned out to be shared across assessment methods. In teasing out variance components, research typically identified person-by-item interaction as the most dominant source of the greatest variance, much greater than the variance associated with the purpose of the measurement, i.e. to discriminate between learners. This implied that to make generalisable inferences, samples had to be large, often unfeasibly so. This Maastricht, the Netherlands
Medical Education 2014: 48: 234–235 doi: 10.1111/medu.12263
234
insight heralded major changes in certification assessment programmes all around the world. The findings from expertise research and psychometrics were replicated in virtually all areas of medical education. Even generic skills, such as professionalism, communication, team performance and leadership proved context specific. Eva2 aptly concluded: ‘context specificity is a profoundly general phenomenon’. Today, context specificity is almost a platitude, but have we really grasped the complexity and magnitude of its theoretical consequences? In psychometric theory, context specificity is dismissed as ‘noise’. But is it? Can we be sure it is ‘noise’ when a doctor fits his actions to the patient or the environment (a hectic emergency department, being on call at night)? What if this is a sign of the doctor’s professional expertise? Durning et al.3 have called for serious study of the multifaceted interactions between the doctor, the patient and the context, based on the assumption that observations may be hard to predict, but nevertheless consistent within certain margins. For example, if we empty a bag of sand on to the ground, we cannot predict where each grain of sand will end up, but we can roughly predict the shape of the heap. The question is do we translate our new understandings back to theory? Has the time come for a new psychometric theory, for probabilistic modelling of outcomes of patient-by-doctor-by-context interactions? Should we explore the nature of contexts? Should we look for a signal in the noise? The assessment community is beginning to endorse the notion that it is safe to rely on professional judgement for capturing complex performances. Correspondence: Cees PM van der Vleuten, Department of Educational Development and Research, Maastricht University, P.O. Box 616, 6200 MD Maastricht, the Netherlands. Tel: 00 31 43 388 5725; E-mail: [email protected]
ª 2014 John Wiley & Sons Ltd. MEDICAL EDUCATION 2014; 48: 234–235
C P M van der Vleuten The new understanding that we are currently documenting is that assessors are not passive, perfectly calibrated measurement instruments, but active agents constructing judgements using information from situational factors and personal experiences.4 From this perspective, different assessors are not expected to make similar judgements, and this may actually be desirable! Indeed, from a learning orientation, different perspectives on performance may be beneficial. Also, we are moving away from scoring by numbers. Complex skills and performances are elusive and need the wealth of information that only narrative can provide. Perhaps we should invoke qualitative methodologies to give meaning to our interpretations. Purposeful sampling, member checking, triangulation and saturation may be sound strategies for making predictions based on complex performances. Should we replace scores by words? So, should we replace psychometric theories with an interpretative theory? Context specificity may be a general phenomenon, but its implications remain to be fully grasped. We will have to study person-by-context interaction and observe its (in)consistencies. We should also re-orientate our theoretical stances to accommodate
context specificity as an integral component. Are we up for the challenge? If it were tomorrow, what would I have written? I wonder…
REFERENCES 1 Ross L, Nisbett RE. The Person and the Situation: Perspectives in Social Psychology. New York, NY: McGraw-Hill 1991. 2 Eva KW. On the generality of specificity. Med Educ 2003;37:587–8. 3 Durning S, Artino AR Jr, Pangaro L, van der Vleuten CPM, Schuwirth L. Context and clinical reasoning: understanding the perspective of the expert’s voice. Med Educ 2011;45:927–38. 4 Govaerts MJ, van der Vleuten CP, Schuwirth LW, Muijtjens AM. Broadening perspectives on clinical performance assessment: rethinking the nature of in-training assessment. Adv Health Sci Educ Theory Pract 2007;12(2):239–60. Received 5 April 2013; editorial comments to author 24 April 2013, accepted for publication 29 April 2013
ª 2014 John Wiley & Sons Ltd. MEDICAL EDUCATION 2014; 48: 234–235
235
