This article was downloaded by: [UZH Hauptbibliothek / Zentralbibliothek Zürich] On: 07 January 2015, At: 02:14 Publisher: Routledge Informa Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK
Teaching and Learning in Medicine: An International Journal Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/htlm20
Theory Development and Application in Medical Education a
b
Anna T. Cianciolo , Kevin W. Eva & Jerry A. Colliver
a
a
Department of Medical Education , Southern Illinois University School of Medicine , Springfield , Illinois , USA b
Department of Medicine , University of British Columbia , Vancouver , British Columbia , Canada Published online: 18 Nov 2013.
To cite this article: Anna T. Cianciolo , Kevin W. Eva & Jerry A. Colliver (2013) Theory Development and Application in Medical Education, Teaching and Learning in Medicine: An International Journal, 25:sup1, S75-S80, DOI: 10.1080/10401334.2013.842907 To link to this article: http://dx.doi.org/10.1080/10401334.2013.842907
PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content. This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http:// www.tandfonline.com/page/terms-and-conditions
Teaching and Learning in Medicine, 25(S1), S75–S80 C 2013, Taylor & Francis Group, LLC Copyright ISSN: 1040-1334 print / 1532-8015 online DOI: 10.1080/10401334.2013.842907
Theory Development and Application in Medical Education Anna T. Cianciolo Department of Medical Education, Southern Illinois University School of Medicine, Springfield, Illinois, USA
Kevin W. Eva Department of Medicine, University of British Columbia, Vancouver, British Columbia, Canada
Downloaded by [UZH Hauptbibliothek / Zentralbibliothek Zürich] at 02:14 07 January 2015
Jerry A. Colliver Department of Medical Education, Southern Illinois University School of Medicine, Springfield, Illinois, USA
The role and status of theory is by no means a new topic in medical education. Yet summarizing where we have been and where we are going with respect to theory development and application is difficult because our community has not yet fully elucidated what constitutes medical education theory. In this article, we explore the idea of conceptualizing theory as an effect on scholarly dialogue among medical educators. We describe theory-enabled conversation as argumentation, which frames inquiry, permits the evaluation of evidence, and enables the acquisition of community understanding that has utility beyond investigators’ local circumstances. We present ideas for assessing argumentation quality and suggest approaches to increasing the frequency and quality of argumentation in the exchange among diverse medical education scholars. Keywords
theory development, theory application
INTRODUCTION Because medical education seeks to promote the development of health care professionals, an important problem for our field to solve is that of producing consistent and generalizable effects on learning and performance.1,2 That is, we have one indication our field is advancing when we understand how to adapt the characteristics of a given educational activity to diverse situational conditions with beneficial outcomes.3,4 This is an important exercise in a complex domain such as education, where causal relationships are inevitably probabilistic rather than absolute, and preserving an effect (e.g., higher levels of thinking, more skilled performance) involves sophisticated modification of implementation design in response to
Correspondence may be sent to Anna T. Cianciolo, Department of Medical Education, SIU School of Medicine, 913 North Rutledge Street, Room 1205A, Springfield, IL 62794-9681, USA. E-mail: [email protected]
contextual factors.3 Understanding how the findings and conclusions of scholarly work apply across variable circumstances requires theory, an explication of the nature of phenomena, their observable characteristics, and situational constraints on their expression. The role and status of theory in medical education is by no means a new topic.2,4–8 Calls for greater rigor in research design, conduct, and analysis so as to enhance theory are numerous and persistent in our field.4,9–12 Yet dialogue among medical educators about effects and their probabilistic causes can be nebulous, lacking the structure of shared or integrated conceptual models that enable the growth of sophisticated explanations.10,13 For example, a much-discussed topic in our literature has been the effectiveness of problem-based learning (PBL), yet many years passed before the field began to turn from tests of differences between PBL and conventional forms of curriculum delivery to research dedicated to answering questions such as, What works in PBL? Why does it work? How can it be improved?14,15 Delving deeper into such questions prompts our community to think more critically about what we are trying to achieve with instruction and promotes exploration of how different circumstances, free of a method label, shape the accomplishment, and demonstration of learning. As a further example, a recent article illustrates how even the conceptual clarity of clinical reasoning, a subject studied for decades due to its centrality to medical practice could be improved by incorporating diverse perspectives into an integrated narrative about what constitutes this phenomenon and how it is observed.16 Summarizing where we have been and where we are going with respect to theory development and application is difficult because our community has not yet fully elucidated what constitutes medical education theory1,2,5 and because the term “theory” often carries undesirable baggage implying “ivory tower” research that has little practical relevance. To overcome these problems requires consideration of many questions, including, What is the scope and content of theory in our field? To what
S75
Downloaded by [UZH Hauptbibliothek / Zentralbibliothek Zürich] at 02:14 07 January 2015
S76
A. T. CIANCIOLO, K. W. EVA, J. A. COLLIVER
degree is it distinct from educational or behavioral theory more generally? How do we know when it is sufficiently complete? This theoretical identity crisis persists at a time when phenomena in medical education have become increasingly complex. Interprofessional teams, information technology, and patient involvement, among other things, are now recognized as central to effective health care. The pursuit of comprehensive, mechanistic understanding of phenomena important to medical education must be broader in scope than a single investigators’ research program, context of study, or mode of inquiry.17 The perspectives that medical educators bring to research vary widely, as do the answers they seek and the tools they use.18 The outcome of theoretical pursuit will continuously evolve as new tools, analytical methods, and educational conditions emerge. Envisioning theory in medical education must involve conceptualizing theory development and application as a collective endeavor, one that fosters dialogue among scholars that is centered, articulate, and productive.5,17 In this article, we explore the idea of conceptualizing theory as an effect on scholarly dialogue among medical educators, as an integrated narrative through which the community evaluates whether the findings of scholarly work are logically consistent and reveal holes in shared understanding. We describe theoryenabled conversation as argumentation, which frames inquiry, permits the evaluation of evidence, and enables the acquisition of sophisticated community understanding. By “argumentation” we do not mean to imply the existence of winners and losers, or that efforts to prove one theory right and others wrong is a productive way forward. Rather, we see a diversity of perspectives and understandings as necessary to provide unique and meaningful insights into the complex problems relevant to educating health professionals.19 Argument is a useful analogy when we conceptualize theory development as the result of testing whether premises (i.e., empirical data) give good reason to believe that a conclusion (e.g., the adoption of particular educational practices) is useful and under what circumstances. Finally, we present ideas for assessing argumentation quality and suggest approaches to increasing the frequency and quality of argumentation in the exchange among diverse medical educators.
WHAT DO WE MEAN BY “THEORY”? At its most basic level, theory is an explanation. It is an account of a phenomenon: its nature, its causes, its function, and its observable properties. It is “an organized, coherent, and systematic articulation of a set of issues that are communicated as a meaningful whole.”8 This articulation specifies the relationships among theoretical constructs in a way that is relevant to application.20 According to this perspective, a hallmark of good theoretical explanation is functional relevance. For example, a theoretical understanding of clinical reasoning is expected to account for variation in case, setting, and physician so as to provide guidance regarding what might be tried to improve practice
across these factors.16 This is not to say that theory must provide absolute predictive capacity for it to be useful. Rather, for a theory to be valuable it needs to yield confidence that one knows something actionable about the world. To return to the example of clinical reasoning, theoretical explanations should enable educators to more precisely assess and accelerate medical students’ attunement to patterns of signs and symptoms and base rates of illness, even though the probabilistic nature of human behavior will prevent any intervention from being successful for every student in every circumstance.16 Theory may also be viewed as a set of beliefs that frame one’s approach to inquiry.4,7,8,21,22 Theory shapes one’s conception of the phenomenon of interest, the questions one seeks to answer, the methods of inquiry chosen, the selection and interpretation of findings, and so forth. Indeed, it may affect investigation in important and often unrecognized ways.18,21,23 For example, studies of professionalism that seek to account for disciplinary actions may focus on quantifying the impact of various antecedents of such actions.24 In contrast, studies of this same construct that seek to account for how attitudes about professional behavior develop may focus on qualitatively investigating the drivers of different belief systems.25 Professionalism may look different from various perspectives, and as a result, research findings may inform diverse aspects of educational practice without trying to determine a “winner” to resolve theoretical tensions. Indeed, when integrated, diverse findings and perspectives may illuminate the multifaceted nature of professional behavior, its development and expression, and its environmental influences, providing a “field’s-eye” view of this phenomenon. In this way, diversity of theory can be considered a particular strength of our field.26 Theory, then, may be thought of as an explanation of a phenomenon that is bounded by the lens through which the inquiry was conducted. Scholarly exchange about a phenomenon is promoted when the necessary evidence to justify the explanation is articulated and the framework of inquiry illuminates the explanation’s boundary conditions and applicability.17 Improving the comprehensiveness and scalability of an explanation—theory development—depends on how effectively the community integrates inquiry frameworks to achieve practical relevance. INVESTIGATION AS ARGUMENTATION How, then, should we lay an argument out, if we want to show the sources of its validity? And in what sense does the acceptability or unacceptability of arguments depend upon their “formal” merits and defects?—Stephen E. Toulmin
Regardless of a scholar’s explicit intention, an investigation reflects a collection of positions. Positions on the nature of knowledge and truth (e.g., truth is socially constructed) influence positions on methodology (e.g., ethnographic study captures a truth in a given context).18 Positions on the nature of the variables studied and their interrelation are reflected in the conceptual models that shape or result from the investigation.20 Data are
S77
THEORY IN MEDICAL EDUCATION
Qualifier
A. Except when So, presumably Data
Claim
Because Unless
Warrant On the basis of
Downloaded by [UZH Hauptbibliothek / Zentralbibliothek Zürich] at 02:14 07 January 2015
Rebual Backing
B. Cross-Cultural Interacon Except when Increased Communicaon Rangs
So, presumably
Effecve Intervenon
Because Focus on Communicaon Behavior Literature Review
FIG. 1.
On the basis of
Unless Other Developmental Opportunies
Qualitave Analysis
An argumentation framework (based on Toulmin, 1958).
gathered in accordance with positions and used as evidence to justify their existence. Positions even influence the criteria used to evaluate the validity of evidence. The more explicit a scholar is in stating and defending his or her positions (i.e., making an argument), the more useful the investigation is for building an integrated narrative about the phenomenon studied and the easier it is for others to determine how the effort might be adapted effectively for local use.27,28 This specificity gives the scholarly community a shared language for discussing phenomena, for collectively determining which explanations stand and which must be revised or discarded altogether. In this way, the adoption of common language evolves into shared explanations.29 Put another way, the sound of scholarly community building, of constructing a shared understanding of “an organized, coherent, and systematic articulation of a set of issues,” is the sound of argumentation. It is the call of “How do you know?” and the response of “Here is my evidence.” Although argument emerges from ongoing community activity, it need not be disorderly. Arguments are in fact structured, and understanding this structure serves as a way of organizing one’s thoughts in advance; it enables defenders to anticipate challenges and provides recognizable opportunities for challengers to question defenders’ positions. When conceptualizing
theory as argument, it is useful to understand the components of argumentation and their interplay so as to better understand how to engage effectively in scholarly dialogue and how to recognize quality contributions from others. Argument frameworks help us to recognize the sound of an argument in ongoing scholarly dialogue and give us the language we need to contribute to the discussion. One framework30 commonly used by educational scholars to represent and assess argumentation is illustrated in Figure 1. As shown in Figure 1, the foundation of an argument is the assertion of a claim and the presentation of evidence, or data, to justify the validity of this claim. For example (illustrated in Panel B of Figure 1), an investigator may claim that a particular intervention improved physician–patient communication and may present increased standardized patient ratings of communication skills following the intervention as data. The link between a claim and its supporting data is one focal point for challenging the argument. Challengers may ask, How can we conclude that the intervention (and not something else) caused the ratings to improve? Warrants provide this justification by indicating why the data support the claim. For example, the investigator may assert that the intervention was conceptually based and that it was designed specifically to increase the frequency of behavior
Downloaded by [UZH Hauptbibliothek / Zentralbibliothek Zürich] at 02:14 07 January 2015
S78
A. T. CIANCIOLO, K. W. EVA, J. A. COLLIVER
thought to be relevant to effective communication. Backing justifies a warrant by providing supporting evidence for its validity. The investigator in our example may present a literature review of extant communication theory and data from a qualitative analysis demonstrating the exercise of communication behavior during the intervention as evidence of the strength of the conceptual basis adopted. Qualifiers delineate when a data-claim linkage may not hold, despite a valid warrant. The conceptually based intervention may not produce improved ratings, for instance, if postintervention performance requires behaviors not addressed by the conceptual model, such as dealing with language barriers or other cultural differences.31 Rebuttals articulate the possible conditions under which a warrant fails to justify a data-claim linkage. In our example, a challenger may assert that improved ratings reflect a successful intervention unless the trainees had other developmental opportunities to interact with patients between pre- and postintervention performance assessment and, thus, may have developed their skills independently of the intervention.
ARGUMENT QUALITY Investigation that promotes theory development can be thought of as featuring the components of argument, making their interplay clear and providing supporting evidence for their validity. Assessment approaches based on the argumentation framework described in this article30 have been applied to evaluating whether dialogue contains argument components (i.e., whether exchange can be considered argumentation) and, if so, how comprehensively the argument structure is contained therein.27,29 Published recommendations for reporting medical education research suggest how this might look in experimental and quasi-experimental studies.10 Specifically, these recommendations include clearly stating the study intent and design to explicitly indicate the claim the investigator expects to make and the data he or she intends to produce to support this claim. These recommendations also include presenting a literature review and conceptual framework, which articulate warrants and their backing. Finally, these recommendations assert the importance of clearly specifying the experimental intervention and the comparison case, which brings to light the qualifiers or conditions under which the findings and conclusions of the experiment hold. Adhering to recommendations such as these promotes the relevance of research findings to others, even as they modify the conditions of an investigation to their local circumstances. However, not all components of a high-quality argument may be exhaustively present in one study.30 Indeed, no one investigation can fully address all positions possible in a comprehensive argument. For instance, an ethnographic study may use a detailed set of observations to generate a claim, whereas refinement of this data-claim linkage becomes the topic of further research that constructs and tests a more comprehensive argument. A single experiment may test data-claim linkages without
explicitly investigating qualifiers. Aggregating findings across studies into a more sophisticated whole, carefully considering if and how the pieces fit together (and where gaps in the puzzle still remain), and evaluating the practical relevance of the explanation that results, in our opinion, constitutes “theory development” and illustrates the value inherent in considering theory explicitly—argument quality can only be judged when all of the pieces are considered cumulatively. Another complexity is that the components of argumentation can be difficult to identify definitively; arguments may be multilayered and what constitutes a claim in one sense may serve as a warrant or backing in another.29 As a result, difficulties arise when trying to develop a reliable, consistently applied structural assessment framework that employs a “checklist” approach. Simply assessing the frequency and comprehensiveness of a single argument’s components fails to fully capture the essence of productive argumentation.20,26,32 Rather, judgment that is informed by the state of the field is required, as it is in other domains.33 Assessment of argumentation quality, therefore, should emphasize the potential impact of investigations on fostering shared, integrated conceptual models.13 A quality argument becomes embedded in ongoing scholarly dialogue, and its thesis and evidence advances the dialogue in substantive ways and fosters the dialogue’s evolution by providing a means to evaluate ideas through application. In this light, quality argumentation is seen when the design and reporting of investigations demonstrate awareness of alternative positions and counterarguments that could be posed by other community members.29 This awareness is reflected in the production of evidence that rebuts, integrates, or synthesizes these differing perspectives.34 It also is reflected in appeals to causal mechanisms as evidence versus analogies or metaphors. The practice of anticipating challengers has been termed “anticipatory dialogue”28 and may be considered forethought regarding “What’s next?”—a question that is generally taken to be the hallmark of scholarly dialogue and “knowledge-building conversation.”35 The production of evidence and appeals to causal mechanisms assumes a scholarly dialogue that has the evaluation of validity at its core, rather than simply the exchange of unsupported claims. Critiques noting the lack of conceptual models present in medical education publications10,13 suggest that we must continue to work to foster more effective argumentation in our scholarly dialogue. A lack of conceptual models is one indication that investigators interested in the same phenomena are not engaged in exchange because they are not referring to what each other has said or what the community has come to think. The Durning et al. article mentioned earlier illustrates, alternatively, how quality collaborative argumentation can look.16 In this article, the authors explore the epistemological orientations (objectivist and constructivist) and learning and assessment perspectives (behaviorist, information processing, psychometric, and situative) from which clinical reasoning has been studied and highlight the implications of an integrated perspective for
THEORY IN MEDICAL EDUCATION
Downloaded by [UZH Hauptbibliothek / Zentralbibliothek Zürich] at 02:14 07 January 2015
understanding the interplay of context and mental processes in the achievement of diagnoses. Promoting Quality Argumentation Conceptualizing theory as enabling argumentation in scholarly dialogue allows us to distinguish between rigorous, theorybuilding scholarship, on one hand, and particular modes of inquiry, on the other. In other words, one investigatory approach (e.g., qualitative or quantitative) or type of answer sought (“basic” or “applied”) is not superior to another as long as the work conducted in either instance enables collaborative argumentation among community members and the development of sophisticated, actionable explanations.1,17,20 Understanding the structure of argumentation helps to clarify how different investigations can contribute to theory. Different modes of inquiry, such as case studies, laboratory experiments, ethnographic analysis, quasi-experiments, correlational studies, and so forth, may target different components of argumentation and variously serve to generate, test, and/or bound theory. Laboratory studies, for instance, commonly test competing claims made about a given data set. Qualifiers that bound the generalizability and applicability of laboratory findings may be discovered when explanations produced by experiments are tested in quasi-experimental or other applied studies.36 Community understanding becomes more sophisticated as these boundaries take shape. Qualitative investigation helps to generate theory by producing detailed data about which various claims may be made and tested. It may also support theory testing and refinement by providing the evidence necessary to justify or question warrants supporting data-claim linkages. This objective, explicit or otherwise, underlies many mixed-method studies published in medical education journals. Quality argumentation in our field, then, is not fostered by choosing the “correct” method or positing the “right” kind of answer.37 Rather, quality is fostered through the practical exercise of working and reworking one’s scholarship with attention to how it contributes to collaborative argumentation, specifically how it refutes or defends various components of shared understanding versus simply stating an alternative claim.27,28 The exercise of argumentation as continuous professional development parallels the formal learning process for behavioral scientists, where a thesis is developed, refined, and defended over a long period of focused effort and the field-dependent criteria for evaluating evidence are acquired through close work with intellectual mentors. For educators interested in participating in medical education research, pairing with colleagues who have behavioral science degrees may provide just such a mentor.12 Routine and informal peer review by such colleagues is critical to identifying weakness in the positions one has taken, and the diversity of behavioral scientists involved in medical education stimulates the construction of sophisticated explanations of the phenomenon of interest.26 Argumentation process and argument content are only acquired over a considerable period of immersion and benefits
S79
greatly from the advisement of colleagues.38 Here, reviewers (formal and informal) become critically important stewards of knowledge-building conversations by providing thoughtful and constructive feedback on the quality of argumentation in the manuscripts they read and by promoting authors’ awareness of and engagement with the scholarly community in their work.20,27,28,39 To summarize, the value of theory has been discussed extensively in our field over the past two and a half decades as we have sought to continuously improve teaching and learning in medicine. Attempts to understand what constitutes theory in our field fail when we conceptualize theory as a single, comprehensive explanation that is perfectly predictive under all circumstances. The phenomena of interest to medical educators are complex and evolving and causal relationships are inevitably probabilistic. As an alternative, we have proposed conceptualizing theory as an effect on scholarly dialogue, one that enables the acquisition of community understanding that has utility beyond investigators’ local circumstances. This effect may be described as argumentation, which is structured in a way that fosters productive interaction among interlocutors, permits evaluation of evidence, and leads to the construction of sophisticated explanations. Assessing and fostering argument quality is a community process characterized by the formal and informal collaboration of medical educators and a diversity of behavioral scientists. Such collaboration is invaluable even as it makes our collective work more challenging as we grow shared understanding together.
REFERENCES 1. Colliver JA. Constructivism with a dose of pragmatism: A cure for what ails educational research. Advances in Health Science Education 1999;4:187–90. 2. Colliver JA. Full-curriculum interventions and small-scale studies of transferImplications for psychology-type theory. Medical Education 2004;38:1212–18. 3. Yeager DS, Walton GM. Social-psychological interventions in education: They’re not magic. Review of Educational Research 2011;81:267–301. 4. Regehr G. It’s NOT rocket science: Rethinking our metaphors for research in health professions education. Medical Education 2010;44:31–9. 5. Colliver JA. Educational theory and medical education practice: A cautionary note for medical school faculty. Academic Medicine 2002;77:1217–20. 6. Colliver JA, Conlee MJ, Verhulst SJ. From test validity to construct validity . . . and back?. Medical Education 2012;46:366–71. 7. Rees CE, Monrouxe LV. Theory in medical education research: How do we get there?. Medical Education 2010;44:334–9. 8. Reeves S, Mathieu A, Kuper A, Hodges BD. Why use theories in qualitative research?. British Medical Journal 2008;337:631–4. 9. Albert M, Hodges B, Regehr G. Research in medical education: Balancing service and science. Advances in Health Sciences Education: Theory and Practice 2007; 12:103–15. 10. Cook DA, Beckman TJ, Bordage G. Quality of reporting of experimental studies in medical education: A systematic review. Medical Education 2007;41:737–45. 11. Markert RJ. Enhancing medical education by improving statistical methodology in journal articles. Teaching and Learning in Medicine 2013;2592:159–64.
Downloaded by [UZH Hauptbibliothek / Zentralbibliothek Zürich] at 02:14 07 January 2015
S80
A. T. CIANCIOLO, K. W. EVA, J. A. COLLIVER
12. Rogers DA. Advancing surgical education scholarship. American Journal of Surgery 2011;201:1–6. 13. Bordage G. Conceptual frameworks to illuminate and magnify. Medical Education 2009;43:312–9. 14. Rotgans JI. The themes, institutions, and people of medical education research 1988–2010: Content analysis of abstracts from six journals. Advances in Health Science Education 2012;17:515–7. 15. Schmidt HG, Rotgans JI, Yew EH. The process of problem-based learning: What works and why. Medical Education 2011;45:792–806. 16. Durning SJ, Artino Jr AR, Schuwirth L, van der Vleuten C. Clarifying assumptions to enhance our understanding and assessment of clinical reasoning. Academic Medicine 2013;88(4):1–7. 17. Eva KW. The State of the Science 2012: Building blocks for the future. Medical Education 2012;46:1–2. 18. Bunniss S, Kelly DR. Research paradigms in medical education research. Medical Education 2010;44:358–66. 19. Eva KW. Dialogue in medical education: Enabling the academic voyeur that lurks inside us all. Medical Education 2012;46:826–7. 20. Colliver JA. Commentary on “Review Criteria for Research Manuscripts.”. Teaching and Learning in Medicine 2002;14:75–6. 21. Denzin NK, Lincoln YS. Introduction: Entering the field of qualitative research. In NK Denzin & YS Lincoln (Eds.), Handbook of qualitative research (pp. 1–17). Thousand Oaks, CA: Sage, 1994. 22. Giacomini M. Theory matters in qualitative research. In I Bourgeault, R Dingwall, R De Vries (Eds.), The SAGE handbook of qualitative methods in health research (pp. 125–156). Los Angeles: Sage, 2010. 23. Gould SJ. The mismeasure of man. Rev. ed. New York: W. W. Norton, 1996. 24. Teherani A, Hodgson C, Banach M, Papadakis M. Domains of unprofessional behavior during medical school associated with future disciplinary action by a state medical board. Academic Medicine 2005;80(10 Suppl):17–20. 25. Ginsburg S, Bernabeo E, Ross KM, Holmboe ES. “It depends”: Results of a qualitative study investigating how practicing internists approach professional dilemmas. Academic Medicine 2012;87:1685–93.
26. Norman G. Fifty years of medical education research: Waves of migration. Medical Education 2011;45:785–91. 27. Erduran S, Simon S, Osborne J. Tapping into argumentation: Developments in the application of Toulmin’s argument pattern for studying science discourse. Science Education 2004;88:915–33. 28. Kuhn D. Teaching and learning science as argument. Science Education 2010;94:810–24. 29. Duschl RA. Quality argumentation and epistemic criteria. In S Erduran, MP Jim´enez-Aleixandre (Eds.), Argumentation in science education (pp. 159–75). Dordrecht: Springer, 2008. 30. Toulmin SE. The uses of argument. Cambridge, England: Cambridge University Press, 1958. 31. Betancourt JR, Carrillo JE, Green AR. Hypertension in multicultural and minority populations: Linking communication to compliance. Current Hypertension Reports 1999;1:482–8. 32. Eva KW. On the limits of systematicity. Medical Education 2008;42: 852–3. 33. Eva KW, Hodges BD. Scylla or Charybdis? Can we navigate between objectification and judgement in assessment?. Medical Education 2012;46:914–9. 34. Nussbaum EM, Schraw G. Promoting argument-counterargument integration in students’ writing. Journal of Experimental Education 2007;76:59–92. 35. Eva KW, Lingard L. What’s next? A guiding question for educators engaged in educational research. Medical Education 2008;42:752–4. 36. Chapanis A. The relevance of laboratory studies to practical situations. Ergonomics 1967;10:557–77. 37. Eva KW. Broadening the debate about quality in medical education research. Medical Education 2009;43:294–6. 38. O’Sullivan PS, Stoddard HA, Kalishman S. Collaborative research in medical education: A discussion of theory and practice. Medical Education 2010;44:1175–84. 39. Eva KW. The reviewer is always right: Peer review of research in medical education. Medical Education 2009;43:2–4.
Teaching and Learning in Medicine: An International Journal Publication details, including instructions for authors and subscription information: http://www.tandfonline.com/loi/htlm20
Theory Development and Application in Medical Education a
b
Anna T. Cianciolo , Kevin W. Eva & Jerry A. Colliver
a
a
Department of Medical Education , Southern Illinois University School of Medicine , Springfield , Illinois , USA b
Department of Medicine , University of British Columbia , Vancouver , British Columbia , Canada Published online: 18 Nov 2013.
To cite this article: Anna T. Cianciolo , Kevin W. Eva & Jerry A. Colliver (2013) Theory Development and Application in Medical Education, Teaching and Learning in Medicine: An International Journal, 25:sup1, S75-S80, DOI: 10.1080/10401334.2013.842907 To link to this article: http://dx.doi.org/10.1080/10401334.2013.842907
PLEASE SCROLL DOWN FOR ARTICLE Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in the publications on our platform. However, Taylor & Francis, our agents, and our licensors make no representations or warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Any opinions and views expressed in this publication are the opinions and views of the authors, and are not the views of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should be independently verified with primary sources of information. Taylor and Francis shall not be liable for any losses, actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoever caused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content. This article may be used for research, teaching, and private study purposes. Any substantial or systematic reproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in any form to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http:// www.tandfonline.com/page/terms-and-conditions
Teaching and Learning in Medicine, 25(S1), S75–S80 C 2013, Taylor & Francis Group, LLC Copyright ISSN: 1040-1334 print / 1532-8015 online DOI: 10.1080/10401334.2013.842907
Theory Development and Application in Medical Education Anna T. Cianciolo Department of Medical Education, Southern Illinois University School of Medicine, Springfield, Illinois, USA
Kevin W. Eva Department of Medicine, University of British Columbia, Vancouver, British Columbia, Canada
Downloaded by [UZH Hauptbibliothek / Zentralbibliothek Zürich] at 02:14 07 January 2015
Jerry A. Colliver Department of Medical Education, Southern Illinois University School of Medicine, Springfield, Illinois, USA
The role and status of theory is by no means a new topic in medical education. Yet summarizing where we have been and where we are going with respect to theory development and application is difficult because our community has not yet fully elucidated what constitutes medical education theory. In this article, we explore the idea of conceptualizing theory as an effect on scholarly dialogue among medical educators. We describe theory-enabled conversation as argumentation, which frames inquiry, permits the evaluation of evidence, and enables the acquisition of community understanding that has utility beyond investigators’ local circumstances. We present ideas for assessing argumentation quality and suggest approaches to increasing the frequency and quality of argumentation in the exchange among diverse medical education scholars. Keywords
theory development, theory application
INTRODUCTION Because medical education seeks to promote the development of health care professionals, an important problem for our field to solve is that of producing consistent and generalizable effects on learning and performance.1,2 That is, we have one indication our field is advancing when we understand how to adapt the characteristics of a given educational activity to diverse situational conditions with beneficial outcomes.3,4 This is an important exercise in a complex domain such as education, where causal relationships are inevitably probabilistic rather than absolute, and preserving an effect (e.g., higher levels of thinking, more skilled performance) involves sophisticated modification of implementation design in response to
Correspondence may be sent to Anna T. Cianciolo, Department of Medical Education, SIU School of Medicine, 913 North Rutledge Street, Room 1205A, Springfield, IL 62794-9681, USA. E-mail: [email protected]
contextual factors.3 Understanding how the findings and conclusions of scholarly work apply across variable circumstances requires theory, an explication of the nature of phenomena, their observable characteristics, and situational constraints on their expression. The role and status of theory in medical education is by no means a new topic.2,4–8 Calls for greater rigor in research design, conduct, and analysis so as to enhance theory are numerous and persistent in our field.4,9–12 Yet dialogue among medical educators about effects and their probabilistic causes can be nebulous, lacking the structure of shared or integrated conceptual models that enable the growth of sophisticated explanations.10,13 For example, a much-discussed topic in our literature has been the effectiveness of problem-based learning (PBL), yet many years passed before the field began to turn from tests of differences between PBL and conventional forms of curriculum delivery to research dedicated to answering questions such as, What works in PBL? Why does it work? How can it be improved?14,15 Delving deeper into such questions prompts our community to think more critically about what we are trying to achieve with instruction and promotes exploration of how different circumstances, free of a method label, shape the accomplishment, and demonstration of learning. As a further example, a recent article illustrates how even the conceptual clarity of clinical reasoning, a subject studied for decades due to its centrality to medical practice could be improved by incorporating diverse perspectives into an integrated narrative about what constitutes this phenomenon and how it is observed.16 Summarizing where we have been and where we are going with respect to theory development and application is difficult because our community has not yet fully elucidated what constitutes medical education theory1,2,5 and because the term “theory” often carries undesirable baggage implying “ivory tower” research that has little practical relevance. To overcome these problems requires consideration of many questions, including, What is the scope and content of theory in our field? To what
S75
Downloaded by [UZH Hauptbibliothek / Zentralbibliothek Zürich] at 02:14 07 January 2015
S76
A. T. CIANCIOLO, K. W. EVA, J. A. COLLIVER
degree is it distinct from educational or behavioral theory more generally? How do we know when it is sufficiently complete? This theoretical identity crisis persists at a time when phenomena in medical education have become increasingly complex. Interprofessional teams, information technology, and patient involvement, among other things, are now recognized as central to effective health care. The pursuit of comprehensive, mechanistic understanding of phenomena important to medical education must be broader in scope than a single investigators’ research program, context of study, or mode of inquiry.17 The perspectives that medical educators bring to research vary widely, as do the answers they seek and the tools they use.18 The outcome of theoretical pursuit will continuously evolve as new tools, analytical methods, and educational conditions emerge. Envisioning theory in medical education must involve conceptualizing theory development and application as a collective endeavor, one that fosters dialogue among scholars that is centered, articulate, and productive.5,17 In this article, we explore the idea of conceptualizing theory as an effect on scholarly dialogue among medical educators, as an integrated narrative through which the community evaluates whether the findings of scholarly work are logically consistent and reveal holes in shared understanding. We describe theoryenabled conversation as argumentation, which frames inquiry, permits the evaluation of evidence, and enables the acquisition of sophisticated community understanding. By “argumentation” we do not mean to imply the existence of winners and losers, or that efforts to prove one theory right and others wrong is a productive way forward. Rather, we see a diversity of perspectives and understandings as necessary to provide unique and meaningful insights into the complex problems relevant to educating health professionals.19 Argument is a useful analogy when we conceptualize theory development as the result of testing whether premises (i.e., empirical data) give good reason to believe that a conclusion (e.g., the adoption of particular educational practices) is useful and under what circumstances. Finally, we present ideas for assessing argumentation quality and suggest approaches to increasing the frequency and quality of argumentation in the exchange among diverse medical educators.
WHAT DO WE MEAN BY “THEORY”? At its most basic level, theory is an explanation. It is an account of a phenomenon: its nature, its causes, its function, and its observable properties. It is “an organized, coherent, and systematic articulation of a set of issues that are communicated as a meaningful whole.”8 This articulation specifies the relationships among theoretical constructs in a way that is relevant to application.20 According to this perspective, a hallmark of good theoretical explanation is functional relevance. For example, a theoretical understanding of clinical reasoning is expected to account for variation in case, setting, and physician so as to provide guidance regarding what might be tried to improve practice
across these factors.16 This is not to say that theory must provide absolute predictive capacity for it to be useful. Rather, for a theory to be valuable it needs to yield confidence that one knows something actionable about the world. To return to the example of clinical reasoning, theoretical explanations should enable educators to more precisely assess and accelerate medical students’ attunement to patterns of signs and symptoms and base rates of illness, even though the probabilistic nature of human behavior will prevent any intervention from being successful for every student in every circumstance.16 Theory may also be viewed as a set of beliefs that frame one’s approach to inquiry.4,7,8,21,22 Theory shapes one’s conception of the phenomenon of interest, the questions one seeks to answer, the methods of inquiry chosen, the selection and interpretation of findings, and so forth. Indeed, it may affect investigation in important and often unrecognized ways.18,21,23 For example, studies of professionalism that seek to account for disciplinary actions may focus on quantifying the impact of various antecedents of such actions.24 In contrast, studies of this same construct that seek to account for how attitudes about professional behavior develop may focus on qualitatively investigating the drivers of different belief systems.25 Professionalism may look different from various perspectives, and as a result, research findings may inform diverse aspects of educational practice without trying to determine a “winner” to resolve theoretical tensions. Indeed, when integrated, diverse findings and perspectives may illuminate the multifaceted nature of professional behavior, its development and expression, and its environmental influences, providing a “field’s-eye” view of this phenomenon. In this way, diversity of theory can be considered a particular strength of our field.26 Theory, then, may be thought of as an explanation of a phenomenon that is bounded by the lens through which the inquiry was conducted. Scholarly exchange about a phenomenon is promoted when the necessary evidence to justify the explanation is articulated and the framework of inquiry illuminates the explanation’s boundary conditions and applicability.17 Improving the comprehensiveness and scalability of an explanation—theory development—depends on how effectively the community integrates inquiry frameworks to achieve practical relevance. INVESTIGATION AS ARGUMENTATION How, then, should we lay an argument out, if we want to show the sources of its validity? And in what sense does the acceptability or unacceptability of arguments depend upon their “formal” merits and defects?—Stephen E. Toulmin
Regardless of a scholar’s explicit intention, an investigation reflects a collection of positions. Positions on the nature of knowledge and truth (e.g., truth is socially constructed) influence positions on methodology (e.g., ethnographic study captures a truth in a given context).18 Positions on the nature of the variables studied and their interrelation are reflected in the conceptual models that shape or result from the investigation.20 Data are
S77
THEORY IN MEDICAL EDUCATION
Qualifier
A. Except when So, presumably Data
Claim
Because Unless
Warrant On the basis of
Downloaded by [UZH Hauptbibliothek / Zentralbibliothek Zürich] at 02:14 07 January 2015
Rebual Backing
B. Cross-Cultural Interacon Except when Increased Communicaon Rangs
So, presumably
Effecve Intervenon
Because Focus on Communicaon Behavior Literature Review
FIG. 1.
On the basis of
Unless Other Developmental Opportunies
Qualitave Analysis
An argumentation framework (based on Toulmin, 1958).
gathered in accordance with positions and used as evidence to justify their existence. Positions even influence the criteria used to evaluate the validity of evidence. The more explicit a scholar is in stating and defending his or her positions (i.e., making an argument), the more useful the investigation is for building an integrated narrative about the phenomenon studied and the easier it is for others to determine how the effort might be adapted effectively for local use.27,28 This specificity gives the scholarly community a shared language for discussing phenomena, for collectively determining which explanations stand and which must be revised or discarded altogether. In this way, the adoption of common language evolves into shared explanations.29 Put another way, the sound of scholarly community building, of constructing a shared understanding of “an organized, coherent, and systematic articulation of a set of issues,” is the sound of argumentation. It is the call of “How do you know?” and the response of “Here is my evidence.” Although argument emerges from ongoing community activity, it need not be disorderly. Arguments are in fact structured, and understanding this structure serves as a way of organizing one’s thoughts in advance; it enables defenders to anticipate challenges and provides recognizable opportunities for challengers to question defenders’ positions. When conceptualizing
theory as argument, it is useful to understand the components of argumentation and their interplay so as to better understand how to engage effectively in scholarly dialogue and how to recognize quality contributions from others. Argument frameworks help us to recognize the sound of an argument in ongoing scholarly dialogue and give us the language we need to contribute to the discussion. One framework30 commonly used by educational scholars to represent and assess argumentation is illustrated in Figure 1. As shown in Figure 1, the foundation of an argument is the assertion of a claim and the presentation of evidence, or data, to justify the validity of this claim. For example (illustrated in Panel B of Figure 1), an investigator may claim that a particular intervention improved physician–patient communication and may present increased standardized patient ratings of communication skills following the intervention as data. The link between a claim and its supporting data is one focal point for challenging the argument. Challengers may ask, How can we conclude that the intervention (and not something else) caused the ratings to improve? Warrants provide this justification by indicating why the data support the claim. For example, the investigator may assert that the intervention was conceptually based and that it was designed specifically to increase the frequency of behavior
Downloaded by [UZH Hauptbibliothek / Zentralbibliothek Zürich] at 02:14 07 January 2015
S78
A. T. CIANCIOLO, K. W. EVA, J. A. COLLIVER
thought to be relevant to effective communication. Backing justifies a warrant by providing supporting evidence for its validity. The investigator in our example may present a literature review of extant communication theory and data from a qualitative analysis demonstrating the exercise of communication behavior during the intervention as evidence of the strength of the conceptual basis adopted. Qualifiers delineate when a data-claim linkage may not hold, despite a valid warrant. The conceptually based intervention may not produce improved ratings, for instance, if postintervention performance requires behaviors not addressed by the conceptual model, such as dealing with language barriers or other cultural differences.31 Rebuttals articulate the possible conditions under which a warrant fails to justify a data-claim linkage. In our example, a challenger may assert that improved ratings reflect a successful intervention unless the trainees had other developmental opportunities to interact with patients between pre- and postintervention performance assessment and, thus, may have developed their skills independently of the intervention.
ARGUMENT QUALITY Investigation that promotes theory development can be thought of as featuring the components of argument, making their interplay clear and providing supporting evidence for their validity. Assessment approaches based on the argumentation framework described in this article30 have been applied to evaluating whether dialogue contains argument components (i.e., whether exchange can be considered argumentation) and, if so, how comprehensively the argument structure is contained therein.27,29 Published recommendations for reporting medical education research suggest how this might look in experimental and quasi-experimental studies.10 Specifically, these recommendations include clearly stating the study intent and design to explicitly indicate the claim the investigator expects to make and the data he or she intends to produce to support this claim. These recommendations also include presenting a literature review and conceptual framework, which articulate warrants and their backing. Finally, these recommendations assert the importance of clearly specifying the experimental intervention and the comparison case, which brings to light the qualifiers or conditions under which the findings and conclusions of the experiment hold. Adhering to recommendations such as these promotes the relevance of research findings to others, even as they modify the conditions of an investigation to their local circumstances. However, not all components of a high-quality argument may be exhaustively present in one study.30 Indeed, no one investigation can fully address all positions possible in a comprehensive argument. For instance, an ethnographic study may use a detailed set of observations to generate a claim, whereas refinement of this data-claim linkage becomes the topic of further research that constructs and tests a more comprehensive argument. A single experiment may test data-claim linkages without
explicitly investigating qualifiers. Aggregating findings across studies into a more sophisticated whole, carefully considering if and how the pieces fit together (and where gaps in the puzzle still remain), and evaluating the practical relevance of the explanation that results, in our opinion, constitutes “theory development” and illustrates the value inherent in considering theory explicitly—argument quality can only be judged when all of the pieces are considered cumulatively. Another complexity is that the components of argumentation can be difficult to identify definitively; arguments may be multilayered and what constitutes a claim in one sense may serve as a warrant or backing in another.29 As a result, difficulties arise when trying to develop a reliable, consistently applied structural assessment framework that employs a “checklist” approach. Simply assessing the frequency and comprehensiveness of a single argument’s components fails to fully capture the essence of productive argumentation.20,26,32 Rather, judgment that is informed by the state of the field is required, as it is in other domains.33 Assessment of argumentation quality, therefore, should emphasize the potential impact of investigations on fostering shared, integrated conceptual models.13 A quality argument becomes embedded in ongoing scholarly dialogue, and its thesis and evidence advances the dialogue in substantive ways and fosters the dialogue’s evolution by providing a means to evaluate ideas through application. In this light, quality argumentation is seen when the design and reporting of investigations demonstrate awareness of alternative positions and counterarguments that could be posed by other community members.29 This awareness is reflected in the production of evidence that rebuts, integrates, or synthesizes these differing perspectives.34 It also is reflected in appeals to causal mechanisms as evidence versus analogies or metaphors. The practice of anticipating challengers has been termed “anticipatory dialogue”28 and may be considered forethought regarding “What’s next?”—a question that is generally taken to be the hallmark of scholarly dialogue and “knowledge-building conversation.”35 The production of evidence and appeals to causal mechanisms assumes a scholarly dialogue that has the evaluation of validity at its core, rather than simply the exchange of unsupported claims. Critiques noting the lack of conceptual models present in medical education publications10,13 suggest that we must continue to work to foster more effective argumentation in our scholarly dialogue. A lack of conceptual models is one indication that investigators interested in the same phenomena are not engaged in exchange because they are not referring to what each other has said or what the community has come to think. The Durning et al. article mentioned earlier illustrates, alternatively, how quality collaborative argumentation can look.16 In this article, the authors explore the epistemological orientations (objectivist and constructivist) and learning and assessment perspectives (behaviorist, information processing, psychometric, and situative) from which clinical reasoning has been studied and highlight the implications of an integrated perspective for
THEORY IN MEDICAL EDUCATION
Downloaded by [UZH Hauptbibliothek / Zentralbibliothek Zürich] at 02:14 07 January 2015
understanding the interplay of context and mental processes in the achievement of diagnoses. Promoting Quality Argumentation Conceptualizing theory as enabling argumentation in scholarly dialogue allows us to distinguish between rigorous, theorybuilding scholarship, on one hand, and particular modes of inquiry, on the other. In other words, one investigatory approach (e.g., qualitative or quantitative) or type of answer sought (“basic” or “applied”) is not superior to another as long as the work conducted in either instance enables collaborative argumentation among community members and the development of sophisticated, actionable explanations.1,17,20 Understanding the structure of argumentation helps to clarify how different investigations can contribute to theory. Different modes of inquiry, such as case studies, laboratory experiments, ethnographic analysis, quasi-experiments, correlational studies, and so forth, may target different components of argumentation and variously serve to generate, test, and/or bound theory. Laboratory studies, for instance, commonly test competing claims made about a given data set. Qualifiers that bound the generalizability and applicability of laboratory findings may be discovered when explanations produced by experiments are tested in quasi-experimental or other applied studies.36 Community understanding becomes more sophisticated as these boundaries take shape. Qualitative investigation helps to generate theory by producing detailed data about which various claims may be made and tested. It may also support theory testing and refinement by providing the evidence necessary to justify or question warrants supporting data-claim linkages. This objective, explicit or otherwise, underlies many mixed-method studies published in medical education journals. Quality argumentation in our field, then, is not fostered by choosing the “correct” method or positing the “right” kind of answer.37 Rather, quality is fostered through the practical exercise of working and reworking one’s scholarship with attention to how it contributes to collaborative argumentation, specifically how it refutes or defends various components of shared understanding versus simply stating an alternative claim.27,28 The exercise of argumentation as continuous professional development parallels the formal learning process for behavioral scientists, where a thesis is developed, refined, and defended over a long period of focused effort and the field-dependent criteria for evaluating evidence are acquired through close work with intellectual mentors. For educators interested in participating in medical education research, pairing with colleagues who have behavioral science degrees may provide just such a mentor.12 Routine and informal peer review by such colleagues is critical to identifying weakness in the positions one has taken, and the diversity of behavioral scientists involved in medical education stimulates the construction of sophisticated explanations of the phenomenon of interest.26 Argumentation process and argument content are only acquired over a considerable period of immersion and benefits
S79
greatly from the advisement of colleagues.38 Here, reviewers (formal and informal) become critically important stewards of knowledge-building conversations by providing thoughtful and constructive feedback on the quality of argumentation in the manuscripts they read and by promoting authors’ awareness of and engagement with the scholarly community in their work.20,27,28,39 To summarize, the value of theory has been discussed extensively in our field over the past two and a half decades as we have sought to continuously improve teaching and learning in medicine. Attempts to understand what constitutes theory in our field fail when we conceptualize theory as a single, comprehensive explanation that is perfectly predictive under all circumstances. The phenomena of interest to medical educators are complex and evolving and causal relationships are inevitably probabilistic. As an alternative, we have proposed conceptualizing theory as an effect on scholarly dialogue, one that enables the acquisition of community understanding that has utility beyond investigators’ local circumstances. This effect may be described as argumentation, which is structured in a way that fosters productive interaction among interlocutors, permits evaluation of evidence, and leads to the construction of sophisticated explanations. Assessing and fostering argument quality is a community process characterized by the formal and informal collaboration of medical educators and a diversity of behavioral scientists. Such collaboration is invaluable even as it makes our collective work more challenging as we grow shared understanding together.
REFERENCES 1. Colliver JA. Constructivism with a dose of pragmatism: A cure for what ails educational research. Advances in Health Science Education 1999;4:187–90. 2. Colliver JA. Full-curriculum interventions and small-scale studies of transferImplications for psychology-type theory. Medical Education 2004;38:1212–18. 3. Yeager DS, Walton GM. Social-psychological interventions in education: They’re not magic. Review of Educational Research 2011;81:267–301. 4. Regehr G. It’s NOT rocket science: Rethinking our metaphors for research in health professions education. Medical Education 2010;44:31–9. 5. Colliver JA. Educational theory and medical education practice: A cautionary note for medical school faculty. Academic Medicine 2002;77:1217–20. 6. Colliver JA, Conlee MJ, Verhulst SJ. From test validity to construct validity . . . and back?. Medical Education 2012;46:366–71. 7. Rees CE, Monrouxe LV. Theory in medical education research: How do we get there?. Medical Education 2010;44:334–9. 8. Reeves S, Mathieu A, Kuper A, Hodges BD. Why use theories in qualitative research?. British Medical Journal 2008;337:631–4. 9. Albert M, Hodges B, Regehr G. Research in medical education: Balancing service and science. Advances in Health Sciences Education: Theory and Practice 2007; 12:103–15. 10. Cook DA, Beckman TJ, Bordage G. Quality of reporting of experimental studies in medical education: A systematic review. Medical Education 2007;41:737–45. 11. Markert RJ. Enhancing medical education by improving statistical methodology in journal articles. Teaching and Learning in Medicine 2013;2592:159–64.
Downloaded by [UZH Hauptbibliothek / Zentralbibliothek Zürich] at 02:14 07 January 2015
S80
A. T. CIANCIOLO, K. W. EVA, J. A. COLLIVER
12. Rogers DA. Advancing surgical education scholarship. American Journal of Surgery 2011;201:1–6. 13. Bordage G. Conceptual frameworks to illuminate and magnify. Medical Education 2009;43:312–9. 14. Rotgans JI. The themes, institutions, and people of medical education research 1988–2010: Content analysis of abstracts from six journals. Advances in Health Science Education 2012;17:515–7. 15. Schmidt HG, Rotgans JI, Yew EH. The process of problem-based learning: What works and why. Medical Education 2011;45:792–806. 16. Durning SJ, Artino Jr AR, Schuwirth L, van der Vleuten C. Clarifying assumptions to enhance our understanding and assessment of clinical reasoning. Academic Medicine 2013;88(4):1–7. 17. Eva KW. The State of the Science 2012: Building blocks for the future. Medical Education 2012;46:1–2. 18. Bunniss S, Kelly DR. Research paradigms in medical education research. Medical Education 2010;44:358–66. 19. Eva KW. Dialogue in medical education: Enabling the academic voyeur that lurks inside us all. Medical Education 2012;46:826–7. 20. Colliver JA. Commentary on “Review Criteria for Research Manuscripts.”. Teaching and Learning in Medicine 2002;14:75–6. 21. Denzin NK, Lincoln YS. Introduction: Entering the field of qualitative research. In NK Denzin & YS Lincoln (Eds.), Handbook of qualitative research (pp. 1–17). Thousand Oaks, CA: Sage, 1994. 22. Giacomini M. Theory matters in qualitative research. In I Bourgeault, R Dingwall, R De Vries (Eds.), The SAGE handbook of qualitative methods in health research (pp. 125–156). Los Angeles: Sage, 2010. 23. Gould SJ. The mismeasure of man. Rev. ed. New York: W. W. Norton, 1996. 24. Teherani A, Hodgson C, Banach M, Papadakis M. Domains of unprofessional behavior during medical school associated with future disciplinary action by a state medical board. Academic Medicine 2005;80(10 Suppl):17–20. 25. Ginsburg S, Bernabeo E, Ross KM, Holmboe ES. “It depends”: Results of a qualitative study investigating how practicing internists approach professional dilemmas. Academic Medicine 2012;87:1685–93.
26. Norman G. Fifty years of medical education research: Waves of migration. Medical Education 2011;45:785–91. 27. Erduran S, Simon S, Osborne J. Tapping into argumentation: Developments in the application of Toulmin’s argument pattern for studying science discourse. Science Education 2004;88:915–33. 28. Kuhn D. Teaching and learning science as argument. Science Education 2010;94:810–24. 29. Duschl RA. Quality argumentation and epistemic criteria. In S Erduran, MP Jim´enez-Aleixandre (Eds.), Argumentation in science education (pp. 159–75). Dordrecht: Springer, 2008. 30. Toulmin SE. The uses of argument. Cambridge, England: Cambridge University Press, 1958. 31. Betancourt JR, Carrillo JE, Green AR. Hypertension in multicultural and minority populations: Linking communication to compliance. Current Hypertension Reports 1999;1:482–8. 32. Eva KW. On the limits of systematicity. Medical Education 2008;42: 852–3. 33. Eva KW, Hodges BD. Scylla or Charybdis? Can we navigate between objectification and judgement in assessment?. Medical Education 2012;46:914–9. 34. Nussbaum EM, Schraw G. Promoting argument-counterargument integration in students’ writing. Journal of Experimental Education 2007;76:59–92. 35. Eva KW, Lingard L. What’s next? A guiding question for educators engaged in educational research. Medical Education 2008;42:752–4. 36. Chapanis A. The relevance of laboratory studies to practical situations. Ergonomics 1967;10:557–77. 37. Eva KW. Broadening the debate about quality in medical education research. Medical Education 2009;43:294–6. 38. O’Sullivan PS, Stoddard HA, Kalishman S. Collaborative research in medical education: A discussion of theory and practice. Medical Education 2010;44:1175–84. 39. Eva KW. The reviewer is always right: Peer review of research in medical education. Medical Education 2009;43:2–4.
