AACN Advanced Critical Care Volume 25, Number 3, pp. 230-236 © 2014 AACN
Differential Diagnosis Correctly Putting the Pieces of the Puzzle Together Kristine Anne Scordo, RN, PhD, ACNP-BC
ABSTRACT Each day, we generate hypotheses about our environment—our perceptions of people, our expectations of events, and our interpretation of images. These hypotheses provide a framework by which we interpret our experiences. The same is true for differential diagnosis by which health care practitioners develop hypotheses or diagnoses from a set of cues provided during an encounter with a patient. For clinicians to be successful at differential diagnosis, they must use a multidimensional and complex process involving
Medicine is a science of uncertainty and an art of probability. Sir William Osler1
E
ach day, we generate differential diagnoses about our patients that are formulated from a set of cues provided during encounters with patients. These cues take many shapes: the patient’s age, race, sex, symptom(s), history, diagnostic tests, laboratory values, and physical examination. From these cues, differential diagnoses that include the most important possible causes are formulated. In essence, we take pieces of a jigsaw puzzle that are provided to us without a picture, and we successfully put the puzzle together. Multiple steps are taken in developing a diagnosis; to succeed, we must use a multidimensional and complex process that involves nonanalytic and analytic cognitive processes along with metacognition—thinking about thinking.2,3 Thus, the purpose of this article is to discuss how
nonanalytic and analytic cognitive processes and metacognition—thinking about thinking. Our conclusions, however, can lead to errors in diagnosis. Many of these errors are due to errors in cognition. The purpose of this article is to discuss this complex process, identify common errors in cognition, and offer strategies to prevent these common errors in differential diagnosis. Keywords: cognitive bias, differential diagnosis, heuristics
clinicians use these processes to formulate differential diagnoses and how these processes can lead to mistaken diagnoses and to provide tips for clinicians to prevent making erroneous diagnoses. Nonanalytic Processes Versus Analytic Reasoning Clinicians use 3 common diagnostic reasoning strategies: pattern recognition, deductive reasoning (hypothetico-deductive), and inductive reasoning (scheme-inductive).4 Simply defined, deductive reasoning moves from more general information to more specific information, whereas inductive reasoning moves from more
Kristine Anne Scordo is Professor and Director, Adult-Gero Acute Care Nurse Practitioner Program, College of Nursing, Wright State University, Dayton, OH 45324 (kscordo@cinci .rr.com). The author declares no conflicts of interest. DOI: 10.1097/NCI.0000000000000035
230 Copyright © 2014 American Association of Critical-Care Nurses. Unauthorized reproduction of this article is prohibited.
NCI-D-14-00015.indd 230
09/07/14 11:29 AM
VOL UME 2 5 • N U MBER 3 • JULY–SEPTEM BER 2014
D IFFE RE NT IA L D IAGNOSIS
specific information to broader generalizations. Deductive reasoning and inductive reasoning are considered analytic strategies, whereas pattern recognition is considered nonanalytic. Nonanalytic Processing
Nonanalytic reasoning is a subconscious, automatic process that is driven from similarities between present and past encounters with patients.4 Nonanalytic processes include tacit or intuitive (“know-how”) knowing. These processes rely on pattern recognition and clinicians’ previous experiences. They are used in a preconscious, nonconscious, automatic, or reflexive manner and often are viewed as subjective and prone to error.3,5 Tacit or intuitive knowing “refers to knowledge that functions at the periphery of attention and makes possible the conventionally recognized explicit domains of human knowledge.”6(p293) Tacit knowing is akin to “knowing how” and represents practical understanding of how things are done. This knowledge encompasses the intricacy of different experiences that are acquired over time, which clinicians use to effectively complete tasks—such as diagnosing, prescribing, and choosing diagnostic tests.4 The key to tacit knowledge is experience. An example of how students acquire tacit knowledge is their interpretation of chest x-ray changes used to diagnose pulmonary edema. At the beginning, students focus on basic radiographic knowledge. Later, with time and experience reading radiographs, students interiorize the knowledge and enter a “new world,”7 becoming more confident in their ability to use this diagnostic tool. Tacit knowledge depends on pattern recognition. Pattern recognition involves comparing the current presentation of a patient with a previous case stored in memory, and making a judgment on the basis of the likelihood of a match with similar experiential cues, schema, or scripts, that is, a classic presentation. Examples of these presentations in the critical care arena might include meningitis, hypertensive crisis, acute myocardial infarction, acute pulmonary edema, and cerebral vascular accident. From experience, the clinician recognizes characteristic findings that match the pattern of a preliminary diagnosis without considering alternative diagnoses. These knowledge structures, or patterns, that contain information
about risk factors and manifestations of a disease are called illness scripts. These scripts allow clinicians to quickly recognize patterns of diseases while filtering out extraneous and irrelevant information, thus narrowing possible diagnoses.8 New clinicians have limited knowledge of illness scripts to filter in reaching a differential diagnosis. With repeated exposure to various patient encounters, or simulation experiences, clinicians gradually develop multiple illness scripts. Although pattern recognition and the application of illness scripts are usually a rapid and easy process, they are not without issue—not all patients fit into a previously recognized pattern. Thus, other types of reasoning must be used. Analytic (Diagnostic) Reasoning Hypothetico-Deductive Reasoning
Whereas nonanalytic reasoning is subconscious, analytic or diagnostic reasoning is a conscious, active, and analytic process.4 This approach consists of cognitive elements, such as logic, which represent a complex process of critical thinking, objective analysis, and reflection.3,9 The classic diagnostic strategy is hypothetico-deductive reasoning. This process begins with the generation of several differential diagnoses that, followed by data collection, drive further inquiry aimed at revealing additional information that either confirms or refutes the differentials generated. Using the hypothetico-deductive method, clinicians must consider diseases that are imperative not to miss.10 As with nonanalytic reasoning, beginning students are taught to take a thorough history and perform a detailed physical examination. Clinical judgments are then made during and after data collection that culminate in various differential diagnoses. In contrast to novices, experienced clinicians will use a few key findings to focus on the most likely diagnoses in a relatively short period of time. They will elicit information about the patient’s main complaint while they listen, observe, and formulate potential diagnoses from which more specific questions are asked—all of which clarify the problem the patient is describing. These differentials then direct subsequent data collection, such as diagnostic testing. A “think aloud” strategy, in which expert clinicians verbalize as they reason, can be helpful in mentoring novice clinicians.
231 Copyright © 2014 American Association of Critical-Care Nurses. Unauthorized reproduction of this article is prohibited.
NCI-D-14-00015.indd 231
09/07/14 11:29 AM
S C OR D O
W W W.A ACNA DVA NCE D CRIT ICA LCA RE .COM
Scheme-Inductive Reasoning
Another analytic reasoning strategy is schemeinductive reasoning (rule-based reasoning).4,11 Schemes reflect an organized knowledge structure for diagnostic reasoning. “The term scheme denotes a conceptual framework that is used in the reasoning process.” 4(p454) These schemes are similar to algorithms or decision trees, whereby the clinician proceeds down several branches that reduce the number of diagnoses. In summary, developing expertise in diagnostic reasoning requires experience with a range of analytic and nonanalytic strategies. Clinicians with an ability to use multiple methods to critically reason make better decisions, are better problem solvers, and are professionally more competent.11–13 History taking and the physical examination form the foundation of analytic reasoning. Regarding history, a wise professor once said, “If you carefully listen to the patient, the patient will usually let you know what is wrong with them” (Fuheid S. Daoud, MD, personal oral communication, September 1980). Determining which objective findings are pertinent to the primary diagnosis is the challenge in physical examination. Experience-Based Techniques (Heuristics) One factor thought to be important in the generation of diagnostic hypotheses is heuristics (experience-based techniques).14 Heuristics are shortcut mental strategies, for example, educated guess, common sense, or rule of thumb, that streamline information.15 These shortcuts can lead to efficient and correct decisions, often greatly simplifying the task of judging probability. As a result, they are frequently used by experienced clinicians. The use of heuristics, however, also can lead to predictable and recurrent errors, which will be discussed later. Commonly used heuristics include representativeness, availability, anchoring/adjustment, and premature closure16–19 (see Table 1). Representativeness can be described by the question, “How closely does patient A match patient B?” Consider a patient with symptoms of acute pancreatitis. The clinician’s estimate of the probability of acute pancreatitis will depend on how closely the patient matches previous encounters or the textbook picture. Availability is described as a situation in which a clinician ascribes a higher probability to the patient’s diagnosis because she or he vividly remembers a recent case that was very simi-
lar, which is a function of familiarity with a clinical entity. For example, the clinician recently diagnosed Exserohilum rostratum meningitis in a patient with lumbar-sacral back pain and now suspects fungal meningitis in a patient with headache. Anchoring/adjustment is a heuristic in which clinicians first use an anchor (diagnostic hypothesis) and adjust their estimates until a final answer is reached. For example, a 65-year-old man with multiple cardiovascular risk factors and chest pain presents to the emergency department (ED). After ruling out an acute event, the clinician decides to perform a graded exercise stress test to aid in the diagnosis of myocardial ischemia. Although this test is positive for ischemia at the early stage of a standard Bruce protocol, the clinician now orders a nuclear scan. The probability of a positive test is high with a nuclear scan if the patient has coronary artery disease. In this case, the clinician underestimates the probability of an abnormal exercise electrocardiogram and a diagnosis of coronary artery disease. Always remember to ask, particularly in this economic environment, “What added value will the test bring—will the results alter the treatment decisions?” If not, then reconsider the purpose of the test. Premature closure represents a situation in which a clinician settles on a diagnosis with insufficient evidence and without considering contradictory evidence. For example, the clinician mistakes a woman with a long history of migraine presenting with a severe headache for another attack of migraine, when in fact she has a subarachnoid hemorrhage. The simplest explanation for any clinical presentation should be considered first (according to the principle of Occam’s razor)2; that is, when in Kansas and you hear hoof beats, think horses not zebras. In acute and critical care settings, however, multiple illnesses may coexist. As such, Hickam’s dictum may prevail: “Patients can have as many diseases as they damn well please.”20(p682) When enough information is accumulated, the clinician will then determine the probability or the likelihood of a given disease. Estimates of probability account for the clinician’s personal experience, published literature, and attributes of the patient. If the probability or the likelihood of the diagnosis is high, such as an electrocardiogram in a 62-year-old man with chest pain that demonstrates ST-segment elevation
232 Copyright © 2014 American Association of Critical-Care Nurses. Unauthorized reproduction of this article is prohibited.
NCI-D-14-00015.indd 232
09/07/14 11:29 AM
VOL UME 2 5 • N U MBER 3 • JULY–SEPTEM BER 2014
D IFFE RE NT IA L D IAGNOSIS
Table 1: Diagnostic Cognitive Biasesa Bias Anchoring
Attribution
Availability
Description Staying with 1 diagnosis
Stereotyping a patient or gender bias
Familiarity with a clinical entity
Example
Corrective Strategy
A clinician treats a patient who presents during flu season with nausea and vomiting for gastroenteritis, but the patient later presents with appendicitis.
Ask:
An intoxicated homeless man presents with a large ulcer on his foot. The assumption is some sort of trauma and unhealthy lifestyle, when he actually has uncontrolled diabetes.
Ask:
A clinician suspects appendicitis in a patient with pancreatitis.
Ask:
What else can this be? Is there some other explanation for these data? Could 2 things be going on at once? Am I stereotyping this person? Am I biased because of his appearance? How should I acknowledge my interactions? What is the likelihood of the diagnosis? What is the prevalence of the diagnosis?
Confirmation
Diagnostic momentum
Framing
Premature closure
a
Tendency to look for confirming evidence to support a diagnosis rather than looking for evidence to refute it
A clinician treats a young female patient who takes birth control pills for calf muscle strain, and she presents later with a pulmonary embolus.
Ask:
Once diagnostic labels are attached to a patient they stick
Nurses in the emergency department ask you to see a “frequent flyer” who seeks narcotics for abdominal pain. All believe him to be seeking drugs when in essence he has an acute abdomen.
Ask:
Assembling elements that support a diagnosis
A clinician assumes pustules are the result of poison ivy in a patient who recently returned from a camping trip.
Ask:
Failing to search for additional information after reaching a diagnostic conclusion
A clinician assumes that a patient who presents with sudden abdominal pain with vomiting after attending a party where others became ill portends food poisoning, whereas in reality, the patient has a small bowel obstruction.
Ask:
What is the worst-case scenario? What other differentials should I consider to evaluate whether my diagnosis correlates with my findings? Did I perform a thorough history and physical examination? Did I arrive at my decision independent of others? Is a diagnostic time out in order?
What might be other causes? If the patient was not camping, then what would the diagnosis be?
What other information do I need? What other differential diagnoses could this be? What is the worst-case scenario? Whose opinion should I seek?
Based on data from Engel, 16 Croskerry,17,18 and Wellbery.19
in the inferior leads along with elevated troponin (treatment threshold), no further testing is needed. Conversely, if the probability is low, such as a 21-year-old man with chest pain with
normal electrocardiographic finding, further testing may be required. Using Bayes theorem2 allows clinicians to combine the objective results of a test, or a test equivalent, with their
233 Copyright © 2014 American Association of Critical-Care Nurses. Unauthorized reproduction of this article is prohibited.
NCI-D-14-00015.indd 233
09/07/14 11:29 AM
S C OR D O
W W W.A ACNA DVA NCE D CRIT ICA LCA RE .COM
prior clinical suspicions to hone the probability of a particular disease.
and can lead to erroneous conclusions (see Table 1).
Errors in Differential Diagnosis As previously mentioned, the process of differential diagnosis is not without error. Errors in the health care arena continue to capture the attention of the public, policy makers, and providers.21 Although treatment errors comprise most mistakes, errors related to the diagnostic process substantially contribute to total medical errors.22 In a systematic review of 31 studies covering 5863 autopsies, 28% reported at least 1 misdiagnosis in the intensive care unit (ICU).23 These authors estimated that as many as 40 500 adult patients annually may die in the ICU because of a misdiagnosis. Commonly missed diagnoses included pulmonary embolism, myocardial infarction, pneumonia, and aspergillosis. It may seem counterintuitive to think that patients in the ICU environment, who are closely monitored and frequently tested, are more commonly misdiagnosed. The ICU, however, is well known to be a very complex environment. Furthermore, some estimates suggest that diagnostic errors in EDs occur up to 10% of the time.24,25 In a review of 122 closed malpractice claims in which patients had alleged a missed or delayed diagnosis in the ED, 79 claims (65%) involved missed diagnoses that harmed patients.26 Of these 79 claims, 58% of the errors were due to a failure to order an appropriate diagnostic test, 42% were due to failure to perform an adequate medical history or physical examination, 37% were due to incorrect interpretation of a diagnostic test, and 33% were due to failure to order an appropriate consultation. Although missed diagnoses in the ED are multifactorial, the authors noted that most errors were due to cognitive factors—commonly mistakes in judgment or flaws in clinical reasoning. These cognitive causes of diagnostic errors can be categorized into 3 types: inadequate or faulty knowledge, faulty data gathering, and faulty synthesis (inaccurate clinical reasoning and erroneous verification of diagnostic hypotheses), with the latter accounting for the vast majority of errors.24 Although complex in nature, errors in diagnostic reasoning often are attributed to diagnostic biases or heuristics. As previously mentioned, when used correctly, heuristics expedite clinical decision making. At times, however, these biases become pitfalls
Metacognition How then do practitioners avoid errors in differential diagnosis? Metacognition—or thinking about thinking—represents a cognitive means to monitor and regulate reasoning.3 In essence, metacognition “is the capacity to think about [nonanalytic] and analytic processes at a higher level of cognitive reflection.”3(p955) The following are metacognitive techniques for clinicians and educators that may help avoid diagnostic errors.27 1. Explicitly describe heuristics and how they affect clinical reasoning. Most clinicians are unaware of these biases, let alone how they influence clinical reasoning. By developing an awareness of the types of heuristics and potential issues, clinicians may gain insight into their clinical reasoning behaviors. 2. Promote the use of “diagnostic timeouts.” Similar to having a colleague review the case by taking a fresh look, take a step back to review supporting diagnostic evidence. 3. Promote the practice of “worst-case scenario.” Consider the patient’s worst possible diagnosis. 4. Promote the use of a systematic approach to problems. For instance, use an anatomical approach to acute abdominal pain, considering the structures within and proximate to the abdomen and working through how each structure could be the source of pain. 5. Ask why. That is, ask “Why did this happen?” For example, “Why did a stable patient with a reduced ejection fraction heart failure suddenly become tachypneic?” “What precipitated this problem?” “Why did the patient with chronic obstructive pulmonary disease have an exacerbation?” “Why is a 21-year-old nonsmoker admitted with a third episode of pneumonia?” 6. Emphasize the value of the clinical history and physical examination. In this age of technology, many clinicians rely on expensive testing when, in fact, more information
234 Copyright © 2014 American Association of Critical-Care Nurses. Unauthorized reproduction of this article is prohibited.
NCI-D-14-00015.indd 234
09/07/14 11:29 AM
VOL UME 2 5 • N U MBER 3 • JULY–SEPTEM BER 2014
7.
8.
9.
10.
D IFFE RE NT IA L D IAGNOSIS
can be obtained by a thorough history and physical examination. For educators, this technique implies teaching at the bedside, modeling the diagnostic process, and incorporating various readings about the how-tos of an examination. Learn Bayesian theory as a way to direct the clinical evaluation and avoid premature closure. Bayes’ theorem is a means of quantifying uncertainty and is based on probability.3 For example, consider the likelihood that an 18-year-old with chest pain is having a myocardial infarction, compared with a 56-year-old with hypertension, hyperlipidemia, and diabetes mellitus. Although this process can be time consuming, errors in premature closure can be demonstrated by applying Bayesian reasoning to the use of pretest and posttest probabilities. Acknowledge how patients make the clinician feel. Think of the “frequent flyer” in the ED who has been identified as drug seeking and now presents with acute appendicitis, or the heroin addict whom you have treated for multiple abscess and who now presents with sudden onset of erythema and swelling in his arm. Practitioners are not without feelings and biases and may discount data in patients perceived as annoying, difficult, or frequent consumers of care. In contrast, if the practitioner has a personal relationship with the patient, data may be overlooked; that is, you cannot see the forest for the trees. Encourage learners to find clinical data that do not fit with a provisional diagnosis: Ask, “What can’t we explain?” Although Occam’s razor is a commonly invoked diagnostic rule, many patients have more than 1 diagnosis or may have a rare or complex disease. Closely examine any and all available data for any findings that the preliminary or established diagnosis cannot explain. Embrace zebras. Consider less common disorders, as they do occur. For example, a 36-year-old woman has shortness of breath and chest pain after
an appendectomy. The clinician may initially diagnose pulmonary emboli, when in reality the patient has primary pulmonary hypertension. 11. Encourage learners to slow down. Take time to think, talk out loud, and consider alternatives, which, admittedly, can be difficult in emergency situations. Getting the diagnosis right the first time, however, is not only beneficial to the patient but also can be quite economical. 12. Admit one’s own mistakes. An important part of critical thinking is reflection. Consider what worked and what did not work. Ask yourself, “How can I do better the next time?” These are important questions that improve decision making. Conclusion Acquiring a well-defined, structured body of knowledge is necessary for a clinician to become an expert.13,28,29 This knowledge results from many encounters with patients. No one is born an expert diagnostician. Nobody becomes an outstanding professional without experience, but extensive experience does not invariably lead people to become experts.30,31 Make a habit of conducting regular and frequent surveillance of your intuitive and cognitive behaviors.28 Remember that clinical expertise is a process, not a product. REFERENCES 1. Bean W, ed. Aphorisms From His Bedside Teaching and Writings. New York: Schuman; 1950:125. 2. Kassirer J, Wong J, Kopelman R. Learning Clinical Reasoning. 2nd ed. Philadelphia, PA: Lippincott Williams & Wilkins; 2010. 3. Marcum J. An integrated model of clinical reasoning: dual-process theory of cognition and metacognition. J Eval Clin Pract. 2012;18:954–961. 4. Heemskerk L, Norman G, Chou S, Mintz M, Mandin HJ, McLaughlin K. The effect of question format and task difficulty on reasoning strategies and diagnostic performance in internal medicine residents. Adv Health Sci Educ. 2008;13:453–462. 5. Offredy M. The application of decision-making concepts by nurse practitioners in general practice. Aust J Adv Nurs. 1998;28(5):532–541. 6. Henry S. Polanyi’s tacit knowing and the relevance of epistemology to clinical medicine. J Eval Clin Pract. 2010;16:293. 7. Polanyi M. Personnel Knowledge. Towards a Post-critical Philosophy (Reprinted 2002). London: Routledge, Taylor and Francis Group; 2002:101. 8. Schmidt J, Rikers R. How expertise develops in medicine: knowledge encapsulation and illness script formation. Med Educ. 2007;41:1133–1139.
235 Copyright © 2014 American Association of Critical-Care Nurses. Unauthorized reproduction of this article is prohibited.
NCI-D-14-00015.indd 235
09/07/14 11:29 AM
S C OR D O
W W W.A ACNA DVA NCE D CRIT ICA LCA RE .COM
9. Rashotte J, Carnevale F. Medical and nursing clinical decision making: a comparative epistemological analysis. Nurs Philos. 2004;5:160–174. 10. Sox H, Higgins M, Owens D. Medical Decision Making. 2nd ed. West Sussex, UK: Wiley-Blackwell; 2013. 11. Coderre S, Mandin H, Harasym P, Fick G. Diagnostic reasoning strategies and diagnostic success. Med Educ. 2003;37(8):695–703. 12. Benner PE. Educating Nurses: A Call for Radical Transformation. San Francisco, CA: Jossey-Bass; 2010. 13. Harasym P, Tsai T, Hemmait P. Current trends in developing medical students’ critical thinking abilities. Kaohsiung J Med Sci. 2008;24:341–354. 14. Kassirer J, Wong J, Kopelman R. Learning Clinical Reasoning. 2nd ed. Baltimore, MD: Wolters Kluwer, Lippincott Williams & Wilkins; 2009. 15. O’Neil E. Heuristics reasoning in diagnostic judgment. J Prof Nurs. 1995;11:239–245. 16. Engel P. Tacit knowledge and visual expertise in medical diagnostic reasoning: implications for medical education. Med Teach. 2008;30(7):184–188. 17. Croskerry P. From mindless to mindful practice—cognitive bias and clinical decision making. N Engl J Med. 2013;368(26):2445–2448. 18. Croskerry P. The importance of cognitive errors in diagnosis and strategies to minimize them. Acad Med. 2003;78(8):775–780. 19. Wellbery C. Flaws in clinical reasoning. Am Fam Physician. 2011;84:1042–1043. 20. Pate J, Kindermann D, Hudson K. A case of Hickam’s dictum: concurrent appendicitis and ectopic pregnancy. J Emerg Med. 2013;45(5):679–682. 21. Koah KT, Corigan JM, Donnaldson MS. To Err Is Human: Building a Safe Health System. Washington, DC: National Academies Press; 1999.
22. van den Berge K, Mamede S. Cognitive diagnostic error in internal medicine. Eur J Intern Med. 2013;24(6):525– 529. Available from: Science Citation Index, Ipswich, MA. Accessed November 11, 2013. 23. Winters B, Custer J, Newman-Toker D, et al. Diagnostic errors in the intensive care unit: a systematic review of autopsy studies. BMJ Qual Saf. 2012;21(11):894–902. Available from: CINAHL Plus with Full Text, Ipswich, MA. Accessed December 1, 2013. 24. Scott IA. Errors in clinical reasoning: causes and remedial strategies. BMJ. 2009;338:b1860. 25. Normal GR, Eva KW. Diagnostic error and clinical reasoning. Ed Educ. 2010;44(1):94–100. 26. Kachalia A, Gandhi T, Studdert D, et al. Missed and delayed diagnoses in the emergency department: a study of closed malpractice claims from 4 liability insurers. Ann Emerg Med. 2007;49(2):196–205. Available from: Science Citation Index, Ipswich, MA. 27. Trowbridge R. Twelve tips for teaching avoidance of diagnostic errors. Med Teach. 2008;30(5):496–500. Available from: Academic Search Complete, Ipswich, MA. 28. Ericsson K. Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Acad Med. 2004;70(10): S70–S81 29. Norman G, Eva K. Doggie diagnosis, diagnostic success and diagnostic reasoning strategies: an alternative view. Med Educ. 2003;37(8):676–677. 30. Alexander E. Perspective: moving students beyond an organ-based approach when teaching medical interviewing and physical examination. Acad Med. 2008;83 (10):906–909. 31. Graber M, Gordon R, Franklin N. Reducing diagnostic errors in medicine: what’s the goal? Acad Med. 2002;77(10):981–992.
236 Copyright © 2014 American Association of Critical-Care Nurses. Unauthorized reproduction of this article is prohibited.
NCI-D-14-00015.indd 236
09/07/14 11:29 AM
Differential Diagnosis Correctly Putting the Pieces of the Puzzle Together Kristine Anne Scordo, RN, PhD, ACNP-BC
ABSTRACT Each day, we generate hypotheses about our environment—our perceptions of people, our expectations of events, and our interpretation of images. These hypotheses provide a framework by which we interpret our experiences. The same is true for differential diagnosis by which health care practitioners develop hypotheses or diagnoses from a set of cues provided during an encounter with a patient. For clinicians to be successful at differential diagnosis, they must use a multidimensional and complex process involving
Medicine is a science of uncertainty and an art of probability. Sir William Osler1
E
ach day, we generate differential diagnoses about our patients that are formulated from a set of cues provided during encounters with patients. These cues take many shapes: the patient’s age, race, sex, symptom(s), history, diagnostic tests, laboratory values, and physical examination. From these cues, differential diagnoses that include the most important possible causes are formulated. In essence, we take pieces of a jigsaw puzzle that are provided to us without a picture, and we successfully put the puzzle together. Multiple steps are taken in developing a diagnosis; to succeed, we must use a multidimensional and complex process that involves nonanalytic and analytic cognitive processes along with metacognition—thinking about thinking.2,3 Thus, the purpose of this article is to discuss how
nonanalytic and analytic cognitive processes and metacognition—thinking about thinking. Our conclusions, however, can lead to errors in diagnosis. Many of these errors are due to errors in cognition. The purpose of this article is to discuss this complex process, identify common errors in cognition, and offer strategies to prevent these common errors in differential diagnosis. Keywords: cognitive bias, differential diagnosis, heuristics
clinicians use these processes to formulate differential diagnoses and how these processes can lead to mistaken diagnoses and to provide tips for clinicians to prevent making erroneous diagnoses. Nonanalytic Processes Versus Analytic Reasoning Clinicians use 3 common diagnostic reasoning strategies: pattern recognition, deductive reasoning (hypothetico-deductive), and inductive reasoning (scheme-inductive).4 Simply defined, deductive reasoning moves from more general information to more specific information, whereas inductive reasoning moves from more
Kristine Anne Scordo is Professor and Director, Adult-Gero Acute Care Nurse Practitioner Program, College of Nursing, Wright State University, Dayton, OH 45324 (kscordo@cinci .rr.com). The author declares no conflicts of interest. DOI: 10.1097/NCI.0000000000000035
230 Copyright © 2014 American Association of Critical-Care Nurses. Unauthorized reproduction of this article is prohibited.
NCI-D-14-00015.indd 230
09/07/14 11:29 AM
VOL UME 2 5 • N U MBER 3 • JULY–SEPTEM BER 2014
D IFFE RE NT IA L D IAGNOSIS
specific information to broader generalizations. Deductive reasoning and inductive reasoning are considered analytic strategies, whereas pattern recognition is considered nonanalytic. Nonanalytic Processing
Nonanalytic reasoning is a subconscious, automatic process that is driven from similarities between present and past encounters with patients.4 Nonanalytic processes include tacit or intuitive (“know-how”) knowing. These processes rely on pattern recognition and clinicians’ previous experiences. They are used in a preconscious, nonconscious, automatic, or reflexive manner and often are viewed as subjective and prone to error.3,5 Tacit or intuitive knowing “refers to knowledge that functions at the periphery of attention and makes possible the conventionally recognized explicit domains of human knowledge.”6(p293) Tacit knowing is akin to “knowing how” and represents practical understanding of how things are done. This knowledge encompasses the intricacy of different experiences that are acquired over time, which clinicians use to effectively complete tasks—such as diagnosing, prescribing, and choosing diagnostic tests.4 The key to tacit knowledge is experience. An example of how students acquire tacit knowledge is their interpretation of chest x-ray changes used to diagnose pulmonary edema. At the beginning, students focus on basic radiographic knowledge. Later, with time and experience reading radiographs, students interiorize the knowledge and enter a “new world,”7 becoming more confident in their ability to use this diagnostic tool. Tacit knowledge depends on pattern recognition. Pattern recognition involves comparing the current presentation of a patient with a previous case stored in memory, and making a judgment on the basis of the likelihood of a match with similar experiential cues, schema, or scripts, that is, a classic presentation. Examples of these presentations in the critical care arena might include meningitis, hypertensive crisis, acute myocardial infarction, acute pulmonary edema, and cerebral vascular accident. From experience, the clinician recognizes characteristic findings that match the pattern of a preliminary diagnosis without considering alternative diagnoses. These knowledge structures, or patterns, that contain information
about risk factors and manifestations of a disease are called illness scripts. These scripts allow clinicians to quickly recognize patterns of diseases while filtering out extraneous and irrelevant information, thus narrowing possible diagnoses.8 New clinicians have limited knowledge of illness scripts to filter in reaching a differential diagnosis. With repeated exposure to various patient encounters, or simulation experiences, clinicians gradually develop multiple illness scripts. Although pattern recognition and the application of illness scripts are usually a rapid and easy process, they are not without issue—not all patients fit into a previously recognized pattern. Thus, other types of reasoning must be used. Analytic (Diagnostic) Reasoning Hypothetico-Deductive Reasoning
Whereas nonanalytic reasoning is subconscious, analytic or diagnostic reasoning is a conscious, active, and analytic process.4 This approach consists of cognitive elements, such as logic, which represent a complex process of critical thinking, objective analysis, and reflection.3,9 The classic diagnostic strategy is hypothetico-deductive reasoning. This process begins with the generation of several differential diagnoses that, followed by data collection, drive further inquiry aimed at revealing additional information that either confirms or refutes the differentials generated. Using the hypothetico-deductive method, clinicians must consider diseases that are imperative not to miss.10 As with nonanalytic reasoning, beginning students are taught to take a thorough history and perform a detailed physical examination. Clinical judgments are then made during and after data collection that culminate in various differential diagnoses. In contrast to novices, experienced clinicians will use a few key findings to focus on the most likely diagnoses in a relatively short period of time. They will elicit information about the patient’s main complaint while they listen, observe, and formulate potential diagnoses from which more specific questions are asked—all of which clarify the problem the patient is describing. These differentials then direct subsequent data collection, such as diagnostic testing. A “think aloud” strategy, in which expert clinicians verbalize as they reason, can be helpful in mentoring novice clinicians.
231 Copyright © 2014 American Association of Critical-Care Nurses. Unauthorized reproduction of this article is prohibited.
NCI-D-14-00015.indd 231
09/07/14 11:29 AM
S C OR D O
W W W.A ACNA DVA NCE D CRIT ICA LCA RE .COM
Scheme-Inductive Reasoning
Another analytic reasoning strategy is schemeinductive reasoning (rule-based reasoning).4,11 Schemes reflect an organized knowledge structure for diagnostic reasoning. “The term scheme denotes a conceptual framework that is used in the reasoning process.” 4(p454) These schemes are similar to algorithms or decision trees, whereby the clinician proceeds down several branches that reduce the number of diagnoses. In summary, developing expertise in diagnostic reasoning requires experience with a range of analytic and nonanalytic strategies. Clinicians with an ability to use multiple methods to critically reason make better decisions, are better problem solvers, and are professionally more competent.11–13 History taking and the physical examination form the foundation of analytic reasoning. Regarding history, a wise professor once said, “If you carefully listen to the patient, the patient will usually let you know what is wrong with them” (Fuheid S. Daoud, MD, personal oral communication, September 1980). Determining which objective findings are pertinent to the primary diagnosis is the challenge in physical examination. Experience-Based Techniques (Heuristics) One factor thought to be important in the generation of diagnostic hypotheses is heuristics (experience-based techniques).14 Heuristics are shortcut mental strategies, for example, educated guess, common sense, or rule of thumb, that streamline information.15 These shortcuts can lead to efficient and correct decisions, often greatly simplifying the task of judging probability. As a result, they are frequently used by experienced clinicians. The use of heuristics, however, also can lead to predictable and recurrent errors, which will be discussed later. Commonly used heuristics include representativeness, availability, anchoring/adjustment, and premature closure16–19 (see Table 1). Representativeness can be described by the question, “How closely does patient A match patient B?” Consider a patient with symptoms of acute pancreatitis. The clinician’s estimate of the probability of acute pancreatitis will depend on how closely the patient matches previous encounters or the textbook picture. Availability is described as a situation in which a clinician ascribes a higher probability to the patient’s diagnosis because she or he vividly remembers a recent case that was very simi-
lar, which is a function of familiarity with a clinical entity. For example, the clinician recently diagnosed Exserohilum rostratum meningitis in a patient with lumbar-sacral back pain and now suspects fungal meningitis in a patient with headache. Anchoring/adjustment is a heuristic in which clinicians first use an anchor (diagnostic hypothesis) and adjust their estimates until a final answer is reached. For example, a 65-year-old man with multiple cardiovascular risk factors and chest pain presents to the emergency department (ED). After ruling out an acute event, the clinician decides to perform a graded exercise stress test to aid in the diagnosis of myocardial ischemia. Although this test is positive for ischemia at the early stage of a standard Bruce protocol, the clinician now orders a nuclear scan. The probability of a positive test is high with a nuclear scan if the patient has coronary artery disease. In this case, the clinician underestimates the probability of an abnormal exercise electrocardiogram and a diagnosis of coronary artery disease. Always remember to ask, particularly in this economic environment, “What added value will the test bring—will the results alter the treatment decisions?” If not, then reconsider the purpose of the test. Premature closure represents a situation in which a clinician settles on a diagnosis with insufficient evidence and without considering contradictory evidence. For example, the clinician mistakes a woman with a long history of migraine presenting with a severe headache for another attack of migraine, when in fact she has a subarachnoid hemorrhage. The simplest explanation for any clinical presentation should be considered first (according to the principle of Occam’s razor)2; that is, when in Kansas and you hear hoof beats, think horses not zebras. In acute and critical care settings, however, multiple illnesses may coexist. As such, Hickam’s dictum may prevail: “Patients can have as many diseases as they damn well please.”20(p682) When enough information is accumulated, the clinician will then determine the probability or the likelihood of a given disease. Estimates of probability account for the clinician’s personal experience, published literature, and attributes of the patient. If the probability or the likelihood of the diagnosis is high, such as an electrocardiogram in a 62-year-old man with chest pain that demonstrates ST-segment elevation
232 Copyright © 2014 American Association of Critical-Care Nurses. Unauthorized reproduction of this article is prohibited.
NCI-D-14-00015.indd 232
09/07/14 11:29 AM
VOL UME 2 5 • N U MBER 3 • JULY–SEPTEM BER 2014
D IFFE RE NT IA L D IAGNOSIS
Table 1: Diagnostic Cognitive Biasesa Bias Anchoring
Attribution
Availability
Description Staying with 1 diagnosis
Stereotyping a patient or gender bias
Familiarity with a clinical entity
Example
Corrective Strategy
A clinician treats a patient who presents during flu season with nausea and vomiting for gastroenteritis, but the patient later presents with appendicitis.
Ask:
An intoxicated homeless man presents with a large ulcer on his foot. The assumption is some sort of trauma and unhealthy lifestyle, when he actually has uncontrolled diabetes.
Ask:
A clinician suspects appendicitis in a patient with pancreatitis.
Ask:
What else can this be? Is there some other explanation for these data? Could 2 things be going on at once? Am I stereotyping this person? Am I biased because of his appearance? How should I acknowledge my interactions? What is the likelihood of the diagnosis? What is the prevalence of the diagnosis?
Confirmation
Diagnostic momentum
Framing
Premature closure
a
Tendency to look for confirming evidence to support a diagnosis rather than looking for evidence to refute it
A clinician treats a young female patient who takes birth control pills for calf muscle strain, and she presents later with a pulmonary embolus.
Ask:
Once diagnostic labels are attached to a patient they stick
Nurses in the emergency department ask you to see a “frequent flyer” who seeks narcotics for abdominal pain. All believe him to be seeking drugs when in essence he has an acute abdomen.
Ask:
Assembling elements that support a diagnosis
A clinician assumes pustules are the result of poison ivy in a patient who recently returned from a camping trip.
Ask:
Failing to search for additional information after reaching a diagnostic conclusion
A clinician assumes that a patient who presents with sudden abdominal pain with vomiting after attending a party where others became ill portends food poisoning, whereas in reality, the patient has a small bowel obstruction.
Ask:
What is the worst-case scenario? What other differentials should I consider to evaluate whether my diagnosis correlates with my findings? Did I perform a thorough history and physical examination? Did I arrive at my decision independent of others? Is a diagnostic time out in order?
What might be other causes? If the patient was not camping, then what would the diagnosis be?
What other information do I need? What other differential diagnoses could this be? What is the worst-case scenario? Whose opinion should I seek?
Based on data from Engel, 16 Croskerry,17,18 and Wellbery.19
in the inferior leads along with elevated troponin (treatment threshold), no further testing is needed. Conversely, if the probability is low, such as a 21-year-old man with chest pain with
normal electrocardiographic finding, further testing may be required. Using Bayes theorem2 allows clinicians to combine the objective results of a test, or a test equivalent, with their
233 Copyright © 2014 American Association of Critical-Care Nurses. Unauthorized reproduction of this article is prohibited.
NCI-D-14-00015.indd 233
09/07/14 11:29 AM
S C OR D O
W W W.A ACNA DVA NCE D CRIT ICA LCA RE .COM
prior clinical suspicions to hone the probability of a particular disease.
and can lead to erroneous conclusions (see Table 1).
Errors in Differential Diagnosis As previously mentioned, the process of differential diagnosis is not without error. Errors in the health care arena continue to capture the attention of the public, policy makers, and providers.21 Although treatment errors comprise most mistakes, errors related to the diagnostic process substantially contribute to total medical errors.22 In a systematic review of 31 studies covering 5863 autopsies, 28% reported at least 1 misdiagnosis in the intensive care unit (ICU).23 These authors estimated that as many as 40 500 adult patients annually may die in the ICU because of a misdiagnosis. Commonly missed diagnoses included pulmonary embolism, myocardial infarction, pneumonia, and aspergillosis. It may seem counterintuitive to think that patients in the ICU environment, who are closely monitored and frequently tested, are more commonly misdiagnosed. The ICU, however, is well known to be a very complex environment. Furthermore, some estimates suggest that diagnostic errors in EDs occur up to 10% of the time.24,25 In a review of 122 closed malpractice claims in which patients had alleged a missed or delayed diagnosis in the ED, 79 claims (65%) involved missed diagnoses that harmed patients.26 Of these 79 claims, 58% of the errors were due to a failure to order an appropriate diagnostic test, 42% were due to failure to perform an adequate medical history or physical examination, 37% were due to incorrect interpretation of a diagnostic test, and 33% were due to failure to order an appropriate consultation. Although missed diagnoses in the ED are multifactorial, the authors noted that most errors were due to cognitive factors—commonly mistakes in judgment or flaws in clinical reasoning. These cognitive causes of diagnostic errors can be categorized into 3 types: inadequate or faulty knowledge, faulty data gathering, and faulty synthesis (inaccurate clinical reasoning and erroneous verification of diagnostic hypotheses), with the latter accounting for the vast majority of errors.24 Although complex in nature, errors in diagnostic reasoning often are attributed to diagnostic biases or heuristics. As previously mentioned, when used correctly, heuristics expedite clinical decision making. At times, however, these biases become pitfalls
Metacognition How then do practitioners avoid errors in differential diagnosis? Metacognition—or thinking about thinking—represents a cognitive means to monitor and regulate reasoning.3 In essence, metacognition “is the capacity to think about [nonanalytic] and analytic processes at a higher level of cognitive reflection.”3(p955) The following are metacognitive techniques for clinicians and educators that may help avoid diagnostic errors.27 1. Explicitly describe heuristics and how they affect clinical reasoning. Most clinicians are unaware of these biases, let alone how they influence clinical reasoning. By developing an awareness of the types of heuristics and potential issues, clinicians may gain insight into their clinical reasoning behaviors. 2. Promote the use of “diagnostic timeouts.” Similar to having a colleague review the case by taking a fresh look, take a step back to review supporting diagnostic evidence. 3. Promote the practice of “worst-case scenario.” Consider the patient’s worst possible diagnosis. 4. Promote the use of a systematic approach to problems. For instance, use an anatomical approach to acute abdominal pain, considering the structures within and proximate to the abdomen and working through how each structure could be the source of pain. 5. Ask why. That is, ask “Why did this happen?” For example, “Why did a stable patient with a reduced ejection fraction heart failure suddenly become tachypneic?” “What precipitated this problem?” “Why did the patient with chronic obstructive pulmonary disease have an exacerbation?” “Why is a 21-year-old nonsmoker admitted with a third episode of pneumonia?” 6. Emphasize the value of the clinical history and physical examination. In this age of technology, many clinicians rely on expensive testing when, in fact, more information
234 Copyright © 2014 American Association of Critical-Care Nurses. Unauthorized reproduction of this article is prohibited.
NCI-D-14-00015.indd 234
09/07/14 11:29 AM
VOL UME 2 5 • N U MBER 3 • JULY–SEPTEM BER 2014
7.
8.
9.
10.
D IFFE RE NT IA L D IAGNOSIS
can be obtained by a thorough history and physical examination. For educators, this technique implies teaching at the bedside, modeling the diagnostic process, and incorporating various readings about the how-tos of an examination. Learn Bayesian theory as a way to direct the clinical evaluation and avoid premature closure. Bayes’ theorem is a means of quantifying uncertainty and is based on probability.3 For example, consider the likelihood that an 18-year-old with chest pain is having a myocardial infarction, compared with a 56-year-old with hypertension, hyperlipidemia, and diabetes mellitus. Although this process can be time consuming, errors in premature closure can be demonstrated by applying Bayesian reasoning to the use of pretest and posttest probabilities. Acknowledge how patients make the clinician feel. Think of the “frequent flyer” in the ED who has been identified as drug seeking and now presents with acute appendicitis, or the heroin addict whom you have treated for multiple abscess and who now presents with sudden onset of erythema and swelling in his arm. Practitioners are not without feelings and biases and may discount data in patients perceived as annoying, difficult, or frequent consumers of care. In contrast, if the practitioner has a personal relationship with the patient, data may be overlooked; that is, you cannot see the forest for the trees. Encourage learners to find clinical data that do not fit with a provisional diagnosis: Ask, “What can’t we explain?” Although Occam’s razor is a commonly invoked diagnostic rule, many patients have more than 1 diagnosis or may have a rare or complex disease. Closely examine any and all available data for any findings that the preliminary or established diagnosis cannot explain. Embrace zebras. Consider less common disorders, as they do occur. For example, a 36-year-old woman has shortness of breath and chest pain after
an appendectomy. The clinician may initially diagnose pulmonary emboli, when in reality the patient has primary pulmonary hypertension. 11. Encourage learners to slow down. Take time to think, talk out loud, and consider alternatives, which, admittedly, can be difficult in emergency situations. Getting the diagnosis right the first time, however, is not only beneficial to the patient but also can be quite economical. 12. Admit one’s own mistakes. An important part of critical thinking is reflection. Consider what worked and what did not work. Ask yourself, “How can I do better the next time?” These are important questions that improve decision making. Conclusion Acquiring a well-defined, structured body of knowledge is necessary for a clinician to become an expert.13,28,29 This knowledge results from many encounters with patients. No one is born an expert diagnostician. Nobody becomes an outstanding professional without experience, but extensive experience does not invariably lead people to become experts.30,31 Make a habit of conducting regular and frequent surveillance of your intuitive and cognitive behaviors.28 Remember that clinical expertise is a process, not a product. REFERENCES 1. Bean W, ed. Aphorisms From His Bedside Teaching and Writings. New York: Schuman; 1950:125. 2. Kassirer J, Wong J, Kopelman R. Learning Clinical Reasoning. 2nd ed. Philadelphia, PA: Lippincott Williams & Wilkins; 2010. 3. Marcum J. An integrated model of clinical reasoning: dual-process theory of cognition and metacognition. J Eval Clin Pract. 2012;18:954–961. 4. Heemskerk L, Norman G, Chou S, Mintz M, Mandin HJ, McLaughlin K. The effect of question format and task difficulty on reasoning strategies and diagnostic performance in internal medicine residents. Adv Health Sci Educ. 2008;13:453–462. 5. Offredy M. The application of decision-making concepts by nurse practitioners in general practice. Aust J Adv Nurs. 1998;28(5):532–541. 6. Henry S. Polanyi’s tacit knowing and the relevance of epistemology to clinical medicine. J Eval Clin Pract. 2010;16:293. 7. Polanyi M. Personnel Knowledge. Towards a Post-critical Philosophy (Reprinted 2002). London: Routledge, Taylor and Francis Group; 2002:101. 8. Schmidt J, Rikers R. How expertise develops in medicine: knowledge encapsulation and illness script formation. Med Educ. 2007;41:1133–1139.
235 Copyright © 2014 American Association of Critical-Care Nurses. Unauthorized reproduction of this article is prohibited.
NCI-D-14-00015.indd 235
09/07/14 11:29 AM
S C OR D O
W W W.A ACNA DVA NCE D CRIT ICA LCA RE .COM
9. Rashotte J, Carnevale F. Medical and nursing clinical decision making: a comparative epistemological analysis. Nurs Philos. 2004;5:160–174. 10. Sox H, Higgins M, Owens D. Medical Decision Making. 2nd ed. West Sussex, UK: Wiley-Blackwell; 2013. 11. Coderre S, Mandin H, Harasym P, Fick G. Diagnostic reasoning strategies and diagnostic success. Med Educ. 2003;37(8):695–703. 12. Benner PE. Educating Nurses: A Call for Radical Transformation. San Francisco, CA: Jossey-Bass; 2010. 13. Harasym P, Tsai T, Hemmait P. Current trends in developing medical students’ critical thinking abilities. Kaohsiung J Med Sci. 2008;24:341–354. 14. Kassirer J, Wong J, Kopelman R. Learning Clinical Reasoning. 2nd ed. Baltimore, MD: Wolters Kluwer, Lippincott Williams & Wilkins; 2009. 15. O’Neil E. Heuristics reasoning in diagnostic judgment. J Prof Nurs. 1995;11:239–245. 16. Engel P. Tacit knowledge and visual expertise in medical diagnostic reasoning: implications for medical education. Med Teach. 2008;30(7):184–188. 17. Croskerry P. From mindless to mindful practice—cognitive bias and clinical decision making. N Engl J Med. 2013;368(26):2445–2448. 18. Croskerry P. The importance of cognitive errors in diagnosis and strategies to minimize them. Acad Med. 2003;78(8):775–780. 19. Wellbery C. Flaws in clinical reasoning. Am Fam Physician. 2011;84:1042–1043. 20. Pate J, Kindermann D, Hudson K. A case of Hickam’s dictum: concurrent appendicitis and ectopic pregnancy. J Emerg Med. 2013;45(5):679–682. 21. Koah KT, Corigan JM, Donnaldson MS. To Err Is Human: Building a Safe Health System. Washington, DC: National Academies Press; 1999.
22. van den Berge K, Mamede S. Cognitive diagnostic error in internal medicine. Eur J Intern Med. 2013;24(6):525– 529. Available from: Science Citation Index, Ipswich, MA. Accessed November 11, 2013. 23. Winters B, Custer J, Newman-Toker D, et al. Diagnostic errors in the intensive care unit: a systematic review of autopsy studies. BMJ Qual Saf. 2012;21(11):894–902. Available from: CINAHL Plus with Full Text, Ipswich, MA. Accessed December 1, 2013. 24. Scott IA. Errors in clinical reasoning: causes and remedial strategies. BMJ. 2009;338:b1860. 25. Normal GR, Eva KW. Diagnostic error and clinical reasoning. Ed Educ. 2010;44(1):94–100. 26. Kachalia A, Gandhi T, Studdert D, et al. Missed and delayed diagnoses in the emergency department: a study of closed malpractice claims from 4 liability insurers. Ann Emerg Med. 2007;49(2):196–205. Available from: Science Citation Index, Ipswich, MA. 27. Trowbridge R. Twelve tips for teaching avoidance of diagnostic errors. Med Teach. 2008;30(5):496–500. Available from: Academic Search Complete, Ipswich, MA. 28. Ericsson K. Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Acad Med. 2004;70(10): S70–S81 29. Norman G, Eva K. Doggie diagnosis, diagnostic success and diagnostic reasoning strategies: an alternative view. Med Educ. 2003;37(8):676–677. 30. Alexander E. Perspective: moving students beyond an organ-based approach when teaching medical interviewing and physical examination. Acad Med. 2008;83 (10):906–909. 31. Graber M, Gordon R, Franklin N. Reducing diagnostic errors in medicine: what’s the goal? Acad Med. 2002;77(10):981–992.
236 Copyright © 2014 American Association of Critical-Care Nurses. Unauthorized reproduction of this article is prohibited.
NCI-D-14-00015.indd 236
09/07/14 11:29 AM
