Needs Assessment for Diagnostic Decision Support Systems (DDSS) Eta S. Bemer, Ed.D. and Alwyn A. Shugerman, M.D. University of Alabama School of Medicine Diagnostic decision support systems are often developed without a clear idea of how well the system will meet the needs of its users. The present study was designed to assess the information needs of clinicians. A set of questions submitted to an information service by family physicians was used to detennine how much need there wasfordiagnosticdecision support, the types ofsupport needed, and the general content areas of their questions. Results showed that less than half of the questions were related to diagnosis and that most of those were requests for general information about a given condition. In addition, the fewest diagnosis questions were for conditions that were seen frequently in ambulatory care in a survey offamily practitioners. In assessing the usefulness of DDSS one key issue is how well they meet the needs of the potential users, e.g. physicians in need of information to support clinical decision making. Several studies have examined the sources that practitioners used to gain information [1-7]. The most frequently used sources were texts and colleagues. One of the advantages of DDSS is that they may contain information that may be more comprehensive than an individual consultant can provide. Furthermore, the DDSS can be more quickly accessed and usable than a textbook or a collection of research articles. DDSS differ, however, not only in the specific diseases contained in their knowledge base, but in the various functions that they can perform with the diseases. Some DDSS can only provide differential diagnoses of findings. Others can make work-up suggestions, display lists of findings, or even provide actual text and references. As more computer-based information resources become available, it will be important to assess how well they address practitioner needs. One method of determining practitioners' needs is to examine the kinds of questions they ask and determine how well a given information source can answer them. A few studies have been done that address these issues.
In 1988, Berner and Brooks examined the correspondence between the knowledge base of QMR, a diagnostic support system for Internal Medicine, and the requests from internists and family physicians to MIST, a telephone consultation service based at an academic medical center [8]. Results showed that the 0195-4210/91/$5.00 ©D 1992 AMIA, Inc.
605
QMR knowledge base contained some information about the majority of consultation requests made by the callers and that a small number of conditions account for most of the requests. There were problems with that study, however. The data on the questions only contained the disease or finding name and it was difficult to determine whether the information really was contained in the knowledge base, even if the particular condition were in the knowledge base. It was clear from the previous study that more detailed information on the specific questions that practitioners ask was needed. Jennett et al. [9,10] as part of a continuing medical education program offered family practitioners in Canada the opportunity to receive literature searches on clinical questions of interest. The physicians submitted their questions to a central location and trained librarians conducted the search. The results were reviewed by University-based clinical experts, additional reference materials were added if necessary, and the results were retumed to the requesting physician. Ihe researchers evaluated the practitioners' satisfaction with the information they received, as well as the time required to address their requests. Practitioners were very satisfied with the information they received. The response time to provide answers to their requests, however, ranged from minutes to weeks. If the requesting physician had conducted the search him/herself, the response time would obviously have been faster. Similarly, if the practitioner had immediate access to a computer-based decision support tool, he/she might be able to obtain answers to the questions even more efficiently, provided that the content was contained in the decision support system and that the decision support system had the functions to manipulate the content to answer the questions. An analysis of questions generated by physicians can provide data on their information needs. These data can also be used to inform the development and evaluation of DDSS. The present study used the questions submitted to the Medical Information Service of the University of Calgary Faculty of Medicine. This service became operational subsequent to the Jennett et al. study. We used these questions to determine (1) how many of
them related to diagnosis; (2) what types of fimctions in a DDSS would be needed to answer them; and, (3) the relative emphases of various content areas.
Method
Questions The questions submitted by 101 family physicians from 17 rural communities in Canada and 15 physicians from the city of Calgary were analyzed in the present study to determine the features needed by DDSS to address those needs. A total of 507 questions were analyzed. Ninety-one percent of the questions related to adult problems and 9 % related to pediatric issues.
Categorization procedure The authors determined which questions related to diagnosis and which were questions pertaining to treatment. Of those that related to diagnosis, the authors independently categorized the questions into the following five major categories: (1) general information about a disease or condition; (2) information about the relationship of a particular sign, symptom or disease to other findings or other diseases; (3) differential diagnosis of a single finding or a set of findings; (4) work-up for a given disease or finding; and, (5) miscellaneous or vague questions related to diagnosis. Questions of iatrogenic illness, e.g., the relationship between a given treatment (usually medication, but occasionally surgery) and a particular clinical finding was considered a subset of the second category. After the initial independent categorization, the authors arrived at a consensus for the appropriate category for each question. The comparison with characteristics of DDSS was then determined on the basis of the final agreed upon categories.
Question content In addition to the functional capacity of the DDSS, the specific content of the knowledge base is obviously an important determinant of how well it would meet the practitioners' needs. However, because the knowledge base of any given DDSS is constantly being increased and updated, the specific disease content of the questions in our data set may not be as useful as the relative emphasis of broader content areas. Content categories from the National Ambulatory Medical Care Survey (NAMCS) were used [11]. This survey was a nationwide survey and the data are based on over 300,000 patients and were classified into categories
606
based on ICD-9 CM codes [12]. The results of the survey show the frequency with which different conditions were seen by family practitioners. The set of questions from the information service had originally been classified by the type of specialist who reviewed the information to be provided as an answer to the questions. In some cases these subspecialties were combined to make them comparable to the NAMCS
categories. Although the NAMCS data are based on a United States sample and the questions come from Canada, they provide a basis for comparing the question content and the actual content of clinical practice. A ratio was computed of the percentage of patients seen for each category in the NAMCS data to the percentage of diagnostic questions in similar categories from the present study. Ratios greater than 1.0 indicate that there was a greater percentage of patients compared to the percentage of questions in that category, while ratios less than 1.0 indicate the reverse. Results and Discussion
After discussing the criteria, both raters agreed that diagnostically-related questions accounted for 222 (44%) of the total 507 questions, a similar percentage to those found by Jennett et al. [10]. Of these diagnostic questions, the largest category ( 96 questions) was for general information about a given
condition. It is possible that the physicians really had patient-specific questions, but did not frame them as such for this study. However, it is also possible that what they really did need was compiled information about a given topic that might be more up-to-date than a textbook. If so, text descriptions, references, access to recent literature, or lists of signs and symptoms that the user can access might be necessary to build into a DDSS to address this type of need.
The next largest category was the relationship of signs and symptoms to each other or to a disease (50 questions). The subcategory relating to manifestations of iatrogenic illness, i.e., whether a given treatment could cause specific symptoms accounted for almost half of this category. A DDSS that does not include medications as possible causes of clinical conditions might not be able to provide useful information for this category of question. Mabry and Miller [13] have suggested a mechanism for incorporating some of this drug information into an existing DDSS. Developers of new DDSS should consider including this information in the initial design.
The categories of (1) differential diagnosis of a single or a set of findings together (33 questions) and (2) work-up of a given finding, which included procedus to work-up a disease or finding, how various work-up strategies compare and how to scree for a given condition a disease (30 questions), together generated only slightly more questions than the relationships category alone. Many DDSS incorporate these differential diagnosis and work-up functions, but in this set of questions they make up a minority of the questions. It should be recognized that a DDSS that focuses on these functions may have limited use since these types of 'diagnostic puzzles" may not occur frequently in practice. Thus, limited use of DDSS may only reflect a limited need for diagnostic assistance, rather than indicg anything about the value of the system. The remaining questions were classified as miscellaneous.
As can be seen from Table 1, the top three categories (respiratory and cardiovascular disease and injuries and poisonings) were the only categories with a ratio much greater than 1.0 These categories cover approximately 40% of the patients, but less than ten percent of the questions. Thus, the conditions that make up a large part of family practitioners' caseload, e.g., cardiovascular and respiratory conditions, generated the fewest diagnostically-related questions. If the purpose of the DDSS is to provide diagnostic decision support to primary care practitioners, rather than to replace the physician, it might be best to emphasize in the kmowledge base some of the subspecialty areas that occur less frequently in primary care practice and which might need the most diagnostic decision support. A DDSS with a large knowledge base that consists mainly of commonly occumng conditions may not be much more helpfil than a DDSS with a smaller klowledge base that emphasized diagnostically difficult areas.
Table 1 shows the ratio of percentage of patients to percentage of diagnostically related questions. The categories are arranged from most to least fiequently seen patient categories. These categories covered 16.3% of the NAMCS patients and 16.8% of the 222 diagnostic questions.
The data from this study provide useful data on the information needs of practitioners. However, there are limitations to the conclusions that can be drawn. One would expect that different specialties would have somewhat different needs. Internists would probably not need information related to obstetric or pediatric questions, and subspecialists' needs would obviously be different from those of generalists. In addition, the type of secondary analysis conducted in this study has other limitations. If the respondents had been told that they could have access to information from a DDSS with the capabilities to address patient specific inquiries, they might have asked more specific and fewer general questions.
TABLE 1
Percentage of Patients and Questions for Specified Conditions Condition
Patients Ouestions Ratio*
Respiratory Disease 17.3 Cardiovascular Disease 13.2 Injuries and Poisonings 9.8 Musculoskeletal Disease 7.6 Endocrine Disease 6.2 Digestive Disease 5.6 Genito-urinary Disease 5.3 Neurological Disease 5.1 Dermatological Disease 4.0 Infectious Disease 3.3 Obstetrics 2.8 Mental Disorders 2.6 1.2 Neoplasms
3.6 .9 3.2 8.6 11.3 4.1 11.3 3.1 4.5 9.0 6.8 5.0 2.2
MISCELLANEOUS
16.8
*
16.3
4.8 14.7 3.1 .9 "'.5 1.4 .5 .4 .9 .4 .4 .5 .5
Despite these limitations, the data on the preponderance of questions on iatrogenic disease and the emphasis on the less commonly occurring content areas can provide important information for both the design and evaluation of DDSS.
Acknowledgements The authors wish to acknowledge the assistance of the University of Calgary Faculty of Medicine Office of Continuing Medical Education for providing the data that were used in this study and the Association of Canadian Medical Colleges for their support. This work was supported in part by the National Library of Medicine resarch grant RO1 LMOS125.
% of Patients % of Questions
607
7. Stinson, E. R. & Mueller, D. A. Survey of Health Professionals' Information Habits and Needs. JAMA, 1980, 2(2), 140-143.
References 1.
2.
Covell, D. G., Uman, G. C., & Manning, P. R. Information Needs in Office Practice: Are They Being Met? Annals of Internal Medicine, 1985, 103(4), 596-599.
8. Bemer, E. S. & Brooks, C. M. Needs assessment for computer-based medical decision support systems, in Proceedings of the Twelfth Annual Symposium of Computer Applications in Medical Cu. Washington, DC: Computer Society of the IEEE, 1988, 232-236.
Egan, K. L. Practicing Physician Access to Biomedical Knowledge: A National Study. Research in Medical Education: 1987, Proceedings of the Twenty-SixthAnnual Conference, sponsored by the Association of American Medical Colleges, 209-214.
9. Jeamett, P.A., Lockyer, J.M., Maes, W., Parboosingh, I.J. & Lawson, D. Providing relevant information to rural practitioners: a study of a medical information system. Teaching and Learing in Medicine, 1990, j(4), 200-204.
3. Gruppen, L. D., Wolf, F. M., Van Voorhees, C., & Stross, J. K. The Influence of Prior Experience and Confidence on Physician Preferences for Information Sources and Continuity of Care. Research in Medical Education: 1986, Proceedings of the Twenty-FifthAnnual Conference, sponsored by the Association of American Medical Colleges, 177-182. 4. Gruppen, L. D., Wolf, F. M., Van Voorhees, C., & Stross, J. K. Information Sng Strategies and Treatment Decision Maling. Reseach in Medical Education: 1987, Proceedings of the Twenty-Sixth Annual Conference, sponsored by the Association of American Medical Colleges, 203-208.
5. Moore-West, M., Northup, D., Skipper, B., & Teaf, D. Information-Seeldng Behavior Among Physicians Practicing in Urban and Nonurban Areas. Research in Medical Education: 1984, Proceedings of the Twenty-Third Annual Conference, sponsored by the Association of American Medical Colleges, 237-242. 6. Northup, D. E., Moore-West, M., Skipper, B., & Teaf, S. R. Characteristics of Clinical Information-Searching: Investigation Using Critical Incident Technique. Joural of Medical Education, 1983, 58(11), 873-881.
10. Jennett, P. A., Kishinevsky, M., Parboosingh, I.T., Lockyer, J. M., & Maes, W. R. Responses to non-emergency questions in nrual medicine: their usefulness to practice decisions. Medical Education, 1991, 259 238-242. 11. Cypress, B.K. Pattems of ambulatory care in general and family practice: the National Ambulatory Medical Care Survey. United States, January 1980-December 1981. Vital Health Stat
1(73), 1-60. 12. International Classification of Diseases - 9th Revision - Clinical Modification [ICD(CM)]. (1978) Volume 1 (Disease Tabular List), Volume 2 (Disease Index). Ann Arbor, Michigan: Commission on Professional and Hospital Activities.
13. Mabry, M.E. and Miller, R.A. Distinguishing drug toxicity syndromes from medical diseases:a QMR computer-based approach. Proceedings of thfe Fourteenth Annual Symposium of Computer Applications in Medical Care. Washington, DC: Computer Society of the IEEE, 1990, 65-71.
608
In 1988, Berner and Brooks examined the correspondence between the knowledge base of QMR, a diagnostic support system for Internal Medicine, and the requests from internists and family physicians to MIST, a telephone consultation service based at an academic medical center [8]. Results showed that the 0195-4210/91/$5.00 ©D 1992 AMIA, Inc.
605
QMR knowledge base contained some information about the majority of consultation requests made by the callers and that a small number of conditions account for most of the requests. There were problems with that study, however. The data on the questions only contained the disease or finding name and it was difficult to determine whether the information really was contained in the knowledge base, even if the particular condition were in the knowledge base. It was clear from the previous study that more detailed information on the specific questions that practitioners ask was needed. Jennett et al. [9,10] as part of a continuing medical education program offered family practitioners in Canada the opportunity to receive literature searches on clinical questions of interest. The physicians submitted their questions to a central location and trained librarians conducted the search. The results were reviewed by University-based clinical experts, additional reference materials were added if necessary, and the results were retumed to the requesting physician. Ihe researchers evaluated the practitioners' satisfaction with the information they received, as well as the time required to address their requests. Practitioners were very satisfied with the information they received. The response time to provide answers to their requests, however, ranged from minutes to weeks. If the requesting physician had conducted the search him/herself, the response time would obviously have been faster. Similarly, if the practitioner had immediate access to a computer-based decision support tool, he/she might be able to obtain answers to the questions even more efficiently, provided that the content was contained in the decision support system and that the decision support system had the functions to manipulate the content to answer the questions. An analysis of questions generated by physicians can provide data on their information needs. These data can also be used to inform the development and evaluation of DDSS. The present study used the questions submitted to the Medical Information Service of the University of Calgary Faculty of Medicine. This service became operational subsequent to the Jennett et al. study. We used these questions to determine (1) how many of
them related to diagnosis; (2) what types of fimctions in a DDSS would be needed to answer them; and, (3) the relative emphases of various content areas.
Method
Questions The questions submitted by 101 family physicians from 17 rural communities in Canada and 15 physicians from the city of Calgary were analyzed in the present study to determine the features needed by DDSS to address those needs. A total of 507 questions were analyzed. Ninety-one percent of the questions related to adult problems and 9 % related to pediatric issues.
Categorization procedure The authors determined which questions related to diagnosis and which were questions pertaining to treatment. Of those that related to diagnosis, the authors independently categorized the questions into the following five major categories: (1) general information about a disease or condition; (2) information about the relationship of a particular sign, symptom or disease to other findings or other diseases; (3) differential diagnosis of a single finding or a set of findings; (4) work-up for a given disease or finding; and, (5) miscellaneous or vague questions related to diagnosis. Questions of iatrogenic illness, e.g., the relationship between a given treatment (usually medication, but occasionally surgery) and a particular clinical finding was considered a subset of the second category. After the initial independent categorization, the authors arrived at a consensus for the appropriate category for each question. The comparison with characteristics of DDSS was then determined on the basis of the final agreed upon categories.
Question content In addition to the functional capacity of the DDSS, the specific content of the knowledge base is obviously an important determinant of how well it would meet the practitioners' needs. However, because the knowledge base of any given DDSS is constantly being increased and updated, the specific disease content of the questions in our data set may not be as useful as the relative emphasis of broader content areas. Content categories from the National Ambulatory Medical Care Survey (NAMCS) were used [11]. This survey was a nationwide survey and the data are based on over 300,000 patients and were classified into categories
606
based on ICD-9 CM codes [12]. The results of the survey show the frequency with which different conditions were seen by family practitioners. The set of questions from the information service had originally been classified by the type of specialist who reviewed the information to be provided as an answer to the questions. In some cases these subspecialties were combined to make them comparable to the NAMCS
categories. Although the NAMCS data are based on a United States sample and the questions come from Canada, they provide a basis for comparing the question content and the actual content of clinical practice. A ratio was computed of the percentage of patients seen for each category in the NAMCS data to the percentage of diagnostic questions in similar categories from the present study. Ratios greater than 1.0 indicate that there was a greater percentage of patients compared to the percentage of questions in that category, while ratios less than 1.0 indicate the reverse. Results and Discussion
After discussing the criteria, both raters agreed that diagnostically-related questions accounted for 222 (44%) of the total 507 questions, a similar percentage to those found by Jennett et al. [10]. Of these diagnostic questions, the largest category ( 96 questions) was for general information about a given
condition. It is possible that the physicians really had patient-specific questions, but did not frame them as such for this study. However, it is also possible that what they really did need was compiled information about a given topic that might be more up-to-date than a textbook. If so, text descriptions, references, access to recent literature, or lists of signs and symptoms that the user can access might be necessary to build into a DDSS to address this type of need.
The next largest category was the relationship of signs and symptoms to each other or to a disease (50 questions). The subcategory relating to manifestations of iatrogenic illness, i.e., whether a given treatment could cause specific symptoms accounted for almost half of this category. A DDSS that does not include medications as possible causes of clinical conditions might not be able to provide useful information for this category of question. Mabry and Miller [13] have suggested a mechanism for incorporating some of this drug information into an existing DDSS. Developers of new DDSS should consider including this information in the initial design.
The categories of (1) differential diagnosis of a single or a set of findings together (33 questions) and (2) work-up of a given finding, which included procedus to work-up a disease or finding, how various work-up strategies compare and how to scree for a given condition a disease (30 questions), together generated only slightly more questions than the relationships category alone. Many DDSS incorporate these differential diagnosis and work-up functions, but in this set of questions they make up a minority of the questions. It should be recognized that a DDSS that focuses on these functions may have limited use since these types of 'diagnostic puzzles" may not occur frequently in practice. Thus, limited use of DDSS may only reflect a limited need for diagnostic assistance, rather than indicg anything about the value of the system. The remaining questions were classified as miscellaneous.
As can be seen from Table 1, the top three categories (respiratory and cardiovascular disease and injuries and poisonings) were the only categories with a ratio much greater than 1.0 These categories cover approximately 40% of the patients, but less than ten percent of the questions. Thus, the conditions that make up a large part of family practitioners' caseload, e.g., cardiovascular and respiratory conditions, generated the fewest diagnostically-related questions. If the purpose of the DDSS is to provide diagnostic decision support to primary care practitioners, rather than to replace the physician, it might be best to emphasize in the kmowledge base some of the subspecialty areas that occur less frequently in primary care practice and which might need the most diagnostic decision support. A DDSS with a large knowledge base that consists mainly of commonly occumng conditions may not be much more helpfil than a DDSS with a smaller klowledge base that emphasized diagnostically difficult areas.
Table 1 shows the ratio of percentage of patients to percentage of diagnostically related questions. The categories are arranged from most to least fiequently seen patient categories. These categories covered 16.3% of the NAMCS patients and 16.8% of the 222 diagnostic questions.
The data from this study provide useful data on the information needs of practitioners. However, there are limitations to the conclusions that can be drawn. One would expect that different specialties would have somewhat different needs. Internists would probably not need information related to obstetric or pediatric questions, and subspecialists' needs would obviously be different from those of generalists. In addition, the type of secondary analysis conducted in this study has other limitations. If the respondents had been told that they could have access to information from a DDSS with the capabilities to address patient specific inquiries, they might have asked more specific and fewer general questions.
TABLE 1
Percentage of Patients and Questions for Specified Conditions Condition
Patients Ouestions Ratio*
Respiratory Disease 17.3 Cardiovascular Disease 13.2 Injuries and Poisonings 9.8 Musculoskeletal Disease 7.6 Endocrine Disease 6.2 Digestive Disease 5.6 Genito-urinary Disease 5.3 Neurological Disease 5.1 Dermatological Disease 4.0 Infectious Disease 3.3 Obstetrics 2.8 Mental Disorders 2.6 1.2 Neoplasms
3.6 .9 3.2 8.6 11.3 4.1 11.3 3.1 4.5 9.0 6.8 5.0 2.2
MISCELLANEOUS
16.8
*
16.3
4.8 14.7 3.1 .9 "'.5 1.4 .5 .4 .9 .4 .4 .5 .5
Despite these limitations, the data on the preponderance of questions on iatrogenic disease and the emphasis on the less commonly occurring content areas can provide important information for both the design and evaluation of DDSS.
Acknowledgements The authors wish to acknowledge the assistance of the University of Calgary Faculty of Medicine Office of Continuing Medical Education for providing the data that were used in this study and the Association of Canadian Medical Colleges for their support. This work was supported in part by the National Library of Medicine resarch grant RO1 LMOS125.
% of Patients % of Questions
607
7. Stinson, E. R. & Mueller, D. A. Survey of Health Professionals' Information Habits and Needs. JAMA, 1980, 2(2), 140-143.
References 1.
2.
Covell, D. G., Uman, G. C., & Manning, P. R. Information Needs in Office Practice: Are They Being Met? Annals of Internal Medicine, 1985, 103(4), 596-599.
8. Bemer, E. S. & Brooks, C. M. Needs assessment for computer-based medical decision support systems, in Proceedings of the Twelfth Annual Symposium of Computer Applications in Medical Cu. Washington, DC: Computer Society of the IEEE, 1988, 232-236.
Egan, K. L. Practicing Physician Access to Biomedical Knowledge: A National Study. Research in Medical Education: 1987, Proceedings of the Twenty-SixthAnnual Conference, sponsored by the Association of American Medical Colleges, 209-214.
9. Jeamett, P.A., Lockyer, J.M., Maes, W., Parboosingh, I.J. & Lawson, D. Providing relevant information to rural practitioners: a study of a medical information system. Teaching and Learing in Medicine, 1990, j(4), 200-204.
3. Gruppen, L. D., Wolf, F. M., Van Voorhees, C., & Stross, J. K. The Influence of Prior Experience and Confidence on Physician Preferences for Information Sources and Continuity of Care. Research in Medical Education: 1986, Proceedings of the Twenty-FifthAnnual Conference, sponsored by the Association of American Medical Colleges, 177-182. 4. Gruppen, L. D., Wolf, F. M., Van Voorhees, C., & Stross, J. K. Information Sng Strategies and Treatment Decision Maling. Reseach in Medical Education: 1987, Proceedings of the Twenty-Sixth Annual Conference, sponsored by the Association of American Medical Colleges, 203-208.
5. Moore-West, M., Northup, D., Skipper, B., & Teaf, D. Information-Seeldng Behavior Among Physicians Practicing in Urban and Nonurban Areas. Research in Medical Education: 1984, Proceedings of the Twenty-Third Annual Conference, sponsored by the Association of American Medical Colleges, 237-242. 6. Northup, D. E., Moore-West, M., Skipper, B., & Teaf, S. R. Characteristics of Clinical Information-Searching: Investigation Using Critical Incident Technique. Joural of Medical Education, 1983, 58(11), 873-881.
10. Jennett, P. A., Kishinevsky, M., Parboosingh, I.T., Lockyer, J. M., & Maes, W. R. Responses to non-emergency questions in nrual medicine: their usefulness to practice decisions. Medical Education, 1991, 259 238-242. 11. Cypress, B.K. Pattems of ambulatory care in general and family practice: the National Ambulatory Medical Care Survey. United States, January 1980-December 1981. Vital Health Stat
1(73), 1-60. 12. International Classification of Diseases - 9th Revision - Clinical Modification [ICD(CM)]. (1978) Volume 1 (Disease Tabular List), Volume 2 (Disease Index). Ann Arbor, Michigan: Commission on Professional and Hospital Activities.
13. Mabry, M.E. and Miller, R.A. Distinguishing drug toxicity syndromes from medical diseases:a QMR computer-based approach. Proceedings of thfe Fourteenth Annual Symposium of Computer Applications in Medical Care. Washington, DC: Computer Society of the IEEE, 1990, 65-71.
608
