*MEDLEARN* A Computer-Assisted Instruction (CAI) Program for MEDLARS* BY LAURA J. EISENBERG, On-Line Training Coordinator National
ibrarv, Medicine Bethesda, Marvland
RoY A. STANDING, Programminier
Conputer A ssis ted Education and Servicies George Washington Universit Y Medical C'enter Washington, D.C.
Office oJ
CHARLES S. TIDBA.L, PH.D., M.D., Director Computer A ssis ted Educ ation and Services George Washington UniversitY Medical ('enter Washington, D.C.
Of/ice of
JOSEPH LEITFR, PH.D, Associate Director L ibrarv Operations National LibrarY oJ Medicine Bethesda, MarYland
ABSTRACT
*MEDLEARN*, a second-generation computerassisted instruction (CAI) program available (nationally) since October 1976, provides on-line training for MEDLINE, one of the National Librarv of Medicine's (NLM) Medical Literature Analysis and Retrieval System (MEDLARS) data bases. *MEDLEARN* was developed as ajoint efrort between NLM and The George Washington University Medical Center. Using MEDLINE formats throughout, *MEDLEARN* combines tutorial dialogue, drill and practice, testing, and simulation. The program was designed in three tracks oriented to basic methods, advanced techniques, and new developments. Each topic is presented on two levels, permitting an alternate explanation for users encountering difficulty. *MEDLEARN*, coded in the cornputer language PILOT, was developed with a modular structure which promotes ease ot writing and revision. A versatile control structure maximizes student control. Frequent interactions check immediate recall, general comprehension, and integration of knowledge. Two MEDLINE simulations are included, providing the student an opportunity to formulate and execute a search, have it evaluated, and then perform the search in MEDLINE. Commenting, news broadcasting, and monitoring (with permission only) capabilities are also available. Subjective field appraisals
*This project was supported in part by Contract No. NOI LM 4 4701 from the National Library of Medicine.
6
have been positive and NLM plans to expand *MEDLEARN* and producc similar programs for other data bases.
THE Medical Literature Analysis and Retrieval System (MEDLARS) of the National Library of Medicine (NLM) provides on-line retrieval of citations to the biomedical literature for health science researchers, practitioners, and students. To facilitate access to and use of the data bases which constitute MEDLARS, the library has developed a variety of instructional tools and training seminars. This educational enfort has attempted to meet the diverse needs and backgrounds of medical librarians, technical information specialists, and library school students, as well as the ultimate users of the MEDLARS bibliographies. TRADITIONAL METHODS
The library's educational program has in the past taken traditional forms, including printed materials (manuals and fact sheets), audiovisuals (slides, audio- and videocassettes, movies), and personal instruction (workshops, courses, tuBull. Med. Libr.
A.ssoc. 66(1 )Januarr 1978
*MEDLEARN*
torials, and demonstrations). The experience of those associated with the program has revealed inherent limitations in each of these instructional methodologies. Printed materials which attempt to describe a developing and dynamic computer system are, by nature of the medium, always out-of-date. Further, no matter how well written and supplemented with illustrative examples, manuals cannot give the student a full appreciation of the intricacies of actually using the system. Audiovisuals typically seek to demonstrate the computer system by guiding the viewer through a sample interaction. Such simulation is instructive, but cannot go far enough. Audiovisuals are limited by format and time constraints which preclude the use of multiple examples and extensive user feedback, and therefore present a narrow view of the complexities of on-line searching. In addition, an AV presentation cannot adapt to the proficiency or information requirements of a diverse audience. Personal instruction has been the most successful of the traditional educational methods. Perhaps the major reason for this success is that it is interactive and therefore can be tailored to the particular needs and abilities of each student. The disadvantages of this technique are the time constraints, financial considerations, and geographic limitations. Notably, personal instruction (unless augmented by "hands-on" experience) fails to expose the student to the practical aspects of using the system. The preceding discussion presents only a sampling of the difficulties encountered in providing training for users of an interactive computer system by traditional means. These problems are not unique to the National Library of Medicine; indeed, they have been experienced by others engaged in various educational programs [1,2,3].
7. guided and realistic experience with the hardware mechanics and search characteristics of the retrieval system 8. utility as a reference tool and for updating skills 9. widespread availability with respect to time, location, and size of audience 10. reliability and ease of access 11. ease of development and modification 12. comprehensive evaluation, record keeping, and feedback on student progress 13. cost and resource effectiveness. The National Library of Medicine has invested considerable time and effort in the exploration of various educational methods which might meet the above requirements. In 1972 the library contracted with The George Washington University Medical Center (GWUMC) to produce a prototype learning package to augment the NLM's training courses on MEDLINE and related data bases. The instructional medium which was selected is called computer-assisted instruction; there is an extensive literature on the use of CAI in other settings [4-10]. A teaching program for MEDLINE was chosen as the first programming effort, to be known as ***MEDLEARN*** [11]. This program was composed of approximately forty sections, including two MEDLINE search simulations. It provided a capability (branching) which allowed the selection of individualized learning sequences, as well as a facility for recording users' comments. This program was available through the TYMSHARE Communications Network, on its PDP-10 computer in California. Following its experiences with this first program, the NLM supported an additional effort with GWUMC to design and develop a CAI package on the MEDLARS system, to be resident on the library's in-house computer system, and inA NEW INSTRUCTIONAL APPROACH tegrally coordinated with NLM's training proAn ideal instructional program for a com- grams. The programming language PILOT, puterized system, regardless of the medium, developed for CAI by the University of California, San Francisco, was selected for this project. A would include the following characteristics: 1. adjustability to the previous experience and new and greatly enhanced teaching program for MEDLINE has been developed and is the subject cognitive level of each student 2. adjustability to the learning capability of of the remainder of this paper; this program is named *MEDLEARN*.t each student 3. abundant patience tThe original ***MEDLEARN*** had three 4. capability of motivating the student 5. flexibility in length and sequence of instruc- asterisks on each side of its name to distinguish it from MEDLINE and thereby prevent possible confusion. The tional material current *MEDLEARN* has one asterisk preceding and 6. requisite variety of responses to deal with following the name to distinguish it from both MEDthe diversity of student input LINE and the earlier CAI program. Bull. Med. Libr. Assoc. 66(1 )January 1978
7
EISENBERG AND OTHERS
Track A
Chapter I
Chapter 2
Chapter 3
Chapter 4
Using the Terminal
General Statement on MEDLINE
First Simulated Search
MeSH
Chapter 10
Chapter 12
Stringsearch
Track B
Track C
Chapter 11
Chapter 20
Chapter 21
Offsearch
Storesearch
Chapter 13
Explosions
Chapter 22
Chapter 23 Other NLM Data Bases
FIG. I.-*MEDLEARN* tracks.
THE TEAM APPROACH such dissimilar user populations, *MEDLEARN* In recognition of the need for expertise in was designed to have three distinct tracks (see programming, CAI methodology, educational Figure 1). Track A (see Table 1) was intended for philosophy, and subject content, a joint newcomers to both computerized systems and NLM/GWUMC *MEDLEARN* team was MEDLINE. Upon completion of Track A, the formed, including supervisory as well as develop- user should be able to perform simple and ment and programming staff. One member of the straightforward searches. The instructional design team concentrated primarily on the textual of Track B presumed successful completion of content of *MEDLEARN*. This included de- Track A or equivalent experience, and would termination of the subject matter, level of instruc- provide training in additional search capabilities of tion, presentation sequence, user interaction, and the NLM system. Track C was designed to educational approach. Another team member describe specialized system capabilities, new provided the control programming which endows developments and update information, and ma*MEDLEARN* with such technical capabilities terial on other MEDLARS data bases. (Only as of this writing.) as flexible sequencing, recursive review, discre- Track A has been completed Track A offers sufficient instruction for those tionary session,control by the student, simulation of MEDLINE; error recovery, *MEDLEARN* biomedical scientists and students who may be inNEWS broadcasting, record keeping, and student progress evaluation. The *MEDLEARN* TABLE I instructional methodology was developed from the *MEDLEARN* PRESENTATION SEQUENCE, TRACK A principles of computer-assisted instruction and Chapter Subject the experiences gained from ***MEDLEARN*** and evolved through many hours of discussion, I Using the computer terminal continual testing, and a modicum of healthy 2 General statement on MEDLINE disagreement. 3 First simulated search 4
ARCHITECTURE OF *MEDLEARN* The primary audience for *MEDLEARN* includes three diverse groups: librarians and information specialists, health scientists, and students in both these fields. To accommodate
8
5 6 7 8 9
Medical Subject Headings (MeSH) Boolean logic Commands and program messages Print commands Second simulated search Track A cumulative quiz Bull. Med. Libr. Assoc. 66(1 )January 1978
*MEDLEARN*
frequent users of MEDLINE and who may wish to do their own searching. It is a prerequisite for the NLM's Initial On-Line Services Training course and provides trained MEDLINE analysts with a helpful tool to augment their training of backup personnel. It also serves library school students as an introduction to on-line retrieval systems. Track B will be a prerequisite for the NLM's Advanced On-Line Services Training course, and a reference and review source for analysts in the field. Track C will offer practicing searchers a convenient means of updating their skills and developing new ones. It will therefore enable analysts to familiarize themselves with new capabilities before incorporating them in their search strategies. Within each track, subject matter is arranged in a modular structure of chapters, sections, and paragraphs (see Figure 2). This modularization, while transparent to the user, allows ease of modification and revision of *MEDLEARN* as necessitated by system changes and step-by-step implementation of new modules as they are written. In addition, this structure gives the student the opportunity to tailor the instructional sequence to match his particular cognitive level and instructional needs. For example, a biomedical
scientist with limited time to devote to *MEDLEARN* can learn a little about MEDLINE, its scope and capabilities, by completing only the first few chapters of the CAI program. This would require about an hour at the terminal and would provide the scientist with sufficient information to perform simple searches and to submit more meaningful search requests to the trained analyst. On the other hand, a person familiar with on-line retrieval systems may choose to utilize just those *MEDLEARN* chapters which deal with the capabilities and vocabulary specific to the NLM system. Thus, while the novice searcher is guided through the logical sequence of topics in Track A, users with different types and degrees of previous experience can select appropriate chapters from the *MEDLEARN* Presentation Sequence to produce a tailored instructional program. Cumulative quizzes and other evaluation techniques provide feedback on the user's proficiency and serve as a check on capricious use of the branching capability. In addition to the arrangements of tracks and the modularization of material within each track, *MEDLEARN* provides a third means of dealing with variations in learning abilities. At the conclusion of a topic, the student is offered the option of
FIG. 2.-Modular structure.
Bull. Med. Libr. Assoc. 66(1 )January 1978
EISENBERG AND OTHERS
C 'hapter 2
Chapter 4
Chapter 3
Chapter 5
User Progress
t Lesel 0
'-
A
'V
.\
I
o
\
o
1A 'Vt 'Vt I"
1I
Level I
A
t t
* Medlearni * Track A FcI. 3. Alternate discussions.
receiving an alternative discussion of that subject for further clarification (see Figure 3). This alternate explanation contains no new information; it is merely a restatement with additional examples and interactions. A further method by which *MEDLEARN* meets the diverse needs of its user population is the capability of terminating a session at the discretion of the user. The program may be completed over any number of sessions, the length of which are determined by the individual student's optimal learning time, comfort, time constraints, and so forth. Reentry into the program can be made at precisely that point at which the previous session was terminated by using a reentry code which *MEDLEARN* automatically provides at the end of each session. While the typical user requires three to four hours to complete Track A in its entirety, this is rarely accomplished in one sitting. It is more common, and probably more advisable, for the user to cover the material in a number of sessions. EDUCATIONAL METHODOLOGY
*MEDLEARN* combines the educational methodologies of tutorials, drill and practice, testing, and simulation. In a typical learning sequence, *MEDLEARN* presents a brief, concise tutorial on a particular topic, including one or more examples and some type of user interaction. This interaction may be in the form of a question
10
which checks the user's immediate recall of the information presented, or it may give the student an opportunity to practice what has just been learned (see Figure 4). As the chapter proceeds and the student's knowledge of the subject increases, queries and interactions test for comprehension and assimilation of material. Rather than merely remembering or reiterating what has just been presented, the student must use the information in an analogous situation. Cumulative quizzes interspersed throughout *MEDLEARN* include comprehension questions, as well as application questions, which require the student to apply the knowledge in a new and different context. Each of these question types provides both feedback and reinforcement of learning to the user. Further, the cumulative quizzes give the user progress reports and recommend necessary review. In some CAI methodologies, a student is not allowed to proceed to a new topic until competence in present material is demonstrated. This approach is used in *MEDLEARN* in the cumulative quiz at the end of Track A. However, it is believed that within a track such rigorous control of the student's progress is detrimental to the learning process. This belief is based on two observations: first, a person can read and reread material and nevertheless fail to notice or comprehend even the most explicit instructions; and secondly, repeated failure can only succeed in disenchanting, discouraging, and intimidating the Bull. Med. Libr. Assoc-. 66(1 )Januarv
1/978
*MEDLEARN* PROG: THE BACK ARROW (-) OR UNDERSCORE (_) IS USED TO DELETE THE IMMEDIATELY PRECEDING CHARACTER OR SPACE. A SECOND BACK ARROW WILL DELETE THE NEXT CHARACTER OR SPACE, AND SO FORTH. AFTER TYPING THE APPROPRIATE NUMBER OF BACK ARROWS TO DELETE EACH OF THE CHARACTERS YOU WISH TO "ERASE," TYPE THE CORRECT CHARACTERS, AS SHOWN BELOW: HYPERTENTION ----SION FOR PRACTICE, TRY THIS. AFTER THE JUST USER: CUE BELOW, TYPE LIBER (INSTEAD OF LIVER) THEN TYPE THREE BACK ARROWS TO DELETE THE ERROR AND INPUT THE CORRECT CHARACTERS.
USER: liver liber PROG: NO, THAT'S INCORRECT. THE COMPUTER RECEIVED LILIVER. REMEMBER TO TYPE ONE BACK ARROW FOR EACH CHARACTER OR SPACE YOU WISH TO DELETE, FOLLOWED BY YOUR CORRECTION. CHECK THE EXAMPLE. PLEASE TRY AGAIN. USER: liber ver PROG: Note: *MEDLEARN* accepts user input in upper or lower case. -
for or analysis of an incorrect response. (See Figure 5.) To allow the student to practice the techniques and capabilities taught in *MEDLEARN* and to provide a realistic experience in on-line searching, *MEDLEARN* provides two simulated MEDLINE searches. The first simulation, presented early in Track A, is designed to acquaint the user with the conventions of searching MEDLINE. The user follows explicit instructions and receives extensive explanations on the resulting interactions. The second simulated search is the culmination of instruction in Track A and requires the user to formulate and execute a search on a miniature version of the MEDLINE data base. The user is given a search topic and must apply all the knowledge gained in Track A to retrieve a complete and relevant bibliography. Upon completion of the search, *MEDLEARN* provides an evaluation of the retrieval. The user is encouraged to perform this search in both the *MEDLEARN* simulation and MEDLINE. Although these simulations do not and can not duplicate all the capabilities of MEDLINE, those which are included are true representations of their MEDLINE counterparts. On the other hand, the protected environment created by the
----
FIG. 4.-Sample learning
sequence.
student. Therefore, when a user experiences repeated difficulty with a particular question or topic, *MEDLEARN* takes one of the following actions: 1. provides the correct answer and an alternate question, 2. refers the user to an external source and permits one more attempt at the original question, or 3. gives the correct answer with an explanation. In every case, after invoking one (or a combination) of these methods, *MEDLEARN* permits the user to proceed to the next topic. At all times, feedback to the user is designed to be of a positive nature. It is never intended to be belittling, as this would annoy the student and detract from the program's goals. At a minimum, the user is given confirmation of a correct response and feedback Bull. Med. Libr. Assoc. 66(1)January 1978
PROG: HOW WOULD YOU SEARCH FOR CITATIONS TO JOURNAL ARTICLES OR MONOGRAPH CHAPTERS BY DR. ROBERT M. BLAKE? USER: rm blake PROG: SORRY, THAT'S NOT RIGHT. THE CORRECT ANSWER IS: BLAKE RM THE LAST NAME, FOLLOWED BY A SINGLE SPACE, FOLLOWED BY THE INITIALS OF THE FIRST AND MIDDLE NAME. THE DESIGNATION DR. IS IGNORED.
TRY ANOTHER: RETRIEVE CITATIONS FOR ARTICLES WRITTEN BY ARLENE F. SMITH, PH.D.
USER: smith af PROG: NOW YOU'VE GOT IT! FIG. 5.-Sample question
sequence.
I1I
EISENBERG AND OTHERS
simulation shelters the new user from many areas of potential confusion. Throughout the program, all interactions between the student and *MEDLEARN* mimic the format and syntax of the MEDLARS system. Therefore, almost unconsciously, the *MEDLEARN* student becomes familiar and comfortable with the conventions of communicating with MEDLINE. ADDED BENEFITS *MEDLEARN* provides three capabilities which are rarely available in instructional media such as printed texts and audiovisuals: commenting, news broadcasting, and record keeping. At any point during the use of *MEDLEARN*, the student may send a comment to the NLM, and may even request a written reply. These comments may be on any subject, such as requests for further information or clarification. The news broadcast facility enables *MEDLEARN* to inform all users of new capabilities, additions to the program, changes in availability of the system, or any other message of general interest. The recordkeeping capability provides detailed accounts from which one can determine total lesson time and time per module, areas of difficulty or confusion, speed of learning, level of comprehension, and so forth. These records are kept only in those cases where permission to monitor an individual student's interactions is requested and granted.
of responding appropriately to all correct student input, and, to a more limited degree, to incorrect user input; however, it cannot deal with the person who wishes to have a question rephrased or clarified. Finally, *MEDLEARN* must be updated regularly as MEDLARS changes. Such changes occur frequently as the system continues to develop and *MEDLEARN* must be updated diligently to maintain its currency and therefore its usefulness. The need for continual investment of effort for maintenance and further development of *MEDLEARN* is somewhat offset by the fact that there is only one copy of the program which needs to be changed. All changes are immediate and leave no outmoded versions to be recalled.
*MEDLEARN* AVAILABILITY Since November 1975, *MEDLEARN* has been used in conjunction with the NLM training program. On October 1, 1976, Track A was made available six days per week to MEDLINE centers, library schools, and health-related institutions throughout the country. In its first full month of availability, 156 hours of use were logged. (See Table 2 for usage since October 1976.) It is difficult to evaluate educational packages of this kind because there are few, if any, objective criteria on which to base an assessment. Nevertheless, many *MEDLELARN* users have responded to questionnaires which have provided anecdotal reactions. A tabulation of more than seventy such responses indicates that most users have found the program to be intelligible, the sequence of topics logical, instructions clear, and both questions and responses appropriate and sufficient for the material presented. Respondents also indicated that after completing *MEDLEARN*, they
LIMITATIONS OF *MEDLEARN* As with any mode of instruction, *MEDLEARN*, and CAI in general, has its limitations. Although the cost of maintaining, improving, and updating the program is quite low, the investment required for initial development and implementation is relatively high as compared TABLE 2 with other teaching methods. Further, a CAI *MEDLEARN* USAGE system requires a significant degree of technical Month No. hourst sophistication. This includes the hardware components, that is, terminals, communications October 1976 155 network, and central computing facility, as well as November 1976 158 the support personnel who keep it all functioning. December 1976 186 Prepared materials, whether written or audioJanuary 1977 194 visual, have a built-in rigidity, and this is true of February 1977 198 *MEDLEARN* as well. The subject content is March 1977 203 restricted to a defined scope and cannot address April 1977 208 May 1977 215 topics beyond that scope regardless of the desires June 1977 305 of its audience. In terms of the individual user, *MEDLEARN* lacks the requisite variety of tAccess to *MEDLEARN* is subject to a limitation responses necessitated by the unpredictability of a on the maximum number of simultaneous users and student. For example, *MEDLEARN* is capable hours of availability. 12
Bull. Med. Libr. Assoc.
66(1 )JanuarY 1978
*MEDLEARN* felt sufficiently confident to perform simple their contributions and efforts during the developsearches of the MEDLINE data base. MED- ment of *MEDLEARN*, and to Davis B. McLINE search analysts expressed interest in using Carn, Special Assistant to the Director for *MEDLEARN* in the in-service training of their Program Planning, NLM, for initiating and supporting our effort. backup personnel. *MEDLEARN* was also utilized in a Medical REFERENCES Library Association continuing education course for librarians from organizations that do not have 1. DAHMER, J.; PIETZKA, B.; AND WALTHER, R. Evaluation of an integrated multi-media learning access to MEDLINE. During the first part of the system. Med. Educ. 10: 491-495, Nov. 1976. course, half of the participants used *MEDCHARLES B. Instructional Media in the LEARN* while the other half received the same 2. KLASEK, Modern School. Lincoln, Nebraska, Professional instruction in a lecture and discussion session. Educators Publications, 1972. p. 47-66. Again, objective data were difficult to obtain, but 3. THE CARNEGIE COMMISSION ON HIGHER EDUCATION. The Fourth Revolution; Instructional observation of the two groups indicated that those Technology in Higher Education. New York, students in the *MEDLEARN* group McGraw-Hill, 1972. demonstrated greater ease and confidence in 4. BRIDEGAM, W. E., JR., AND MEYERHOFF, E. Library performing the MEDLINE searches assigned participation in a biomedical communication and information network. Bull. Med. Libr. Assoc. 58: during the second part of the course. 103-111, Apr. 1970. Criticisms and suggestions received from users 5. CARUSO, D. E. Teaching with and about interactive A of number typically fell into two categories. computer-based document retrieval. In: Berton, persons requested more student interaction with Alberta D., ed. Proceedings of the Seventh Anthe program; others wanted additional feedback nual National Colloquium on Information Retrieval. Philadelphia, The College of Physicians for correct responses to program queries. of Philadelphia Medical Documentation Service, The National Library of Medicine is pursuing 1970. p. 416-427. the further development of *MEDLEARN* and 6. . Tutorial programs for operation of on-line corresponding applications of the CAI system to retrieval systems. J. Chem. Doc. 10: 98-105, May 1970. other MEDLARS data bases. A *TOXLEARN* program on TOXLINE (the library's toxicology 7. EICHHORN, M. M., AND REINECKE, R. D. Vision Information Center: a user-oriented data base. data base) should be available during 1978. Science 169: 29 3 1, July 3, 1970. The library encourages active use of *MED- 8. LYMAN, E. R. An on-line document retrieval LEARN* by novices to experience computerized strategy using the PLATO laboratory. Urbana, Computer-based Education Research Laborainstruction and learn about interactive retrieval tory, University of Illinois, 1971. (CERL Rep. no. systems. With respect to the timidity which X 21.) novices commonly feel toward computers, one last 9. HUNTER, B., et al. Learning Alternatives in U.S. (and unexpected) benefit of *MEDLEARN* must Education: Where Student and Computer Meet. New York, Educational Technology Publications, be mentioned. *MEDLEARN* users have in1975. variably been so engrossed in the learning process 10. JOHN E., ed. Programmed Learning and COULSON, that, despite themselves, they have overcome Computer-based Instruction. Proceedings of the their computer angst. Conference on Application of Digital Computers Ac KNOWL E DGM ENT
The authors wish to express their appreciation to Rose Marie Woodsmall and Julie Fuller for
Bull. Med. Libr. Assoc. 66(1l)January 1978
to Automated Instruction, 10-12 October 1961. New York, Wiley, 1962. 11. SOBEN, P., AND TIDBALL, C. S. ***MEDLEARN***: an orientation to MEDLINE. Bull. Med. Libr. Assoc. 62: 92-94, Apr. 1974.
13
ibrarv, Medicine Bethesda, Marvland
RoY A. STANDING, Programminier
Conputer A ssis ted Education and Servicies George Washington Universit Y Medical C'enter Washington, D.C.
Office oJ
CHARLES S. TIDBA.L, PH.D., M.D., Director Computer A ssis ted Educ ation and Services George Washington UniversitY Medical ('enter Washington, D.C.
Of/ice of
JOSEPH LEITFR, PH.D, Associate Director L ibrarv Operations National LibrarY oJ Medicine Bethesda, MarYland
ABSTRACT
*MEDLEARN*, a second-generation computerassisted instruction (CAI) program available (nationally) since October 1976, provides on-line training for MEDLINE, one of the National Librarv of Medicine's (NLM) Medical Literature Analysis and Retrieval System (MEDLARS) data bases. *MEDLEARN* was developed as ajoint efrort between NLM and The George Washington University Medical Center. Using MEDLINE formats throughout, *MEDLEARN* combines tutorial dialogue, drill and practice, testing, and simulation. The program was designed in three tracks oriented to basic methods, advanced techniques, and new developments. Each topic is presented on two levels, permitting an alternate explanation for users encountering difficulty. *MEDLEARN*, coded in the cornputer language PILOT, was developed with a modular structure which promotes ease ot writing and revision. A versatile control structure maximizes student control. Frequent interactions check immediate recall, general comprehension, and integration of knowledge. Two MEDLINE simulations are included, providing the student an opportunity to formulate and execute a search, have it evaluated, and then perform the search in MEDLINE. Commenting, news broadcasting, and monitoring (with permission only) capabilities are also available. Subjective field appraisals
*This project was supported in part by Contract No. NOI LM 4 4701 from the National Library of Medicine.
6
have been positive and NLM plans to expand *MEDLEARN* and producc similar programs for other data bases.
THE Medical Literature Analysis and Retrieval System (MEDLARS) of the National Library of Medicine (NLM) provides on-line retrieval of citations to the biomedical literature for health science researchers, practitioners, and students. To facilitate access to and use of the data bases which constitute MEDLARS, the library has developed a variety of instructional tools and training seminars. This educational enfort has attempted to meet the diverse needs and backgrounds of medical librarians, technical information specialists, and library school students, as well as the ultimate users of the MEDLARS bibliographies. TRADITIONAL METHODS
The library's educational program has in the past taken traditional forms, including printed materials (manuals and fact sheets), audiovisuals (slides, audio- and videocassettes, movies), and personal instruction (workshops, courses, tuBull. Med. Libr.
A.ssoc. 66(1 )Januarr 1978
*MEDLEARN*
torials, and demonstrations). The experience of those associated with the program has revealed inherent limitations in each of these instructional methodologies. Printed materials which attempt to describe a developing and dynamic computer system are, by nature of the medium, always out-of-date. Further, no matter how well written and supplemented with illustrative examples, manuals cannot give the student a full appreciation of the intricacies of actually using the system. Audiovisuals typically seek to demonstrate the computer system by guiding the viewer through a sample interaction. Such simulation is instructive, but cannot go far enough. Audiovisuals are limited by format and time constraints which preclude the use of multiple examples and extensive user feedback, and therefore present a narrow view of the complexities of on-line searching. In addition, an AV presentation cannot adapt to the proficiency or information requirements of a diverse audience. Personal instruction has been the most successful of the traditional educational methods. Perhaps the major reason for this success is that it is interactive and therefore can be tailored to the particular needs and abilities of each student. The disadvantages of this technique are the time constraints, financial considerations, and geographic limitations. Notably, personal instruction (unless augmented by "hands-on" experience) fails to expose the student to the practical aspects of using the system. The preceding discussion presents only a sampling of the difficulties encountered in providing training for users of an interactive computer system by traditional means. These problems are not unique to the National Library of Medicine; indeed, they have been experienced by others engaged in various educational programs [1,2,3].
7. guided and realistic experience with the hardware mechanics and search characteristics of the retrieval system 8. utility as a reference tool and for updating skills 9. widespread availability with respect to time, location, and size of audience 10. reliability and ease of access 11. ease of development and modification 12. comprehensive evaluation, record keeping, and feedback on student progress 13. cost and resource effectiveness. The National Library of Medicine has invested considerable time and effort in the exploration of various educational methods which might meet the above requirements. In 1972 the library contracted with The George Washington University Medical Center (GWUMC) to produce a prototype learning package to augment the NLM's training courses on MEDLINE and related data bases. The instructional medium which was selected is called computer-assisted instruction; there is an extensive literature on the use of CAI in other settings [4-10]. A teaching program for MEDLINE was chosen as the first programming effort, to be known as ***MEDLEARN*** [11]. This program was composed of approximately forty sections, including two MEDLINE search simulations. It provided a capability (branching) which allowed the selection of individualized learning sequences, as well as a facility for recording users' comments. This program was available through the TYMSHARE Communications Network, on its PDP-10 computer in California. Following its experiences with this first program, the NLM supported an additional effort with GWUMC to design and develop a CAI package on the MEDLARS system, to be resident on the library's in-house computer system, and inA NEW INSTRUCTIONAL APPROACH tegrally coordinated with NLM's training proAn ideal instructional program for a com- grams. The programming language PILOT, puterized system, regardless of the medium, developed for CAI by the University of California, San Francisco, was selected for this project. A would include the following characteristics: 1. adjustability to the previous experience and new and greatly enhanced teaching program for MEDLINE has been developed and is the subject cognitive level of each student 2. adjustability to the learning capability of of the remainder of this paper; this program is named *MEDLEARN*.t each student 3. abundant patience tThe original ***MEDLEARN*** had three 4. capability of motivating the student 5. flexibility in length and sequence of instruc- asterisks on each side of its name to distinguish it from MEDLINE and thereby prevent possible confusion. The tional material current *MEDLEARN* has one asterisk preceding and 6. requisite variety of responses to deal with following the name to distinguish it from both MEDthe diversity of student input LINE and the earlier CAI program. Bull. Med. Libr. Assoc. 66(1 )January 1978
7
EISENBERG AND OTHERS
Track A
Chapter I
Chapter 2
Chapter 3
Chapter 4
Using the Terminal
General Statement on MEDLINE
First Simulated Search
MeSH
Chapter 10
Chapter 12
Stringsearch
Track B
Track C
Chapter 11
Chapter 20
Chapter 21
Offsearch
Storesearch
Chapter 13
Explosions
Chapter 22
Chapter 23 Other NLM Data Bases
FIG. I.-*MEDLEARN* tracks.
THE TEAM APPROACH such dissimilar user populations, *MEDLEARN* In recognition of the need for expertise in was designed to have three distinct tracks (see programming, CAI methodology, educational Figure 1). Track A (see Table 1) was intended for philosophy, and subject content, a joint newcomers to both computerized systems and NLM/GWUMC *MEDLEARN* team was MEDLINE. Upon completion of Track A, the formed, including supervisory as well as develop- user should be able to perform simple and ment and programming staff. One member of the straightforward searches. The instructional design team concentrated primarily on the textual of Track B presumed successful completion of content of *MEDLEARN*. This included de- Track A or equivalent experience, and would termination of the subject matter, level of instruc- provide training in additional search capabilities of tion, presentation sequence, user interaction, and the NLM system. Track C was designed to educational approach. Another team member describe specialized system capabilities, new provided the control programming which endows developments and update information, and ma*MEDLEARN* with such technical capabilities terial on other MEDLARS data bases. (Only as of this writing.) as flexible sequencing, recursive review, discre- Track A has been completed Track A offers sufficient instruction for those tionary session,control by the student, simulation of MEDLINE; error recovery, *MEDLEARN* biomedical scientists and students who may be inNEWS broadcasting, record keeping, and student progress evaluation. The *MEDLEARN* TABLE I instructional methodology was developed from the *MEDLEARN* PRESENTATION SEQUENCE, TRACK A principles of computer-assisted instruction and Chapter Subject the experiences gained from ***MEDLEARN*** and evolved through many hours of discussion, I Using the computer terminal continual testing, and a modicum of healthy 2 General statement on MEDLINE disagreement. 3 First simulated search 4
ARCHITECTURE OF *MEDLEARN* The primary audience for *MEDLEARN* includes three diverse groups: librarians and information specialists, health scientists, and students in both these fields. To accommodate
8
5 6 7 8 9
Medical Subject Headings (MeSH) Boolean logic Commands and program messages Print commands Second simulated search Track A cumulative quiz Bull. Med. Libr. Assoc. 66(1 )January 1978
*MEDLEARN*
frequent users of MEDLINE and who may wish to do their own searching. It is a prerequisite for the NLM's Initial On-Line Services Training course and provides trained MEDLINE analysts with a helpful tool to augment their training of backup personnel. It also serves library school students as an introduction to on-line retrieval systems. Track B will be a prerequisite for the NLM's Advanced On-Line Services Training course, and a reference and review source for analysts in the field. Track C will offer practicing searchers a convenient means of updating their skills and developing new ones. It will therefore enable analysts to familiarize themselves with new capabilities before incorporating them in their search strategies. Within each track, subject matter is arranged in a modular structure of chapters, sections, and paragraphs (see Figure 2). This modularization, while transparent to the user, allows ease of modification and revision of *MEDLEARN* as necessitated by system changes and step-by-step implementation of new modules as they are written. In addition, this structure gives the student the opportunity to tailor the instructional sequence to match his particular cognitive level and instructional needs. For example, a biomedical
scientist with limited time to devote to *MEDLEARN* can learn a little about MEDLINE, its scope and capabilities, by completing only the first few chapters of the CAI program. This would require about an hour at the terminal and would provide the scientist with sufficient information to perform simple searches and to submit more meaningful search requests to the trained analyst. On the other hand, a person familiar with on-line retrieval systems may choose to utilize just those *MEDLEARN* chapters which deal with the capabilities and vocabulary specific to the NLM system. Thus, while the novice searcher is guided through the logical sequence of topics in Track A, users with different types and degrees of previous experience can select appropriate chapters from the *MEDLEARN* Presentation Sequence to produce a tailored instructional program. Cumulative quizzes and other evaluation techniques provide feedback on the user's proficiency and serve as a check on capricious use of the branching capability. In addition to the arrangements of tracks and the modularization of material within each track, *MEDLEARN* provides a third means of dealing with variations in learning abilities. At the conclusion of a topic, the student is offered the option of
FIG. 2.-Modular structure.
Bull. Med. Libr. Assoc. 66(1 )January 1978
EISENBERG AND OTHERS
C 'hapter 2
Chapter 4
Chapter 3
Chapter 5
User Progress
t Lesel 0
'-
A
'V
.\
I
o
\
o
1A 'Vt 'Vt I"
1I
Level I
A
t t
* Medlearni * Track A FcI. 3. Alternate discussions.
receiving an alternative discussion of that subject for further clarification (see Figure 3). This alternate explanation contains no new information; it is merely a restatement with additional examples and interactions. A further method by which *MEDLEARN* meets the diverse needs of its user population is the capability of terminating a session at the discretion of the user. The program may be completed over any number of sessions, the length of which are determined by the individual student's optimal learning time, comfort, time constraints, and so forth. Reentry into the program can be made at precisely that point at which the previous session was terminated by using a reentry code which *MEDLEARN* automatically provides at the end of each session. While the typical user requires three to four hours to complete Track A in its entirety, this is rarely accomplished in one sitting. It is more common, and probably more advisable, for the user to cover the material in a number of sessions. EDUCATIONAL METHODOLOGY
*MEDLEARN* combines the educational methodologies of tutorials, drill and practice, testing, and simulation. In a typical learning sequence, *MEDLEARN* presents a brief, concise tutorial on a particular topic, including one or more examples and some type of user interaction. This interaction may be in the form of a question
10
which checks the user's immediate recall of the information presented, or it may give the student an opportunity to practice what has just been learned (see Figure 4). As the chapter proceeds and the student's knowledge of the subject increases, queries and interactions test for comprehension and assimilation of material. Rather than merely remembering or reiterating what has just been presented, the student must use the information in an analogous situation. Cumulative quizzes interspersed throughout *MEDLEARN* include comprehension questions, as well as application questions, which require the student to apply the knowledge in a new and different context. Each of these question types provides both feedback and reinforcement of learning to the user. Further, the cumulative quizzes give the user progress reports and recommend necessary review. In some CAI methodologies, a student is not allowed to proceed to a new topic until competence in present material is demonstrated. This approach is used in *MEDLEARN* in the cumulative quiz at the end of Track A. However, it is believed that within a track such rigorous control of the student's progress is detrimental to the learning process. This belief is based on two observations: first, a person can read and reread material and nevertheless fail to notice or comprehend even the most explicit instructions; and secondly, repeated failure can only succeed in disenchanting, discouraging, and intimidating the Bull. Med. Libr. Assoc-. 66(1 )Januarv
1/978
*MEDLEARN* PROG: THE BACK ARROW (-) OR UNDERSCORE (_) IS USED TO DELETE THE IMMEDIATELY PRECEDING CHARACTER OR SPACE. A SECOND BACK ARROW WILL DELETE THE NEXT CHARACTER OR SPACE, AND SO FORTH. AFTER TYPING THE APPROPRIATE NUMBER OF BACK ARROWS TO DELETE EACH OF THE CHARACTERS YOU WISH TO "ERASE," TYPE THE CORRECT CHARACTERS, AS SHOWN BELOW: HYPERTENTION ----SION FOR PRACTICE, TRY THIS. AFTER THE JUST USER: CUE BELOW, TYPE LIBER (INSTEAD OF LIVER) THEN TYPE THREE BACK ARROWS TO DELETE THE ERROR AND INPUT THE CORRECT CHARACTERS.
USER: liver liber PROG: NO, THAT'S INCORRECT. THE COMPUTER RECEIVED LILIVER. REMEMBER TO TYPE ONE BACK ARROW FOR EACH CHARACTER OR SPACE YOU WISH TO DELETE, FOLLOWED BY YOUR CORRECTION. CHECK THE EXAMPLE. PLEASE TRY AGAIN. USER: liber ver PROG: Note: *MEDLEARN* accepts user input in upper or lower case. -
for or analysis of an incorrect response. (See Figure 5.) To allow the student to practice the techniques and capabilities taught in *MEDLEARN* and to provide a realistic experience in on-line searching, *MEDLEARN* provides two simulated MEDLINE searches. The first simulation, presented early in Track A, is designed to acquaint the user with the conventions of searching MEDLINE. The user follows explicit instructions and receives extensive explanations on the resulting interactions. The second simulated search is the culmination of instruction in Track A and requires the user to formulate and execute a search on a miniature version of the MEDLINE data base. The user is given a search topic and must apply all the knowledge gained in Track A to retrieve a complete and relevant bibliography. Upon completion of the search, *MEDLEARN* provides an evaluation of the retrieval. The user is encouraged to perform this search in both the *MEDLEARN* simulation and MEDLINE. Although these simulations do not and can not duplicate all the capabilities of MEDLINE, those which are included are true representations of their MEDLINE counterparts. On the other hand, the protected environment created by the
----
FIG. 4.-Sample learning
sequence.
student. Therefore, when a user experiences repeated difficulty with a particular question or topic, *MEDLEARN* takes one of the following actions: 1. provides the correct answer and an alternate question, 2. refers the user to an external source and permits one more attempt at the original question, or 3. gives the correct answer with an explanation. In every case, after invoking one (or a combination) of these methods, *MEDLEARN* permits the user to proceed to the next topic. At all times, feedback to the user is designed to be of a positive nature. It is never intended to be belittling, as this would annoy the student and detract from the program's goals. At a minimum, the user is given confirmation of a correct response and feedback Bull. Med. Libr. Assoc. 66(1)January 1978
PROG: HOW WOULD YOU SEARCH FOR CITATIONS TO JOURNAL ARTICLES OR MONOGRAPH CHAPTERS BY DR. ROBERT M. BLAKE? USER: rm blake PROG: SORRY, THAT'S NOT RIGHT. THE CORRECT ANSWER IS: BLAKE RM THE LAST NAME, FOLLOWED BY A SINGLE SPACE, FOLLOWED BY THE INITIALS OF THE FIRST AND MIDDLE NAME. THE DESIGNATION DR. IS IGNORED.
TRY ANOTHER: RETRIEVE CITATIONS FOR ARTICLES WRITTEN BY ARLENE F. SMITH, PH.D.
USER: smith af PROG: NOW YOU'VE GOT IT! FIG. 5.-Sample question
sequence.
I1I
EISENBERG AND OTHERS
simulation shelters the new user from many areas of potential confusion. Throughout the program, all interactions between the student and *MEDLEARN* mimic the format and syntax of the MEDLARS system. Therefore, almost unconsciously, the *MEDLEARN* student becomes familiar and comfortable with the conventions of communicating with MEDLINE. ADDED BENEFITS *MEDLEARN* provides three capabilities which are rarely available in instructional media such as printed texts and audiovisuals: commenting, news broadcasting, and record keeping. At any point during the use of *MEDLEARN*, the student may send a comment to the NLM, and may even request a written reply. These comments may be on any subject, such as requests for further information or clarification. The news broadcast facility enables *MEDLEARN* to inform all users of new capabilities, additions to the program, changes in availability of the system, or any other message of general interest. The recordkeeping capability provides detailed accounts from which one can determine total lesson time and time per module, areas of difficulty or confusion, speed of learning, level of comprehension, and so forth. These records are kept only in those cases where permission to monitor an individual student's interactions is requested and granted.
of responding appropriately to all correct student input, and, to a more limited degree, to incorrect user input; however, it cannot deal with the person who wishes to have a question rephrased or clarified. Finally, *MEDLEARN* must be updated regularly as MEDLARS changes. Such changes occur frequently as the system continues to develop and *MEDLEARN* must be updated diligently to maintain its currency and therefore its usefulness. The need for continual investment of effort for maintenance and further development of *MEDLEARN* is somewhat offset by the fact that there is only one copy of the program which needs to be changed. All changes are immediate and leave no outmoded versions to be recalled.
*MEDLEARN* AVAILABILITY Since November 1975, *MEDLEARN* has been used in conjunction with the NLM training program. On October 1, 1976, Track A was made available six days per week to MEDLINE centers, library schools, and health-related institutions throughout the country. In its first full month of availability, 156 hours of use were logged. (See Table 2 for usage since October 1976.) It is difficult to evaluate educational packages of this kind because there are few, if any, objective criteria on which to base an assessment. Nevertheless, many *MEDLELARN* users have responded to questionnaires which have provided anecdotal reactions. A tabulation of more than seventy such responses indicates that most users have found the program to be intelligible, the sequence of topics logical, instructions clear, and both questions and responses appropriate and sufficient for the material presented. Respondents also indicated that after completing *MEDLEARN*, they
LIMITATIONS OF *MEDLEARN* As with any mode of instruction, *MEDLEARN*, and CAI in general, has its limitations. Although the cost of maintaining, improving, and updating the program is quite low, the investment required for initial development and implementation is relatively high as compared TABLE 2 with other teaching methods. Further, a CAI *MEDLEARN* USAGE system requires a significant degree of technical Month No. hourst sophistication. This includes the hardware components, that is, terminals, communications October 1976 155 network, and central computing facility, as well as November 1976 158 the support personnel who keep it all functioning. December 1976 186 Prepared materials, whether written or audioJanuary 1977 194 visual, have a built-in rigidity, and this is true of February 1977 198 *MEDLEARN* as well. The subject content is March 1977 203 restricted to a defined scope and cannot address April 1977 208 May 1977 215 topics beyond that scope regardless of the desires June 1977 305 of its audience. In terms of the individual user, *MEDLEARN* lacks the requisite variety of tAccess to *MEDLEARN* is subject to a limitation responses necessitated by the unpredictability of a on the maximum number of simultaneous users and student. For example, *MEDLEARN* is capable hours of availability. 12
Bull. Med. Libr. Assoc.
66(1 )JanuarY 1978
*MEDLEARN* felt sufficiently confident to perform simple their contributions and efforts during the developsearches of the MEDLINE data base. MED- ment of *MEDLEARN*, and to Davis B. McLINE search analysts expressed interest in using Carn, Special Assistant to the Director for *MEDLEARN* in the in-service training of their Program Planning, NLM, for initiating and supporting our effort. backup personnel. *MEDLEARN* was also utilized in a Medical REFERENCES Library Association continuing education course for librarians from organizations that do not have 1. DAHMER, J.; PIETZKA, B.; AND WALTHER, R. Evaluation of an integrated multi-media learning access to MEDLINE. During the first part of the system. Med. Educ. 10: 491-495, Nov. 1976. course, half of the participants used *MEDCHARLES B. Instructional Media in the LEARN* while the other half received the same 2. KLASEK, Modern School. Lincoln, Nebraska, Professional instruction in a lecture and discussion session. Educators Publications, 1972. p. 47-66. Again, objective data were difficult to obtain, but 3. THE CARNEGIE COMMISSION ON HIGHER EDUCATION. The Fourth Revolution; Instructional observation of the two groups indicated that those Technology in Higher Education. New York, students in the *MEDLEARN* group McGraw-Hill, 1972. demonstrated greater ease and confidence in 4. BRIDEGAM, W. E., JR., AND MEYERHOFF, E. Library performing the MEDLINE searches assigned participation in a biomedical communication and information network. Bull. Med. Libr. Assoc. 58: during the second part of the course. 103-111, Apr. 1970. Criticisms and suggestions received from users 5. CARUSO, D. E. Teaching with and about interactive A of number typically fell into two categories. computer-based document retrieval. In: Berton, persons requested more student interaction with Alberta D., ed. Proceedings of the Seventh Anthe program; others wanted additional feedback nual National Colloquium on Information Retrieval. Philadelphia, The College of Physicians for correct responses to program queries. of Philadelphia Medical Documentation Service, The National Library of Medicine is pursuing 1970. p. 416-427. the further development of *MEDLEARN* and 6. . Tutorial programs for operation of on-line corresponding applications of the CAI system to retrieval systems. J. Chem. Doc. 10: 98-105, May 1970. other MEDLARS data bases. A *TOXLEARN* program on TOXLINE (the library's toxicology 7. EICHHORN, M. M., AND REINECKE, R. D. Vision Information Center: a user-oriented data base. data base) should be available during 1978. Science 169: 29 3 1, July 3, 1970. The library encourages active use of *MED- 8. LYMAN, E. R. An on-line document retrieval LEARN* by novices to experience computerized strategy using the PLATO laboratory. Urbana, Computer-based Education Research Laborainstruction and learn about interactive retrieval tory, University of Illinois, 1971. (CERL Rep. no. systems. With respect to the timidity which X 21.) novices commonly feel toward computers, one last 9. HUNTER, B., et al. Learning Alternatives in U.S. (and unexpected) benefit of *MEDLEARN* must Education: Where Student and Computer Meet. New York, Educational Technology Publications, be mentioned. *MEDLEARN* users have in1975. variably been so engrossed in the learning process 10. JOHN E., ed. Programmed Learning and COULSON, that, despite themselves, they have overcome Computer-based Instruction. Proceedings of the their computer angst. Conference on Application of Digital Computers Ac KNOWL E DGM ENT
The authors wish to express their appreciation to Rose Marie Woodsmall and Julie Fuller for
Bull. Med. Libr. Assoc. 66(1l)January 1978
to Automated Instruction, 10-12 October 1961. New York, Wiley, 1962. 11. SOBEN, P., AND TIDBALL, C. S. ***MEDLEARN***: an orientation to MEDLINE. Bull. Med. Libr. Assoc. 62: 92-94, Apr. 1974.
13
