Resource
Strategies for the assessment of competence in laboratory animal science courses Axel Kornerup Hansen, DVSc, DVM, DipECLAM & Dorte Bratbo Sørensen, PhD, DVM
© 2014 Nature America, Inc. All rights reserved.
Evaluation of skills, knowledge and competencies is an essential part of education in laboratory animal science. In Europe, a greater emphasis will be placed on such evaluations going forward, because the European Union will base its education and training framework on learning outcomes rather than on course time and syllabuses, as done previously. The authors present their experiences administering different written, oral and practical examinations for Federation of European Laboratory Animal Science Associations categories B, C and D courses. Examinations can be administered online as well as on campus, if time constraints are provided to compensate for the advantage of being able to use external resources. Overall, students benefit from exposure to multiple types of exams over the course of their education because each type prepares students for different situations.
npg
One way of ensuring the welfare of laboratory animals is to provide sufficient education to practitioners of laboratory animal science. Since 1986, educational standards for persons performing experiments with animals in Europe have been developed by both governmental and non-governmental organizations1–10 (Table 1). These educational standards have been based on the competencies required of personnel and are categorized into four levels (A, B, C and D; Table 2), as proposed by the Federation of European Laboratory Animal Science Associations (FELASA)4,5,7 and partially accepted by the Council of Europe3. For more than 15 years, our section at the University of Copenhagen has offered category B training for persons carrying out experiments with animals. These courses have a traditional structure, consisting of lectures as well as both theoretical and practical exercises. For the same period of time, we have offered category C training for persons responsible for directing animal experiments. Over the last 5 years, we have implemented a model of blended learning for these courses, which include a combination of web-based ‘e-learning’ and on-campus instruction of various types. The web-based component of the course allows students to access lectures and materials for group work in an electronic format from an online server and to raise questions on a discussion board. We also hold
plenary sessions on selected topics to allow the students to engage with the teacher as needed. A recent meta-analysis concluded that blended learning conditions result in an improvement in student achievement outcomes by approximately one-third of a standard deviation compared with classroom instruction conditions11. For more than 10 years, we have offered category D training for specialists including veterinarians, managers and animal welfare officers as a post-graduate Master’s degree in laboratory animal science. This program includes a number of courses that are structured into three phases: an initial ‘pre-teaching’ phase, during which lectures in an electronic format, materials for group work and a discussion board are provided online; an intensive teaching phase, provided on campus and consisting of plenary sessions, group work and practical demonstrations; and a ‘post-teaching’ phase, provided online and concluding with an online examination. The categories C and D educational programs offered at the university have been accredited by FELASA, the Danish Governmental Administration and study boards at University of Copenhagen. In the past, the FELASA educational guidelines for each competence category have been based on course syllabuses, requiring that a certain number of hours of training (e.g., at least 80 h for category C courses) be
Section of Experimental Animal Models, Department of Veterinary Disease Biology, University of Copenhagen, Denmark. Correspondence should be addressed to A.K.H. ([email protected]).
LAB ANIMAL
Volume 43, No. 10 | OCTOBER 2014 359
Resource
npg
© 2014 Nature America, Inc. All rights reserved.
TABLE 1 | European regulatory standards for education and training in laboratory animal science Year
Organization
Regulatory action
1986
Council of Europe (CoE)
“Authorization [to carry out a procedure] shall be granted only to persons deemed to be competent by the responsible authority”1
1986
European Economic Community (EEC)
“Experiments shall be performed solely by competent authorized persons, or under the direct responsibility of such a person”2
1993
CoE
Presents requirements for establishing appropriate training in competencies defined under the categories A, B, C and D (ref. 3)
1995
Federation of Laboratory Animal Science Associations (FELASA)
Presents detailed guidelines for training in competencies defined under the categories A and C (ref. 4)
1999
FELASA
Presents detailed guidelines for training in competencies defined under category D (ref. 5)
1999
European Union (EU)
“The [European] Community shall approve the European Convention for the protection of vertebrate animals used for experimental and other scientific purposes”6
2000
FELASA
Presents detailed guidelines for training of persons defined under category B (ref. 7)
2003
FELASA
Establishes that any European laboratory animal science course can achieve official FELASA accreditation through fulfillment of FELASA guidelines for education and training, subject to assessment and approval by a FELASA Accreditation Board8
2010
EU
“Member States should ensure through authorisation or by other means that staff are adequately educated, trained and competent. Furthermore, it is important that staff are supervised until they have obtained and demonstrated the requisite competence”9
2013
EU
Designates new categories for competence of personnel10 Proposes a training framework with a focus on learning outcomes10
completed and that certain topics be included in that training4,5,7. Any European course that adhered to the FELASA guidelines could achieve official accreditation8. As a consequence of the revised EU directive6, however, the European Union Commission issued the National Authority Consensus Paper (EU consensus paper)10, which places a greater emphasis on learning outcomes than on course time and syllabuses and, for the first time, sets standards for the assessment of learning outcomes. It states that the assessment of learning outcomes should “be objective and transparent; be comprehensible and clear without ambiguity; have clear pass-fail criteria; provide reliable results; ensure students have achieved an acceptable level of understanding of the subject—suitable to proceed to working under supervision, such that no unnecessary pain, suffering, distress or lasting harm is inflicted; [and] identify, where appropriate, critical elements that cannot be failed”10. The EU consensus paper also describes the achievement of competence within laboratory animal science as a process consisting of three steps: (i) acquisition of knowledge, (ii) practical experience and (iii) development 360 Volume 43, No. 10 | OCTOBER 2014
of deeper learning and critical thinking10. The FELASA accreditation board for education and training (E&T Board) views its role in providing accreditation for courses in laboratory animal science as ensuring the acquisition of knowledge and practical training, which should be accurately assessed through course examinations12. The current accreditation scheme includes consideration of “the method of student assessment, the range and depth of evaluation of learning outcomes and the way in which practical skills are assessed”12. In addition to the guidelines for course assessments established by FELASA and the EU commission, we have established additional defined learning outcomes and requirements that must be fulfilled in order to achieve credit for completing courses in laboratory animal science. Here, we provide our experiences with administering various types of examinations for the assessment of competence in laboratory animal science courses. Examination types We use a range of different types of examinations for category D courses, some of which are also used for www.labanimal.com
Resource
TABLE 2 | Categories for competence of personnel as previously defined by FELASA3–5 and as defined presently by the EU10 Personnel
FELASA category
EU category
Persons taking care of animals
A
C
Persons carrying out animal experiments/procedures on animals
B
A
Persons responsible for directing animal experiments/designing procedures and projects
C
B
Persons killing animals
–
D
Laboratory animal science specialists (e.g., veterinarians, managers, animal welfare officers)
D
–
npg
© 2014 Nature America, Inc. All rights reserved.
category B courses and category C courses. Written examinations consist of long essay questions (requiring responses up to four pages in length), short essay questions (requiring responses less than one-half page in length), short open questions (requiring responses up to five lines in length), multiple-choice questions, problem-solving exercises and picture or slide descriptions. In contrast with the written examination given for category B courses, which is given on campus, we offer an online written examination for category D courses (‘Laboratory animal health and pathology’ and ‘Laboratory animal anesthesia and experimental surgery’). Both examinations are given for a period of 2–4 h, but, whereas the on-campus examination is administered in a closed room with proctors present, the online examination can be taken at home, with students having access to the examination online within a fixed period of time. Another difference is that participants taking the online examination may consult external resources in order to answer the questions. They must attest that they wrote their own replies, however. The written report, which is the examination type used for category D courses ‘Ethics and legislation in relation to laboratory animals’, ‘Animal models’ and ‘Design and evaluation of animal experiments,’ is an extended version of the essay portion of a written examination. Participants must write one or more reports, each on a clearly defined subject. These reports may be written at home with the help of external resources and must be turned in by an assigned deadline. For the ‘Ethics and legislation’ course, the challenge has been to teach and assess students from EU member states with differing national legislations; although legal principles do not necessarily differ, law formats do. This challenge has been overcome by assigning the students to write three shorter reports on their own national legislation, to be evaluated by a local supervisor hired by the course leader, and then subsequently write longer reports on a chosen topic as well as an assigned topic. For all of our category C courses, the examination consists of a short (15–20 min) oral presentation and a practical examination. One of the course teachers acts LAB ANIMAL
as the examiner, and a proctor attends the examination. In order to qualify for the examination, students must have been present and must have taken part in practical exercises during the majority of course sessions. For the oral presentation, a question is drawn at random by the student from a list of 25–30 theoretical questions that cover the entire curriculum. The list is made available to the students during the course, prior to the examination. Each student is expected to engage in a short discussion on the question at hand. An oral examination is used also for the category D course ‘Veterinary biomedicine.’ These examinations differ from those given for category C courses in that students are given some preparation time prior to giving their presentation. For the category D course ‘Laboratory animal maintenance, breeding and biology,’ the students are given 48 h to prepare a 20-min oral presentation on each of eight different topics. During the examination, one presentation topic is randomly drawn by the student, and the student must present on this topic. After the presentation, the examiner and the proctor may ask a few clarifying questions of the student, but the entire examination grade is based on the quality and the content of the presentation. A similar examination format is used for the category D course ‘In vivo pharmacology.’ In some cases, a poster or a report must be produced and presented during the oral examination. For courses with a larger international student population, these exams may be administered both on campus and online. Practical examinations are used for the category D courses ‘Laboratory animal anesthesia and experimental surgery’ and ‘Microsurgery’ as well as for category C courses. For category C courses, one of four practical exercises is drawn at random by the student, and the student is asked to carry out the exercise using an artificial animal model. For category D course practical examinations, live animals are used. In most cases, the practical examination is completed on campus, but in some rare cases it has been completed online under supervision of a local senior scientist. To graduate with a category D certificate, students must complete a small original research project and Volume 43, No. 10 | OCTOBER 2014 361
Resource
npg
© 2014 Nature America, Inc. All rights reserved.
write a thesis. In most cases, the project is carried out at the animal facility of the student’s employer rather than at the university. The thesis is first evaluated by the course supervisor and a proctor and is then presented to and discussed with the supervisor and the proctor during a 1-h seminar. Grading of performance For written examinations, we inform students of the number of possible points awarded for each question in the evaluation. To grade essay-type questions, we typically use the European Credit Transfer and Accumulation System (ECTS) scale, which assigns grades of ‘A’ to the best 10% of responses, ‘B’ to the next best 25% of responses, ‘C’ to the following 30% of responses, ‘D’ to the following 25% of responses and ‘E’ to the final 10% of responses; quality of responses is determined historically on the basis of all examinations for all courses. Short questions are typically graded as correct (3 points awarded), not entirely correct (2 points awarded), not entirely incorrect (1 point awarded) or incorrect (0 points awarded). Multiple choice questions are graded as either correct or incorrect. Fifty percent of the maximum number of points must be earned in order to pass the exam. In principle, the oral examinations are similarly evaluated on the basis of defined performance criteria. For example, for oral examinations given for category C courses, the student must demonstrate proficiency in species-specific, basic handling techniques on artificial animal models; describe in detail minor procedures (such as blood sampling and injections techniques) and demonstrate them on artificial animal models; discuss ethical and legal principles of experimental animal use (including the principles of the 3Rs: replacement, reduction and refinement) and demonstrate knowledge of how to submit an animal experiment for ethical approval; demonstrate knowledge of principles and methods of experimental animal use including those relating to breeding, reproduction, housing, maintenance, health monitoring, genetic monitoring and genetic modification techniques, anesthesia, analgesia (intraand postoperative), evaluation of pain- and stressrelated behaviors and basic biology of laboratory animals including behavioral and physiological needs; demonstrate knowledge of basic surgical and animal handling techniques pre-, intra- and postoperatively; and demonstrate knowledge of design and evaluation of animal experiments. In practice, the evaluation of oral presentations and practical examinations is far more subjective. The level of subjectivity is comparable to that of the evaluation of essay questions on a written examination. For the oral examinations given for category C courses, the average failure rate is estimated to be 362 Volume 43, No. 10 | OCTOBER 2014
~5–10%. However, it has been our experience that, if the failure rate is lower in a given year, it increases the following year, perhaps because the students do not take the examination as seriously. In general, the failure rate is low for category D courses, because the students tend to postpone the examination if they are not ready. On the other hand, most students in category D courses earn only average grades; in particular, we rarely assign ‘A’ grades. We have found that we need to be more conscious of complying with the EU guideline of identifying “critical elements that cannot be failed”10 and making these elements clear to the students at the beginning of the course. Challenges in assessment for laboratory animal science courses We have encountered three key challenges in developing assessments of competence in laboratory animal science courses. The first challenge is the evaluation of whether practical skills are sufficient to be used with live animals without supervision. We found that the category C practical examinations in which artificial animal models are used provide examiners with an opportunity for such evaluation, although they do not provide the same accuracy of assessment as if live animals were used. This limitation must be balanced, however, with the ethical consideration that students in category C level examinations occasionally use handling approaches that are unacceptable, which cannot be allowed with live animals. In contrast, category D level courses are only offered to students that have already achieved category C accreditation and therefore have a higher level of experience with animals. Therefore, we are ethically able to use live animals for category D practical examinations. The second challenge is the examination of geographically dispersed students on very specialized courses. The online examination is an attractive option, but students taking the examination at home, have the advantage of being able to use external resources. To compensate for this advantage, the examiner should calculate the amount of time it should take to complete the examination using only acquired knowledge and impose time limits that allow minimal time to spend on researching information. Of course, it is not possible to ensure that a student’s responses in an online written examination were not written by someone else, which is why we require students to sign a legally binding document attesting that the work is theirs alone. Incidences of copying from electronic media are easily discovered using electronic plagiarism software. Therefore, we find that such examinations “provide reliable results”10 as recommended by the EU. The oral online examinations do not pose the same challenges, because the examiner and proctor are able to observe the student during the entire exam. www.labanimal.com
Resource
© 2014 Nature America, Inc. All rights reserved.
The third challenge, which applies to all examinations no matter the subject, is implementing an examination that assesses the expected learning outcomes of the course. Written examinations have the advantage of being more “comprehensible”10 as well as “objective and transparent”10 as recommended by the EU for laboratory animal science assessments. But as students are not provided the opportunity for further discussion and clarification, written examinations may not fulfill the EU guideline of providing “reliable results”10, especially in relation to whether students’ skills are “suitable to proceed to working under supervision, such that no unnecessary pain, suffering, distress or lasting harm is inflicted”10. This suitability is easier to assess during an oral examination, which more accurately reflects real-life situations, especially when combined with a practical examination. Another advantage of the written examination is that scoring of responses to short essay and multiple-choice questions can be more or less automated, fulfilling the EU guideline of establishing clear pass-fail criteria. However, such questions can only test the student’s fact-based, indisputable knowledge, rather than fulfill the other EU recommendation to “ensure students have achieved an acceptable level of understanding of the subject”10. Basing too much of the examination on fact-based, short-answer questions may therefore generate course graduates whose capacity for interacting in a more complicated world and for proceeding to work without supervision at a later time, as recommended by the EU10, is not known.
npg
Conclusions A single examination strategy can never entirely assess the various competencies of individual students or evaluate all learning outcomes. It is valuable in itself, however, for a longer education consisting of several courses to expose graduates to multiple types of exams, each of which prepares students for different types of situations. The FELASA E&T Board requirements for course accreditation state that “course organizers must have implemented a mechanism by which students can reflect and report on their learning experience, and must provide examples of course evaluation to provide reassurance that the course is balanced, effective and worthwhile”12. Whereas the current accreditation scheme addresses only the training component of the educational process, the FELASA E&T Board has stated that “assessment of competences… may be considered for accreditation at a later stage”12. Meanwhile, course organizers must develop their course assessments in such a way as to fulfill the EU guidelines10. Further discussion of experiences with administering assessments for laboratory animal science courses will be an important part of future course development and improvement. LAB ANIMAL
COMPETING FINANCIAL INTERESTS The authors declare no competing financial interests. Received 8 January 2014; accepted 29 May 2014 Published online at http://www.labanimal.com/ 1. Council of Europe. European Convention for the Protection of Vertebrate Animals used for Experimental and other Scientific Purposes (ETS No. 123). Council of Europe, Strasbourg, France (1986). 2. EEC. Directive 86/609/EEC of 24 November 1986 on the approximation of laws, regulations and administrative provisions of the Member States regarding the protection of animals used for experimental and other scientific purposes. EEC Official Journal L358, 1–28 (1986). 3. Council of Europe. Multilateral consultation of parties to the European Convention for the protection of vertebrate animals used for experimental and other scientific purposes (ETS No. 123). Resolution on education and training of persons working with laboratory animals adopted by the Multilateral Consultation on 3 December 1993. Council of Europe, Strasbourg, France, 1993. 4. Wilson, M.S. et al. FELASA recommendations on the education and training of persons working with laboratory animals: Categories A and C. Reports of the Federation of European Laboratory Animal Science Associations Working Group on Education accepted by the FELASA Board of Management. Lab. Anim. 29, 121–131 (1995). 5. Nevalainen, T. et al. FELASA guidelines for education of specialists in laboratory animal science (Category D): Report of the Federation of Laboratory Animal Science Associations Working Group on Education of Specialists (Category D) accepted by the FELASA Board of Management. Lab. Anim. 33, 1–15 (1999). 6. European Council. 1999/575/EC: Council Decision of 23 March 1998 concerning the conclusion by the Community of the European Convention for the protection of vertebrate animals used for experimental and other scientific purposes. Official Journal of the European Union L222, 29–30 (1999). 7. Nevalainen, T. et al. FELASA recommendations for the education and training of persons carrying out animal experiments (Category B). Report of the Federation of European Laboratory Animal Science Associations Working Group on Education of Persons Carrying out Animal Experiments (Category B) accepted by the FELASA Board of Management. Lab. Anim. 34, 229–235 (2000). 8. Nevalainen, T. et al. FELASA recommendations for the accreditation of laboratory animal science education and training: Report of the Federation of European Laboratory Animal Science Associations Working Group on Accreditation of Laboratory Animal Science Education and Training. Lab. Anim. 36, 373–377 (2002). 9. European Union. Directive 2010/63/EU of the European Parliament and of the Council of 22 September 2010 on the protection of animals used for scientific purposes. Official Journal of the European Union L276, 33–79 (2010). 10. European Union. National Competent Authorities for the implementation of Directive 2010/63/EU on the protection of animals used for scientific purposes. A working document on the development of a common education and training framework to fulfill the requirements under the Directive (replacing consensus document of 18–19 September 2013). 19–20 February 2014, Brussels, Belgium. 11. Bernard, R.M., Borokhovski, E., Schmid, R.F., Tamim, R.M. & Abrami, P.C. A meta-analysis of blended learning and technology use in higher education: from the general to the applied. J. Comput. Higher Ed. 26, 87–122 (2014). 12. FELASA Education and Training Board. FELASA recommendations for the accreditation of Education and Training courses in Laboratory Animal Science. Revised 21 May 2014.
Volume 43, No. 10 | OCTOBER 2014 363
Strategies for the assessment of competence in laboratory animal science courses Axel Kornerup Hansen, DVSc, DVM, DipECLAM & Dorte Bratbo Sørensen, PhD, DVM
© 2014 Nature America, Inc. All rights reserved.
Evaluation of skills, knowledge and competencies is an essential part of education in laboratory animal science. In Europe, a greater emphasis will be placed on such evaluations going forward, because the European Union will base its education and training framework on learning outcomes rather than on course time and syllabuses, as done previously. The authors present their experiences administering different written, oral and practical examinations for Federation of European Laboratory Animal Science Associations categories B, C and D courses. Examinations can be administered online as well as on campus, if time constraints are provided to compensate for the advantage of being able to use external resources. Overall, students benefit from exposure to multiple types of exams over the course of their education because each type prepares students for different situations.
npg
One way of ensuring the welfare of laboratory animals is to provide sufficient education to practitioners of laboratory animal science. Since 1986, educational standards for persons performing experiments with animals in Europe have been developed by both governmental and non-governmental organizations1–10 (Table 1). These educational standards have been based on the competencies required of personnel and are categorized into four levels (A, B, C and D; Table 2), as proposed by the Federation of European Laboratory Animal Science Associations (FELASA)4,5,7 and partially accepted by the Council of Europe3. For more than 15 years, our section at the University of Copenhagen has offered category B training for persons carrying out experiments with animals. These courses have a traditional structure, consisting of lectures as well as both theoretical and practical exercises. For the same period of time, we have offered category C training for persons responsible for directing animal experiments. Over the last 5 years, we have implemented a model of blended learning for these courses, which include a combination of web-based ‘e-learning’ and on-campus instruction of various types. The web-based component of the course allows students to access lectures and materials for group work in an electronic format from an online server and to raise questions on a discussion board. We also hold
plenary sessions on selected topics to allow the students to engage with the teacher as needed. A recent meta-analysis concluded that blended learning conditions result in an improvement in student achievement outcomes by approximately one-third of a standard deviation compared with classroom instruction conditions11. For more than 10 years, we have offered category D training for specialists including veterinarians, managers and animal welfare officers as a post-graduate Master’s degree in laboratory animal science. This program includes a number of courses that are structured into three phases: an initial ‘pre-teaching’ phase, during which lectures in an electronic format, materials for group work and a discussion board are provided online; an intensive teaching phase, provided on campus and consisting of plenary sessions, group work and practical demonstrations; and a ‘post-teaching’ phase, provided online and concluding with an online examination. The categories C and D educational programs offered at the university have been accredited by FELASA, the Danish Governmental Administration and study boards at University of Copenhagen. In the past, the FELASA educational guidelines for each competence category have been based on course syllabuses, requiring that a certain number of hours of training (e.g., at least 80 h for category C courses) be
Section of Experimental Animal Models, Department of Veterinary Disease Biology, University of Copenhagen, Denmark. Correspondence should be addressed to A.K.H. ([email protected]).
LAB ANIMAL
Volume 43, No. 10 | OCTOBER 2014 359
Resource
npg
© 2014 Nature America, Inc. All rights reserved.
TABLE 1 | European regulatory standards for education and training in laboratory animal science Year
Organization
Regulatory action
1986
Council of Europe (CoE)
“Authorization [to carry out a procedure] shall be granted only to persons deemed to be competent by the responsible authority”1
1986
European Economic Community (EEC)
“Experiments shall be performed solely by competent authorized persons, or under the direct responsibility of such a person”2
1993
CoE
Presents requirements for establishing appropriate training in competencies defined under the categories A, B, C and D (ref. 3)
1995
Federation of Laboratory Animal Science Associations (FELASA)
Presents detailed guidelines for training in competencies defined under the categories A and C (ref. 4)
1999
FELASA
Presents detailed guidelines for training in competencies defined under category D (ref. 5)
1999
European Union (EU)
“The [European] Community shall approve the European Convention for the protection of vertebrate animals used for experimental and other scientific purposes”6
2000
FELASA
Presents detailed guidelines for training of persons defined under category B (ref. 7)
2003
FELASA
Establishes that any European laboratory animal science course can achieve official FELASA accreditation through fulfillment of FELASA guidelines for education and training, subject to assessment and approval by a FELASA Accreditation Board8
2010
EU
“Member States should ensure through authorisation or by other means that staff are adequately educated, trained and competent. Furthermore, it is important that staff are supervised until they have obtained and demonstrated the requisite competence”9
2013
EU
Designates new categories for competence of personnel10 Proposes a training framework with a focus on learning outcomes10
completed and that certain topics be included in that training4,5,7. Any European course that adhered to the FELASA guidelines could achieve official accreditation8. As a consequence of the revised EU directive6, however, the European Union Commission issued the National Authority Consensus Paper (EU consensus paper)10, which places a greater emphasis on learning outcomes than on course time and syllabuses and, for the first time, sets standards for the assessment of learning outcomes. It states that the assessment of learning outcomes should “be objective and transparent; be comprehensible and clear without ambiguity; have clear pass-fail criteria; provide reliable results; ensure students have achieved an acceptable level of understanding of the subject—suitable to proceed to working under supervision, such that no unnecessary pain, suffering, distress or lasting harm is inflicted; [and] identify, where appropriate, critical elements that cannot be failed”10. The EU consensus paper also describes the achievement of competence within laboratory animal science as a process consisting of three steps: (i) acquisition of knowledge, (ii) practical experience and (iii) development 360 Volume 43, No. 10 | OCTOBER 2014
of deeper learning and critical thinking10. The FELASA accreditation board for education and training (E&T Board) views its role in providing accreditation for courses in laboratory animal science as ensuring the acquisition of knowledge and practical training, which should be accurately assessed through course examinations12. The current accreditation scheme includes consideration of “the method of student assessment, the range and depth of evaluation of learning outcomes and the way in which practical skills are assessed”12. In addition to the guidelines for course assessments established by FELASA and the EU commission, we have established additional defined learning outcomes and requirements that must be fulfilled in order to achieve credit for completing courses in laboratory animal science. Here, we provide our experiences with administering various types of examinations for the assessment of competence in laboratory animal science courses. Examination types We use a range of different types of examinations for category D courses, some of which are also used for www.labanimal.com
Resource
TABLE 2 | Categories for competence of personnel as previously defined by FELASA3–5 and as defined presently by the EU10 Personnel
FELASA category
EU category
Persons taking care of animals
A
C
Persons carrying out animal experiments/procedures on animals
B
A
Persons responsible for directing animal experiments/designing procedures and projects
C
B
Persons killing animals
–
D
Laboratory animal science specialists (e.g., veterinarians, managers, animal welfare officers)
D
–
npg
© 2014 Nature America, Inc. All rights reserved.
category B courses and category C courses. Written examinations consist of long essay questions (requiring responses up to four pages in length), short essay questions (requiring responses less than one-half page in length), short open questions (requiring responses up to five lines in length), multiple-choice questions, problem-solving exercises and picture or slide descriptions. In contrast with the written examination given for category B courses, which is given on campus, we offer an online written examination for category D courses (‘Laboratory animal health and pathology’ and ‘Laboratory animal anesthesia and experimental surgery’). Both examinations are given for a period of 2–4 h, but, whereas the on-campus examination is administered in a closed room with proctors present, the online examination can be taken at home, with students having access to the examination online within a fixed period of time. Another difference is that participants taking the online examination may consult external resources in order to answer the questions. They must attest that they wrote their own replies, however. The written report, which is the examination type used for category D courses ‘Ethics and legislation in relation to laboratory animals’, ‘Animal models’ and ‘Design and evaluation of animal experiments,’ is an extended version of the essay portion of a written examination. Participants must write one or more reports, each on a clearly defined subject. These reports may be written at home with the help of external resources and must be turned in by an assigned deadline. For the ‘Ethics and legislation’ course, the challenge has been to teach and assess students from EU member states with differing national legislations; although legal principles do not necessarily differ, law formats do. This challenge has been overcome by assigning the students to write three shorter reports on their own national legislation, to be evaluated by a local supervisor hired by the course leader, and then subsequently write longer reports on a chosen topic as well as an assigned topic. For all of our category C courses, the examination consists of a short (15–20 min) oral presentation and a practical examination. One of the course teachers acts LAB ANIMAL
as the examiner, and a proctor attends the examination. In order to qualify for the examination, students must have been present and must have taken part in practical exercises during the majority of course sessions. For the oral presentation, a question is drawn at random by the student from a list of 25–30 theoretical questions that cover the entire curriculum. The list is made available to the students during the course, prior to the examination. Each student is expected to engage in a short discussion on the question at hand. An oral examination is used also for the category D course ‘Veterinary biomedicine.’ These examinations differ from those given for category C courses in that students are given some preparation time prior to giving their presentation. For the category D course ‘Laboratory animal maintenance, breeding and biology,’ the students are given 48 h to prepare a 20-min oral presentation on each of eight different topics. During the examination, one presentation topic is randomly drawn by the student, and the student must present on this topic. After the presentation, the examiner and the proctor may ask a few clarifying questions of the student, but the entire examination grade is based on the quality and the content of the presentation. A similar examination format is used for the category D course ‘In vivo pharmacology.’ In some cases, a poster or a report must be produced and presented during the oral examination. For courses with a larger international student population, these exams may be administered both on campus and online. Practical examinations are used for the category D courses ‘Laboratory animal anesthesia and experimental surgery’ and ‘Microsurgery’ as well as for category C courses. For category C courses, one of four practical exercises is drawn at random by the student, and the student is asked to carry out the exercise using an artificial animal model. For category D course practical examinations, live animals are used. In most cases, the practical examination is completed on campus, but in some rare cases it has been completed online under supervision of a local senior scientist. To graduate with a category D certificate, students must complete a small original research project and Volume 43, No. 10 | OCTOBER 2014 361
Resource
npg
© 2014 Nature America, Inc. All rights reserved.
write a thesis. In most cases, the project is carried out at the animal facility of the student’s employer rather than at the university. The thesis is first evaluated by the course supervisor and a proctor and is then presented to and discussed with the supervisor and the proctor during a 1-h seminar. Grading of performance For written examinations, we inform students of the number of possible points awarded for each question in the evaluation. To grade essay-type questions, we typically use the European Credit Transfer and Accumulation System (ECTS) scale, which assigns grades of ‘A’ to the best 10% of responses, ‘B’ to the next best 25% of responses, ‘C’ to the following 30% of responses, ‘D’ to the following 25% of responses and ‘E’ to the final 10% of responses; quality of responses is determined historically on the basis of all examinations for all courses. Short questions are typically graded as correct (3 points awarded), not entirely correct (2 points awarded), not entirely incorrect (1 point awarded) or incorrect (0 points awarded). Multiple choice questions are graded as either correct or incorrect. Fifty percent of the maximum number of points must be earned in order to pass the exam. In principle, the oral examinations are similarly evaluated on the basis of defined performance criteria. For example, for oral examinations given for category C courses, the student must demonstrate proficiency in species-specific, basic handling techniques on artificial animal models; describe in detail minor procedures (such as blood sampling and injections techniques) and demonstrate them on artificial animal models; discuss ethical and legal principles of experimental animal use (including the principles of the 3Rs: replacement, reduction and refinement) and demonstrate knowledge of how to submit an animal experiment for ethical approval; demonstrate knowledge of principles and methods of experimental animal use including those relating to breeding, reproduction, housing, maintenance, health monitoring, genetic monitoring and genetic modification techniques, anesthesia, analgesia (intraand postoperative), evaluation of pain- and stressrelated behaviors and basic biology of laboratory animals including behavioral and physiological needs; demonstrate knowledge of basic surgical and animal handling techniques pre-, intra- and postoperatively; and demonstrate knowledge of design and evaluation of animal experiments. In practice, the evaluation of oral presentations and practical examinations is far more subjective. The level of subjectivity is comparable to that of the evaluation of essay questions on a written examination. For the oral examinations given for category C courses, the average failure rate is estimated to be 362 Volume 43, No. 10 | OCTOBER 2014
~5–10%. However, it has been our experience that, if the failure rate is lower in a given year, it increases the following year, perhaps because the students do not take the examination as seriously. In general, the failure rate is low for category D courses, because the students tend to postpone the examination if they are not ready. On the other hand, most students in category D courses earn only average grades; in particular, we rarely assign ‘A’ grades. We have found that we need to be more conscious of complying with the EU guideline of identifying “critical elements that cannot be failed”10 and making these elements clear to the students at the beginning of the course. Challenges in assessment for laboratory animal science courses We have encountered three key challenges in developing assessments of competence in laboratory animal science courses. The first challenge is the evaluation of whether practical skills are sufficient to be used with live animals without supervision. We found that the category C practical examinations in which artificial animal models are used provide examiners with an opportunity for such evaluation, although they do not provide the same accuracy of assessment as if live animals were used. This limitation must be balanced, however, with the ethical consideration that students in category C level examinations occasionally use handling approaches that are unacceptable, which cannot be allowed with live animals. In contrast, category D level courses are only offered to students that have already achieved category C accreditation and therefore have a higher level of experience with animals. Therefore, we are ethically able to use live animals for category D practical examinations. The second challenge is the examination of geographically dispersed students on very specialized courses. The online examination is an attractive option, but students taking the examination at home, have the advantage of being able to use external resources. To compensate for this advantage, the examiner should calculate the amount of time it should take to complete the examination using only acquired knowledge and impose time limits that allow minimal time to spend on researching information. Of course, it is not possible to ensure that a student’s responses in an online written examination were not written by someone else, which is why we require students to sign a legally binding document attesting that the work is theirs alone. Incidences of copying from electronic media are easily discovered using electronic plagiarism software. Therefore, we find that such examinations “provide reliable results”10 as recommended by the EU. The oral online examinations do not pose the same challenges, because the examiner and proctor are able to observe the student during the entire exam. www.labanimal.com
Resource
© 2014 Nature America, Inc. All rights reserved.
The third challenge, which applies to all examinations no matter the subject, is implementing an examination that assesses the expected learning outcomes of the course. Written examinations have the advantage of being more “comprehensible”10 as well as “objective and transparent”10 as recommended by the EU for laboratory animal science assessments. But as students are not provided the opportunity for further discussion and clarification, written examinations may not fulfill the EU guideline of providing “reliable results”10, especially in relation to whether students’ skills are “suitable to proceed to working under supervision, such that no unnecessary pain, suffering, distress or lasting harm is inflicted”10. This suitability is easier to assess during an oral examination, which more accurately reflects real-life situations, especially when combined with a practical examination. Another advantage of the written examination is that scoring of responses to short essay and multiple-choice questions can be more or less automated, fulfilling the EU guideline of establishing clear pass-fail criteria. However, such questions can only test the student’s fact-based, indisputable knowledge, rather than fulfill the other EU recommendation to “ensure students have achieved an acceptable level of understanding of the subject”10. Basing too much of the examination on fact-based, short-answer questions may therefore generate course graduates whose capacity for interacting in a more complicated world and for proceeding to work without supervision at a later time, as recommended by the EU10, is not known.
npg
Conclusions A single examination strategy can never entirely assess the various competencies of individual students or evaluate all learning outcomes. It is valuable in itself, however, for a longer education consisting of several courses to expose graduates to multiple types of exams, each of which prepares students for different types of situations. The FELASA E&T Board requirements for course accreditation state that “course organizers must have implemented a mechanism by which students can reflect and report on their learning experience, and must provide examples of course evaluation to provide reassurance that the course is balanced, effective and worthwhile”12. Whereas the current accreditation scheme addresses only the training component of the educational process, the FELASA E&T Board has stated that “assessment of competences… may be considered for accreditation at a later stage”12. Meanwhile, course organizers must develop their course assessments in such a way as to fulfill the EU guidelines10. Further discussion of experiences with administering assessments for laboratory animal science courses will be an important part of future course development and improvement. LAB ANIMAL
COMPETING FINANCIAL INTERESTS The authors declare no competing financial interests. Received 8 January 2014; accepted 29 May 2014 Published online at http://www.labanimal.com/ 1. Council of Europe. European Convention for the Protection of Vertebrate Animals used for Experimental and other Scientific Purposes (ETS No. 123). Council of Europe, Strasbourg, France (1986). 2. EEC. Directive 86/609/EEC of 24 November 1986 on the approximation of laws, regulations and administrative provisions of the Member States regarding the protection of animals used for experimental and other scientific purposes. EEC Official Journal L358, 1–28 (1986). 3. Council of Europe. Multilateral consultation of parties to the European Convention for the protection of vertebrate animals used for experimental and other scientific purposes (ETS No. 123). Resolution on education and training of persons working with laboratory animals adopted by the Multilateral Consultation on 3 December 1993. Council of Europe, Strasbourg, France, 1993. 4. Wilson, M.S. et al. FELASA recommendations on the education and training of persons working with laboratory animals: Categories A and C. Reports of the Federation of European Laboratory Animal Science Associations Working Group on Education accepted by the FELASA Board of Management. Lab. Anim. 29, 121–131 (1995). 5. Nevalainen, T. et al. FELASA guidelines for education of specialists in laboratory animal science (Category D): Report of the Federation of Laboratory Animal Science Associations Working Group on Education of Specialists (Category D) accepted by the FELASA Board of Management. Lab. Anim. 33, 1–15 (1999). 6. European Council. 1999/575/EC: Council Decision of 23 March 1998 concerning the conclusion by the Community of the European Convention for the protection of vertebrate animals used for experimental and other scientific purposes. Official Journal of the European Union L222, 29–30 (1999). 7. Nevalainen, T. et al. FELASA recommendations for the education and training of persons carrying out animal experiments (Category B). Report of the Federation of European Laboratory Animal Science Associations Working Group on Education of Persons Carrying out Animal Experiments (Category B) accepted by the FELASA Board of Management. Lab. Anim. 34, 229–235 (2000). 8. Nevalainen, T. et al. FELASA recommendations for the accreditation of laboratory animal science education and training: Report of the Federation of European Laboratory Animal Science Associations Working Group on Accreditation of Laboratory Animal Science Education and Training. Lab. Anim. 36, 373–377 (2002). 9. European Union. Directive 2010/63/EU of the European Parliament and of the Council of 22 September 2010 on the protection of animals used for scientific purposes. Official Journal of the European Union L276, 33–79 (2010). 10. European Union. National Competent Authorities for the implementation of Directive 2010/63/EU on the protection of animals used for scientific purposes. A working document on the development of a common education and training framework to fulfill the requirements under the Directive (replacing consensus document of 18–19 September 2013). 19–20 February 2014, Brussels, Belgium. 11. Bernard, R.M., Borokhovski, E., Schmid, R.F., Tamim, R.M. & Abrami, P.C. A meta-analysis of blended learning and technology use in higher education: from the general to the applied. J. Comput. Higher Ed. 26, 87–122 (2014). 12. FELASA Education and Training Board. FELASA recommendations for the accreditation of Education and Training courses in Laboratory Animal Science. Revised 21 May 2014.
Volume 43, No. 10 | OCTOBER 2014 363
