Nurse Education Today 35 (2015) 25–31
Contents lists available at ScienceDirect
Nurse Education Today journal homepage: www.elsevier.com/nedt
Evaluation of disaster preparedness training and disaster drill for nursing students Syahirul Alim a,b,1, Masato Kawabata c,2, Minato Nakazawa d,3 a
Graduate School of Health Sciences, Kobe University, Japan School of Nursing, Faculty of Medicine, Universitas Gadjah Mada, Indonesia Graduate School of Medicine, Kobe University, Japan d Graduate School of Health Sciences, Kobe University, Japan b c
a r t i c l e
i n f o
Article history: Accepted 23 April 2014
Keywords: Disaster preparedness training Nursing students Disaster drill Training evaluation
s u m m a r y Background: Preparedness and preventive measures are needed to reduce the impact of disasters. Disaster preparedness training for nurses has a long history. However, the effectiveness of disaster preparedness training for nursing students has been limited, to some extent, since they have been based on self-evaluation. Objective: The study attempts to evaluate the effectiveness of a disaster preparedness training program followed by a disaster drill designed for nursing students. Participants: Participants were undergraduate students from Universitas Gadjah Mada and diploma students from four randomly chosen nursing colleges located in Yogyakarta, Indonesia. 309 students participated in the training program whereas 225 students participated in the disaster drill. Methods: The present study conducted in-class training followed by a disaster drill and evaluated using 3 components: pre-test and post-test evaluation of knowledge (score range: 0–20), observation of skills during disaster drill (5-point Likert scale), and a structured one-to-one interview of participants' responses to the training and drill process. Results: Pre-test and post-test evaluation scores showed significant improvement (P b 0.01, paired t-test) for both university and diploma students. Almost all observation items during the disaster drill were above 4.0 (on 5-point Likert scale). Interview results showed that most participants responded positively. Conclusions: The present study completely evaluated the effectiveness of a disaster preparedness training and disaster drill: The training and drill improved the knowledge and ability of disaster preparedness for both undergraduate and diploma students. © 2014 Elsevier Ltd. All rights reserved.
Introduction An impact resulting from a catastrophic condition caused by natural disasters affected not only buildings and the surrounding infrastructure but also human life and society itself. In order to reduce the impact of disasters, the World Health Organization (WHO) has stressed that despite the importance of relief supplies in emergencies, preventive measures and preparedness are of equal and perhaps more fundamental importance. In the health sector, the availability of suitably equipped health providers who are able to respond when needed is considered vital. (WHO, 2013)
E-mail addresses: [email protected] (S. Alim), [email protected] (M. Kawabata), [email protected] (M. Nakazawa). 1 Tel.: +81 80 4646 4944. 2 Tel.: +81 78 382 5681. 3 Tel.: +81 78 796 4551.
http://dx.doi.org/10.1016/j.nedt.2014.04.016 0260-6917/© 2014 Elsevier Ltd. All rights reserved.
Nurses, the main source of healthcare providers, are indispensable workforce in disaster situations. It is nurses' responsibility to understand their role in preparedness as well as to prepare in advance in order to be available if necessary (ICN, 2009). In this sense, disaster preparedness education is vital for nurses in order to be sufficient in knowledge and skills to respond in any disaster. Disaster preparedness training for nurses had increasingly been reported in America since the end of World War II. It was reported by Cole (1960) that in the United States between 1951 and 1960, there were 656,333 home nurses and first aiders trained under the disaster plan project and 14,405 registered nurses had been given courses in disaster nursing. Exercise London, conducted in London, Ontario, Canada in 1974, was the first successful large-scale disaster drill. It was a largescale drill to assess the ability of healthcare providers and other community components to overcome catastrophic conditions using 150 casualties and involving a large-scale community (Cole, 1960). Despite the efforts to train healthcare providers, there is a gap between what nursing students are taught and what practitioners in
26
S. Alim et al. / Nurse Education Today 35 (2015) 25–31
disaster preparedness know (Jennings-Sanders et al., 2005). In addition, the same study also found that students have limitations in finding information related to disasters and this leads to inadequate knowledge regarding disaster nursing such as: (1) incomplete definitions of disaster nursing; (2) lack of knowledge about community resources, mock disaster drills, and how to utilize disaster planning models; and (3) unawareness of the significant role nurses could play in disaster situations. In order to overcome these conditions nursing education institutions initiated to incorporate disaster-related subjects into their curriculums. Disaster-related subjects are being included as a part of Emergency Nursing and/or Community Health Nursing Programs. Another possibility is to develop disaster nursing as an independent subject. In recent years, many academic institutions introduced disaster topics for students in disaster or emergency-related training programs. Following in-class training they practiced their new skills in disaster drills. A study conducted by Kaplan et al. revealed a positive response from the implementation of Emergency Preparedness Disaster Simulation (EPDS) for nursing students (Kaplan et al., 2012) However, the effectiveness of these programs has not been properly evaluated yet. It is necessary to have a comprehensive evaluation of both the effectiveness of the training and the disaster drill itself. Feedback and input from the participants could provide additional data to better design and develop curriculums for training and drills. Methods All participants in the study were undergraduate students from Universitas Gadjah Mada, and diploma students from four randomly chosen nursing colleges in Yogyakarta, Indonesia. The undergraduate students were fourth grade students who had zero experience in actual hospital practice as part of their academic program. While the diploma students were third grade students from nursing colleges who already been exposed to hospital practices for their academic program. Both undergraduate and diploma students were recruited firstly by giving them information related to this study through the teachers. Participants voluntarily agreed to participate in this study. A-one day disaster preparedness training and a disaster drill were conducted in the study. In-class training enabled students to gain knowledge and understanding in the context of disaster preparedness. The training sessions were conducted based on a schedule that had been arranged and agreed by all institutions. Each nursing college was responsible for choosing students who participated in this training. The disaster drill stimulated participants to apply their knowledge and skills they had learned in a “real” case condition. In the disaster drill all participants worked in small groups, which gave them the opportunity to develop the sense of teamwork and collaboration needed to respond in a disaster scenario. The scenario was based on real events experienced by the authors of the 2006 earthquake in Yogyakarta. The situation and description of the environment, survivors and availability of resources were all explained in detail. The study evaluated the effectiveness of the training and drill using 3 evaluation components: (1) Pre-test and post-test evaluation to measure the degree of knowledge achieved in the training; (2) observation to evaluate skills during the disaster drill; and (3) interview to obtain participants' responses and feedbacks on both the training and drill. Disaster Preparedness Training A big challenge in conducting this study was that there was no standard guideline related to disaster preparedness training for nursing students. In lieu of this, we followed the International Council of Nurses framework of disaster nursing competency (Dorsey et al., 2009) to develop the curriculum for our training. In addition, the study also incorporated, ‘cross-cutting competencies for healthcare workers in disaster training’, developed by Hsu et al. (2006) and Core
Competencies for Nurses in Emergency and Disaster Preparedness by Gebbie and Qureshi (2002). The training was held over 8 h. The course was divided into 4 topics (I) Introduction to disaster preparedness for nurses, which covered a general description of health problems in disasters, the role of nurses in every phase of a disaster and nursing care principles in a disaster; (II) Command and management for healthcare, which covered incident command systems, multiagency coordination systems and public information systems; (III) Basic life support I, which covered triage methods in the field and cardio-pulmonary resuscitation; (IV) Basic life support II, which covered management for head trauma, bleeding and fracture (Table 1). Trainers and instructors for this course were nursing practitioners from hospitals and universities who had expertise in disaster, emergency and critical care. The trainers also had a great deal of actual experience in responding to disaster situations both nationally and internationally. Each trainer was assigned to be responsible for one topic. The guideline book and manual were distributed to all trainers prior to the training session, so that all the trainers had the same competencies related to the process of training and topics. Twenty minute pre-test and post-test were conducted before and after completing the training session for every participant. There were 20 multiple-choice questions based on topics covering all aspects of the training. Each question was scored as one for a correct answer and zero for an incorrect one. Thus the total score ranged from zero to twenty. The evaluation had five questions from each topic of training. The evaluation sheet was first read and evaluated by all trainers and instructors for content validity and consistency before the training session began. In the beginning of training, participants filled a sheet of demographic data survey. We also asked information about participants' previous training experience since this information could influence the result of study. Participants might have had previous experience in disaster or emergency response training. All participants received handouts on the topics and were allowed to take notes during the training sessions. At the end of each topic session delivered by a trainer, a question period ensued and was encouraged. Disaster Drill Only participants, who completed the in-class training of disaster preparedness, participated in the disaster drill. A number of undergraduate students could not participate in this drill due to their academic schedules while only one diploma student was absent for the disaster drill. The total number of participants in the disaster drill was 225 students. It consisted of 138 diploma students and 87 undergraduate students. Prior to the drill the participants were divided into small groups of 10–15 students. In this disaster drill, the participants were challenged to respond to the disaster situation written on the scenario sheet (Fig. 2, scenario of the disaster drill). They were encouraged to work together as a group to respond to the disaster situation. Each student was assigned a different task according to the specific role. Some were team leaders, some triage staff, and others treatment, and evacuation staff. (See Fig. 1.) This disaster drill used actor-patients as well as patient simulators. Actor-patients were non-participant students who played the role of victim patients. They were dressed and made up according to the scenario they were in using artificial wounds and blood. They were then coached by a skillful instructor in disaster, to mimic the real sounds and condition of patients in the field. Patient simulators were used for practicing cardio-pulmonary resuscitation and other treatments that were not suitable to be used on actor-patients. Disaster drills were conducted in the parking lot of the university and the emergency nursing practice room served as the health facility. All groups prepared medications and equipment necessary for initial
S. Alim et al. / Nurse Education Today 35 (2015) 25–31
27
Table 1 Topics of training. Topics
Sub-topics
Duration (hours)
Topic I: Introduction to the disaster preparedness for nurses
• • • • • • • • •
2
Topic II: Command and management for healthcare Topic III: Basic life support treatment in disaster 1 Topic IV: Basic life support treatment in disaster 2
A general description of health problems in disasters; The role of nurses in every phase of a disaster; Nursing care principles in a disaster. Incident Command System (ICS); Multiagency coordination systems; Public information systems. Triage methods in the field; Cardio-pulmonary resuscitation. Management for head trauma, chest injury, bleeding, fracture.
treatment and to assist in transporting patients before the drill started. Participants began the drill from practice room after being briefed about the steps of the drill and the scenario that they would be responding to. (See Fig. 2.) Instructors measured performance of the group by observing how each group responded to the disaster situation during the drill. Particular attention was also paid to how well they worked and collaborated with other members. There were 4 main items of evaluation: Obs.1) the ability to do triage and to identify the survivors' health problem; Obs.2) the ability to do initial treatment; Obs.3) the ability to transport survivors safely to the nearest health facility; and Obs.4) the ability to work with other team members. Results of the observations of all evaluation items were based on a 5-point Likert scale (5 = very proficient, 4 = proficient, 3 = fairly proficient, 2 = moderately proficient, 1 = poorly proficient). Debriefing procedures were conducted after completing the drill in order to have a clear understanding related to the drill process. In this session, participants shared their experience during the drill, how they felt about the role and task they had conducted, and how they managed to work together in a team to respond in a disaster situation. Instructors
Disaster Preparedness Training and Disaster Drill Formation
Forty participants (15%) who completed in-class training and the disaster drill were randomly chosen, using computergenerated randomized number, to participate in structured oneto-one interview. The purpose of the interview, to evaluate the whole process of training and drill, was explained to the participants. Each participant was interviewed for 5–10 min by the principal author (SA). Responses in the interview session were recorded for later data analysis. Four main questions were asked (1) “What benefits might you have from this training and drill?”; (2) “What do you think were the advantages and disadvantages of this training and drill?”; (3) “What did you think about the training and drill material, and equipment?”; and (4) “What do you think about trainers and instructors?”.
Diploma Students In-class Training: 139 students Disaster Drill: 138 students
Group A Day 1: Training (27 students) Day 2: Drill (27 students)
Coordinating with research assistants: schedule and activities.
Assessing the availability of equipment and materials for training and disaster drill.
Group B Day 1: Training (51 students) Day 2: Drill (51 students)
Group C Day 1: Training (35 students) Day 2: Drill (34 students)
Group D Day 1: Training (26 students) Day 2: Drill (26 students)
Pre and Post-Test for In-class Training Observation for Disaster Drill In depth Interview (Feedback & Input): 1. In-Class Training 2. Disaster Drill
2
Interview for Training and Disaster Drill
Disaster Preparedness Training & Disaster Drill
Completing proposal and requirements for ethical permission.
2
then provided clarifications and explanations to the participants for misconducts during the drill.
Preparation for Training & Disaster Drill
Developing guideline and materials for training and drill.
2
Evaluation Components
Fig. 1. Study process chart.
Undergraduate Students In-class Training: 170 students Disaster Drill: 87 students
Group A Day 1: Training (44 students) Day 2: Drill (44 students)
Group B Day 1: Training (43 students) Day 2: Drill (43 students)
Group C Day 1: Training (42 students) Day 2: Drill (0 students)
Group A Day 1: Training (41 students) Day 2: Drill (0 students)
28
S. Alim et al. / Nurse Education Today 35 (2015) 25–31
“You are a nursing student who do the community health program practice in the village together with 9 other nursing students. Early in the morning, while preparing for the activity in Primary Health Care, the 7.5 magnitude scale earthquake struck the area. After the quake, having checked your condition and other students are safe. Unfortunately, the house next to the Primary Health Care is nearly breakdown. The owner of the house is an elderly couple, Mr. and Mrs. Joko. Mr. Joko is unconscious, no pulse, there is bruise sign at the back of his neck. Mrs. Joko is crying, with open wound and fracture at her right arm. You and other students do the initial treatment to the victims”. 1. Within the group choose one leader and distribute the role to do the task! 2. Do the initial treatment for Mr. and Mrs. Joko! 3. Do the transport and refer to the health care facility appropriately! Fig. 2. Scenario of the disaster drill.
was voluntary and students were not compensated for their participation.
Statistical Analysis Statistical analysis was conducted using R Version 3.0.0 software (R Foundation for Statistical Computing, Vienna, Austria). Paired t-test was used to compare the mean difference between the pre-test and posttest results. The t-test was used to compare the mean score of observation in disaster drill between undergraduate and diploma students. In the performance of each group during the disaster drill stepwise linear regression analysis was performed. This was used to find out factors, which may influence the observation results, including strata of students (undergraduate and diploma), post-test score (lowest, mean, and highest), the mean difference of pre-test and post-test scores, and the proportion of students' previous training experience. Alpha (α) level for analysis was set at 0.05.
Ethical Consideration The present study was approved by the ethical committee of the School of Nursing, Faculty of Medicine, Universitas Gadjah Mada, and obtained consent from the nursing colleges, as well as the Yogyakarta Provincial Government. Participation of students
Results Demographic Data A total of 309 nursing students participated in the disaster preparedness training, whereas only 225 students took part in the disaster drill (see Table 2). The percentage of undergraduate students was 55% of the total number of in-class training participants, but decreased to 39% of those in the disaster drill. This condition was due to the fact that the academic schedule of the university was very tight, and did not allow some students to meet the schedule of the drill. Mean participants' age was 20 years old, whether undergraduate or diploma student for in-class training, whereas in the disaster drill it was 21 years old. In terms of gender, the majority of participants were female. More than 20% of participants had one of previous training experiences in emergency, basic life support, basic trauma life support, or disaster response as part of their curriculum or extracurricular program.
Pre-Test & Post-Test Evaluation for In-Class Training Table 2 Demographic data. In-class training
Disaster drill
Number of Participants (n, %)
n = 309
%
n = 225a
%
Undergraduate students Diploma students
170 139
55.02 44.98
87 138
38.67 61.33
Age (Range, Mean)
Range
Mean
Range
Mean
All participants Undergraduate students Diploma students
18–29 18–22 19–29
20.66 19–29 20.44 20–22 20.92 19–29
21 20.92 21.05
Sex (n, %)
n
%
n
%
Female Male
271 38
87.70 12.30
194 31
66.22 33.78
Previous disaster/emergency training participation
n
%
n
%
Undergraduate students
47 123 35 104
27.65 72.35 25.18 74.82
28 59 34 104
32.18 67.82 24.64 75.36
Diploma students a
With Without With Without
84 undergraduate students could not participate in disaster drill for academic schedule reason.
The mean pre-test scores for undergraduate students and diploma students were 9.84 and 10.38, respectively (P = 0.001, t-test) (see Tables 3 and 4). However, the mean post-test scores were not significantly different between the two groups. This caused a slightly higher mean difference (D) score in undergraduate students (4.63) in comparison to diploma students (4.29), whereas the scores were significantly improved in both groups (P b 0.001 for both, paired t-test). The study found that diploma students, without previous training experience, have a significantly higher difference score (4.90) than groups who have previous training experience (3.30) (P b 0.001, t-test) (Fig. 3).
Table 3 Pre-test & post-test result.
Pre-test Post-test Difference
Undergraduate students
Diploma students
P
9.84 14.46 4.63
10.38 14.68 4.29
0.01 0.45 0.34
S. Alim et al. / Nurse Education Today 35 (2015) 25–31
Interview Results
Table 4 Pre-test & post-test result categorized by institution. Institution
Pretest
Posttest
Difference
University Aa B C D
9.84 10.65 11.31 10.14 9.37
14.46 12.19 12.94 16.25 16.33
4.63 1.54 1.63 6.12 6.96
a
29
A–D refer to the 4 randomly chosen nursing colleges in Yogyakarta, Indonesia.
Observation Data of Disaster Drill For the 225 participants, a mean score of 4.39 was reported for the ability to work with other team members (Obs.4), mean score of 4.10, for the ability to transport survivors safely to the nearest health facility (Obs.3), mean score of 4.05 for the ability to do initial treatment (Obs.2) and the lowest reported mean score of 3.70 for the ability to do triage and identify health problems (Obs.1). The results of the observations are presented in Table 5. Mean scores for Obs.1 and Obs.4 in undergraduate students were 4.38 and 4.87, which were significantly higher than diploma students 3.28 and 4.09 respectively, whereas mean scores for Obs.2 and Obs.3, were almost the same for both undergraduate students (4.13 and 4.24, respectively) and diploma students 4.00 for both items. In order to find out the factors that could significantly influence the observation result (Obs.1, Obs.2, Obs.3, and Obs.4) among groups, we analyzed all possible factors including strata of students (undergraduate or diploma), post-test score (lowest and highest among each group), post-test mean of each group, mean difference of score (D) for each group, and training experience using stepwise linear regression analysis. We found that several factors were significantly correlated with Obs.1 and Obs.4 (see Tables 6 and 7). The most influential factor correlated with these two evaluation items, was the difference level of undergraduate or diploma. For Obs.4, post-test score negatively influenced the result. It showed that participants with lower post-test scores had higher difference scores.
Fig. 3. t-Test analysis of difference score in relation to training experience.
One of the authors (SA) conducted focused coding and analyzed all data collected from interview. Table 8 shows the extent to which participants responded positively to statements related to benefits and advantages of training and disaster drill. The response by most of the participants (73%) can be summarized as “I could understand what to do in disaster situations or emergency conditions”. Whereas only some participants (27.5%) can be summarized as “I can have benefit to support my academic class”. Related to the evaluation of content and process, the response from most participants can be summarized, as “the training and disaster drill materials were interesting and up to date” (80%); and “there was limited time for disaster drill and training” (83%). With regard to instructors, most participants' response (83%) can be summarized as “the trainers and instructors were clear and easy to understand”. Participants' response in the drill (20%) can be summarized as “the drill should have more of cases added to the scenario”. Discussion The effectiveness of the present study was demonstrated by the result of three components: pre-test and post-test, observation during disaster drill, and interview. Pre-Test and Post-Test In the present study we used pre-test and post-test evaluations to assess knowledge achievement after completing the in-class disaster preparedness training. However, the value of the pre-test and posttest evaluation tends to be influenced by many external factors, including the Hawthorne effect, the halo effect and the practice effect. The Hawthorne effect refers to the phenomenon of altered behavior or performance resulting from awareness of being a part of an experimental study (Campbell et al., 1995), the halo effect is defined as the tendency to let the assessment of an individual on one trait, influence evaluation of that person on other specific traits (Darby, 2007), and the practice effect is a phenomenon where the performance of a person is influenced by the learning that results from repeated exposure to the testing materials (Beglinger et al., 2005). In order to provide accurate pre-test and post-test results without the influence of the halo effect, the present study used an anonymous-unique code number for every participant's evaluation result. The first evaluation of the pre-test and post-test results, showed that participants' mean different score (D) was positive in both undergraduate and diploma students. This result means that the participants gained a better understanding of disaster preparedness after completing the training. In measuring the effectiveness of training, Palameta et al. described that “an effective training can be defined as one which is designed and delivered in a way that is well aligned with the learning needs of its target population such that it enables participation and produces relevant knowledge and skill gains” (Palameta et al., 2011, p.2). In this sense, our study proved to be effective. The findings also confirmed that the mean difference score in pretest and post-test evaluation for undergraduate student was slightly higher than diploma students. The other finding revealed that the group with previous training experience had on average a lower difference score than those without, in diploma students. This finding raised an issue related to the benefits of previous training. Whereas the group with previous training experience was expected to have a better result in evaluation in this instance it didn't. The possible reason might be that most diploma students have previous training experience in a specific subject related to emergency response rather than disaster preparedness. The other reason is that in diploma students the group with previous training on the average already achieved higher pre-test
30
S. Alim et al. / Nurse Education Today 35 (2015) 25–31
Table 5 Observation result. Observation items
All participants (n = 225), SD
Diploma students (n = 138), SD
Undergraduate students (n = 87), SD
Obs.1. The ability to do the triage and to identify the survivors' health problems in disaster situation. Obs.2. The ability to do the initial treatment for survivors. Obs.3. The ability to transport survivors safely from the field to the nearest health facility. Obs.4. The ability to work with other members.
3.7 (0.71) 4.05 (0.51) 4.10 (0.43) 4.39 (0.82)
3.28 (0.45) 4.00 (0.43) 4.00 (0.08) 4.09 (0.89)
4.38 (0.49) 4.13 (0.61) 4.24 (0.67) 4.87 (0.34)
5 point Likert scale (5 = very proficient).
scores than those without, so that there was a small possibility of gaining further increased scores.
confidence in dealing with emergency preparedness situations and in working within a team. Structured One-to-One Interview
Observation During Disaster Drill
The effectiveness of the present study was also evaluated based on the results of the interview with 40 participants. Most participants responded positively, that this training and drill improved knowledge and skills of disaster preparedness. They felt that they could understand and do actions to respond in disaster situations. The study from Kaplan et al. (2012) also found that most participants commented positively on the emergency preparedness disaster drill. Another study conducted by Ireland et al. (2006) in integrating disaster preparedness into a community health nursing course found that by participating in a mock drill, nursing students were given the opportunity to apply the learning and to develop additional skills required to respond to a variety of threats. The students also commented that the program had been helpful in developing understanding and experiencing the nature of catastrophic events.
The second variable demonstrating the effectiveness of the study was based on observations during the disaster drill to evaluate the ability to respond in a disaster event. Consistent with the findings in the pre-test and post-test evaluation, the mean score was higher for undergraduate students than those for diploma students (Table 5). The findings proved that most students were able to respond to the scenarios of disaster drill and able to perform procedures, starting from conducting of triage, to identifying health problems, performing initial treatment and transporting patients safely to the healthcare facility. Previous studies have suggested that conducting disaster drills for nursing students is essential for building capacity in the early stage of professional education. It could serve as a means for students to gain experience, develop concrete ideas and know what to do in dealing with catastrophic situations (Kako et al., 2012). All participants scored higher than average in relation to teamwork. In their study of the use of simulation for training teamwork skills in health care, Beaubien & Baker (2004) defined teamwork as behaviors such as dynamic interaction, having a shared past, having a foreseeable shared future, and sharing a common fate that facilitates effective team member interaction. Furthermore, in order to achieve an effective result from teamwork, each member must possess teamwork competency components such as knowledge, skills, and attitudes. Teamwork knowledge refers to factual information that the team members must possess, such as information about the mission, or information about the roles and responsibilities of individual team members. Teamwork skills refer to the learned capacity to perform some type of task, such as being able to communicate clearly, concisely, and using the proper phraseology. Teamwork attitudes refer to mental states that influence the team members to behave in a particular way. Teamwork effectiveness seemed to be achieved by the process of gaining new knowledge in the training session and could be directly put into action in the disaster drill. Participants could also play particular roles within a group during the drill. The findings were also consistent with the study from Kaplan et al. (2012) who investigated the use of emergency preparedness disaster drills on undergraduate nursing students. The study found that the majority of participants (95%) stated, “agreed” and “strongly agreed” that the drill increased knowledge and
The present study conducted a disaster preparedness training program and disaster simulation. The study successfully evaluated the effectiveness of the disaster preparedness training and disaster drill: The program improved the ability of disaster preparedness in nursing students, both undergraduate and diploma students; the extent of improvement depended on previous experience of training. Further studies are needed to develop more extensive training curricula and design for disaster preparedness, as well as specific evaluation of posttraining for both the short term and the long term.
Table 6 Stepwise linear regression analysis result for Obs.1.
Table 7 Stepwise linear regression analysis result for Obs.4.
Limitations Our study has several limitations. First, our in-class training program was designed as a one day eight hour course. This was a very limited time to cover all the subjects required in disaster preparedness training along with pre-test and post-test evaluation. Second, the concurrent schedule of regular academic studies made it impossible for some undergraduate students to participate in the disaster drill. Third, the present study was able to assess short-term achievement of knowledge and skills, but was unable to evaluate long-term knowledge and skill retention. Conclusions
Variable
β⁎
Standard error
T value
P
Variable
βa
Standard error
t value
P
Mean difference Previous training experience
0.21 0.01
0.07 0.004
2.88 2.58
0.01 0.02
Mean difference Post-test score
0.36 −0.39
0.09 0.14
3.67 −2.85
0.002 0.01
Adj-R2. ⁎ Standardized partial regression coefficient.
Adj-R2. a Standardized partial regression coefficient.
S. Alim et al. / Nurse Education Today 35 (2015) 25–31
References
Table 8 Interview result. n = 40, % I. Benefits and advantages of training and disaster drill: 1. I could understand what to do in disaster or emergency situations 2. I could improve my knowledge in disaster response 3. It could support my academic class 4. I think this training had an advantage for practical session (disaster drill) II. Evaluation of content and process 1. The training materials were interesting 2. The training materials were up to date 3. I think the time for training and drill were limited 4. The trainers and instructors were clear and easy to understand 5. I think the drill should have more cases
31
75 60 27.5 65
75 80 83 83 20
Acknowledgments 1. This study was supported by program funding from the Research Center for Urban Safety and Security (RCUSS), Kobe University, Japan. 2. We would like to thank Sutono, Sri Setiyarini, Happy Kusumawati, and Eri Yanuar in Emergency and Critical Care Nursing Department, School of Nursing, Faculty of Medicine, Universitas Gadjah Mada for all the hard work to assist this program. 3. We would like to thank to all trainers and instructors (Al Afik, and Sumartinah) for their assistance in this study. 4. We would like to thank Don Moore for his assistance with the English phrasing of our ideas.
Beaubien, J.M., Baker, D.P., 2004. The use of simulation for training teamwork skills in health care: how low can you go? Qual. Saf. Health Care 13 (Suppl. 1), i51–i56. Beglinger, L.J., Gaydos, B., Tangphao-Daniels, O., Duff, K., Kareken, D.A., Crawford, J., et al., 2005. Practice effects and the use of alternate forms in serial neuropsychological testing. Arch. Clin. Neuropsychol. 20 (4), 517–529. Campbell, J.P., Maxey, V.A., Watson, W.A., 1995. Hawthorne effect: implications for prehospital research. Ann. Emerg. Med. 26 (5), 590–594. Cole, F.L., 1960. Preparation for disaster in California. Calif. Med. 93 (2), 90–92. Darby, J.A., 2007. Are course evaluations subject to a ‘halo effect’? Res. Ed. (77), 46–55. Dorsey, D.M., et al., 2009. ICN Framework of Disaster Nursing Competencies. World Health Organization and International Council of Nurses, Geneva, Switzerland pp. 47–60. Gebbie, K., Qureshi, K., 2002. Emergency and disaster preparedness: core competencies for nurses: what every nurse should but may not know. Am. J. Nurs. 102 (1), 46–51. Hsu, E., Thomas, T., Bass, E., Whyne, D., Kelen, G., Green, G., 2006. Healthcare worker competencies for disaster training. BMC Med. Educ. 6 (1), 19. Ireland, M., Ea, E., Kontzamanis, E., Michel, C., 2006. Integrating disaster preparedness into a community health nursing course: one school's experience. Disaster Manag. Response 4, 72–76. Jennings-Sanders, A., Frisch, N., Wing, S., 2005. Nursing students' perceptions about disaster nursing. Disaster Manag. Response 3, 6. Kako, M., Mitani, S., Arbon, P., 2012. Literature review of disaster health research in Japan: focusing on disaster nursing education. Prehosp. Disaster Med. 27 (2), 1–6. Kaplan, B.G., Connor, A., Ferranti, E.P., Holmes, L., Spencer, L., 2012. Use of an emergency preparedness disaster simulation with undergraduate nursing students. Public Health Nurse 29 (1), 44–45. Palameta, B., et al., 2011. Understanding Training Program Effectiveness: A Comprehensive framework. Social Research and Demonstration Corporation, Ontario, Canada pp. 2–20. World Health Organization Regional Office for Eastern Mediterranean, Strategy-resolution. Emergency Preparedness and Humanitarian Actions. Resolutions. http://www.emro. who.int/emergency/strategy-resolutions/ (Accessed July 9, 2013). World Health Organization and International Council of Nurses, ICN Framework of Disaster Nursing Competencies. Geneva, Switzerland.
Contents lists available at ScienceDirect
Nurse Education Today journal homepage: www.elsevier.com/nedt
Evaluation of disaster preparedness training and disaster drill for nursing students Syahirul Alim a,b,1, Masato Kawabata c,2, Minato Nakazawa d,3 a
Graduate School of Health Sciences, Kobe University, Japan School of Nursing, Faculty of Medicine, Universitas Gadjah Mada, Indonesia Graduate School of Medicine, Kobe University, Japan d Graduate School of Health Sciences, Kobe University, Japan b c
a r t i c l e
i n f o
Article history: Accepted 23 April 2014
Keywords: Disaster preparedness training Nursing students Disaster drill Training evaluation
s u m m a r y Background: Preparedness and preventive measures are needed to reduce the impact of disasters. Disaster preparedness training for nurses has a long history. However, the effectiveness of disaster preparedness training for nursing students has been limited, to some extent, since they have been based on self-evaluation. Objective: The study attempts to evaluate the effectiveness of a disaster preparedness training program followed by a disaster drill designed for nursing students. Participants: Participants were undergraduate students from Universitas Gadjah Mada and diploma students from four randomly chosen nursing colleges located in Yogyakarta, Indonesia. 309 students participated in the training program whereas 225 students participated in the disaster drill. Methods: The present study conducted in-class training followed by a disaster drill and evaluated using 3 components: pre-test and post-test evaluation of knowledge (score range: 0–20), observation of skills during disaster drill (5-point Likert scale), and a structured one-to-one interview of participants' responses to the training and drill process. Results: Pre-test and post-test evaluation scores showed significant improvement (P b 0.01, paired t-test) for both university and diploma students. Almost all observation items during the disaster drill were above 4.0 (on 5-point Likert scale). Interview results showed that most participants responded positively. Conclusions: The present study completely evaluated the effectiveness of a disaster preparedness training and disaster drill: The training and drill improved the knowledge and ability of disaster preparedness for both undergraduate and diploma students. © 2014 Elsevier Ltd. All rights reserved.
Introduction An impact resulting from a catastrophic condition caused by natural disasters affected not only buildings and the surrounding infrastructure but also human life and society itself. In order to reduce the impact of disasters, the World Health Organization (WHO) has stressed that despite the importance of relief supplies in emergencies, preventive measures and preparedness are of equal and perhaps more fundamental importance. In the health sector, the availability of suitably equipped health providers who are able to respond when needed is considered vital. (WHO, 2013)
E-mail addresses: [email protected] (S. Alim), [email protected] (M. Kawabata), [email protected] (M. Nakazawa). 1 Tel.: +81 80 4646 4944. 2 Tel.: +81 78 382 5681. 3 Tel.: +81 78 796 4551.
http://dx.doi.org/10.1016/j.nedt.2014.04.016 0260-6917/© 2014 Elsevier Ltd. All rights reserved.
Nurses, the main source of healthcare providers, are indispensable workforce in disaster situations. It is nurses' responsibility to understand their role in preparedness as well as to prepare in advance in order to be available if necessary (ICN, 2009). In this sense, disaster preparedness education is vital for nurses in order to be sufficient in knowledge and skills to respond in any disaster. Disaster preparedness training for nurses had increasingly been reported in America since the end of World War II. It was reported by Cole (1960) that in the United States between 1951 and 1960, there were 656,333 home nurses and first aiders trained under the disaster plan project and 14,405 registered nurses had been given courses in disaster nursing. Exercise London, conducted in London, Ontario, Canada in 1974, was the first successful large-scale disaster drill. It was a largescale drill to assess the ability of healthcare providers and other community components to overcome catastrophic conditions using 150 casualties and involving a large-scale community (Cole, 1960). Despite the efforts to train healthcare providers, there is a gap between what nursing students are taught and what practitioners in
26
S. Alim et al. / Nurse Education Today 35 (2015) 25–31
disaster preparedness know (Jennings-Sanders et al., 2005). In addition, the same study also found that students have limitations in finding information related to disasters and this leads to inadequate knowledge regarding disaster nursing such as: (1) incomplete definitions of disaster nursing; (2) lack of knowledge about community resources, mock disaster drills, and how to utilize disaster planning models; and (3) unawareness of the significant role nurses could play in disaster situations. In order to overcome these conditions nursing education institutions initiated to incorporate disaster-related subjects into their curriculums. Disaster-related subjects are being included as a part of Emergency Nursing and/or Community Health Nursing Programs. Another possibility is to develop disaster nursing as an independent subject. In recent years, many academic institutions introduced disaster topics for students in disaster or emergency-related training programs. Following in-class training they practiced their new skills in disaster drills. A study conducted by Kaplan et al. revealed a positive response from the implementation of Emergency Preparedness Disaster Simulation (EPDS) for nursing students (Kaplan et al., 2012) However, the effectiveness of these programs has not been properly evaluated yet. It is necessary to have a comprehensive evaluation of both the effectiveness of the training and the disaster drill itself. Feedback and input from the participants could provide additional data to better design and develop curriculums for training and drills. Methods All participants in the study were undergraduate students from Universitas Gadjah Mada, and diploma students from four randomly chosen nursing colleges in Yogyakarta, Indonesia. The undergraduate students were fourth grade students who had zero experience in actual hospital practice as part of their academic program. While the diploma students were third grade students from nursing colleges who already been exposed to hospital practices for their academic program. Both undergraduate and diploma students were recruited firstly by giving them information related to this study through the teachers. Participants voluntarily agreed to participate in this study. A-one day disaster preparedness training and a disaster drill were conducted in the study. In-class training enabled students to gain knowledge and understanding in the context of disaster preparedness. The training sessions were conducted based on a schedule that had been arranged and agreed by all institutions. Each nursing college was responsible for choosing students who participated in this training. The disaster drill stimulated participants to apply their knowledge and skills they had learned in a “real” case condition. In the disaster drill all participants worked in small groups, which gave them the opportunity to develop the sense of teamwork and collaboration needed to respond in a disaster scenario. The scenario was based on real events experienced by the authors of the 2006 earthquake in Yogyakarta. The situation and description of the environment, survivors and availability of resources were all explained in detail. The study evaluated the effectiveness of the training and drill using 3 evaluation components: (1) Pre-test and post-test evaluation to measure the degree of knowledge achieved in the training; (2) observation to evaluate skills during the disaster drill; and (3) interview to obtain participants' responses and feedbacks on both the training and drill. Disaster Preparedness Training A big challenge in conducting this study was that there was no standard guideline related to disaster preparedness training for nursing students. In lieu of this, we followed the International Council of Nurses framework of disaster nursing competency (Dorsey et al., 2009) to develop the curriculum for our training. In addition, the study also incorporated, ‘cross-cutting competencies for healthcare workers in disaster training’, developed by Hsu et al. (2006) and Core
Competencies for Nurses in Emergency and Disaster Preparedness by Gebbie and Qureshi (2002). The training was held over 8 h. The course was divided into 4 topics (I) Introduction to disaster preparedness for nurses, which covered a general description of health problems in disasters, the role of nurses in every phase of a disaster and nursing care principles in a disaster; (II) Command and management for healthcare, which covered incident command systems, multiagency coordination systems and public information systems; (III) Basic life support I, which covered triage methods in the field and cardio-pulmonary resuscitation; (IV) Basic life support II, which covered management for head trauma, bleeding and fracture (Table 1). Trainers and instructors for this course were nursing practitioners from hospitals and universities who had expertise in disaster, emergency and critical care. The trainers also had a great deal of actual experience in responding to disaster situations both nationally and internationally. Each trainer was assigned to be responsible for one topic. The guideline book and manual were distributed to all trainers prior to the training session, so that all the trainers had the same competencies related to the process of training and topics. Twenty minute pre-test and post-test were conducted before and after completing the training session for every participant. There were 20 multiple-choice questions based on topics covering all aspects of the training. Each question was scored as one for a correct answer and zero for an incorrect one. Thus the total score ranged from zero to twenty. The evaluation had five questions from each topic of training. The evaluation sheet was first read and evaluated by all trainers and instructors for content validity and consistency before the training session began. In the beginning of training, participants filled a sheet of demographic data survey. We also asked information about participants' previous training experience since this information could influence the result of study. Participants might have had previous experience in disaster or emergency response training. All participants received handouts on the topics and were allowed to take notes during the training sessions. At the end of each topic session delivered by a trainer, a question period ensued and was encouraged. Disaster Drill Only participants, who completed the in-class training of disaster preparedness, participated in the disaster drill. A number of undergraduate students could not participate in this drill due to their academic schedules while only one diploma student was absent for the disaster drill. The total number of participants in the disaster drill was 225 students. It consisted of 138 diploma students and 87 undergraduate students. Prior to the drill the participants were divided into small groups of 10–15 students. In this disaster drill, the participants were challenged to respond to the disaster situation written on the scenario sheet (Fig. 2, scenario of the disaster drill). They were encouraged to work together as a group to respond to the disaster situation. Each student was assigned a different task according to the specific role. Some were team leaders, some triage staff, and others treatment, and evacuation staff. (See Fig. 1.) This disaster drill used actor-patients as well as patient simulators. Actor-patients were non-participant students who played the role of victim patients. They were dressed and made up according to the scenario they were in using artificial wounds and blood. They were then coached by a skillful instructor in disaster, to mimic the real sounds and condition of patients in the field. Patient simulators were used for practicing cardio-pulmonary resuscitation and other treatments that were not suitable to be used on actor-patients. Disaster drills were conducted in the parking lot of the university and the emergency nursing practice room served as the health facility. All groups prepared medications and equipment necessary for initial
S. Alim et al. / Nurse Education Today 35 (2015) 25–31
27
Table 1 Topics of training. Topics
Sub-topics
Duration (hours)
Topic I: Introduction to the disaster preparedness for nurses
• • • • • • • • •
2
Topic II: Command and management for healthcare Topic III: Basic life support treatment in disaster 1 Topic IV: Basic life support treatment in disaster 2
A general description of health problems in disasters; The role of nurses in every phase of a disaster; Nursing care principles in a disaster. Incident Command System (ICS); Multiagency coordination systems; Public information systems. Triage methods in the field; Cardio-pulmonary resuscitation. Management for head trauma, chest injury, bleeding, fracture.
treatment and to assist in transporting patients before the drill started. Participants began the drill from practice room after being briefed about the steps of the drill and the scenario that they would be responding to. (See Fig. 2.) Instructors measured performance of the group by observing how each group responded to the disaster situation during the drill. Particular attention was also paid to how well they worked and collaborated with other members. There were 4 main items of evaluation: Obs.1) the ability to do triage and to identify the survivors' health problem; Obs.2) the ability to do initial treatment; Obs.3) the ability to transport survivors safely to the nearest health facility; and Obs.4) the ability to work with other team members. Results of the observations of all evaluation items were based on a 5-point Likert scale (5 = very proficient, 4 = proficient, 3 = fairly proficient, 2 = moderately proficient, 1 = poorly proficient). Debriefing procedures were conducted after completing the drill in order to have a clear understanding related to the drill process. In this session, participants shared their experience during the drill, how they felt about the role and task they had conducted, and how they managed to work together in a team to respond in a disaster situation. Instructors
Disaster Preparedness Training and Disaster Drill Formation
Forty participants (15%) who completed in-class training and the disaster drill were randomly chosen, using computergenerated randomized number, to participate in structured oneto-one interview. The purpose of the interview, to evaluate the whole process of training and drill, was explained to the participants. Each participant was interviewed for 5–10 min by the principal author (SA). Responses in the interview session were recorded for later data analysis. Four main questions were asked (1) “What benefits might you have from this training and drill?”; (2) “What do you think were the advantages and disadvantages of this training and drill?”; (3) “What did you think about the training and drill material, and equipment?”; and (4) “What do you think about trainers and instructors?”.
Diploma Students In-class Training: 139 students Disaster Drill: 138 students
Group A Day 1: Training (27 students) Day 2: Drill (27 students)
Coordinating with research assistants: schedule and activities.
Assessing the availability of equipment and materials for training and disaster drill.
Group B Day 1: Training (51 students) Day 2: Drill (51 students)
Group C Day 1: Training (35 students) Day 2: Drill (34 students)
Group D Day 1: Training (26 students) Day 2: Drill (26 students)
Pre and Post-Test for In-class Training Observation for Disaster Drill In depth Interview (Feedback & Input): 1. In-Class Training 2. Disaster Drill
2
Interview for Training and Disaster Drill
Disaster Preparedness Training & Disaster Drill
Completing proposal and requirements for ethical permission.
2
then provided clarifications and explanations to the participants for misconducts during the drill.
Preparation for Training & Disaster Drill
Developing guideline and materials for training and drill.
2
Evaluation Components
Fig. 1. Study process chart.
Undergraduate Students In-class Training: 170 students Disaster Drill: 87 students
Group A Day 1: Training (44 students) Day 2: Drill (44 students)
Group B Day 1: Training (43 students) Day 2: Drill (43 students)
Group C Day 1: Training (42 students) Day 2: Drill (0 students)
Group A Day 1: Training (41 students) Day 2: Drill (0 students)
28
S. Alim et al. / Nurse Education Today 35 (2015) 25–31
“You are a nursing student who do the community health program practice in the village together with 9 other nursing students. Early in the morning, while preparing for the activity in Primary Health Care, the 7.5 magnitude scale earthquake struck the area. After the quake, having checked your condition and other students are safe. Unfortunately, the house next to the Primary Health Care is nearly breakdown. The owner of the house is an elderly couple, Mr. and Mrs. Joko. Mr. Joko is unconscious, no pulse, there is bruise sign at the back of his neck. Mrs. Joko is crying, with open wound and fracture at her right arm. You and other students do the initial treatment to the victims”. 1. Within the group choose one leader and distribute the role to do the task! 2. Do the initial treatment for Mr. and Mrs. Joko! 3. Do the transport and refer to the health care facility appropriately! Fig. 2. Scenario of the disaster drill.
was voluntary and students were not compensated for their participation.
Statistical Analysis Statistical analysis was conducted using R Version 3.0.0 software (R Foundation for Statistical Computing, Vienna, Austria). Paired t-test was used to compare the mean difference between the pre-test and posttest results. The t-test was used to compare the mean score of observation in disaster drill between undergraduate and diploma students. In the performance of each group during the disaster drill stepwise linear regression analysis was performed. This was used to find out factors, which may influence the observation results, including strata of students (undergraduate and diploma), post-test score (lowest, mean, and highest), the mean difference of pre-test and post-test scores, and the proportion of students' previous training experience. Alpha (α) level for analysis was set at 0.05.
Ethical Consideration The present study was approved by the ethical committee of the School of Nursing, Faculty of Medicine, Universitas Gadjah Mada, and obtained consent from the nursing colleges, as well as the Yogyakarta Provincial Government. Participation of students
Results Demographic Data A total of 309 nursing students participated in the disaster preparedness training, whereas only 225 students took part in the disaster drill (see Table 2). The percentage of undergraduate students was 55% of the total number of in-class training participants, but decreased to 39% of those in the disaster drill. This condition was due to the fact that the academic schedule of the university was very tight, and did not allow some students to meet the schedule of the drill. Mean participants' age was 20 years old, whether undergraduate or diploma student for in-class training, whereas in the disaster drill it was 21 years old. In terms of gender, the majority of participants were female. More than 20% of participants had one of previous training experiences in emergency, basic life support, basic trauma life support, or disaster response as part of their curriculum or extracurricular program.
Pre-Test & Post-Test Evaluation for In-Class Training Table 2 Demographic data. In-class training
Disaster drill
Number of Participants (n, %)
n = 309
%
n = 225a
%
Undergraduate students Diploma students
170 139
55.02 44.98
87 138
38.67 61.33
Age (Range, Mean)
Range
Mean
Range
Mean
All participants Undergraduate students Diploma students
18–29 18–22 19–29
20.66 19–29 20.44 20–22 20.92 19–29
21 20.92 21.05
Sex (n, %)
n
%
n
%
Female Male
271 38
87.70 12.30
194 31
66.22 33.78
Previous disaster/emergency training participation
n
%
n
%
Undergraduate students
47 123 35 104
27.65 72.35 25.18 74.82
28 59 34 104
32.18 67.82 24.64 75.36
Diploma students a
With Without With Without
84 undergraduate students could not participate in disaster drill for academic schedule reason.
The mean pre-test scores for undergraduate students and diploma students were 9.84 and 10.38, respectively (P = 0.001, t-test) (see Tables 3 and 4). However, the mean post-test scores were not significantly different between the two groups. This caused a slightly higher mean difference (D) score in undergraduate students (4.63) in comparison to diploma students (4.29), whereas the scores were significantly improved in both groups (P b 0.001 for both, paired t-test). The study found that diploma students, without previous training experience, have a significantly higher difference score (4.90) than groups who have previous training experience (3.30) (P b 0.001, t-test) (Fig. 3).
Table 3 Pre-test & post-test result.
Pre-test Post-test Difference
Undergraduate students
Diploma students
P
9.84 14.46 4.63
10.38 14.68 4.29
0.01 0.45 0.34
S. Alim et al. / Nurse Education Today 35 (2015) 25–31
Interview Results
Table 4 Pre-test & post-test result categorized by institution. Institution
Pretest
Posttest
Difference
University Aa B C D
9.84 10.65 11.31 10.14 9.37
14.46 12.19 12.94 16.25 16.33
4.63 1.54 1.63 6.12 6.96
a
29
A–D refer to the 4 randomly chosen nursing colleges in Yogyakarta, Indonesia.
Observation Data of Disaster Drill For the 225 participants, a mean score of 4.39 was reported for the ability to work with other team members (Obs.4), mean score of 4.10, for the ability to transport survivors safely to the nearest health facility (Obs.3), mean score of 4.05 for the ability to do initial treatment (Obs.2) and the lowest reported mean score of 3.70 for the ability to do triage and identify health problems (Obs.1). The results of the observations are presented in Table 5. Mean scores for Obs.1 and Obs.4 in undergraduate students were 4.38 and 4.87, which were significantly higher than diploma students 3.28 and 4.09 respectively, whereas mean scores for Obs.2 and Obs.3, were almost the same for both undergraduate students (4.13 and 4.24, respectively) and diploma students 4.00 for both items. In order to find out the factors that could significantly influence the observation result (Obs.1, Obs.2, Obs.3, and Obs.4) among groups, we analyzed all possible factors including strata of students (undergraduate or diploma), post-test score (lowest and highest among each group), post-test mean of each group, mean difference of score (D) for each group, and training experience using stepwise linear regression analysis. We found that several factors were significantly correlated with Obs.1 and Obs.4 (see Tables 6 and 7). The most influential factor correlated with these two evaluation items, was the difference level of undergraduate or diploma. For Obs.4, post-test score negatively influenced the result. It showed that participants with lower post-test scores had higher difference scores.
Fig. 3. t-Test analysis of difference score in relation to training experience.
One of the authors (SA) conducted focused coding and analyzed all data collected from interview. Table 8 shows the extent to which participants responded positively to statements related to benefits and advantages of training and disaster drill. The response by most of the participants (73%) can be summarized as “I could understand what to do in disaster situations or emergency conditions”. Whereas only some participants (27.5%) can be summarized as “I can have benefit to support my academic class”. Related to the evaluation of content and process, the response from most participants can be summarized, as “the training and disaster drill materials were interesting and up to date” (80%); and “there was limited time for disaster drill and training” (83%). With regard to instructors, most participants' response (83%) can be summarized as “the trainers and instructors were clear and easy to understand”. Participants' response in the drill (20%) can be summarized as “the drill should have more of cases added to the scenario”. Discussion The effectiveness of the present study was demonstrated by the result of three components: pre-test and post-test, observation during disaster drill, and interview. Pre-Test and Post-Test In the present study we used pre-test and post-test evaluations to assess knowledge achievement after completing the in-class disaster preparedness training. However, the value of the pre-test and posttest evaluation tends to be influenced by many external factors, including the Hawthorne effect, the halo effect and the practice effect. The Hawthorne effect refers to the phenomenon of altered behavior or performance resulting from awareness of being a part of an experimental study (Campbell et al., 1995), the halo effect is defined as the tendency to let the assessment of an individual on one trait, influence evaluation of that person on other specific traits (Darby, 2007), and the practice effect is a phenomenon where the performance of a person is influenced by the learning that results from repeated exposure to the testing materials (Beglinger et al., 2005). In order to provide accurate pre-test and post-test results without the influence of the halo effect, the present study used an anonymous-unique code number for every participant's evaluation result. The first evaluation of the pre-test and post-test results, showed that participants' mean different score (D) was positive in both undergraduate and diploma students. This result means that the participants gained a better understanding of disaster preparedness after completing the training. In measuring the effectiveness of training, Palameta et al. described that “an effective training can be defined as one which is designed and delivered in a way that is well aligned with the learning needs of its target population such that it enables participation and produces relevant knowledge and skill gains” (Palameta et al., 2011, p.2). In this sense, our study proved to be effective. The findings also confirmed that the mean difference score in pretest and post-test evaluation for undergraduate student was slightly higher than diploma students. The other finding revealed that the group with previous training experience had on average a lower difference score than those without, in diploma students. This finding raised an issue related to the benefits of previous training. Whereas the group with previous training experience was expected to have a better result in evaluation in this instance it didn't. The possible reason might be that most diploma students have previous training experience in a specific subject related to emergency response rather than disaster preparedness. The other reason is that in diploma students the group with previous training on the average already achieved higher pre-test
30
S. Alim et al. / Nurse Education Today 35 (2015) 25–31
Table 5 Observation result. Observation items
All participants (n = 225), SD
Diploma students (n = 138), SD
Undergraduate students (n = 87), SD
Obs.1. The ability to do the triage and to identify the survivors' health problems in disaster situation. Obs.2. The ability to do the initial treatment for survivors. Obs.3. The ability to transport survivors safely from the field to the nearest health facility. Obs.4. The ability to work with other members.
3.7 (0.71) 4.05 (0.51) 4.10 (0.43) 4.39 (0.82)
3.28 (0.45) 4.00 (0.43) 4.00 (0.08) 4.09 (0.89)
4.38 (0.49) 4.13 (0.61) 4.24 (0.67) 4.87 (0.34)
5 point Likert scale (5 = very proficient).
scores than those without, so that there was a small possibility of gaining further increased scores.
confidence in dealing with emergency preparedness situations and in working within a team. Structured One-to-One Interview
Observation During Disaster Drill
The effectiveness of the present study was also evaluated based on the results of the interview with 40 participants. Most participants responded positively, that this training and drill improved knowledge and skills of disaster preparedness. They felt that they could understand and do actions to respond in disaster situations. The study from Kaplan et al. (2012) also found that most participants commented positively on the emergency preparedness disaster drill. Another study conducted by Ireland et al. (2006) in integrating disaster preparedness into a community health nursing course found that by participating in a mock drill, nursing students were given the opportunity to apply the learning and to develop additional skills required to respond to a variety of threats. The students also commented that the program had been helpful in developing understanding and experiencing the nature of catastrophic events.
The second variable demonstrating the effectiveness of the study was based on observations during the disaster drill to evaluate the ability to respond in a disaster event. Consistent with the findings in the pre-test and post-test evaluation, the mean score was higher for undergraduate students than those for diploma students (Table 5). The findings proved that most students were able to respond to the scenarios of disaster drill and able to perform procedures, starting from conducting of triage, to identifying health problems, performing initial treatment and transporting patients safely to the healthcare facility. Previous studies have suggested that conducting disaster drills for nursing students is essential for building capacity in the early stage of professional education. It could serve as a means for students to gain experience, develop concrete ideas and know what to do in dealing with catastrophic situations (Kako et al., 2012). All participants scored higher than average in relation to teamwork. In their study of the use of simulation for training teamwork skills in health care, Beaubien & Baker (2004) defined teamwork as behaviors such as dynamic interaction, having a shared past, having a foreseeable shared future, and sharing a common fate that facilitates effective team member interaction. Furthermore, in order to achieve an effective result from teamwork, each member must possess teamwork competency components such as knowledge, skills, and attitudes. Teamwork knowledge refers to factual information that the team members must possess, such as information about the mission, or information about the roles and responsibilities of individual team members. Teamwork skills refer to the learned capacity to perform some type of task, such as being able to communicate clearly, concisely, and using the proper phraseology. Teamwork attitudes refer to mental states that influence the team members to behave in a particular way. Teamwork effectiveness seemed to be achieved by the process of gaining new knowledge in the training session and could be directly put into action in the disaster drill. Participants could also play particular roles within a group during the drill. The findings were also consistent with the study from Kaplan et al. (2012) who investigated the use of emergency preparedness disaster drills on undergraduate nursing students. The study found that the majority of participants (95%) stated, “agreed” and “strongly agreed” that the drill increased knowledge and
The present study conducted a disaster preparedness training program and disaster simulation. The study successfully evaluated the effectiveness of the disaster preparedness training and disaster drill: The program improved the ability of disaster preparedness in nursing students, both undergraduate and diploma students; the extent of improvement depended on previous experience of training. Further studies are needed to develop more extensive training curricula and design for disaster preparedness, as well as specific evaluation of posttraining for both the short term and the long term.
Table 6 Stepwise linear regression analysis result for Obs.1.
Table 7 Stepwise linear regression analysis result for Obs.4.
Limitations Our study has several limitations. First, our in-class training program was designed as a one day eight hour course. This was a very limited time to cover all the subjects required in disaster preparedness training along with pre-test and post-test evaluation. Second, the concurrent schedule of regular academic studies made it impossible for some undergraduate students to participate in the disaster drill. Third, the present study was able to assess short-term achievement of knowledge and skills, but was unable to evaluate long-term knowledge and skill retention. Conclusions
Variable
β⁎
Standard error
T value
P
Variable
βa
Standard error
t value
P
Mean difference Previous training experience
0.21 0.01
0.07 0.004
2.88 2.58
0.01 0.02
Mean difference Post-test score
0.36 −0.39
0.09 0.14
3.67 −2.85
0.002 0.01
Adj-R2. ⁎ Standardized partial regression coefficient.
Adj-R2. a Standardized partial regression coefficient.
S. Alim et al. / Nurse Education Today 35 (2015) 25–31
References
Table 8 Interview result. n = 40, % I. Benefits and advantages of training and disaster drill: 1. I could understand what to do in disaster or emergency situations 2. I could improve my knowledge in disaster response 3. It could support my academic class 4. I think this training had an advantage for practical session (disaster drill) II. Evaluation of content and process 1. The training materials were interesting 2. The training materials were up to date 3. I think the time for training and drill were limited 4. The trainers and instructors were clear and easy to understand 5. I think the drill should have more cases
31
75 60 27.5 65
75 80 83 83 20
Acknowledgments 1. This study was supported by program funding from the Research Center for Urban Safety and Security (RCUSS), Kobe University, Japan. 2. We would like to thank Sutono, Sri Setiyarini, Happy Kusumawati, and Eri Yanuar in Emergency and Critical Care Nursing Department, School of Nursing, Faculty of Medicine, Universitas Gadjah Mada for all the hard work to assist this program. 3. We would like to thank to all trainers and instructors (Al Afik, and Sumartinah) for their assistance in this study. 4. We would like to thank Don Moore for his assistance with the English phrasing of our ideas.
Beaubien, J.M., Baker, D.P., 2004. The use of simulation for training teamwork skills in health care: how low can you go? Qual. Saf. Health Care 13 (Suppl. 1), i51–i56. Beglinger, L.J., Gaydos, B., Tangphao-Daniels, O., Duff, K., Kareken, D.A., Crawford, J., et al., 2005. Practice effects and the use of alternate forms in serial neuropsychological testing. Arch. Clin. Neuropsychol. 20 (4), 517–529. Campbell, J.P., Maxey, V.A., Watson, W.A., 1995. Hawthorne effect: implications for prehospital research. Ann. Emerg. Med. 26 (5), 590–594. Cole, F.L., 1960. Preparation for disaster in California. Calif. Med. 93 (2), 90–92. Darby, J.A., 2007. Are course evaluations subject to a ‘halo effect’? Res. Ed. (77), 46–55. Dorsey, D.M., et al., 2009. ICN Framework of Disaster Nursing Competencies. World Health Organization and International Council of Nurses, Geneva, Switzerland pp. 47–60. Gebbie, K., Qureshi, K., 2002. Emergency and disaster preparedness: core competencies for nurses: what every nurse should but may not know. Am. J. Nurs. 102 (1), 46–51. Hsu, E., Thomas, T., Bass, E., Whyne, D., Kelen, G., Green, G., 2006. Healthcare worker competencies for disaster training. BMC Med. Educ. 6 (1), 19. Ireland, M., Ea, E., Kontzamanis, E., Michel, C., 2006. Integrating disaster preparedness into a community health nursing course: one school's experience. Disaster Manag. Response 4, 72–76. Jennings-Sanders, A., Frisch, N., Wing, S., 2005. Nursing students' perceptions about disaster nursing. Disaster Manag. Response 3, 6. Kako, M., Mitani, S., Arbon, P., 2012. Literature review of disaster health research in Japan: focusing on disaster nursing education. Prehosp. Disaster Med. 27 (2), 1–6. Kaplan, B.G., Connor, A., Ferranti, E.P., Holmes, L., Spencer, L., 2012. Use of an emergency preparedness disaster simulation with undergraduate nursing students. Public Health Nurse 29 (1), 44–45. Palameta, B., et al., 2011. Understanding Training Program Effectiveness: A Comprehensive framework. Social Research and Demonstration Corporation, Ontario, Canada pp. 2–20. World Health Organization Regional Office for Eastern Mediterranean, Strategy-resolution. Emergency Preparedness and Humanitarian Actions. Resolutions. http://www.emro. who.int/emergency/strategy-resolutions/ (Accessed July 9, 2013). World Health Organization and International Council of Nurses, ICN Framework of Disaster Nursing Competencies. Geneva, Switzerland.
