The Journal of Emergency Medicine, Vol. 48, No. 1, pp. 19–25, 2015 Copyright Ó 2015 Elsevier Inc. Printed in the USA. All rights reserved 0736-4679/$ - see front matter
http://dx.doi.org/10.1016/j.jemermed.2014.06.039
Original Contributions
THE EFFECTIVENESS OF A STUDENT VOLUNTEER PROGRAM FOR RESEARCH IN A PEDIATRIC EMERGENCY DEPARTMENT Patrick E. Steadman, MSC,* Johanna Crudden, RT,† Taline Naranian, MSC,† John Paul Oliveria, BSC,†‡ and Kathy Boutis, MD*† *Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada, †Division of Emergency Medicine, Department of Pediatrics, The Hospital for Sick Children, Toronto, Ontario, Canada, and ‡Department of Medicine, McMaster University, Hamilton, Ontario, Canada Reprint Address: Kathy Boutis, MD, The Hospital for Sick Children, 555 University Avenue, Toronto, ON M5G 1X8, Canada
, Abstract—Background: Emergency Department (ED) student-based research assistant programs have been shown to be effective in enrolling patients when the students receive university course credit or pay. However, the impact on research outcomes when university students act as volunteers in this role is relatively unknown. Objectives: The main objective of this study was to determine how often potentially eligible children were accurately identified by volunteer research assistants for enrollment into prospective research in the ED. We also examined the frequency of successful enrollments and the accuracy of data capture. Methods: This was a prospective crosssectional study of university student volunteer research assistant performance in a tertiary care pediatric ED between March 2011 and July 2013. The participant’s primary role was to screen and facilitate enrollment of ED patients into clinical research. For each volunteer, we recorded demographics, number of screenings, enrollments, and data capture accuracy. Results: Over five 6month sessions, 151 student volunteers participated. Of these, 77.3% were female, 58.8% were undergraduate students, and 61.1% were interested in medical school. Student volunteers accurately screened 11,362/13,067 (87.0%) children, and they accurately identified 4407/ 4984 (88.4%) potentially eligible children for study enrollment. Of the 3805 eligible for enrollment exclusively by the students, 3228 (84.8%) families/children consented and completed all study procedures. Furthermore, students correctly entered 11,660/12,567 (92.8%) data points. Conclusions: Utilizing university student volunteers to facili-
tate research enrollment in the ED is effective and allows for the capture of a high percentage of potentially eligible patients into prospective clinical research studies. Ó 2015 Elsevier Inc. , Keywords—research; data collection; education; funding; volunteer; hospital volunteers; emergency care
INTRODUCTION Prospective research in medicine is very important as it holds the highest potential for changing practice. However, research that requires enrollment of patients from the emergency department (ED) is often difficult and expensive to conduct (1). Therefore, in the last several years there has been the development and implementation of student-based research assistant programs that provide daily on-site research presence in the ED for the purposes of screening and enrolling potentially eligible ED patients into a wide variety of studies (1–4). Students in these programs generally participate for 4-month blocks and obtain university course credit or pay for the experience. Research on these programs has demonstrated that methods used to collect research data were 97% complete and accurate, there were significant increases in publications at the ED with such a program, and there was high student participant satisfaction (1,2,5).
RECEIVED: 18 November 2013; FINAL SUBMISSION RECEIVED: 27 February 2014; ACCEPTED: 30 June 2014 19
20
P. E. Steadman et al.
However, there are some programs where university students participate as volunteers, and therefore are not paid nor receive academic credit. Currently, there is limited evidence on the success of volunteer-based ED research assistant programs. One study included 25 volunteer research assistants and found that participants were reliable and covered 95.6% of shifts (3). Although this work is encouraging, it included a small pool of volunteers for a short period, and this research did not address volunteer effectiveness with respect to the screening and capture of potentially eligible patients into prospective research studies. Further, this research was conducted in a general ED and therefore, volunteer performance in a children’s ED is also relatively unknown. The main objective of this research was to determine how often potentially eligible children were accurately identified by university student volunteer research assistants for the purposes of enrollment into prospective ED research. Secondary objectives included the frequency of successful enrollment by university student volunteer research assistants in studies exclusively administered by these students, and the accuracy of data capture.
each study and liaises between the students, investigators, and research coordinators, and the program director. Study Population
This was a prospective cross-sectional study of university student volunteer research assistants in a tertiary care pediatric ED. The study hospital’s ED treats approximately 65,000 patients per year, and functions as a clinical, teaching, and research hospital. This research was approved by the institution’s research ethics board.
Volunteer participants were recruited by advertising with the local university. Applicants were required to be currently enrolled in a university program, submit a curriculum vita, a transcript, a letter of intent, and a list of references. There were 42 positions available per term, and on average, 20 of the positions were filled by returning high-performing volunteers, and 20 by new volunteers. All participants attended two orientation sessions. The first is to orient the participants to the policies of becoming volunteers at the study institution, procedures and expectations of the program, privacy regulations, and ethical conduct of studies. On the second orientation day, the volunteers got briefed on the purpose and methods of all the studies currently enrolling patients in the ED, and there is an interactive workshop on how to critically think about chief complaints and how they may apply to a particular study. Finally, they also participated in two paired shifts in the ED over 2 weeks, where they learned about how to effectively screen and enroll from out-going student volunteers. The volunteers were all required to complete a minimum of 23 5-h shifts over a 6-month period. All shifts were ideally doublecovered and occurred daily from 8:00 a.m. to 1:00 p.m.; 1:00 p.m.-6:00 p.m.; and 6:00 p.m.–11:00 p.m. All volunteers who participated in this program consented to collection of demographics, performance metrics, participation in surveys during and after exiting the program, and to have their de-identified data used for research purposes. There were no exclusion criteria.
Program Description and Management
Research Studies and Database
Our ED student volunteer program is modeled after previously described programs (1,5,6). The program director is a pediatric emergency physician who is engaged in patient care, education, and patient outcomes research. The director reviews all faculty proposals for participation in the program to ensure that the studies are appropriate for student participation. A full-time program manager with experience as a research coordinator interviews and selects the best candidates for the program. She ensures that the successful students are credentialed, meet all hospital regulatory and security requirements, and have proper research ethics certification prior to starting the program. The manager tracks attendance and a detailed performance evaluation of each volunteer participant. She provides monthly feedback to all volunteers on their performance. Finally, she is trained to perform all the student-required duties for
Studies eligible for participation included all studies that had enrollment facilitated by the student volunteer research assistant program from March 2011 to July 2013. Study data were collected and managed using REDCap electronic data capture tools hosted at the study institution (7). For the purposes of this study, we recorded volunteer participant demographics, attendance, and performance for each study with respect to the following: identification of potentially eligible patients, enrollment into studies where applicable, success of consent where applicable, and accuracy of data capture for each patient screened or enrolled.
MATERIALS AND METHODS Study Design and Setting
Outcomes The main outcome was the proportion of potentially eligible children who were accurately identified by student volunteers for enrollment into prospective research
Effectiveness and Student Volunteer Research Assistant Program
in the ED. Accurately identified was defined as a student volunteer correctly screening (meets inclusion and no exclusion criteria) of children that were eligible for prospective ED studies and following through on studyspecific procedures that allowed families/children to be approached for consent to participate. Secondary outcomes also measured as proportions were: 1) patients who were accurately screened as potentially eligible/ excluded; 2) successfully enrolled patients exclusively by students in studies where this applies; and 3) proportion of shifts successfully attended by student volunteers. Analyses Based on recent data and previous research, we assumed that < 20% of patients will be inaccurately screened, and a sample size of approximately 12,000 screenings (8). This produced a two-sided 95% confidence interval (CI) with a width equal to approximately 6 1.5%, using an alpha of 0.05 and a beta of 0.8. The aggregate data from all student volunteers were used to address all outcomes. Descriptive statistics were used to summarize responses. Medians were reported with respective range (min, max), and proportions with respective 95% CIs. RESULTS From March 2011 through July 2013, student volunteers screened for 12 studies (Table 1) and exclusively completed enrollment for six. There were six prospective cohort studies, five cross-sectional studies, and one randomized control trial enrolling during the study period. The median duration that each study was live during this time period was 12 (min 7, max 38) months. Students exclusively screened, enrolled, and completed all data collection patients for six (50%) studies, whereas for the remaining studies, volunteers screened and identified eligible patients and contacted specific study staff. In addition to recording of patient data, one of the studies included specimen handling. There were a total of 151 university student volunteer participants for each of the five sessions in the study period. Detailed demographics on the participants are reported elsewhere; briefly, 77.3% of the participants were female, 58.8% were undergraduate students, and 61.1% were interested in medical school (9). Further, participants were also interested in graduate school (60.5%), allied health professions (21.5%), or research support jobs such as assistants/coordinators (65.7%). Over the entire study period, attendance with two volunteers per shift was 3853/4816 shifts (80.0%; 95% CI 78.9–81.1), and there was a minimum of one volunteer for 2475 of the 2646 shifts (93.5%; 95% CI 92.5–94.4). Session-specific attendance rates are detailed in Table 2.
21
Student volunteers accurately screened 11,362/13,067 (87.0%; 95% CI 86.4–87.6) children during their shifts as potentially eligible or excluded. Importantly, they accurately identified 4407/4984 (88.4%; 95% CI 87.5–89.3) potentially eligible children for study enrollment by study personnel/student volunteers (primary outcome). Of the 3805 eligible for enrollment exclusively by the student volunteers, 3228 (84.8%; 95% CI 83.7–85.9) families/ children consented and completed all study procedures. Furthermore, student volunteers entered 11,660/12,567 (92.8%; 95% CI 92.4–93.3) data points onto clinical research forms correctly. The mean number of screenings/study/month are reported in Table 3 and ranged from 12.3 to 262.1. DISCUSSION We demonstrated that the university student volunteers in our ED research assistant program were able to accurately identify 88% of patients that were eligible for enrollment into prospective studies. Further, these volunteer participants were able to successfully consent and enroll about 85% of families into the six studies where enrollment was done exclusively by the volunteers. Where volunteer consented and enrolled, almost 93% of the data fields were completed correctly. Thus, these data support that a student-volunteer-based research assistant program in a pediatric ED can be a highly effective mechanism for screening, enrollment, and data collection for prospective research. There have been few other studies of student-based ED research assistant programs that have reported on the accuracy of its participants with respect to screening and identifying eligible patients. One program based in Pennsylvania includes students that participate for course credit, and they reported about a 90% study subject identification and enrollment, and 10,351 of 10,635 (97%) data fields were properly completed (5). Another program hired paramedics and students for pay to identify/enroll patients into ED research daily for 14 h/day. Over 18 months they reported the completion of 17 prospective studies and enrolled 2175 patients (10). These data on program accuracy and productivity are comparable to the results we obtained with our student volunteers. It is important to note that similar to our quality-control methods, these programs provided close supervision and timely identification of errors by research coordinators or investigators, which further enhanced the integrity of data collection and management (5). Therefore, we believe that trained and supervised student volunteers function as accurately as students who participate in these programs for course credit or pay. Because our participants are student volunteers with little training on patient recruitment and are generally
22
Table 1. Research Studies Participating in Student Volunteer Emergency Department Research Assistant Program Study Name
Chief Complaint Eligible for Screening
Dates in program
Type of Study
Implementation of the Low Risk Ankle Rule Parental Perceptions of Risks from Diagnostic Imaging Body Mass Index and the Incidence of Fractures
Ankle injury
March 2011–September 2011
Cohort
Head injury
September 2011–August 2012
Cross-sectional
Extremity injury
March 2012–December 2012
Cross-sectional
Primary Care Physician Follow-up of Distal Radius Buckle Fractures
Wrist injury
September 2012–July 2013
Cohort
Parental Knowledge of Concussion
Head injury
October 2012– July 2013
Cross-sectional
Emergency Response Study
Resuscitation
January 2013–May 2013
Cross-sectional
Impact of Oximetry on Disposition in Bronchiolitis What happens to Oxygen Saturation after Discharge in Patients with Bronchiolitis Electrolyte Maintenance Solution versus Fluid as Tolerated Biomarkers of Traumatic Brain Injury Study
Respiratory distress
March 2011–May 2013
Cohort
Respiratory distress
March 2011–May 2013
Cohort
Vomiting/diarrhea
March 2011–July 2013
Randomized Control trial
Head injury
May 2011–June 2013
Cohort
Salter-Harris I of the Distal Fibula: Clinical Suspicion versus Reality Vascular Effects of Infection in Pediatric Stroke (control group)
Ankle injury
September 2012–July 2013
Cohort
Injury requiring intravenous sedation for repair
November 2013–May 2013
Cross-sectional
Student Volunteer Study-Specific Tasks Verbal consent for follow-up telephone call Full informed consent; completed a 10 -min parental survey Full informed consent; measured height; completed a 10-min parental survey; data collection sheet on current injury Verbal consent for follow-up telephone call; completed a 5-min parental survey; data collection sheet on current injury Full informed consent; completed a 10-min parental survey Completed data collection sheet on emergency responses to patients in the emergency department Screen patients and inform study staff of eligible patients Screen patients and inform study staff of eligible patients Screen patients and inform study staff of eligible patients Collect and process study blood specimens on major trauma patients with head injury; screen patient and inform study staff of eligible patient Screen patients and inform study staff of eligible patients Screen patients and inform study staff of eligible patients
P. E. Steadman et al.
Effectiveness and Student Volunteer Research Assistant Program
23
Table 2. Student Volunteer Attendance Mar 2011–Aug 2012 Sep 2011–Feb 2012 Mar 2012–Aug 2012 Sep 2012–Jan 2013 Feb 2013–July 2013 Attendance shifts double covered/shifts available*, n (%) Attendance shifts single covered/ shifts*, n (%)
821/1092 (75.2)
827/1092 (75.7)
832/1092 (76.2)
826/924 (89.4)
939/1092 (86.0)
512/546 (93.8)
489/546 (90.0)
484/546 (88.6)
455/462 (98.5)
535/546 (98.0)
* There were three shifts per: day 8:00 a.m.–1:00 p.m.; 1:00 p.m.–6:00 p.m.; 6:00–11:00 p.m. and there were two shifts available for each time block. A minimum target was to have one volunteer per shift, but ideally there were two student volunteers available per shift.
performance of our student volunteers in this area was as good as similar programs running as undergraduate courses, and comparable to that achieved by trained study coordinators. In the absence of course credit or financial reimbursement, participant commitment to scheduled shifts may be an issue. A study by Hollander et al. that included students who participated for university course credit found that there was 100% participant attendance, motivated primarily by the program experience and course credit (1). Cobaugh et al. demonstrated the research effectiveness of paid participants (10). One smaller study included 25 student research assistant volunteers for one 4-month
unfamiliar with the clinically stressful environment that the ED provides, approaching families for informed consent for research may be challenging. One study included student research assistants who participated for course credit in their pediatric ED and other clinical departments. They found that consent rates varied from 77% to 88% for various types of studies, and one of the investigators documented that students successfully consented 80% of eligible patients for a particular study, whereas certified research coordinators consented 83% (11). In our study period, student volunteers were required to perform informed consent in 50% of the studies, with successful consent in about 85% of cases. Thus, the Table 3. Student Volunteer Performance by Study Type
Student Volunteer Screens, Consents, and Completes Enrollment of Eligible Patients Data Fields Completed Mean Number of Accurate Screening Consented Eligible Correctly for Screening and Screenings/Month (SD) n/N (%) Successfully n/N (%) Enrollment n/N (%)
Study Name Implementation of the Low Risk Ankle Rule Parental Perceptions of Risks from Diagnostic Imaging Body Mass Index and the Incidence of Fractures Primary Care Physician Follow up of Distal Radius Buckle Fractures Parental Knowledge of Concussion Emergency Response Study
26.2 (3.9)
252/297 (84.8%)
161/196 (82.1%)
182/196 (92.9)
134.3 (3.7)
1528/1609 (95.0%)
742/987 (75.2%)
1464/1609 (91.0%)
262.1 (5.7)
2317/2604 (89.0%)
1570/1803 (87.1%)
2422/2604 (93.0%)
46.2 (3.2)
433/510 (84.9%)
105/110 (95.5%)
449/510 (88.0%)
198.8 (3.9) 48.7 (3.8)
1512/1582 (95.6%) 322/322 (100%)
328/387 (84.8%) 322/322 (100%)
1471/1582 (93.0%) 322/322 (100%)
Student Volunteer Screens and Contacts Study Staff for Potentially Eligible Patients
Study Name Impact of Oximetry on Disposition in Bronchiolitis What happens to Oxygen Saturation after Discharge in Patients with Bronchiolitis Electrolyte Maintenance Solution versus Fluid as Tolerated Biomarkers of Traumatic Brain Injury Study Salter-Harris I of the Distal Fibula: Clinical Suspicion versus Reality Vascular Effects of Infection in Pediatric Stroke
Mean Number of Screenings/Month (SD)
Accurate Screening n/N (%)
Contacted Study Staff for Potentially Eligible n/N (%)
75.1 (39.8)
573/716 (80%)
183/183 (100%)
569/643 (88.5%)
67.8 (37.3)
511/661 (77.3%)
186/186 (100%)
517/578 (89.5%)
145.5 (70.5)
1288/1574 (81.8%)
420/420 (100%)
1388/1507 (92.1%)
125.5 (48)
1331/1678 (79.3%)
146/146 (100%)
1609/1721 (93.5%)
12.3 (3.4)
306/306 (100%)
168/168 (100%)
306/306 (100%)
161.8 (35.3)
989/1208 (81.9%)
76/76 (100%)
961/989 (97.2%)
Data Fields Completed Correctly for Screening n/N (%)
24
term and they found that participants maintained a high rate of shift coverage, with 95.6% of shifts covered (3). Our study builds on the latter volunteer framework, including over 150 volunteers during a 30-month period and demonstrated that volunteers do consistently demonstrate a reliable research presence, despite the lack of university course credit or pay. In our ED research assistant program, about 94% of shifts were filled as scheduled. Over half of the participants voluntarily agreed to participate for at least two 6-month sessions, and about onethird did so for at least three sessions. These results are even better than those reported by one program, with an average attendance of 9/13 (69.2%) shifts over one semester that included 41 students who had the potential for using the experience for course credit (2). Thus, our results speak strongly that factors other than course credit or pay can be motivating to student participants, namely the clinical/research experience and potential gains for their career. Limitations The studies that our student volunteers administer completely are largely limited to low-risk patient outcome studies, and their role in higher-risk studies was limited to screening, identification of potentially eligible patients, and contacting study-specific staff. Therefore, we cannot comment on their effectiveness in having a larger role in higher-risk studies. Because the enrollers are present for 15 h per day from 8:00 a.m. to 11:00 p.m., enrollment of subjects into EDbased studies may represent a convenience sampling and their effectiveness during overnight hours remains relatively unknown. We did not report on changes in research productivity with respect to published manuscripts/abstracts for the physician members because this may be influenced by a number of variables that could confound our results. Further, our program has tracking of volunteer performance that is reported back to the volunteers monthly, and thus these results may not be generalizable to programs without this measure. This program has full-time managerial support and was conducted in a tertiary care pediatric ED with a strong commitment to research by its physicians and nurses, and therefore, our results may not be generalizable to EDs with less support in these areas. Finally, this research did not examine what specific student factors (e.g., long-term goals, economic status) result in high-performing volunteer research assistants in an ED. Although we do describe some of the motivators for participation and performance in a separate paper, specific research examining volunteer demographics and socioeconomic factors to program performance would be an interesting area for future study (9). The
P. E. Steadman et al.
latter information would allow program directors to facilitate selection of applicants best suited to this type of experience. CONCLUSIONS We demonstrated that university student volunteers with career goals that primarily include health care professions or graduate school can have a high degree of success in identifying potentially eligible patients and following through on study procedures that enroll patients into prospective studies in a busy and stressful clinical environment. Therefore, our student-volunteer model can serve as a feasible and potentially highly successful framework for clinician scientists in EDs and other specialties with moderate- to high-volume clinics to implement highquality research at a minimal cost and without the added complexities of administering a university course. Acknowledgment—The Department of Pediatrics, Hospital for Sick Children provides funding for a full-time manager for the program. We would like to acknowledge the efforts of the student volunteer participants of the Pediatric Research Academic Initiative at SickKids Emergency.
REFERENCES 1. Hollander JE. Academic associate program: integrating clinical emergency medicine research with undergraduate education. Acad Emerg Med 1997;4:225–30. 2. Bradley K, Osborn HH, Tang M. College research associates: a program to increase emergency medicine clinical research productivity. Ann Emerg Med 1996;28:328–33. 3. Hollander JE, Reinhard SS, Shofer F. Clinical research assistants in the emergency department: can volunteers be relied upon [Conference abstract]? Acad Emerg Med 1998;5:533. 4. Hollander JE, Reinhard SS, Eskin B, Shofer F, Singer AJ. The use of students to assist with clinical research: feasible in a variety of emergency department settings. Ann Emerg Med 1998;32:S43. 5. Hollander JE, Singer AJ. An innovative strategy for conducting clinical research: the academic associate program. Acad Emerg Med 2002;9:134–7. 6. Davis DP, Poste JC, Kelly D. The UCSD Research Associate Program: a recipe for successfully integrating undergraduates with emergency medicine research. J Emerg Med 2005;28:89–93. 7. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009;42:377–81. 8. Hollander JE, Sparano DM, Karounos M, Sites FD, Shofer FS. Studies in emergency department data collection: shared versus split responsibility for patient enrollment. Acad Emerg Med 2004;11:200–3. 9. Steadman PE, Crudden J, Naranian T, Oliveria JP, Boutis K. Professional benefits for volunteer research assistants in a pediatric emergency department. J Emerg Med 2014. XX: XX–XXcon. 10. Cobaugh DJ, Spillane LL, Schneider SM. Research subject enroller program: a key to successful emergency medicine research. Acad Emerg Med 1997;4:231–3. 11. Holsti M, Adelgais KM, Willis L, Jacobsen K, Clark EB, Byington CL. Developing future clinician scientists while supporting a research infrastructure. Clin Transl Sci 2013;6:94–7.
Effectiveness and Student Volunteer Research Assistant Program
ARTICLE SUMMARY 1. Why is this topic important? Research that requires enrollment of patients from the emergency department (ED) is often difficult and expensive to conduct. In the last several years, several EDs have successfully implemented student-based research assistant programs that provide on-site research presence 15 h per day for screening and enrollment of potentially eligible patients into research studies, and in exchange, the students receive university course credit or pay. 2. What does this study attempt to show? The main objective of this study was to determine how effective similar programs are if they employ university student volunteers, whereby they do not receive added incentive to participate or perform as research assistants. 3. What are the key findings? Student volunteers accurately identified 88.4% potentially eligible children for study enrollment. Further, the successfully consented and enrolled 84.8% of patients in studies where the study procedures were performed exclusively by volunteers. Finally, student volunteers correctly entered 92.8% data points into data collection records. 4. How is patient care impacted? Utilizing student volunteers to facilitate research enrollment in the ED is effective and allows for the capture of a high percentage of potentially eligible patients into prospective clinical research studies. The constant on-site presence of student volunteers permits research to be performed at a high-quality level, while minimizing the burden on clinical staff and costs/administrative burden that may be associated with a university course.
25
http://dx.doi.org/10.1016/j.jemermed.2014.06.039
Original Contributions
THE EFFECTIVENESS OF A STUDENT VOLUNTEER PROGRAM FOR RESEARCH IN A PEDIATRIC EMERGENCY DEPARTMENT Patrick E. Steadman, MSC,* Johanna Crudden, RT,† Taline Naranian, MSC,† John Paul Oliveria, BSC,†‡ and Kathy Boutis, MD*† *Faculty of Medicine, University of Toronto, Toronto, Ontario, Canada, †Division of Emergency Medicine, Department of Pediatrics, The Hospital for Sick Children, Toronto, Ontario, Canada, and ‡Department of Medicine, McMaster University, Hamilton, Ontario, Canada Reprint Address: Kathy Boutis, MD, The Hospital for Sick Children, 555 University Avenue, Toronto, ON M5G 1X8, Canada
, Abstract—Background: Emergency Department (ED) student-based research assistant programs have been shown to be effective in enrolling patients when the students receive university course credit or pay. However, the impact on research outcomes when university students act as volunteers in this role is relatively unknown. Objectives: The main objective of this study was to determine how often potentially eligible children were accurately identified by volunteer research assistants for enrollment into prospective research in the ED. We also examined the frequency of successful enrollments and the accuracy of data capture. Methods: This was a prospective crosssectional study of university student volunteer research assistant performance in a tertiary care pediatric ED between March 2011 and July 2013. The participant’s primary role was to screen and facilitate enrollment of ED patients into clinical research. For each volunteer, we recorded demographics, number of screenings, enrollments, and data capture accuracy. Results: Over five 6month sessions, 151 student volunteers participated. Of these, 77.3% were female, 58.8% were undergraduate students, and 61.1% were interested in medical school. Student volunteers accurately screened 11,362/13,067 (87.0%) children, and they accurately identified 4407/ 4984 (88.4%) potentially eligible children for study enrollment. Of the 3805 eligible for enrollment exclusively by the students, 3228 (84.8%) families/children consented and completed all study procedures. Furthermore, students correctly entered 11,660/12,567 (92.8%) data points. Conclusions: Utilizing university student volunteers to facili-
tate research enrollment in the ED is effective and allows for the capture of a high percentage of potentially eligible patients into prospective clinical research studies. Ó 2015 Elsevier Inc. , Keywords—research; data collection; education; funding; volunteer; hospital volunteers; emergency care
INTRODUCTION Prospective research in medicine is very important as it holds the highest potential for changing practice. However, research that requires enrollment of patients from the emergency department (ED) is often difficult and expensive to conduct (1). Therefore, in the last several years there has been the development and implementation of student-based research assistant programs that provide daily on-site research presence in the ED for the purposes of screening and enrolling potentially eligible ED patients into a wide variety of studies (1–4). Students in these programs generally participate for 4-month blocks and obtain university course credit or pay for the experience. Research on these programs has demonstrated that methods used to collect research data were 97% complete and accurate, there were significant increases in publications at the ED with such a program, and there was high student participant satisfaction (1,2,5).
RECEIVED: 18 November 2013; FINAL SUBMISSION RECEIVED: 27 February 2014; ACCEPTED: 30 June 2014 19
20
P. E. Steadman et al.
However, there are some programs where university students participate as volunteers, and therefore are not paid nor receive academic credit. Currently, there is limited evidence on the success of volunteer-based ED research assistant programs. One study included 25 volunteer research assistants and found that participants were reliable and covered 95.6% of shifts (3). Although this work is encouraging, it included a small pool of volunteers for a short period, and this research did not address volunteer effectiveness with respect to the screening and capture of potentially eligible patients into prospective research studies. Further, this research was conducted in a general ED and therefore, volunteer performance in a children’s ED is also relatively unknown. The main objective of this research was to determine how often potentially eligible children were accurately identified by university student volunteer research assistants for the purposes of enrollment into prospective ED research. Secondary objectives included the frequency of successful enrollment by university student volunteer research assistants in studies exclusively administered by these students, and the accuracy of data capture.
each study and liaises between the students, investigators, and research coordinators, and the program director. Study Population
This was a prospective cross-sectional study of university student volunteer research assistants in a tertiary care pediatric ED. The study hospital’s ED treats approximately 65,000 patients per year, and functions as a clinical, teaching, and research hospital. This research was approved by the institution’s research ethics board.
Volunteer participants were recruited by advertising with the local university. Applicants were required to be currently enrolled in a university program, submit a curriculum vita, a transcript, a letter of intent, and a list of references. There were 42 positions available per term, and on average, 20 of the positions were filled by returning high-performing volunteers, and 20 by new volunteers. All participants attended two orientation sessions. The first is to orient the participants to the policies of becoming volunteers at the study institution, procedures and expectations of the program, privacy regulations, and ethical conduct of studies. On the second orientation day, the volunteers got briefed on the purpose and methods of all the studies currently enrolling patients in the ED, and there is an interactive workshop on how to critically think about chief complaints and how they may apply to a particular study. Finally, they also participated in two paired shifts in the ED over 2 weeks, where they learned about how to effectively screen and enroll from out-going student volunteers. The volunteers were all required to complete a minimum of 23 5-h shifts over a 6-month period. All shifts were ideally doublecovered and occurred daily from 8:00 a.m. to 1:00 p.m.; 1:00 p.m.-6:00 p.m.; and 6:00 p.m.–11:00 p.m. All volunteers who participated in this program consented to collection of demographics, performance metrics, participation in surveys during and after exiting the program, and to have their de-identified data used for research purposes. There were no exclusion criteria.
Program Description and Management
Research Studies and Database
Our ED student volunteer program is modeled after previously described programs (1,5,6). The program director is a pediatric emergency physician who is engaged in patient care, education, and patient outcomes research. The director reviews all faculty proposals for participation in the program to ensure that the studies are appropriate for student participation. A full-time program manager with experience as a research coordinator interviews and selects the best candidates for the program. She ensures that the successful students are credentialed, meet all hospital regulatory and security requirements, and have proper research ethics certification prior to starting the program. The manager tracks attendance and a detailed performance evaluation of each volunteer participant. She provides monthly feedback to all volunteers on their performance. Finally, she is trained to perform all the student-required duties for
Studies eligible for participation included all studies that had enrollment facilitated by the student volunteer research assistant program from March 2011 to July 2013. Study data were collected and managed using REDCap electronic data capture tools hosted at the study institution (7). For the purposes of this study, we recorded volunteer participant demographics, attendance, and performance for each study with respect to the following: identification of potentially eligible patients, enrollment into studies where applicable, success of consent where applicable, and accuracy of data capture for each patient screened or enrolled.
MATERIALS AND METHODS Study Design and Setting
Outcomes The main outcome was the proportion of potentially eligible children who were accurately identified by student volunteers for enrollment into prospective research
Effectiveness and Student Volunteer Research Assistant Program
in the ED. Accurately identified was defined as a student volunteer correctly screening (meets inclusion and no exclusion criteria) of children that were eligible for prospective ED studies and following through on studyspecific procedures that allowed families/children to be approached for consent to participate. Secondary outcomes also measured as proportions were: 1) patients who were accurately screened as potentially eligible/ excluded; 2) successfully enrolled patients exclusively by students in studies where this applies; and 3) proportion of shifts successfully attended by student volunteers. Analyses Based on recent data and previous research, we assumed that < 20% of patients will be inaccurately screened, and a sample size of approximately 12,000 screenings (8). This produced a two-sided 95% confidence interval (CI) with a width equal to approximately 6 1.5%, using an alpha of 0.05 and a beta of 0.8. The aggregate data from all student volunteers were used to address all outcomes. Descriptive statistics were used to summarize responses. Medians were reported with respective range (min, max), and proportions with respective 95% CIs. RESULTS From March 2011 through July 2013, student volunteers screened for 12 studies (Table 1) and exclusively completed enrollment for six. There were six prospective cohort studies, five cross-sectional studies, and one randomized control trial enrolling during the study period. The median duration that each study was live during this time period was 12 (min 7, max 38) months. Students exclusively screened, enrolled, and completed all data collection patients for six (50%) studies, whereas for the remaining studies, volunteers screened and identified eligible patients and contacted specific study staff. In addition to recording of patient data, one of the studies included specimen handling. There were a total of 151 university student volunteer participants for each of the five sessions in the study period. Detailed demographics on the participants are reported elsewhere; briefly, 77.3% of the participants were female, 58.8% were undergraduate students, and 61.1% were interested in medical school (9). Further, participants were also interested in graduate school (60.5%), allied health professions (21.5%), or research support jobs such as assistants/coordinators (65.7%). Over the entire study period, attendance with two volunteers per shift was 3853/4816 shifts (80.0%; 95% CI 78.9–81.1), and there was a minimum of one volunteer for 2475 of the 2646 shifts (93.5%; 95% CI 92.5–94.4). Session-specific attendance rates are detailed in Table 2.
21
Student volunteers accurately screened 11,362/13,067 (87.0%; 95% CI 86.4–87.6) children during their shifts as potentially eligible or excluded. Importantly, they accurately identified 4407/4984 (88.4%; 95% CI 87.5–89.3) potentially eligible children for study enrollment by study personnel/student volunteers (primary outcome). Of the 3805 eligible for enrollment exclusively by the student volunteers, 3228 (84.8%; 95% CI 83.7–85.9) families/ children consented and completed all study procedures. Furthermore, student volunteers entered 11,660/12,567 (92.8%; 95% CI 92.4–93.3) data points onto clinical research forms correctly. The mean number of screenings/study/month are reported in Table 3 and ranged from 12.3 to 262.1. DISCUSSION We demonstrated that the university student volunteers in our ED research assistant program were able to accurately identify 88% of patients that were eligible for enrollment into prospective studies. Further, these volunteer participants were able to successfully consent and enroll about 85% of families into the six studies where enrollment was done exclusively by the volunteers. Where volunteer consented and enrolled, almost 93% of the data fields were completed correctly. Thus, these data support that a student-volunteer-based research assistant program in a pediatric ED can be a highly effective mechanism for screening, enrollment, and data collection for prospective research. There have been few other studies of student-based ED research assistant programs that have reported on the accuracy of its participants with respect to screening and identifying eligible patients. One program based in Pennsylvania includes students that participate for course credit, and they reported about a 90% study subject identification and enrollment, and 10,351 of 10,635 (97%) data fields were properly completed (5). Another program hired paramedics and students for pay to identify/enroll patients into ED research daily for 14 h/day. Over 18 months they reported the completion of 17 prospective studies and enrolled 2175 patients (10). These data on program accuracy and productivity are comparable to the results we obtained with our student volunteers. It is important to note that similar to our quality-control methods, these programs provided close supervision and timely identification of errors by research coordinators or investigators, which further enhanced the integrity of data collection and management (5). Therefore, we believe that trained and supervised student volunteers function as accurately as students who participate in these programs for course credit or pay. Because our participants are student volunteers with little training on patient recruitment and are generally
22
Table 1. Research Studies Participating in Student Volunteer Emergency Department Research Assistant Program Study Name
Chief Complaint Eligible for Screening
Dates in program
Type of Study
Implementation of the Low Risk Ankle Rule Parental Perceptions of Risks from Diagnostic Imaging Body Mass Index and the Incidence of Fractures
Ankle injury
March 2011–September 2011
Cohort
Head injury
September 2011–August 2012
Cross-sectional
Extremity injury
March 2012–December 2012
Cross-sectional
Primary Care Physician Follow-up of Distal Radius Buckle Fractures
Wrist injury
September 2012–July 2013
Cohort
Parental Knowledge of Concussion
Head injury
October 2012– July 2013
Cross-sectional
Emergency Response Study
Resuscitation
January 2013–May 2013
Cross-sectional
Impact of Oximetry on Disposition in Bronchiolitis What happens to Oxygen Saturation after Discharge in Patients with Bronchiolitis Electrolyte Maintenance Solution versus Fluid as Tolerated Biomarkers of Traumatic Brain Injury Study
Respiratory distress
March 2011–May 2013
Cohort
Respiratory distress
March 2011–May 2013
Cohort
Vomiting/diarrhea
March 2011–July 2013
Randomized Control trial
Head injury
May 2011–June 2013
Cohort
Salter-Harris I of the Distal Fibula: Clinical Suspicion versus Reality Vascular Effects of Infection in Pediatric Stroke (control group)
Ankle injury
September 2012–July 2013
Cohort
Injury requiring intravenous sedation for repair
November 2013–May 2013
Cross-sectional
Student Volunteer Study-Specific Tasks Verbal consent for follow-up telephone call Full informed consent; completed a 10 -min parental survey Full informed consent; measured height; completed a 10-min parental survey; data collection sheet on current injury Verbal consent for follow-up telephone call; completed a 5-min parental survey; data collection sheet on current injury Full informed consent; completed a 10-min parental survey Completed data collection sheet on emergency responses to patients in the emergency department Screen patients and inform study staff of eligible patients Screen patients and inform study staff of eligible patients Screen patients and inform study staff of eligible patients Collect and process study blood specimens on major trauma patients with head injury; screen patient and inform study staff of eligible patient Screen patients and inform study staff of eligible patients Screen patients and inform study staff of eligible patients
P. E. Steadman et al.
Effectiveness and Student Volunteer Research Assistant Program
23
Table 2. Student Volunteer Attendance Mar 2011–Aug 2012 Sep 2011–Feb 2012 Mar 2012–Aug 2012 Sep 2012–Jan 2013 Feb 2013–July 2013 Attendance shifts double covered/shifts available*, n (%) Attendance shifts single covered/ shifts*, n (%)
821/1092 (75.2)
827/1092 (75.7)
832/1092 (76.2)
826/924 (89.4)
939/1092 (86.0)
512/546 (93.8)
489/546 (90.0)
484/546 (88.6)
455/462 (98.5)
535/546 (98.0)
* There were three shifts per: day 8:00 a.m.–1:00 p.m.; 1:00 p.m.–6:00 p.m.; 6:00–11:00 p.m. and there were two shifts available for each time block. A minimum target was to have one volunteer per shift, but ideally there were two student volunteers available per shift.
performance of our student volunteers in this area was as good as similar programs running as undergraduate courses, and comparable to that achieved by trained study coordinators. In the absence of course credit or financial reimbursement, participant commitment to scheduled shifts may be an issue. A study by Hollander et al. that included students who participated for university course credit found that there was 100% participant attendance, motivated primarily by the program experience and course credit (1). Cobaugh et al. demonstrated the research effectiveness of paid participants (10). One smaller study included 25 student research assistant volunteers for one 4-month
unfamiliar with the clinically stressful environment that the ED provides, approaching families for informed consent for research may be challenging. One study included student research assistants who participated for course credit in their pediatric ED and other clinical departments. They found that consent rates varied from 77% to 88% for various types of studies, and one of the investigators documented that students successfully consented 80% of eligible patients for a particular study, whereas certified research coordinators consented 83% (11). In our study period, student volunteers were required to perform informed consent in 50% of the studies, with successful consent in about 85% of cases. Thus, the Table 3. Student Volunteer Performance by Study Type
Student Volunteer Screens, Consents, and Completes Enrollment of Eligible Patients Data Fields Completed Mean Number of Accurate Screening Consented Eligible Correctly for Screening and Screenings/Month (SD) n/N (%) Successfully n/N (%) Enrollment n/N (%)
Study Name Implementation of the Low Risk Ankle Rule Parental Perceptions of Risks from Diagnostic Imaging Body Mass Index and the Incidence of Fractures Primary Care Physician Follow up of Distal Radius Buckle Fractures Parental Knowledge of Concussion Emergency Response Study
26.2 (3.9)
252/297 (84.8%)
161/196 (82.1%)
182/196 (92.9)
134.3 (3.7)
1528/1609 (95.0%)
742/987 (75.2%)
1464/1609 (91.0%)
262.1 (5.7)
2317/2604 (89.0%)
1570/1803 (87.1%)
2422/2604 (93.0%)
46.2 (3.2)
433/510 (84.9%)
105/110 (95.5%)
449/510 (88.0%)
198.8 (3.9) 48.7 (3.8)
1512/1582 (95.6%) 322/322 (100%)
328/387 (84.8%) 322/322 (100%)
1471/1582 (93.0%) 322/322 (100%)
Student Volunteer Screens and Contacts Study Staff for Potentially Eligible Patients
Study Name Impact of Oximetry on Disposition in Bronchiolitis What happens to Oxygen Saturation after Discharge in Patients with Bronchiolitis Electrolyte Maintenance Solution versus Fluid as Tolerated Biomarkers of Traumatic Brain Injury Study Salter-Harris I of the Distal Fibula: Clinical Suspicion versus Reality Vascular Effects of Infection in Pediatric Stroke
Mean Number of Screenings/Month (SD)
Accurate Screening n/N (%)
Contacted Study Staff for Potentially Eligible n/N (%)
75.1 (39.8)
573/716 (80%)
183/183 (100%)
569/643 (88.5%)
67.8 (37.3)
511/661 (77.3%)
186/186 (100%)
517/578 (89.5%)
145.5 (70.5)
1288/1574 (81.8%)
420/420 (100%)
1388/1507 (92.1%)
125.5 (48)
1331/1678 (79.3%)
146/146 (100%)
1609/1721 (93.5%)
12.3 (3.4)
306/306 (100%)
168/168 (100%)
306/306 (100%)
161.8 (35.3)
989/1208 (81.9%)
76/76 (100%)
961/989 (97.2%)
Data Fields Completed Correctly for Screening n/N (%)
24
term and they found that participants maintained a high rate of shift coverage, with 95.6% of shifts covered (3). Our study builds on the latter volunteer framework, including over 150 volunteers during a 30-month period and demonstrated that volunteers do consistently demonstrate a reliable research presence, despite the lack of university course credit or pay. In our ED research assistant program, about 94% of shifts were filled as scheduled. Over half of the participants voluntarily agreed to participate for at least two 6-month sessions, and about onethird did so for at least three sessions. These results are even better than those reported by one program, with an average attendance of 9/13 (69.2%) shifts over one semester that included 41 students who had the potential for using the experience for course credit (2). Thus, our results speak strongly that factors other than course credit or pay can be motivating to student participants, namely the clinical/research experience and potential gains for their career. Limitations The studies that our student volunteers administer completely are largely limited to low-risk patient outcome studies, and their role in higher-risk studies was limited to screening, identification of potentially eligible patients, and contacting study-specific staff. Therefore, we cannot comment on their effectiveness in having a larger role in higher-risk studies. Because the enrollers are present for 15 h per day from 8:00 a.m. to 11:00 p.m., enrollment of subjects into EDbased studies may represent a convenience sampling and their effectiveness during overnight hours remains relatively unknown. We did not report on changes in research productivity with respect to published manuscripts/abstracts for the physician members because this may be influenced by a number of variables that could confound our results. Further, our program has tracking of volunteer performance that is reported back to the volunteers monthly, and thus these results may not be generalizable to programs without this measure. This program has full-time managerial support and was conducted in a tertiary care pediatric ED with a strong commitment to research by its physicians and nurses, and therefore, our results may not be generalizable to EDs with less support in these areas. Finally, this research did not examine what specific student factors (e.g., long-term goals, economic status) result in high-performing volunteer research assistants in an ED. Although we do describe some of the motivators for participation and performance in a separate paper, specific research examining volunteer demographics and socioeconomic factors to program performance would be an interesting area for future study (9). The
P. E. Steadman et al.
latter information would allow program directors to facilitate selection of applicants best suited to this type of experience. CONCLUSIONS We demonstrated that university student volunteers with career goals that primarily include health care professions or graduate school can have a high degree of success in identifying potentially eligible patients and following through on study procedures that enroll patients into prospective studies in a busy and stressful clinical environment. Therefore, our student-volunteer model can serve as a feasible and potentially highly successful framework for clinician scientists in EDs and other specialties with moderate- to high-volume clinics to implement highquality research at a minimal cost and without the added complexities of administering a university course. Acknowledgment—The Department of Pediatrics, Hospital for Sick Children provides funding for a full-time manager for the program. We would like to acknowledge the efforts of the student volunteer participants of the Pediatric Research Academic Initiative at SickKids Emergency.
REFERENCES 1. Hollander JE. Academic associate program: integrating clinical emergency medicine research with undergraduate education. Acad Emerg Med 1997;4:225–30. 2. Bradley K, Osborn HH, Tang M. College research associates: a program to increase emergency medicine clinical research productivity. Ann Emerg Med 1996;28:328–33. 3. Hollander JE, Reinhard SS, Shofer F. Clinical research assistants in the emergency department: can volunteers be relied upon [Conference abstract]? Acad Emerg Med 1998;5:533. 4. Hollander JE, Reinhard SS, Eskin B, Shofer F, Singer AJ. The use of students to assist with clinical research: feasible in a variety of emergency department settings. Ann Emerg Med 1998;32:S43. 5. Hollander JE, Singer AJ. An innovative strategy for conducting clinical research: the academic associate program. Acad Emerg Med 2002;9:134–7. 6. Davis DP, Poste JC, Kelly D. The UCSD Research Associate Program: a recipe for successfully integrating undergraduates with emergency medicine research. J Emerg Med 2005;28:89–93. 7. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)—a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform 2009;42:377–81. 8. Hollander JE, Sparano DM, Karounos M, Sites FD, Shofer FS. Studies in emergency department data collection: shared versus split responsibility for patient enrollment. Acad Emerg Med 2004;11:200–3. 9. Steadman PE, Crudden J, Naranian T, Oliveria JP, Boutis K. Professional benefits for volunteer research assistants in a pediatric emergency department. J Emerg Med 2014. XX: XX–XXcon. 10. Cobaugh DJ, Spillane LL, Schneider SM. Research subject enroller program: a key to successful emergency medicine research. Acad Emerg Med 1997;4:231–3. 11. Holsti M, Adelgais KM, Willis L, Jacobsen K, Clark EB, Byington CL. Developing future clinician scientists while supporting a research infrastructure. Clin Transl Sci 2013;6:94–7.
Effectiveness and Student Volunteer Research Assistant Program
ARTICLE SUMMARY 1. Why is this topic important? Research that requires enrollment of patients from the emergency department (ED) is often difficult and expensive to conduct. In the last several years, several EDs have successfully implemented student-based research assistant programs that provide on-site research presence 15 h per day for screening and enrollment of potentially eligible patients into research studies, and in exchange, the students receive university course credit or pay. 2. What does this study attempt to show? The main objective of this study was to determine how effective similar programs are if they employ university student volunteers, whereby they do not receive added incentive to participate or perform as research assistants. 3. What are the key findings? Student volunteers accurately identified 88.4% potentially eligible children for study enrollment. Further, the successfully consented and enrolled 84.8% of patients in studies where the study procedures were performed exclusively by volunteers. Finally, student volunteers correctly entered 92.8% data points into data collection records. 4. How is patient care impacted? Utilizing student volunteers to facilitate research enrollment in the ED is effective and allows for the capture of a high percentage of potentially eligible patients into prospective clinical research studies. The constant on-site presence of student volunteers permits research to be performed at a high-quality level, while minimizing the burden on clinical staff and costs/administrative burden that may be associated with a university course.
25
