Quality Metrics in Neonatal and Pediatric Critical Care Transport: A National Delphi Project* Hamilton P. Schwartz, MD1; Michael T. Bigham, MD2; Pamela J. Schoettker, MS3; Keith Meyer, MD4; Michael S. Trautman, MD5; Robert M. Insoft, MD6; on behalf of the American Academy of Pediatrics Section on Transport Medicine
Objectives: The transport of neonatal and pediatric patients to tertiary care facilities for specialized care demands monitoring the quality of care delivered during transport and its impact on patient outcomes. In 2011, pediatric transport teams in Ohio met to identify quality indicators permitting comparisons among programs. However, no set of national consensus quality metrics exists for benchmarking transport teams. The aim of this project was to achieve national consensus on appropriate neonatal and pediatric transport quality metrics. Design: Modified Delphi technique. Setting: The first round of consensus determination was via electronic mail survey, followed by rounds of consensus determination in-person at the American Academy of Pediatrics Section on Transport Medicine’s 2012 Quality Metrics Summit. *See also p. 775. 1 Division of Emergency Medicine, Department of Pediatrics, Cincinnati Children’s Hospital, University of Cincinnati School of Medicine, Cincinnati, OH. 2 Division of Critical Care Medicine, Department of Pediatrics, Akron Children’s Hospital, Northeast Ohio Medical University, Akron, OH. 3 James M. Anderson Center for Health Systems Excellence, Cincinnati Children’s Hospital, Cincinnati, OH. 4 Division of Critical Care Medicine, Department of Pediatrics, Miami Children’s Hospital, Miami, FL. 5 Division of Neonatology, Department of Pediatrics, James W. Riley Hospital for Children, Indiana University School of Medicine, Indianapolis, IN. 6 Department of Pediatrics, Brown University Alpert School of Medicine, Providence, RI. Dr. Schoettker received support for manuscript writing/review; received provision of writing assistance, medicines, equipment, or administrative support; and received support for manuscript preparation (Dr. Schoettker is employed full-time by the Cincinnati Children's Hospital Medical Center as a medical writer. She works with many clinicians to help them publish their research). Dr. Trautman received support for travel from the American Academy of Pediatrics (lectures not specifically related to this). The remaining authors have disclosed that they do not have any potential conflicts of interest. Address requests for reprints to: Hamilton P. Schwartz, MD, Division of Emergency Medicine, Department of Pediatrics, Cincinnati Children’s Hospital, 3333 Burnet Avenue, Cincinnati, OH 45229. E-mail: hamilton. [email protected] Copyright © 2015 by the Society of Critical Care Medicine and the World Federation of Pediatric Intensive and Critical Care Societies DOI: 10.1097/PCC.0000000000000477
Pediatric Critical Care Medicine
Subjects: All attendees of the American Academy of Pediatrics Section on Transport Medicine Quality Metrics Summit, conducted on October 21–23, 2012, in New Orleans, LA, were eligible to participate. Measurements and Main Results: Candidate quality metrics were identified through literature review and those metrics currently tracked by participating programs. Participants were asked in a series of rounds to identify “very important” quality metrics for transport. It was determined a priori that consensus on a metric’s importance was achieved when at least 70% of respondents were in agreement. This is consistent with other Delphi studies. Eighty-two candidate metrics were considered initially. Ultimately, 12 metrics achieved consensus as “very important” to transport. These include metrics related to airway management, team mobilization time, patient and crew injuries, and adverse patient care events. Definitions were assigned to the 12 metrics to facilitate uniform data tracking among programs. Conclusions: The authors succeeded in achieving consensus among a diverse group of national transport experts on 12 core neonatal and pediatric transport quality metrics. We propose that transport teams across the country use these metrics to benchmark and guide their quality improvement activities. (Pediatr Crit Care Med 2015; 16:711–717) Key Words: outcome measures; pediatrics; process measures; quality improvement; safety; transportation of patients
E
ach year in the United States, more than 200,000 newborns, infants, and children are transported from local hospitals to regional hospitals for inpatient specialty care (1). Despite the increasing national focus on improving the quality of healthcare delivery, tremendous practice variation remains in neonatal and pediatric transport. In response, national organizations, such as the American Academy of Pediatrics (AAP), the Commission on Accreditation of Medical Transport Services, and the Air Medical Physician Association, have published operational and clinical care recommendations (2–4). However, there are few established performance metrics that can be used to benchmark neonatal and pediatric transport organizations to identify best practices for transport operations and clinical care. www.pccmjournal.org
711
Copyright © 2015 by the Society of Critical Care Medicine and the World Federation of Pediatric Intensive and Critical Care Societies. Unauthorized reproduction of this article is prohibited
Schwartz et al
Other organizations and specialties have developed pediatric and neonatal quality metrics, benchmarks, and best practices. Participants in the Pediatric Data Quality Systems Collaborative have developed two consensus measure sets and an assessment of nursing-sensitive indicators in pediatric care (5). In 2011, the Pediatric Emergency Care Applied Research Network published an analysis of 405 performance metrics for pediatric emergency care identified through literature review, surveys of emergency department directors, and consensus methodology (6). The Vermont Oxford Network is a steward for National Quality Forum (NQF)-endorsed performance metrics related to the care of premature infants and very low birth weight neonates (7). The ImproveCareNow pediatric inflammatory bowel disease quality improvement collaborative developed a list of 19 measures through literature review, expert consensus, and discussion, refinement, and agreement by participating collaborative sites (8). We recently published the results of a consensus project involving six neonatal and pediatric transport programs across Ohio that identified and defined critical care transport quality metrics (9). Candidate quality metrics were identified through literature review and current tracking by the participants. Consensus was obtained using the nominal group technique (10–12). Final metrics were categorized according to Institute of Medicine (IOM)-recommended quality care domains (safe, timely, effective, equitable, efficient, and patient-centered) and Donabedian’s quality measure framework (structure, process, and outcome) (13, 14). Although we feel the list of quality metrics established for the state of Ohio is comprehensive and relevant to other transport programs, others have suggested that there may be regional biases that limit its generalizability elsewhere in country. We sought to identify a generalizable core set of neonatal and pediatric transport quality metrics. These recommended quality metrics were derived using the modified Delphi technique, with participants representing various regions and program types from across the United States.
MATERIALS AND METHODS Participants and Setting All attendees at the AAP Section on Transport Medicine (SOTM) Quality Metrics Summit, conducted on October 21–23, 2012, in New Orleans, LA, were eligible to participate. Participants were initially identified from the preregistration list for the above Summit and through Summit advertisements posted on the AAP Transport Listserv. Candidate Quality Metrics Candidate transport quality metrics were identified through four sources: the English-language literature, national transport organizations’ recommendations, metrics already tracked by wellestablished transport programs, and experts in neonatal and pediatric transport medicine (9). An initial list of candidate transport quality metrics was reduced by applying the NQF’s evaluation criteria of metric importance, usability, validity, and feasibility (15). The 70 metrics comprising this list were then grouped according to the six IOM-recommended quality care domains (14). 712
www.pccmjournal.org
Modified Delphi Technique The authors chose the Delphi methodology to reach consensus on the metrics. The Delphi technique has been a popular method for developing indicators reflecting patient and general practitioner perspectives of chronic illness, performance indicators for emergency medicine, and indicators for cardiovascular disease (6, 8, 16–19). Delphi methodology succeeds in converging agreement, not in ensuring the validity of the results. The validity of the results is dependent on the expertise and characteristics of the participants. Delphi technique is popular because large numbers of individuals across diverse locations and areas of expertise can be included anonymously, thus avoiding domination of the consensus process by one or a few experts. The Classical Delphi process involves two or more rounds of questionnaires administered to participants by mail. The first questionnaire is heavily open-ended questions, with the purpose of generating the opinions that will be sent back to participants in subsequent rounds for consideration. Participant “experts” rank or rate the responses of others, which inform development of the next survey. Surveys continue until a predetermined level of consensus, conventionally 70%, is achieved on a set of items (20). Variation to this classical process, while still maintaining the core Delphi principles, allows customization to the individual project. The more common variations are e-Delphi (Delphi surveys done completely via e-mail), real-time Delphi (experts are in the same room with consensus achieved in real time), and modified Delphi (first survey of open-ended questions is augmented or replaced by existing opinions in the literature). This project incorporated aspects of all three of the above variations. Round 1. Two weeks before the start of the Neonatal and Pediatric Transport Medicine Quality Metrics Summit, anticipated participants were contacted by e-mail and invited to complete an online SurveyMonkey survey (SurveyMonkey Inc., Palo Alto, CA; http://www.surveymonkey.com). In addition to demographic questions, participants were asked to rate the importance of 70 candidate metrics on a five-point Likert-type scale, ranging from 1 = not important to 5 = very important. Metrics achieving a score of 5 by at least 70% of participants were definitively accepted. Participants were also asked to contribute suggestions for other candidate metrics. An e-mail reminding participants to complete the survey was sent 1 week prior to the start of the consensus conference. Round 2. During the Transport Medicine Quality Metrics Summit, two neonatologists, two pediatric critical care physicians, and a pediatric emergency medicine physician led an expert panel discussion of those candidate quality metrics that had achieved a rating of 5, “very important,” by more than half of respondents during round 1, but had not reached the 70% agreement cutoff noted in round 1. Additionally, the expert panelists shared their opinions of the new metrics contributed by the participants on the first survey. The participants were then asked to categorize, using an anonymous electronic audience participation system (Turning Technologies, Youngstown, OH), how important it was to include each candidate metric in a minimum standard, core list of metrics that could be used for benchmarking programs’ performances. Participants again used October 2015 • Volume 16 • Number 8
Copyright © 2015 by the Society of Critical Care Medicine and the World Federation of Pediatric Intensive and Critical Care Societies. Unauthorized reproduction of this article is prohibited
Feature Articles
Table 1. Participant Characteristics Including Transport Team Program Types Characteristic
Round 1
Round 2
Table 1. (Continued). Participant Characteristics Including Transport Team Program Types Characteristic
Current/most recent role, % (n)
Round 1
Round 2
Clinical manager/director
21.7 (18)
23.9 (27)
Types of patients, % (n)
Patient care provider
34.9 (29)
36.3 (41)
Strictly pediatric
9.6 (8)
8.9 (10)
Medical director
39.8 (33)
35.4 (40)
Strictly neonatal
22.9 (19)
23.9 (27)
3.6 (3)
4.4 (5)
Both
67.5 (56)
61.1 (69)
Other
0 (0)
6.2 (7)
Other nonclinical personnel Healthcare discipline, % (n) Physician
51.8 (43)
48.7 (56)
50 (21)
42.9 (24)
Pediatric critical care
33.3 (14)
32.1 (18)
Pediatric emergency medicine
14.3 (6)
17.9 (10)
Neonatology
General emergency medicine
0 (0)
0 (0)
Hospital medicine
0 (0)
1.8 (1)
General pediatrics
2.4 (1)
1.8 (1)
Other
2.4 (1)
3.6 (2)
28.9 (24)
33.0 (38)
Respiratory therapist
4.8 (4)
3.5 (4)
Emergency medical technician/paramedic
1.2 (1)
2.6 (3)
Nurse practitioner
10.8 (9)
10.4 (12)
Physician assistant
2.4 (2)
1.7 (2)
United States
93.9 (78)
93.9 (107)
Northeast
16.2 (12)
22.0 (24)
Midwest
23.0 (17)
27.5 (30)
South
47.3 (35)
38.5 (42)
Nurse
Location, % (n)
West
13.5 (10)
11.9 (13)
Canada
2.4 (2)
2.6 (3)
Europe
0 (0)
1.8 (2)
Asia
0 (0)
0.9 (1)
1.2 (1)
0.9 (1)
37.4 (31)
46.9 (53)
2.4 (2)
1.8 (2)
60.2 (50)
51.3 (58)
Stand-alone/dedicated team
59.8 (49)
62.0 (70)
Unit-based team
24.4 (20)
23.0 (26)
Combination of both
13.4 (11)
13.3 (15)
2.4 (2)
1.8 (2)
South Pacific Primary modes of transport, % (n) > 95% ground transport > 95% helicopter/airplane transport Both ground and air transport Program type, % (n)
Other
(Continued)
Pediatric Critical Care Medicine
the aforementioned five-point L ikert-type scale. A weighted rank was then assigned in order to prioritize the importance of each final consensus metric. Weighted rank was calculated for each metric achieving consensus, by averaging its round 2 scores on the five-point scale. Metric definitions were drafted if they had not been previously established in the literature. The Institutional Review Board at Akron Children’s Hospital approved this project. As a part of the online round 1 survey, participants were asked to confirm their understanding of the project and agree to participate. Participants were again asked to agree to participate before beginning the real-time round 2 survey. Responses to the surveys were reported in aggregate, so participants’ identities were not linked to their individual responses.
RESULTS Characteristics of Study Participants Eighty-three respondents completed both rounds of surveys. Thirty additional people participated in round 2 only. There were similar participant demographics and proportions of clinical managers/directors, patient care providers, and medical directors in rounds 1 and 2 (Table 1). The healthcare providers were mostly physicians and nurses, and the largest group of physicians was neonatologists followed by pediatric critical care physicians. Ninety-four percent of the participants at the consensus conference were from the United States, with similar representation from all areas of the country. Approximately 50% of the participants worked with transport services that provided both ground and air transportation. Sixty-two percent of participants worked for dedicated teams whose members’ primary job responsibilities are critical care transport, not inpatient care. Sixty-one percent of teams served both neonatal and pediatric patients. Main Results Round 1 Metrics. Ten of the original 70 candidate metrics were rated as “very important” by at least 70% of the round 1 participants. Thirty additional candidate metrics were rated as “very important” by at least 50% of round 1 participants (Table 2). Most of the metrics rated as “very important” belonged to the IOM safety domain. An additional 12 candidate metrics were suggested by round 1 participants to be included in round 2. Round 2 and Final Metrics. Twelve metrics met at least 70% consensus as “very important” during the round 2 survey. www.pccmjournal.org
713
Copyright © 2015 by the Society of Critical Care Medicine and the World Federation of Pediatric Intensive and Critical Care Societies. Unauthorized reproduction of this article is prohibited
Schwartz et al
Table 2. Metrics Rated as “Very Important” by Participants in Round 1 and Organized by Institute of Medicine Quality Domains Rate “Very Important” (%)
Metric
Effectiveness metrics Tracheal tube placement success rate
92.7
Cardiopulmonary resuscitation performed during transport
78.0
Unintended neonatal hypothermia upon arrival at destination hospital
65.9
Development of hypoxia en route
54.9
Rate of return of spontaneous circulation after cardiopulmonary resuscitation during transport
51.3
Chest tube placement success rate
50.0
Initiation of therapeutic cooling for eligible patients
50.0
Safety metrics Unplanned dislodgement of therapeutic devices, including unplanned extubation
81.0
Verification of tracheal tube placement (confirmatory techniques)
79.7
Medication administration errors
77.5
Patient medical equipment failure during transport
76.3
Patient injuries
74.7
Recognition of abnormalities of vital signs and appropriate treatment
70.9
Rate of cardiac arrest during transport
70.0
Access to medical control during transport
66.2
Maintenance of team clinical care competencies
65.8
Crew injuries
63.3
Use of age-appropriate safety restraining devices
56.4
Development of a pneumothorax en route
55.1
Vehicle failure
53.8
Adverse drug reactions
52.5
Completeness of patient care documentation
50.6
Patient identity confirmed prior to transport
50.6
Efficiency metrics Use of a standardized patient care hand-off
56.3
Family/patient-centeredness Time to pain assessment and treatment
56.3
Timeliness Average mobilization time for transport team
70.0
Time to treat identified arrhythmias
66.3
Time to antibiotic administration for appropriate patients
62.5
Delays in referral hospitals
51.3
Average scene time for various missions (i.e., neonatal and pediatric)
50.0
Seven of the metrics were from the IOM safety domain, three from the effectiveness domain, and one each from the timeliness and efficiency domains. One of the final metrics had been 714
www.pccmjournal.org
contributed by one of the participants as part of the round 1 survey. Final metric draft definitions and their weighted ranks are shown in Table 3. October 2015 • Volume 16 • Number 8
Copyright © 2015 by the Society of Critical Care Medicine and the World Federation of Pediatric Intensive and Critical Care Societies. Unauthorized reproduction of this article is prohibited
Feature Articles
Table 3. Final Consensus Neonatal and Pediatric Transport Quality Metrics Stratified by Weighted Rank and Inclusive of Metrics Definition and Reporting Taxonomy Metric
Weighted Rank Draft Definition
Unplanned dislodgement of therapeutic devices
4.944
The number of documented dislodgements (may be > 1 per transport) while under the care of the transport team of the following devices (intraosseous lines, IVs, umbilical arterial catheters/ umbilical venous catheters, central venous catheters, arterial catheters, TTs, chest tubes, and tracheostomy tubes) divided by the number of transports during the calendar month. This does not include IVs that infiltrate without obvious dislodgement. This metric is reported as “Dislodgements of therapeutic devices per 1,000 transports”
Verification of TT placement
4.900
The number of TTs on transport (regardless of whether or not the transport team placed them themselves) for which there is documentation confirming placement using a minimum of two of the following confirmatory techniques: radiograph, direct visualization through the cords, continuous capnometry or use of a colorimetric capnometer, and assessment for symmetric breath sounds divided by the number of intubated patients transported during the calendar month. This metric is reported as “Percent of intubated patients with documented TT verification”
Average mobilization time of the transport team
4.899
The average time (includes all transports in the calendar month, excluding transports scheduled in advance and patient transports out of the originating facility) in minutes (rounded up to nearest minute) from the start of the referral phone call to the transport team to the time the transport team is en route to the referral facility. “Stacked” trips or transports right after the last during which the team never returns to base should still be included in this count. This metric is reported as “Average mobilization time for an unscheduled transport”
First-attempt TT placement success
4.892
The total number of intubations successful on the first attempt divided by the number of patients on whom intubation was attempted by the transport team during the calendar month. “Intubation attempt” is further defined as laryngoscopy by any member of the transport team regardless of whether there is an attempt to pass a TT. A successful intubation is further defined as that which has been confirmed as described in “Verification of TT tube placement.” First-attempt success should not be disqualified by necessary adjustments to the depth of the TT and resecuring it. This number should be reported separately for patients less than 28 d (neonates) and those 28 d and above (nonneonatal). This metric is reported as “Percent of neonatal intubations with a successful first attempt” and “Percent of nonneonatal intubations with a successful first attempt”
Rate of transportrelated patient injuries
4.879
The number of documented transport-related patient injuries or deaths divided by the number of transports during the calendar month. Excluded are injuries and deaths related to the medical care itself or the omission of medical care. Examples include a patient fall, a loose piece of transport equipment that falls and strikes the patient, and injury suffered in a transport vehicle accident. This metric is reported as a “Rolling 12-mo transport-related patient injury rate per 10,000 transports”
Rate of medication administration errors
4.876
The number of documented medication administration errors during a calendar month divided by the number of transports during the calendar month. Medication administration errors are further defined as drug administrations violating any of the “five rights”—right patient, right medication, right dose, right route, and right time. The metric is reported as “Medication administration errors per 1,000 transports”
Rate of patient medical equipment failure during transport
4.775
The number of documented medical equipment failures (may be > 1 per transport) while under the care of the transport team divided by the number of transports during the calendar month. Examples include IV pumps and ventilators that malfunction during transport, broken monitor leads, and empty medical gas tanks. This metric is reported as “Medical equipment failures per 1,000 transports”
Rate of CPR performed during transport
4.741
The number of transports during which chest compressions are performed from the time the transport team assumes care (“hands on”) until the patient hand-off is completed at the destination facility divided by the number of transports during the calendar month. Multiple episodes of chest compressions in a single transport should only be counted as one episode. If CPR is in progress when the team arrives, this should not be included in this count. This metric is reported as a “Rolling 12-mo CPR rate per 10,000 transports”
Rate of SREs
4.727
The number of SREs during the calendar month divided by the number of transports during the calendar month. An SRE is defined as any unanticipated and largely preventable event involving death, life-threatening consequences, or serious physical or psychological harm. Qualifying events include, but are not limited to, the National Quality Forum’s SREs available at http:// www.qualityforum.org/Topics/SREs/List_of_SREs.aspx. The metric is reported as a “Rolling 12-mo SRE rate per 10,000 transports” (Continued)
Pediatric Critical Care Medicine
www.pccmjournal.org
715
Copyright © 2015 by the Society of Critical Care Medicine and the World Federation of Pediatric Intensive and Critical Care Societies. Unauthorized reproduction of this article is prohibited
Schwartz et al
Table 3. (Continued). Final Consensus Neonatal and Pediatric Transport Quality Metrics Stratified by Weighted Rank and Inclusive of Metrics Definition and Reporting Taxonomy Weighted Rank Draft Definition
Metric
Unintended neonatal hypothermia upon arrival to destination
4.676
The number of neonates (infants < 28 d) with admission temperatures at the destination facility less than 36.5°C axillary (excluding those being actively cooled) divided by the number of neonates transported during the calendar month. This metric is reported as “Percent of transported neonates found hypothermic at admission”
Rate of transportrelated crew injury
4.482
The number of transport-related crew injuries or deaths reported to the institution’s employee health department or equivalent during the calendar month divided by the number of transports during the calendar month. The metric is reported as a “Rolling 12-mo transportrelated crew injury rate per 10,000 transports”
Use of a standardized patient care hand-off
4.464
The number of transports for which there is documented use of a standardized hand-off procedure for turning over patient care at the destination hospital divided by the number of transports during the calendar month. This metric is reported as “Percentage of transports involving a standardized patient care hand-off”
TT = tracheal tube, CPR = cardiopulmonary resuscitation, SRE = serious reportable event. All met at least 70% consensus as “very important.”
DISCUSSION We have identified 12 core quality metrics for neonatal and pediatric transport teams interested in benchmarking their performances and guiding quality improvement efforts. This is the first set of consensus quality metrics established by a large group of participants representative of transport services from around the United States. Central to the Delphi technique is that the validity of the results is dependent on the characteristics of the participants: that they are experts in their fields and that their opinions are representative of different types of programs and locales, etc. We feel that this group comprehensively represents American neonatal and pediatric transport teams, taking into consideration the diversity of leadership structures, team configurations, program models, and geographic locations. The inclusion of both frontline and administrative staff contributed multiple perspectives when weighing the merits and feasibility of individual candidate metrics. We believe this supports the broad applicability of these metrics to transport teams throughout the United States. Not surprisingly, the majority of the final metrics represented the IOM safety domain (14). There were no final metrics from the equitable and patient-centered domains. Similar results have been reported by others developing quality metrics (6). The abundance of safety metrics likely reflects the current national and international focus on improving patient safety and the prevalence of reports on air transport accidents in the scientific and lay press (21–23). Priority metrics were selected naturally by participants for their importance and not with the intent of forcing equal representation across all domains of quality or Donebedian categories. Tremendous care was taken to ensure that the final list of metrics and their definitions met the NQF’s Measure Evaluation Criteria: importance to measure and report, scientific acceptability, feasibility, and usability (15). Ultimately, metrics are only helpful if programs use them. Identifying agreed-upon quality metrics is a critical first step but not sufficient in isolation. Its importance rests in what it enables: benchmarking teams’ performances and guiding 716
www.pccmjournal.org
quality improvement efforts on the local level and industry level as a whole. Benchmarking allows identification of high performers and their best practices. These best practices, customized and spread, will drive quality improvement of the neonatal and pediatric transport industry at large. We understand that this initial set of metrics is a “living document” that will evolve with new research and as clinical priorities change in transport medicine. These metrics and their draft definitions have been placed on the AAP SOTM Metrics Project website (http://www.aap-sotm.org) along with the means for viewers to share their comments and provide feedback. In addition, a downloadable spreadsheet is available to assist teams recording the 12 metrics. Many programs already use these resources; some now have 1–2 years worth of baseline data to analyze and drive improvements on the local level. Work is underway on creating a national electronic database with which programs will be able to contribute metric data and compare their performances to others. This will also guide the establishment of achievable metric goals and the identification of best practices, as described previously.
LIMITATIONS Techniques for conducting Delphi studies are part art and part science. There are no universally accepted requirements for using Delphi technique; experts disagree on variables such as what constitutes group consensus, expert selection, and appropriate reporting of methods and results (11, 20). Unlike many Delphi studies that handpick experts to enhance the validity of the results, the participants in this project were a convenience sample of those attending the neonatal and pediatric transport course hosted by the AAP SOTM every other year. Historically, however, the event is well attended by many of those considered experts in this field, and the participants tend to be a diverse group who represent the field well; the 2012 conference was no different. There was some overrepresentation of participants from the South that may have introduced bias into October 2015 • Volume 16 • Number 8
Copyright © 2015 by the Society of Critical Care Medicine and the World Federation of Pediatric Intensive and Critical Care Societies. Unauthorized reproduction of this article is prohibited
Feature Articles
the selection of the final chosen metrics. The authors do not believe this had a major impact because of the metrics’ similarities to those previously published (9). Although the amount of pediatric transport-related scholarly work registered per year with PubMed has increased over the last 10 years, there is less transport-related literature than for many other fields of medicine. Identification of evidencebased candidate quality metrics for this Delphi process began with what was published in the literature. Although there are large registries such as National Emergency Medical Services Information System, which stores prehospital Emergency Medical Services data from every state for the purposes of quality improvement, no similar database exists for interfacility transport. This limited the availability of evidence-based, data-driven metrics in the literature to guide this project. There were seven participants (6%) from outside the United States. We encouraged their participation and do not feel that this small number makes our conclusions unrepresentative of neonatal and pediatric transport teams in the United States. Lastly, Delphi studies are most valid when the participants do not change from round to round. We did our best to identify conference attendees (round 2) prior to the event and encouraged them to complete the round 1 survey. Seventythree percent of participants contributed to both surveys. The demographics of participants in each round were similar, so we do not believe this adversely impacted the final list of metrics.
CONCLUSIONS This project makes important contributions to the field of neonatal and pediatric transport medicine. The authors succeeded in achieving consensus among a diverse group of national transport experts on 12 core neonatal and pediatric transport quality metrics and their definitions. We propose that transport teams across the country use these quality metrics and their definitions to measure, benchmark, and guide their quality improvement activities.
ACKNOWLEDGMENTS American Academy of Pediatrics Section on Transport Medicine (AAP SOTM) Board Members: M. Michelle Moss, University of Arkansas for Medical Sciences, Arkansas Children’s Hospital, Little Rock, AR; Caraciolo J. Fernandes, Baylor College of Medicine, Texas Children’s Hospital, Houston, TX; Howard S. Heiman, North Shore University Hospital, Manhasset, NY; Nicholas Tsarouhaus, University of Pennsylvania Perelman School of Medicine, Children’s Hospital of Philadelphia, Philadelphia, PA; Webra Price Douglas, Maryland Regional Neonatal Transport Program, Baltimore, MD; Renee McCraine Taylor, Ochsner Medical Center, New Orleans, LA; and Niccole Alexander, Executive Director, AAP SOTM, Elk Grove, IL.
REFERENCES
1. Agency for Healthcare Research and Quality: Healthcare Cost and Utilization Project (HCUP). 2012. Available at: http://hcup-us.ahrq. gov/. Accessed October 28, 2013
Pediatric Critical Care Medicine
2. American Academy of Pediatrics Section on Transport Medicine, Woodward GA, Insoft RM, et al: Guidelines for Air and Ground Transport of Neonatal and Pediatrics. Third Edition. Elk Grove Village, IL, American Academy of Pediatrics, 2007 3. Blumen IJ, Lemkin DL: Principles and Direction of Air Medical Transport. Salt Lake City, UT, Air Medical Physician Association, 2006 4. Commission on Accreditation of Medical Transport Systems (CAMTS): Accreditation Standards of the Commission on Accreditation of Medical Transport Systems. Ninth Edition. 2012. Available at: http:// www.camts.org/Stds_Approved_EditedLL8-30-12.pdf. Accessed November 15, 2012 5. Schwalenstocker E, Bisarya H, Lawless ST, et al: Closing the gap in children’s quality measures: A collaborative model. J Healthc Qual 2008; 30:4–11 6. Alessandrini E, Varadarajan K, Alpern ER, et al; Pediatric Emergency Care Applied Research Network: Emergency department quality: An analysis of existing pediatric measures. Acad Emerg Med 2011; 18:519–526 7. National Quality Forum: Quality Positioning System. Available at: http://www.qualityforum.org/QPS/. Accessed November 16, 2012 8. Crandall WV, Boyle BM, Colletti RB, et al: Development of process and outcome measures for improvement: Lessons learned in a quality improvement collaborative for pediatric inflammatory bowel disease. Inflamm Bowel Dis 2011; 17:2184–2191 9. Bigham MT, Schwartz HP; Ohio Neonatal/Pediatric Transport Quality Collaborative: Quality metrics in neonatal and pediatric critical care transport: A consensus statement. Pediatr Crit Care Med 2013; 14:518–524 10. Delbecq AL, Van de Ven AH: A group process model for problem identification and program planning. J Appl Behav Sci 1971; 7:466–492 11. Delbecq AL, Van de Ven AH, Gustafson DH: Group Techniques for Program Planning: A Guide to Nominal Group and Delphi Processes. Glenview, IL, Scott Foresman, 1975 12. Van de Ven AH, Delbecq AL: The nominal group as a research instrument for exploratory health studies. Am J Public Health 1972; 62:337–342 13. Donabedian A: An Introduction to Quality Assurance in Health Care. New York, NY, Oxford University Press, 2003 14. Institute of Medicine (U.S.) Committee on Quality of Health Care in America: Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC, National Academy Press, 2001 15. National Quality Forum: Measure Evaluation Criteria. 2011. Available at: http://www.qualityforum.org/docs/measure_evaluation_criteria. aspx. Accessed November 29, 2012 16. Rideout C, Gil R, Browne R, et al: Using the Delphi and snow card techniques to build consensus among diverse community and academic stakeholders. Prog Community Health Partnersh 2013; 7:331–339 17. Strober BE, Clay Cather J, Cohen D, et al: A Delphi Consensus approach to challenging case scenarios in moderate-to-severe psoriasis: Part 1. Dermatol Ther (Heidelb) 2012; 2:1 18. Flume PA, Mogayzel PJ Jr, Robinson KA, et al; Clinical Practice Guidelines for Pulmonary Therapies Committee; Cystic Fibrosis Foundation Pulmonary Therapies Committee: Cystic fibrosis pulmonary guidelines: Pulmonary complications: Hemoptysis and pneumothorax. Am J Respir Crit Care Med 2010; 182:298–306 19. Normand SL, McNeil BJ, Peterson LE, et al: Eliciting expert opinion using the Delphi technique: Identifying performance indicators for cardiovascular disease. Int J Qual Health Care 1998; 10:247–260 20. Keeney S, Hasson F, McKenna HP: The Delphi Technique in Nursing and Health Research. West Sussex, UK, Wiley-Blackwell, 2011 21. Handel DA, Yackel TR: Fixed-wing medical transport crashes: Characteristics associated with fatal outcomes. Air Med J 2011; 30:149–152 22. Low RB, Dunne MJ, Blumen IJ, et al: Factors associated with the safety of EMS helicopters. Am J Emerg Med 1991; 9:103–106 23. Maag C: EMS Helicopter Safety: Can New Rules Save Lives? 2009. Available at: http://content.time.com/time/nation/article/0,8599,1876095,00.html. Accessed December 6, 2012
www.pccmjournal.org
717
Copyright © 2015 by the Society of Critical Care Medicine and the World Federation of Pediatric Intensive and Critical Care Societies. Unauthorized reproduction of this article is prohibited
Objectives: The transport of neonatal and pediatric patients to tertiary care facilities for specialized care demands monitoring the quality of care delivered during transport and its impact on patient outcomes. In 2011, pediatric transport teams in Ohio met to identify quality indicators permitting comparisons among programs. However, no set of national consensus quality metrics exists for benchmarking transport teams. The aim of this project was to achieve national consensus on appropriate neonatal and pediatric transport quality metrics. Design: Modified Delphi technique. Setting: The first round of consensus determination was via electronic mail survey, followed by rounds of consensus determination in-person at the American Academy of Pediatrics Section on Transport Medicine’s 2012 Quality Metrics Summit. *See also p. 775. 1 Division of Emergency Medicine, Department of Pediatrics, Cincinnati Children’s Hospital, University of Cincinnati School of Medicine, Cincinnati, OH. 2 Division of Critical Care Medicine, Department of Pediatrics, Akron Children’s Hospital, Northeast Ohio Medical University, Akron, OH. 3 James M. Anderson Center for Health Systems Excellence, Cincinnati Children’s Hospital, Cincinnati, OH. 4 Division of Critical Care Medicine, Department of Pediatrics, Miami Children’s Hospital, Miami, FL. 5 Division of Neonatology, Department of Pediatrics, James W. Riley Hospital for Children, Indiana University School of Medicine, Indianapolis, IN. 6 Department of Pediatrics, Brown University Alpert School of Medicine, Providence, RI. Dr. Schoettker received support for manuscript writing/review; received provision of writing assistance, medicines, equipment, or administrative support; and received support for manuscript preparation (Dr. Schoettker is employed full-time by the Cincinnati Children's Hospital Medical Center as a medical writer. She works with many clinicians to help them publish their research). Dr. Trautman received support for travel from the American Academy of Pediatrics (lectures not specifically related to this). The remaining authors have disclosed that they do not have any potential conflicts of interest. Address requests for reprints to: Hamilton P. Schwartz, MD, Division of Emergency Medicine, Department of Pediatrics, Cincinnati Children’s Hospital, 3333 Burnet Avenue, Cincinnati, OH 45229. E-mail: hamilton. [email protected] Copyright © 2015 by the Society of Critical Care Medicine and the World Federation of Pediatric Intensive and Critical Care Societies DOI: 10.1097/PCC.0000000000000477
Pediatric Critical Care Medicine
Subjects: All attendees of the American Academy of Pediatrics Section on Transport Medicine Quality Metrics Summit, conducted on October 21–23, 2012, in New Orleans, LA, were eligible to participate. Measurements and Main Results: Candidate quality metrics were identified through literature review and those metrics currently tracked by participating programs. Participants were asked in a series of rounds to identify “very important” quality metrics for transport. It was determined a priori that consensus on a metric’s importance was achieved when at least 70% of respondents were in agreement. This is consistent with other Delphi studies. Eighty-two candidate metrics were considered initially. Ultimately, 12 metrics achieved consensus as “very important” to transport. These include metrics related to airway management, team mobilization time, patient and crew injuries, and adverse patient care events. Definitions were assigned to the 12 metrics to facilitate uniform data tracking among programs. Conclusions: The authors succeeded in achieving consensus among a diverse group of national transport experts on 12 core neonatal and pediatric transport quality metrics. We propose that transport teams across the country use these metrics to benchmark and guide their quality improvement activities. (Pediatr Crit Care Med 2015; 16:711–717) Key Words: outcome measures; pediatrics; process measures; quality improvement; safety; transportation of patients
E
ach year in the United States, more than 200,000 newborns, infants, and children are transported from local hospitals to regional hospitals for inpatient specialty care (1). Despite the increasing national focus on improving the quality of healthcare delivery, tremendous practice variation remains in neonatal and pediatric transport. In response, national organizations, such as the American Academy of Pediatrics (AAP), the Commission on Accreditation of Medical Transport Services, and the Air Medical Physician Association, have published operational and clinical care recommendations (2–4). However, there are few established performance metrics that can be used to benchmark neonatal and pediatric transport organizations to identify best practices for transport operations and clinical care. www.pccmjournal.org
711
Copyright © 2015 by the Society of Critical Care Medicine and the World Federation of Pediatric Intensive and Critical Care Societies. Unauthorized reproduction of this article is prohibited
Schwartz et al
Other organizations and specialties have developed pediatric and neonatal quality metrics, benchmarks, and best practices. Participants in the Pediatric Data Quality Systems Collaborative have developed two consensus measure sets and an assessment of nursing-sensitive indicators in pediatric care (5). In 2011, the Pediatric Emergency Care Applied Research Network published an analysis of 405 performance metrics for pediatric emergency care identified through literature review, surveys of emergency department directors, and consensus methodology (6). The Vermont Oxford Network is a steward for National Quality Forum (NQF)-endorsed performance metrics related to the care of premature infants and very low birth weight neonates (7). The ImproveCareNow pediatric inflammatory bowel disease quality improvement collaborative developed a list of 19 measures through literature review, expert consensus, and discussion, refinement, and agreement by participating collaborative sites (8). We recently published the results of a consensus project involving six neonatal and pediatric transport programs across Ohio that identified and defined critical care transport quality metrics (9). Candidate quality metrics were identified through literature review and current tracking by the participants. Consensus was obtained using the nominal group technique (10–12). Final metrics were categorized according to Institute of Medicine (IOM)-recommended quality care domains (safe, timely, effective, equitable, efficient, and patient-centered) and Donabedian’s quality measure framework (structure, process, and outcome) (13, 14). Although we feel the list of quality metrics established for the state of Ohio is comprehensive and relevant to other transport programs, others have suggested that there may be regional biases that limit its generalizability elsewhere in country. We sought to identify a generalizable core set of neonatal and pediatric transport quality metrics. These recommended quality metrics were derived using the modified Delphi technique, with participants representing various regions and program types from across the United States.
MATERIALS AND METHODS Participants and Setting All attendees at the AAP Section on Transport Medicine (SOTM) Quality Metrics Summit, conducted on October 21–23, 2012, in New Orleans, LA, were eligible to participate. Participants were initially identified from the preregistration list for the above Summit and through Summit advertisements posted on the AAP Transport Listserv. Candidate Quality Metrics Candidate transport quality metrics were identified through four sources: the English-language literature, national transport organizations’ recommendations, metrics already tracked by wellestablished transport programs, and experts in neonatal and pediatric transport medicine (9). An initial list of candidate transport quality metrics was reduced by applying the NQF’s evaluation criteria of metric importance, usability, validity, and feasibility (15). The 70 metrics comprising this list were then grouped according to the six IOM-recommended quality care domains (14). 712
www.pccmjournal.org
Modified Delphi Technique The authors chose the Delphi methodology to reach consensus on the metrics. The Delphi technique has been a popular method for developing indicators reflecting patient and general practitioner perspectives of chronic illness, performance indicators for emergency medicine, and indicators for cardiovascular disease (6, 8, 16–19). Delphi methodology succeeds in converging agreement, not in ensuring the validity of the results. The validity of the results is dependent on the expertise and characteristics of the participants. Delphi technique is popular because large numbers of individuals across diverse locations and areas of expertise can be included anonymously, thus avoiding domination of the consensus process by one or a few experts. The Classical Delphi process involves two or more rounds of questionnaires administered to participants by mail. The first questionnaire is heavily open-ended questions, with the purpose of generating the opinions that will be sent back to participants in subsequent rounds for consideration. Participant “experts” rank or rate the responses of others, which inform development of the next survey. Surveys continue until a predetermined level of consensus, conventionally 70%, is achieved on a set of items (20). Variation to this classical process, while still maintaining the core Delphi principles, allows customization to the individual project. The more common variations are e-Delphi (Delphi surveys done completely via e-mail), real-time Delphi (experts are in the same room with consensus achieved in real time), and modified Delphi (first survey of open-ended questions is augmented or replaced by existing opinions in the literature). This project incorporated aspects of all three of the above variations. Round 1. Two weeks before the start of the Neonatal and Pediatric Transport Medicine Quality Metrics Summit, anticipated participants were contacted by e-mail and invited to complete an online SurveyMonkey survey (SurveyMonkey Inc., Palo Alto, CA; http://www.surveymonkey.com). In addition to demographic questions, participants were asked to rate the importance of 70 candidate metrics on a five-point Likert-type scale, ranging from 1 = not important to 5 = very important. Metrics achieving a score of 5 by at least 70% of participants were definitively accepted. Participants were also asked to contribute suggestions for other candidate metrics. An e-mail reminding participants to complete the survey was sent 1 week prior to the start of the consensus conference. Round 2. During the Transport Medicine Quality Metrics Summit, two neonatologists, two pediatric critical care physicians, and a pediatric emergency medicine physician led an expert panel discussion of those candidate quality metrics that had achieved a rating of 5, “very important,” by more than half of respondents during round 1, but had not reached the 70% agreement cutoff noted in round 1. Additionally, the expert panelists shared their opinions of the new metrics contributed by the participants on the first survey. The participants were then asked to categorize, using an anonymous electronic audience participation system (Turning Technologies, Youngstown, OH), how important it was to include each candidate metric in a minimum standard, core list of metrics that could be used for benchmarking programs’ performances. Participants again used October 2015 • Volume 16 • Number 8
Copyright © 2015 by the Society of Critical Care Medicine and the World Federation of Pediatric Intensive and Critical Care Societies. Unauthorized reproduction of this article is prohibited
Feature Articles
Table 1. Participant Characteristics Including Transport Team Program Types Characteristic
Round 1
Round 2
Table 1. (Continued). Participant Characteristics Including Transport Team Program Types Characteristic
Current/most recent role, % (n)
Round 1
Round 2
Clinical manager/director
21.7 (18)
23.9 (27)
Types of patients, % (n)
Patient care provider
34.9 (29)
36.3 (41)
Strictly pediatric
9.6 (8)
8.9 (10)
Medical director
39.8 (33)
35.4 (40)
Strictly neonatal
22.9 (19)
23.9 (27)
3.6 (3)
4.4 (5)
Both
67.5 (56)
61.1 (69)
Other
0 (0)
6.2 (7)
Other nonclinical personnel Healthcare discipline, % (n) Physician
51.8 (43)
48.7 (56)
50 (21)
42.9 (24)
Pediatric critical care
33.3 (14)
32.1 (18)
Pediatric emergency medicine
14.3 (6)
17.9 (10)
Neonatology
General emergency medicine
0 (0)
0 (0)
Hospital medicine
0 (0)
1.8 (1)
General pediatrics
2.4 (1)
1.8 (1)
Other
2.4 (1)
3.6 (2)
28.9 (24)
33.0 (38)
Respiratory therapist
4.8 (4)
3.5 (4)
Emergency medical technician/paramedic
1.2 (1)
2.6 (3)
Nurse practitioner
10.8 (9)
10.4 (12)
Physician assistant
2.4 (2)
1.7 (2)
United States
93.9 (78)
93.9 (107)
Northeast
16.2 (12)
22.0 (24)
Midwest
23.0 (17)
27.5 (30)
South
47.3 (35)
38.5 (42)
Nurse
Location, % (n)
West
13.5 (10)
11.9 (13)
Canada
2.4 (2)
2.6 (3)
Europe
0 (0)
1.8 (2)
Asia
0 (0)
0.9 (1)
1.2 (1)
0.9 (1)
37.4 (31)
46.9 (53)
2.4 (2)
1.8 (2)
60.2 (50)
51.3 (58)
Stand-alone/dedicated team
59.8 (49)
62.0 (70)
Unit-based team
24.4 (20)
23.0 (26)
Combination of both
13.4 (11)
13.3 (15)
2.4 (2)
1.8 (2)
South Pacific Primary modes of transport, % (n) > 95% ground transport > 95% helicopter/airplane transport Both ground and air transport Program type, % (n)
Other
(Continued)
Pediatric Critical Care Medicine
the aforementioned five-point L ikert-type scale. A weighted rank was then assigned in order to prioritize the importance of each final consensus metric. Weighted rank was calculated for each metric achieving consensus, by averaging its round 2 scores on the five-point scale. Metric definitions were drafted if they had not been previously established in the literature. The Institutional Review Board at Akron Children’s Hospital approved this project. As a part of the online round 1 survey, participants were asked to confirm their understanding of the project and agree to participate. Participants were again asked to agree to participate before beginning the real-time round 2 survey. Responses to the surveys were reported in aggregate, so participants’ identities were not linked to their individual responses.
RESULTS Characteristics of Study Participants Eighty-three respondents completed both rounds of surveys. Thirty additional people participated in round 2 only. There were similar participant demographics and proportions of clinical managers/directors, patient care providers, and medical directors in rounds 1 and 2 (Table 1). The healthcare providers were mostly physicians and nurses, and the largest group of physicians was neonatologists followed by pediatric critical care physicians. Ninety-four percent of the participants at the consensus conference were from the United States, with similar representation from all areas of the country. Approximately 50% of the participants worked with transport services that provided both ground and air transportation. Sixty-two percent of participants worked for dedicated teams whose members’ primary job responsibilities are critical care transport, not inpatient care. Sixty-one percent of teams served both neonatal and pediatric patients. Main Results Round 1 Metrics. Ten of the original 70 candidate metrics were rated as “very important” by at least 70% of the round 1 participants. Thirty additional candidate metrics were rated as “very important” by at least 50% of round 1 participants (Table 2). Most of the metrics rated as “very important” belonged to the IOM safety domain. An additional 12 candidate metrics were suggested by round 1 participants to be included in round 2. Round 2 and Final Metrics. Twelve metrics met at least 70% consensus as “very important” during the round 2 survey. www.pccmjournal.org
713
Copyright © 2015 by the Society of Critical Care Medicine and the World Federation of Pediatric Intensive and Critical Care Societies. Unauthorized reproduction of this article is prohibited
Schwartz et al
Table 2. Metrics Rated as “Very Important” by Participants in Round 1 and Organized by Institute of Medicine Quality Domains Rate “Very Important” (%)
Metric
Effectiveness metrics Tracheal tube placement success rate
92.7
Cardiopulmonary resuscitation performed during transport
78.0
Unintended neonatal hypothermia upon arrival at destination hospital
65.9
Development of hypoxia en route
54.9
Rate of return of spontaneous circulation after cardiopulmonary resuscitation during transport
51.3
Chest tube placement success rate
50.0
Initiation of therapeutic cooling for eligible patients
50.0
Safety metrics Unplanned dislodgement of therapeutic devices, including unplanned extubation
81.0
Verification of tracheal tube placement (confirmatory techniques)
79.7
Medication administration errors
77.5
Patient medical equipment failure during transport
76.3
Patient injuries
74.7
Recognition of abnormalities of vital signs and appropriate treatment
70.9
Rate of cardiac arrest during transport
70.0
Access to medical control during transport
66.2
Maintenance of team clinical care competencies
65.8
Crew injuries
63.3
Use of age-appropriate safety restraining devices
56.4
Development of a pneumothorax en route
55.1
Vehicle failure
53.8
Adverse drug reactions
52.5
Completeness of patient care documentation
50.6
Patient identity confirmed prior to transport
50.6
Efficiency metrics Use of a standardized patient care hand-off
56.3
Family/patient-centeredness Time to pain assessment and treatment
56.3
Timeliness Average mobilization time for transport team
70.0
Time to treat identified arrhythmias
66.3
Time to antibiotic administration for appropriate patients
62.5
Delays in referral hospitals
51.3
Average scene time for various missions (i.e., neonatal and pediatric)
50.0
Seven of the metrics were from the IOM safety domain, three from the effectiveness domain, and one each from the timeliness and efficiency domains. One of the final metrics had been 714
www.pccmjournal.org
contributed by one of the participants as part of the round 1 survey. Final metric draft definitions and their weighted ranks are shown in Table 3. October 2015 • Volume 16 • Number 8
Copyright © 2015 by the Society of Critical Care Medicine and the World Federation of Pediatric Intensive and Critical Care Societies. Unauthorized reproduction of this article is prohibited
Feature Articles
Table 3. Final Consensus Neonatal and Pediatric Transport Quality Metrics Stratified by Weighted Rank and Inclusive of Metrics Definition and Reporting Taxonomy Metric
Weighted Rank Draft Definition
Unplanned dislodgement of therapeutic devices
4.944
The number of documented dislodgements (may be > 1 per transport) while under the care of the transport team of the following devices (intraosseous lines, IVs, umbilical arterial catheters/ umbilical venous catheters, central venous catheters, arterial catheters, TTs, chest tubes, and tracheostomy tubes) divided by the number of transports during the calendar month. This does not include IVs that infiltrate without obvious dislodgement. This metric is reported as “Dislodgements of therapeutic devices per 1,000 transports”
Verification of TT placement
4.900
The number of TTs on transport (regardless of whether or not the transport team placed them themselves) for which there is documentation confirming placement using a minimum of two of the following confirmatory techniques: radiograph, direct visualization through the cords, continuous capnometry or use of a colorimetric capnometer, and assessment for symmetric breath sounds divided by the number of intubated patients transported during the calendar month. This metric is reported as “Percent of intubated patients with documented TT verification”
Average mobilization time of the transport team
4.899
The average time (includes all transports in the calendar month, excluding transports scheduled in advance and patient transports out of the originating facility) in minutes (rounded up to nearest minute) from the start of the referral phone call to the transport team to the time the transport team is en route to the referral facility. “Stacked” trips or transports right after the last during which the team never returns to base should still be included in this count. This metric is reported as “Average mobilization time for an unscheduled transport”
First-attempt TT placement success
4.892
The total number of intubations successful on the first attempt divided by the number of patients on whom intubation was attempted by the transport team during the calendar month. “Intubation attempt” is further defined as laryngoscopy by any member of the transport team regardless of whether there is an attempt to pass a TT. A successful intubation is further defined as that which has been confirmed as described in “Verification of TT tube placement.” First-attempt success should not be disqualified by necessary adjustments to the depth of the TT and resecuring it. This number should be reported separately for patients less than 28 d (neonates) and those 28 d and above (nonneonatal). This metric is reported as “Percent of neonatal intubations with a successful first attempt” and “Percent of nonneonatal intubations with a successful first attempt”
Rate of transportrelated patient injuries
4.879
The number of documented transport-related patient injuries or deaths divided by the number of transports during the calendar month. Excluded are injuries and deaths related to the medical care itself or the omission of medical care. Examples include a patient fall, a loose piece of transport equipment that falls and strikes the patient, and injury suffered in a transport vehicle accident. This metric is reported as a “Rolling 12-mo transport-related patient injury rate per 10,000 transports”
Rate of medication administration errors
4.876
The number of documented medication administration errors during a calendar month divided by the number of transports during the calendar month. Medication administration errors are further defined as drug administrations violating any of the “five rights”—right patient, right medication, right dose, right route, and right time. The metric is reported as “Medication administration errors per 1,000 transports”
Rate of patient medical equipment failure during transport
4.775
The number of documented medical equipment failures (may be > 1 per transport) while under the care of the transport team divided by the number of transports during the calendar month. Examples include IV pumps and ventilators that malfunction during transport, broken monitor leads, and empty medical gas tanks. This metric is reported as “Medical equipment failures per 1,000 transports”
Rate of CPR performed during transport
4.741
The number of transports during which chest compressions are performed from the time the transport team assumes care (“hands on”) until the patient hand-off is completed at the destination facility divided by the number of transports during the calendar month. Multiple episodes of chest compressions in a single transport should only be counted as one episode. If CPR is in progress when the team arrives, this should not be included in this count. This metric is reported as a “Rolling 12-mo CPR rate per 10,000 transports”
Rate of SREs
4.727
The number of SREs during the calendar month divided by the number of transports during the calendar month. An SRE is defined as any unanticipated and largely preventable event involving death, life-threatening consequences, or serious physical or psychological harm. Qualifying events include, but are not limited to, the National Quality Forum’s SREs available at http:// www.qualityforum.org/Topics/SREs/List_of_SREs.aspx. The metric is reported as a “Rolling 12-mo SRE rate per 10,000 transports” (Continued)
Pediatric Critical Care Medicine
www.pccmjournal.org
715
Copyright © 2015 by the Society of Critical Care Medicine and the World Federation of Pediatric Intensive and Critical Care Societies. Unauthorized reproduction of this article is prohibited
Schwartz et al
Table 3. (Continued). Final Consensus Neonatal and Pediatric Transport Quality Metrics Stratified by Weighted Rank and Inclusive of Metrics Definition and Reporting Taxonomy Weighted Rank Draft Definition
Metric
Unintended neonatal hypothermia upon arrival to destination
4.676
The number of neonates (infants < 28 d) with admission temperatures at the destination facility less than 36.5°C axillary (excluding those being actively cooled) divided by the number of neonates transported during the calendar month. This metric is reported as “Percent of transported neonates found hypothermic at admission”
Rate of transportrelated crew injury
4.482
The number of transport-related crew injuries or deaths reported to the institution’s employee health department or equivalent during the calendar month divided by the number of transports during the calendar month. The metric is reported as a “Rolling 12-mo transportrelated crew injury rate per 10,000 transports”
Use of a standardized patient care hand-off
4.464
The number of transports for which there is documented use of a standardized hand-off procedure for turning over patient care at the destination hospital divided by the number of transports during the calendar month. This metric is reported as “Percentage of transports involving a standardized patient care hand-off”
TT = tracheal tube, CPR = cardiopulmonary resuscitation, SRE = serious reportable event. All met at least 70% consensus as “very important.”
DISCUSSION We have identified 12 core quality metrics for neonatal and pediatric transport teams interested in benchmarking their performances and guiding quality improvement efforts. This is the first set of consensus quality metrics established by a large group of participants representative of transport services from around the United States. Central to the Delphi technique is that the validity of the results is dependent on the characteristics of the participants: that they are experts in their fields and that their opinions are representative of different types of programs and locales, etc. We feel that this group comprehensively represents American neonatal and pediatric transport teams, taking into consideration the diversity of leadership structures, team configurations, program models, and geographic locations. The inclusion of both frontline and administrative staff contributed multiple perspectives when weighing the merits and feasibility of individual candidate metrics. We believe this supports the broad applicability of these metrics to transport teams throughout the United States. Not surprisingly, the majority of the final metrics represented the IOM safety domain (14). There were no final metrics from the equitable and patient-centered domains. Similar results have been reported by others developing quality metrics (6). The abundance of safety metrics likely reflects the current national and international focus on improving patient safety and the prevalence of reports on air transport accidents in the scientific and lay press (21–23). Priority metrics were selected naturally by participants for their importance and not with the intent of forcing equal representation across all domains of quality or Donebedian categories. Tremendous care was taken to ensure that the final list of metrics and their definitions met the NQF’s Measure Evaluation Criteria: importance to measure and report, scientific acceptability, feasibility, and usability (15). Ultimately, metrics are only helpful if programs use them. Identifying agreed-upon quality metrics is a critical first step but not sufficient in isolation. Its importance rests in what it enables: benchmarking teams’ performances and guiding 716
www.pccmjournal.org
quality improvement efforts on the local level and industry level as a whole. Benchmarking allows identification of high performers and their best practices. These best practices, customized and spread, will drive quality improvement of the neonatal and pediatric transport industry at large. We understand that this initial set of metrics is a “living document” that will evolve with new research and as clinical priorities change in transport medicine. These metrics and their draft definitions have been placed on the AAP SOTM Metrics Project website (http://www.aap-sotm.org) along with the means for viewers to share their comments and provide feedback. In addition, a downloadable spreadsheet is available to assist teams recording the 12 metrics. Many programs already use these resources; some now have 1–2 years worth of baseline data to analyze and drive improvements on the local level. Work is underway on creating a national electronic database with which programs will be able to contribute metric data and compare their performances to others. This will also guide the establishment of achievable metric goals and the identification of best practices, as described previously.
LIMITATIONS Techniques for conducting Delphi studies are part art and part science. There are no universally accepted requirements for using Delphi technique; experts disagree on variables such as what constitutes group consensus, expert selection, and appropriate reporting of methods and results (11, 20). Unlike many Delphi studies that handpick experts to enhance the validity of the results, the participants in this project were a convenience sample of those attending the neonatal and pediatric transport course hosted by the AAP SOTM every other year. Historically, however, the event is well attended by many of those considered experts in this field, and the participants tend to be a diverse group who represent the field well; the 2012 conference was no different. There was some overrepresentation of participants from the South that may have introduced bias into October 2015 • Volume 16 • Number 8
Copyright © 2015 by the Society of Critical Care Medicine and the World Federation of Pediatric Intensive and Critical Care Societies. Unauthorized reproduction of this article is prohibited
Feature Articles
the selection of the final chosen metrics. The authors do not believe this had a major impact because of the metrics’ similarities to those previously published (9). Although the amount of pediatric transport-related scholarly work registered per year with PubMed has increased over the last 10 years, there is less transport-related literature than for many other fields of medicine. Identification of evidencebased candidate quality metrics for this Delphi process began with what was published in the literature. Although there are large registries such as National Emergency Medical Services Information System, which stores prehospital Emergency Medical Services data from every state for the purposes of quality improvement, no similar database exists for interfacility transport. This limited the availability of evidence-based, data-driven metrics in the literature to guide this project. There were seven participants (6%) from outside the United States. We encouraged their participation and do not feel that this small number makes our conclusions unrepresentative of neonatal and pediatric transport teams in the United States. Lastly, Delphi studies are most valid when the participants do not change from round to round. We did our best to identify conference attendees (round 2) prior to the event and encouraged them to complete the round 1 survey. Seventythree percent of participants contributed to both surveys. The demographics of participants in each round were similar, so we do not believe this adversely impacted the final list of metrics.
CONCLUSIONS This project makes important contributions to the field of neonatal and pediatric transport medicine. The authors succeeded in achieving consensus among a diverse group of national transport experts on 12 core neonatal and pediatric transport quality metrics and their definitions. We propose that transport teams across the country use these quality metrics and their definitions to measure, benchmark, and guide their quality improvement activities.
ACKNOWLEDGMENTS American Academy of Pediatrics Section on Transport Medicine (AAP SOTM) Board Members: M. Michelle Moss, University of Arkansas for Medical Sciences, Arkansas Children’s Hospital, Little Rock, AR; Caraciolo J. Fernandes, Baylor College of Medicine, Texas Children’s Hospital, Houston, TX; Howard S. Heiman, North Shore University Hospital, Manhasset, NY; Nicholas Tsarouhaus, University of Pennsylvania Perelman School of Medicine, Children’s Hospital of Philadelphia, Philadelphia, PA; Webra Price Douglas, Maryland Regional Neonatal Transport Program, Baltimore, MD; Renee McCraine Taylor, Ochsner Medical Center, New Orleans, LA; and Niccole Alexander, Executive Director, AAP SOTM, Elk Grove, IL.
REFERENCES
1. Agency for Healthcare Research and Quality: Healthcare Cost and Utilization Project (HCUP). 2012. Available at: http://hcup-us.ahrq. gov/. Accessed October 28, 2013
Pediatric Critical Care Medicine
2. American Academy of Pediatrics Section on Transport Medicine, Woodward GA, Insoft RM, et al: Guidelines for Air and Ground Transport of Neonatal and Pediatrics. Third Edition. Elk Grove Village, IL, American Academy of Pediatrics, 2007 3. Blumen IJ, Lemkin DL: Principles and Direction of Air Medical Transport. Salt Lake City, UT, Air Medical Physician Association, 2006 4. Commission on Accreditation of Medical Transport Systems (CAMTS): Accreditation Standards of the Commission on Accreditation of Medical Transport Systems. Ninth Edition. 2012. Available at: http:// www.camts.org/Stds_Approved_EditedLL8-30-12.pdf. Accessed November 15, 2012 5. Schwalenstocker E, Bisarya H, Lawless ST, et al: Closing the gap in children’s quality measures: A collaborative model. J Healthc Qual 2008; 30:4–11 6. Alessandrini E, Varadarajan K, Alpern ER, et al; Pediatric Emergency Care Applied Research Network: Emergency department quality: An analysis of existing pediatric measures. Acad Emerg Med 2011; 18:519–526 7. National Quality Forum: Quality Positioning System. Available at: http://www.qualityforum.org/QPS/. Accessed November 16, 2012 8. Crandall WV, Boyle BM, Colletti RB, et al: Development of process and outcome measures for improvement: Lessons learned in a quality improvement collaborative for pediatric inflammatory bowel disease. Inflamm Bowel Dis 2011; 17:2184–2191 9. Bigham MT, Schwartz HP; Ohio Neonatal/Pediatric Transport Quality Collaborative: Quality metrics in neonatal and pediatric critical care transport: A consensus statement. Pediatr Crit Care Med 2013; 14:518–524 10. Delbecq AL, Van de Ven AH: A group process model for problem identification and program planning. J Appl Behav Sci 1971; 7:466–492 11. Delbecq AL, Van de Ven AH, Gustafson DH: Group Techniques for Program Planning: A Guide to Nominal Group and Delphi Processes. Glenview, IL, Scott Foresman, 1975 12. Van de Ven AH, Delbecq AL: The nominal group as a research instrument for exploratory health studies. Am J Public Health 1972; 62:337–342 13. Donabedian A: An Introduction to Quality Assurance in Health Care. New York, NY, Oxford University Press, 2003 14. Institute of Medicine (U.S.) Committee on Quality of Health Care in America: Crossing the Quality Chasm: A New Health System for the 21st Century. Washington, DC, National Academy Press, 2001 15. National Quality Forum: Measure Evaluation Criteria. 2011. Available at: http://www.qualityforum.org/docs/measure_evaluation_criteria. aspx. Accessed November 29, 2012 16. Rideout C, Gil R, Browne R, et al: Using the Delphi and snow card techniques to build consensus among diverse community and academic stakeholders. Prog Community Health Partnersh 2013; 7:331–339 17. Strober BE, Clay Cather J, Cohen D, et al: A Delphi Consensus approach to challenging case scenarios in moderate-to-severe psoriasis: Part 1. Dermatol Ther (Heidelb) 2012; 2:1 18. Flume PA, Mogayzel PJ Jr, Robinson KA, et al; Clinical Practice Guidelines for Pulmonary Therapies Committee; Cystic Fibrosis Foundation Pulmonary Therapies Committee: Cystic fibrosis pulmonary guidelines: Pulmonary complications: Hemoptysis and pneumothorax. Am J Respir Crit Care Med 2010; 182:298–306 19. Normand SL, McNeil BJ, Peterson LE, et al: Eliciting expert opinion using the Delphi technique: Identifying performance indicators for cardiovascular disease. Int J Qual Health Care 1998; 10:247–260 20. Keeney S, Hasson F, McKenna HP: The Delphi Technique in Nursing and Health Research. West Sussex, UK, Wiley-Blackwell, 2011 21. Handel DA, Yackel TR: Fixed-wing medical transport crashes: Characteristics associated with fatal outcomes. Air Med J 2011; 30:149–152 22. Low RB, Dunne MJ, Blumen IJ, et al: Factors associated with the safety of EMS helicopters. Am J Emerg Med 1991; 9:103–106 23. Maag C: EMS Helicopter Safety: Can New Rules Save Lives? 2009. Available at: http://content.time.com/time/nation/article/0,8599,1876095,00.html. Accessed December 6, 2012
www.pccmjournal.org
717
Copyright © 2015 by the Society of Critical Care Medicine and the World Federation of Pediatric Intensive and Critical Care Societies. Unauthorized reproduction of this article is prohibited
