Could safety boards provide a valuable tool to enhance the safety of reproductive medicine? Richard T. Scott Jr., M.D.,a,b and Nathalie De Zieglerc a Reproductive Medicine Associates of New Jersey, Basking Ridge, New Jersey; b Department of Obstetrics, Gynecology, and Reproductive Science, Robert Wood Johnson Medical School, Rutgers University, New Brunswick, New Jersey; and c Captain ^te Analyse, Paris, France and Air Operations Advisor, Bureau Enque
Medicine and aviation have a striking number of similarities. Both are led by highly-trained individuals performing complex tasks that are critical to outcomes. They also integrate the efforts of other professionals to assure that the flight, procedures, or processes are completed successfully. Also in common, is the potential for errors to have catastrophic and even life-threatening consequences. Both aviation and medicine have responded to this complex operating environment by building safety programs. Unfortunately, those in medicine have not been optimal in reducing significant adverse outcomes, including deaths. It has been suggested that given the parallels, that aviation safety programs might be adapted to clinical medicine. One such measure would the formation of a Clinical Safety Board (CSB) modeled after the National Transportation Safety Board (NTSB). Such a board would collect data across the nation and determine root causes of errors. They may then provide recommendations to professional societies and regulatory agencies for consideration for implementation. Such programs would be dependent on accurate and thorough reporting. Indemnification, similar to that enacted by the federal government for aviation, would be critical. In the end, a CSB should Use your smartphone empower better patient care with reduced liability to the providers and programs. (Fertil SterilÒ to scan this QR code 2013;100:1518–23. Ó2013 by American Society for Reproductive Medicine.) and connect to the Key Words: Safety, NTSB, ART, IVF, accident investigation Discuss: You can discuss this article with its authors and with other ASRM members at http:// fertstertforum.com/scottr-safety-ntsb-art-ivf/
F
‘‘
irst do no harm’’ is a central tenet in the practice of medicine, and there is little doubt that all clinical programs seek to attain the best possible outcomes for their patients while minimizing complications. Still adverse sequelae do occur. While some complications or sub-optimal outcomes are intrinsic to indicated medical or surgical treatments, others are the result of human errors. The impact of these errors is astounding. The Institute of Medicine estimated that almost a hundred thousand deaths occur annually secondary to medical treatment errors (1). In response to these challenges, a variety of risk-management and safety
programs have been implemented. Examples might include mandatory safety training programs at facilities where the clinicians are credentialed, the use of ‘‘time-out's’’ prior to the initiation of invasive procedures, and tracking of abnormal laboratory results to assure that they are seen quickly by the patients' physician. In spite of these improvements, the overall prevalence of errors has not changed meaningfully over the last decades. Considered in aggregate, safety program effectiveness has been suboptimal (2). Upon initial consideration, these issues might not seem to apply to reproductive medicine. Our patients are young and healthy and deaths are so
Received September 30, 2013; revised October 21, 2013; accepted October 22, 2013. R.T.S. has nothing to disclose. N.D.Z. has nothing to disclose. Reprint requests: Richard T. Scott Jr., M.D.,140 Allen Road, Basking Ridge, New Jersey 07920 (E-mail: [email protected]). Fertility and Sterility® Vol. 100, No. 6, December 2013 0015-0282/$36.00 Copyright ©2013 American Society for Reproductive Medicine, Published by Elsevier Inc. http://dx.doi.org/10.1016/j.fertnstert.2013.10.033 1518
discussion forum for this article now.*
* Download a free QR code scanner by searching for “QR scanner” in your smartphone’s app store or app marketplace.
exceedingly rare that the vast majority of providers will complete their entire career without having one within their practice (3). The reality is that there are genuine safety issues in assisted reproductive technology (ART). Rare catastrophic events such as severe hemorrhage and pelvic abscesses do occasionally occur. Ovarian hyperstimulation syndrome is another obvious example. Medication errors, misidentification of patients, loss of chain of custody of biological specimens, and use of expired media are just a few of many real-world examples.
THE NATURE OF ERRORS To prevent errors you must first understand them. Most things which impact safety involve some component of human error. While equipment may break or a production lot of media may perform sub-optimally even after VOL. 100 NO. 6 / DECEMBER 2013
Fertility and Sterility® passing quality assurance testing, the reality is that most clinical and laboratory care is performed or controlled by humans. As such, deviations often reflect either human error or a failure of a human to recognize and correct a problem. A traditional model for avoiding errors is to clearly define optimal management in a given clinical circumstance, train providers and staff members on how to execute that algorithm, and then follow up to make sure that protocol has been implemented and providers are within compliance. This approach is amongst the guiding principles for ART laboratory certification used by the College of American Pathologists (4). In brief, define the ‘‘right way’’ in a policy and protocol; implement that protocol, and then check back to assure compliance. If deviations are identified, humans are either counseled, retrained, or punished. Whole programs may be decredentialed for noncompliance. No doubt traditional approaches to safety may be both informative and effective. However, many do not consider the inevitability of human imperfection, seek to define paradigms which reduce the opportunities for human error, or try and minimize the impact of any errors which might occur. They have the simple philosophy of ‘‘you are a highly intelligent and extensively trained professional—do the right thing’’. A broader perspective would be to look for those factors which are prone to human error and then try and reduce or eliminate them. This does not replace the need for human vigilance; it simply states that systems may be put in place which meaningfully reduce the opportunity for errors to occur or to produce harm. An example of this comes from aviation. A few decades ago, a large jet aircraft was rolling down the runway and gathering speed in the process of taking off. The pilot of the aircraft looked over and saw that his copilot, who was having personal problems, looked sad and distracted. Using the intercom, he told the co-pilot to ‘‘cheer up.’’ The copilot was not paying sufficient attention to the progress of the takeoff and did not realize that while the nose of the aircraft had been raised that the main wheels were still on the ground. Being distracted, he thought that the pilot had said ‘‘gear up’’ and he reached over and moved the landing gear control lever to the up position. The landing gear retracted before the pilot could react. The plane crashed and burned and all lives were lost. The story was discovered by listening to the ‘‘black box’’ recording during the accident investigation. Obviously, this is a wholly unacceptable error. Pilots should be attentive at all times but particularly during the critical transitions of take-off and landing. They all certainly know better than to retract the landing gear while the plane is still on the ground. If the traditional safety models commonly utilized in medicine had been applied to this tragedy, the following actions would likely result. Local morbidity and mortality conferences would have presented the facts of the tragedy, and been followed by discussions of the deviations in the ‘‘standard of practice/care’’ which had occurred. All pilots would then be encouraged to pay attention to take-off procedures and not to raise the landing gear until after the plane is clearly in the air. Perhaps procedure manuals would be rewritten to VOL. 100 NO. 6 / DECEMBER 2013
provide greater emphasis on vigilance by both the pilot and copilot during take-off. Certifying examinations might have a question or two added which emphasized the importance of landing gear management sequences. Would any of those things have prevented this accident? Of course not, and those were not the recommendations which resulted. Instead, the aviation accident investigation team suggested that a simple pressure monitor be placed in the landing gear system which precludes the pilot from raising the landing gear if there is still any weight on the wheels. Until the plane is actually flying, the landing gear control lever is locked in place and may not move. This means that no matter what, no pilot could ever make this error again. Safety is not just about teaching people the right thing to do, it is also about creating environments and systems which reduce the risk for or consequences of human error. Several authors have pointed out the similarities between aviation and medicine and have suggested that the types of safety program used in the aviation industry might be also be useful in enhancing the safety of medical care.
AVIATION AND MEDICINE The parallels between aviation and medicine are enormous (Table 1) (5–7). Highly-trained individuals perform high complexity tasks. Successful completion of those tasks frequently requires a great deal of sophisticated equipment and the support from a wide variety of staff/team members with different but critical skills sets. When things are proceeding normally, the tasks may be repetitive and even monotonous. Still, individuals performing the procedures must maintain great overall situational awareness and react quickly to any one of a variety of different emergencies. Finally deviations from ideal (whether the fault of the pilot/ physician or not) can produce catastrophic consequences including significant injury and even loss of life. Aviation has powerful safety tools which have produced dramatic reductions in accidents. Given the rarity of some catastrophic events, data is collected from across the aviation industry as a whole. Systematic evaluation of those data provides insight into the sequence of events that lead to the errors. A truly catastrophic accident is usually a result of a confluence of many errors (8). This type of root-cause analyses has led to recommendations for powerful changes in training, equipment, and operations. How is this system wide data collection, analysis, and recommendation paradigm achieved? By implementation of a National Transportation Safety Board (NTSB) or Bureau Enquete et Analyse which is the French equivalent. These two groups investigate the majority of commercial airline accidents throughout the world since they investigate all accidents involvement Boeing and Airbus aircraft, respectively.
WHAT IS AN NTSB? The roots of the NTSB date back to the 1926 with the Air Commerce Act, but a fully functional NTSB was not truly formed until 1974 when the board was made autonomous from the Department of Transportation (9). Among other responsibilities, it is chartered with investigating every aviation accident. 1519
VIEWS AND REVIEWS
TABLE 1 Comparisons between aviation and reproductive medicine. Factor being compared
Level of similarity
Training
High
Complexity Consequences of errors Coordination of information and services from multiple other specialists Commonality of checklists and procedures throughout the industry Number of providers
High High High
Requirement to function under sub-optimal conditions Witness to errors Indemnified self-reporting systems for errors and ‘‘near-misses’’
Low
Low Low
Low Low
Commercial aviation
Reproductive medicine
High levels with frequent recurrence training High Life-threatening to insignificant Extensive (flight controllers, meteorologists, maintenance, etc.)
High levels with ongoing continuing education and re-certification High Life-threatening to insignificant Extensive (embryologists, andrologists, nurses, molecular geneticists, etc)
Dictated principally by manufacturers; high degree of commonality Low; eight airlines dominate the US commercial airline market Rare; exceptions for some helicopter and search and rescue operations Very high Present through FAA's ASAP program
Developed by individual ART programs; very low degree of commonality High; over 400 independent clinical ART programs Common Very low None
Note: ART, assisted reproductive technology; ASAP, Aviation Safety Action Program; FAA, Federal Aviation Agency. Scott. A clinical safety board for reproductive medicine. Fertil Steril 2013.
The NTSB is a special organization whose unique features empower it to be unusually effective (Table 2) (10). The NTSB was chartered by and is funded through the federal government, but functions nearly autonomously. It is not political and is staffed with the full complement of experts with sufficient expertise to investigate the accidents. These include pilots, engineers, maintenance experts, accident investigators, and a variety of other specialists. The leadership of the NTSB is appointed by the executive branch, but the five member team may not have more than 3 members of either party to minimize any political influence. Perhaps most important, all findings and analyses are not admissible during litigation (11). There are also well defined limits on the power and influence of the NTSB (10). The organization makes recommenda-
TABLE 2 What is a National Transportation Safety Board (NTSB)? Characteristics of the United States NTSB National and international investigative agency Federally chartered and funded Functions independently in ‘‘apolitical’’ environment Staffed with high-qualified individuals capable of investigating all mechanical, environmental, and human factors which contributed to accidents No direct regulatory authority Investigates all aviation accidents Investigates all major accidents in other areas of transportation, pipelines, and hazardous chemicals Collects data from individual events and integrates them with data from other occurrences to make recommendations regarding equipment, training, or operations to enhance safety Creates a most-wanted list of which improvements are most important for the future Supports investigations of other national and international agencies in time of crisis or catastrophe Scott. A clinical safety board for reproductive medicine. Fertil Steril 2013.
1520
tions, but has no authority to create policy or set standards regarding any aspect of aviation or the aviation industry. Their recommendations go back to the Federal Aviation Agency (FAA) or to industry for consideration. As such, the NTSB works closely with industry and regulatory aspects of the government to help assure that implementation is optimal. Perhaps most important, the NTSB has been a major component of a safety system which in aggregate has attained unparalled success. During an interval when there was no meaningful reduction in the prevalence of significant medical errors, there was an 83% reduction in fatal aviation accidents (7).
AN NTSB FOR REPRODUCTIVE MEDICINE Reproductive Medicine provides an excellent opportunity for a medical NTSB, also termed a Clinical Safety Board (CSB) (12, 13). The opportunity to integrate data attained from all the programs in the field would allow accumulation of sufficient information to empower evidenced-based recommendations which might aid in the prevention of even the most uncommon hazards. The importance of an integrated national model cannot be over-stated, most complications are rare enough that it is difficult to accrue valuable insight by considering data from even the largest individual centers. The author recently learned of a case which illustrates the importance of sharing data. A patient with one of the inherited thrombophilia's was advised by a hematologist to be on low molecular weight heparin during their in vitro fertilization cycle. The hematologist insisted that the patient continue their anticoagulation throughout the cycle, including the day of retrieval. The reproductive endocrinologist spoke to the hematologist but ultimately relented and agreed to maintain the anticoagulation. The patient ultimately suffered a rare serious complication from internal bleeding. Anecdotally, other practitioners have reported ‘‘severe hemorrhage’’ in patients receiving VOL. 100 NO. 6 / DECEMBER 2013
Fertility and Sterility® anticoagulation at the time of retrieval. A national CSB would allow accrual of information about these complications and the development of evidence-based recommendations which optimize management and improve safety. A CSB has great potential to work quite well because it would be able to evaluate problems in any clinical setting, not just those which occur in inpatients. Most critical and high-risk health care in other medical specialties occurs in the inpatient setting. As such, hospitals typically assume responsibility and provide direction for safety programs. The majority of reproductive medicine care is provided within the office facilities of the individual practices. These are generally not located within a hospitals inpatient or outpatient facilities. While there are myriad standards for the operation of these facilities, comprehensive safety programs for independent offices are less well-defined. Traditional safety programs rightfully focus on clinical care. Even CSB's modeled after those in the aviation industry (such as those implemented in anesthesia and radiology) are focused on actual clinical care. Implementing a CSB for reproductive medicine would also provide an opportunity to develop insight into safety issues in embryology. Embryology laboratories provide a truly unique and highly specialized clinical service, but are outside of the domain of many institutional clinical safety programs. Here again, a national safety board would allow accrual of actual data regarding repetitive errors which might lead to recommendations that improve the safety and precision of embryology practice. Prior efforts through more conventional regulatory mechanisms have led to burdensome and costly safety programs to prevent complications which had never even occurred. At a time of transition in the way health care is being provided and regulated, a CSB would provide means of focusing on all aspects of reproductive medicine.
BARRIERS TO IMPLEMENTATION While there are a number of similarities, aviation and medicine are also different in important ways. A number of factors would need to be addressed when designing a CSB for reproductive medicine.
THE COMPLEX PROVIDER MIX There are relative few airlines compared to the number of ART clinics. Additionally, each airline operates a limited number of aircraft types. This provides a greater opportunity for uniformity. Beyond that, many of the procedural aspects of aviation, including emergency procedures, are worked out by the manufacturers and not the airlines, providing even greater uniformity. If pilots are following similar procedures throughout the industry, it should be considerably easier to identify those procedures or issues which are prone to error or which may have contributed to accidents. It also makes it easier to implement enhanced safety procedures once a problem is identified—define a single procedural or mechanical change and simply allow all the carriers flying that type of aircraft to incorporate it in an essentially identical fashion. VOL. 100 NO. 6 / DECEMBER 2013
Medicine is much more ‘‘atomized’’. There are a much larger number of groups and each provides a relatively low percentage of the care being provided nationwide. While these programs are highly dependent on their equipment, manufacturers do not typically provide checklists for how they should be used when performing actual clinical care. In contrast, the manufacturers of commercial jets provide checklists that operators adopt for actual flight operations. Flight data recorders are provided by the manufacturers and the information collected in a near identical fashion for all flights of that type aircraft. This simplifies investigation and analyses of industry wide procedural issues. In contrast, individual ART programs design their own charts and collect and record data in their own unique ways. As electronic medical records become uniformly implemented, the limited number of commercial electronic medical records will make data collection more uniform. Unfortunately, procedural differences between programs will continue to make these types of investigations most challenging.
OPERATING IN HIGH-RISK ENVIRONMENTS Helicopter operations may be most analogous to the practice of reproductive medicine. There are a relatively large number of small independent operators, each with their own approach to safety, decision making, and operational tempo. Additionally, they commonly fly in ‘‘must-go’’ situations. While the major airlines delay or cancel flights in bad weather or if the full complement of coordination from the FAA is unavailable (as they should), helicopter operators frequently provide rescue and evacuation services. They are more likely to fly when the weather is extremely poor and thus operate in the most difficult and dangerous circumstances. Helicopters also fly intrinsically complicated and risky missions. Monitoring and repairing massive high-voltage power lines from a flying helicopter, with live power in those lines, is high risk no matter the weather. It might be analogous to dealing with a patient whose cervical ectopic pregnancy is rupturing. Medicine is also analogous to helicopter flying because high risk patients still need care. Only rarely is the risk so high that providers withhold care and refuse to attempt to ‘‘rescue’’ the patient. Accident rates for helicopter operations remain 10- to 15-fold higher than the commercial airlines. In spite of this, enhancements in the safety of helicopter operations have been achieved. Nothing of that order of magnitude has been realized in medicine.
EVENT REPORTING The NTSB model for enhancing safety is critically dependent on one thing, identification and reporting of accidents and near-accidents so they may be investigated. Major accidents in aviation are evident. If a plane runs off a runway or crashes enroute to its destination, an enormous number of people are immediately aware. Pilots, passengers, air traffic controllers, a variety of crew members, and emergency response personnel all know when an accident occurs. Activation of the NTSB accident response team is virtually assured. A great deal may also be learned by evaluation of 1521
VIEWS AND REVIEWS ‘‘near-miss’’ events. Situations where a deviation occurred but no accident or harm was done. These events are typically voluntarily reported by the pilots or flight controllers themselves. In contrast, not all accidents in medicine are so evident. In the case of ART, serious clinical complications such as abscess or embolus would be relatively easily identified and reported, but less serious complications, including those in the in vitro fertilization laboratory, may be less evident. Dropping a dish containing embryos is clearly a very serious event, but may not be witnessed. Self-reporting would be required to ‘‘flag’’ the adverse event. In medicine, voluntary reporting of deviations is uncommon. There are issues of liability, credentialing, reviews in very public venues such as departmental or hospital morbidity and mortality meetings which are commonly attended by business competitors, and a system which is focused on punitive measures once any event has occurred. So what motivates aviation professionals to ‘‘self-identify’’ deviations which occur? The FAA, the regulator of all air operations in the United States, commercial or private, has indeed enacted a non-punitive reporting system of errors and incidents, which is instrumental for enhancing safety. The system known as Aviation Safety Action Program (ASAP) allows pilots (and airline controllers) to report errors committed on the job. These reports guarantee no punitive consequences as long as no rules were blatantly and knowingly infringed (14). Practically, this has provided the FAA with a trove of information on errors in flying (and controlling flying). As might be expected, many errors committed while flying are menial in nature, sporadic in their occurrences, and display no particular pattern of recurrence. Other errors may be repetitive and represent a genuine and recurring risk to air safety. For example, a pilot turning left after departure when a right turn is expected might be at risk of flying into other aircraft in the congested airspace near an airport. Unfortunately, this may rarely occur just because it is in the human nature to make mistakes. Yet, if wrong side turns occur with any frequency, the situation becomes different. The root of the error is likely that something is unclear in the departure instructions and/or charts, which evidently needs prompt attention and remedy. Without implementing ASAP the regulator, the FAA, would never have known that it needed to amend the departure charts at the given airport, unless an accident ultimately occurred. Non-punitive reporting such as enacted by ASAP thus helps prevent accidents by identifying problems and implementing solutions before accidents occur. The incentive for pilots and controllers to file reports with the FAA within the ASAP format is to gain indemnification from possible sanctions for that error. ASAP works because in aviation the risk of sanction is high in case of errors. This stems from the fact that aviation is highly monitored including by automatic reporting of deviations in flight through the Flight Operational Quality Assurance program (15). The latter automatically investigates deviations in flight. Pilots may not use ASAP if they failed to notify in advance of the FAA advising them of an incident or investi1522
gation. It is safer to report the deviation and be protected from punitive measures unless a truly reckless violation occurred. The protection from sanctions is therefore the motivator which propels information into the ASAP system and in turn allowed the FAA, NTSB, and manufacturers to evaluate errors and enact meaningful safety solutions prior to catastrophe.
ENACTING NON-PUNITIVE REPORTING IN MEDICINE Systems enabling non-punitive reporting of errors are urgently needed in medicine. For such system to be effective however, a powerful incentive needs to exist (to be created) as doctors are not inclined to report their errors. The day to day routine practice of medicine is much less supervised than aviation. Every step of every procedure is not continuously monitored. As such, doctors don't fear sanctions for ‘‘near-miss’’ type errors simply because such sanctions rarely occur. If programs like ASAP were translated directly into medicine it is likely that utilization would still be low. A promise of no sanction has little effect when no one with the authority to levy a sanction typically even knows what happened. Selfreporting simply invites additional and unwanted scrutiny. One practical way for effectively creating an incentive for self-reporting errors in medicine in an ASAP-like way would be to make the self-reported data on errors inaccessible (and not usable) in court in case of litigation or disciplinary actions. A well-established mechanism for secluding information related to self-reporting of errors is the attorney-client privilege, which protects clients for all the sensitive information exchanged with their lawyers. The attorney-client privilege is the oldest privilege recognized by Anglo-American jurisprudence. There are precise definitions of its nature and strict definitions about circumstances which constitute exceptions when the privilege may be breached. This would provide protection for providers from regulatory issues, credentialing boards, and civil litigation. Without such protections, self-reporting would not make sense for the clinician and all the insight and opportunities to improve safety will continue to be lost.
FUNDING AND ADMINISTRATION A CSB for reproductive medicine would have real expenses and would likely require either federal or substantial professional society support. While the core staff might be fairly small, funding for travel, investigative tools, and administrative support might still be substantial. It might be possible for the project to be initiated with limited funds from professional societies, industry, or the ART programs themselves. It is likely that legislation would be needed to provide the privacy and indemnification which aviators enjoy at the current time and which is key to meaningful implementation. The NTSB grew out from the Department of Transportation, an organization with an excellent understanding of the issues faced by aviation. Similarly, a CSB for reproductive medicine will likely need affiliation with a professional organization with a deep understanding of this field of VOL. 100 NO. 6 / DECEMBER 2013
Fertility and Sterility® medicine, the safety issues, and the barriers to meaningful improvements. The American Society for Reproductive Medicine and the European Society for Human Reproduction and Embryology would be excellent choices for the United States and Europe, respectively. The focus of these groups need not be restricted to North America or Europe, but could also collaborate with programs around the world.
4.
5. 6. 7. 8.
SUMMARY As reproductive medicine and ART continue to evolve, enhanced safety programs should be integral to efforts to optimize outcomes. Creation of a CSB adapted from the aviation safety model provided by the NTSB offers a great opportunity to enhance patient safety while optimizing clinical practice.
REFERENCES 1. 2.
3.
Kohn LT, Corrigan JM, Donaldson MS. To err is human: building a safer health system. Washington, DC: National Academy Press; 2000. Phillips DP, Broeder CC. Morbidity and mortality from medical errors: an increasingly serious public health problem. Annu Rev Public Health 2002; 23:135–50. Land JA, Evers JLH. Risks and complications in assisted reproduction techniques: Report of an ESHRE consensus meeting. Humanit Rep 2003;18: 455–7.
VOL. 100 NO. 6 / DECEMBER 2013
9.
10.
11.
12.
13. 14.
15.
College of American Pathologists. Guidelines of embryology and andrology laboratories. Available at: http://www.cap.org/apps/docs/laboratory_accre ditation/build/pdf/standards_repro.pdf. Accessed August 10, 2013. Helmreich RL. On error management: lessons from aviation. BMJ 2000;18: 781–5. Ricci M, Panos AL, Lincoln J, Salerno TA, Warshauer L. Is aviation a good model to study human errors in health care? Am J Surg 2012;203:798–801. The Commercial Aviation Safety Team. CAST. Available at: http://www.castsafety.org/about_background.cfm. Accessed August 15, 2013. Dismukes K, Loukopoulos L. The limits of expertise: the misunderstood role of pilot error in airline accidents. Available at: http://humansystems.arc.na sa.gov/flightcognition/article2.htm. Accessed August 9, 2013. National Transportation Safety Board. History of the National Transportation Safety Board. Available at: http://www.ntsb.gov/about/history.html. Accessed August 9, 2013. National Transportation Safety Board. Structure of the National Transportation Safety Board. Available at: http://www.ntsb.gov/about/organization. html. Accessed August 9, 2013. National Transportation Safety Board. Legal issues regarding the National Transportation Safety Board. Available at: http://www.ntsb.gov/le gal/gc.html. Accessed August 13, 2013. Denham CR, Sullenberger CB, Quaid DW, Nance JJ. An NTSB for health care learning from innovation: debate and innovate or capitulate. Patient Saf 2012;8:3–14. Hall J. First, make no mistakes. The New York Times 2009:A23. Federal Aviation Administration. Federal Aviation Administration aviation safety action program (ASAP). Available at: http://www.faa.gov/about/ini tiatives/asap/. Accessed August 19, 2013. Federal Aviation Administration. Federal Aviation Administration flight operational quality assurance program (FOQA). Available at: http://www.faa.gov/ about/initiatives/atos/air_carrier/foqa/. Accessed August 19, 2013.
1523
Medicine and aviation have a striking number of similarities. Both are led by highly-trained individuals performing complex tasks that are critical to outcomes. They also integrate the efforts of other professionals to assure that the flight, procedures, or processes are completed successfully. Also in common, is the potential for errors to have catastrophic and even life-threatening consequences. Both aviation and medicine have responded to this complex operating environment by building safety programs. Unfortunately, those in medicine have not been optimal in reducing significant adverse outcomes, including deaths. It has been suggested that given the parallels, that aviation safety programs might be adapted to clinical medicine. One such measure would the formation of a Clinical Safety Board (CSB) modeled after the National Transportation Safety Board (NTSB). Such a board would collect data across the nation and determine root causes of errors. They may then provide recommendations to professional societies and regulatory agencies for consideration for implementation. Such programs would be dependent on accurate and thorough reporting. Indemnification, similar to that enacted by the federal government for aviation, would be critical. In the end, a CSB should Use your smartphone empower better patient care with reduced liability to the providers and programs. (Fertil SterilÒ to scan this QR code 2013;100:1518–23. Ó2013 by American Society for Reproductive Medicine.) and connect to the Key Words: Safety, NTSB, ART, IVF, accident investigation Discuss: You can discuss this article with its authors and with other ASRM members at http:// fertstertforum.com/scottr-safety-ntsb-art-ivf/
F
‘‘
irst do no harm’’ is a central tenet in the practice of medicine, and there is little doubt that all clinical programs seek to attain the best possible outcomes for their patients while minimizing complications. Still adverse sequelae do occur. While some complications or sub-optimal outcomes are intrinsic to indicated medical or surgical treatments, others are the result of human errors. The impact of these errors is astounding. The Institute of Medicine estimated that almost a hundred thousand deaths occur annually secondary to medical treatment errors (1). In response to these challenges, a variety of risk-management and safety
programs have been implemented. Examples might include mandatory safety training programs at facilities where the clinicians are credentialed, the use of ‘‘time-out's’’ prior to the initiation of invasive procedures, and tracking of abnormal laboratory results to assure that they are seen quickly by the patients' physician. In spite of these improvements, the overall prevalence of errors has not changed meaningfully over the last decades. Considered in aggregate, safety program effectiveness has been suboptimal (2). Upon initial consideration, these issues might not seem to apply to reproductive medicine. Our patients are young and healthy and deaths are so
Received September 30, 2013; revised October 21, 2013; accepted October 22, 2013. R.T.S. has nothing to disclose. N.D.Z. has nothing to disclose. Reprint requests: Richard T. Scott Jr., M.D.,140 Allen Road, Basking Ridge, New Jersey 07920 (E-mail: [email protected]). Fertility and Sterility® Vol. 100, No. 6, December 2013 0015-0282/$36.00 Copyright ©2013 American Society for Reproductive Medicine, Published by Elsevier Inc. http://dx.doi.org/10.1016/j.fertnstert.2013.10.033 1518
discussion forum for this article now.*
* Download a free QR code scanner by searching for “QR scanner” in your smartphone’s app store or app marketplace.
exceedingly rare that the vast majority of providers will complete their entire career without having one within their practice (3). The reality is that there are genuine safety issues in assisted reproductive technology (ART). Rare catastrophic events such as severe hemorrhage and pelvic abscesses do occasionally occur. Ovarian hyperstimulation syndrome is another obvious example. Medication errors, misidentification of patients, loss of chain of custody of biological specimens, and use of expired media are just a few of many real-world examples.
THE NATURE OF ERRORS To prevent errors you must first understand them. Most things which impact safety involve some component of human error. While equipment may break or a production lot of media may perform sub-optimally even after VOL. 100 NO. 6 / DECEMBER 2013
Fertility and Sterility® passing quality assurance testing, the reality is that most clinical and laboratory care is performed or controlled by humans. As such, deviations often reflect either human error or a failure of a human to recognize and correct a problem. A traditional model for avoiding errors is to clearly define optimal management in a given clinical circumstance, train providers and staff members on how to execute that algorithm, and then follow up to make sure that protocol has been implemented and providers are within compliance. This approach is amongst the guiding principles for ART laboratory certification used by the College of American Pathologists (4). In brief, define the ‘‘right way’’ in a policy and protocol; implement that protocol, and then check back to assure compliance. If deviations are identified, humans are either counseled, retrained, or punished. Whole programs may be decredentialed for noncompliance. No doubt traditional approaches to safety may be both informative and effective. However, many do not consider the inevitability of human imperfection, seek to define paradigms which reduce the opportunities for human error, or try and minimize the impact of any errors which might occur. They have the simple philosophy of ‘‘you are a highly intelligent and extensively trained professional—do the right thing’’. A broader perspective would be to look for those factors which are prone to human error and then try and reduce or eliminate them. This does not replace the need for human vigilance; it simply states that systems may be put in place which meaningfully reduce the opportunity for errors to occur or to produce harm. An example of this comes from aviation. A few decades ago, a large jet aircraft was rolling down the runway and gathering speed in the process of taking off. The pilot of the aircraft looked over and saw that his copilot, who was having personal problems, looked sad and distracted. Using the intercom, he told the co-pilot to ‘‘cheer up.’’ The copilot was not paying sufficient attention to the progress of the takeoff and did not realize that while the nose of the aircraft had been raised that the main wheels were still on the ground. Being distracted, he thought that the pilot had said ‘‘gear up’’ and he reached over and moved the landing gear control lever to the up position. The landing gear retracted before the pilot could react. The plane crashed and burned and all lives were lost. The story was discovered by listening to the ‘‘black box’’ recording during the accident investigation. Obviously, this is a wholly unacceptable error. Pilots should be attentive at all times but particularly during the critical transitions of take-off and landing. They all certainly know better than to retract the landing gear while the plane is still on the ground. If the traditional safety models commonly utilized in medicine had been applied to this tragedy, the following actions would likely result. Local morbidity and mortality conferences would have presented the facts of the tragedy, and been followed by discussions of the deviations in the ‘‘standard of practice/care’’ which had occurred. All pilots would then be encouraged to pay attention to take-off procedures and not to raise the landing gear until after the plane is clearly in the air. Perhaps procedure manuals would be rewritten to VOL. 100 NO. 6 / DECEMBER 2013
provide greater emphasis on vigilance by both the pilot and copilot during take-off. Certifying examinations might have a question or two added which emphasized the importance of landing gear management sequences. Would any of those things have prevented this accident? Of course not, and those were not the recommendations which resulted. Instead, the aviation accident investigation team suggested that a simple pressure monitor be placed in the landing gear system which precludes the pilot from raising the landing gear if there is still any weight on the wheels. Until the plane is actually flying, the landing gear control lever is locked in place and may not move. This means that no matter what, no pilot could ever make this error again. Safety is not just about teaching people the right thing to do, it is also about creating environments and systems which reduce the risk for or consequences of human error. Several authors have pointed out the similarities between aviation and medicine and have suggested that the types of safety program used in the aviation industry might be also be useful in enhancing the safety of medical care.
AVIATION AND MEDICINE The parallels between aviation and medicine are enormous (Table 1) (5–7). Highly-trained individuals perform high complexity tasks. Successful completion of those tasks frequently requires a great deal of sophisticated equipment and the support from a wide variety of staff/team members with different but critical skills sets. When things are proceeding normally, the tasks may be repetitive and even monotonous. Still, individuals performing the procedures must maintain great overall situational awareness and react quickly to any one of a variety of different emergencies. Finally deviations from ideal (whether the fault of the pilot/ physician or not) can produce catastrophic consequences including significant injury and even loss of life. Aviation has powerful safety tools which have produced dramatic reductions in accidents. Given the rarity of some catastrophic events, data is collected from across the aviation industry as a whole. Systematic evaluation of those data provides insight into the sequence of events that lead to the errors. A truly catastrophic accident is usually a result of a confluence of many errors (8). This type of root-cause analyses has led to recommendations for powerful changes in training, equipment, and operations. How is this system wide data collection, analysis, and recommendation paradigm achieved? By implementation of a National Transportation Safety Board (NTSB) or Bureau Enquete et Analyse which is the French equivalent. These two groups investigate the majority of commercial airline accidents throughout the world since they investigate all accidents involvement Boeing and Airbus aircraft, respectively.
WHAT IS AN NTSB? The roots of the NTSB date back to the 1926 with the Air Commerce Act, but a fully functional NTSB was not truly formed until 1974 when the board was made autonomous from the Department of Transportation (9). Among other responsibilities, it is chartered with investigating every aviation accident. 1519
VIEWS AND REVIEWS
TABLE 1 Comparisons between aviation and reproductive medicine. Factor being compared
Level of similarity
Training
High
Complexity Consequences of errors Coordination of information and services from multiple other specialists Commonality of checklists and procedures throughout the industry Number of providers
High High High
Requirement to function under sub-optimal conditions Witness to errors Indemnified self-reporting systems for errors and ‘‘near-misses’’
Low
Low Low
Low Low
Commercial aviation
Reproductive medicine
High levels with frequent recurrence training High Life-threatening to insignificant Extensive (flight controllers, meteorologists, maintenance, etc.)
High levels with ongoing continuing education and re-certification High Life-threatening to insignificant Extensive (embryologists, andrologists, nurses, molecular geneticists, etc)
Dictated principally by manufacturers; high degree of commonality Low; eight airlines dominate the US commercial airline market Rare; exceptions for some helicopter and search and rescue operations Very high Present through FAA's ASAP program
Developed by individual ART programs; very low degree of commonality High; over 400 independent clinical ART programs Common Very low None
Note: ART, assisted reproductive technology; ASAP, Aviation Safety Action Program; FAA, Federal Aviation Agency. Scott. A clinical safety board for reproductive medicine. Fertil Steril 2013.
The NTSB is a special organization whose unique features empower it to be unusually effective (Table 2) (10). The NTSB was chartered by and is funded through the federal government, but functions nearly autonomously. It is not political and is staffed with the full complement of experts with sufficient expertise to investigate the accidents. These include pilots, engineers, maintenance experts, accident investigators, and a variety of other specialists. The leadership of the NTSB is appointed by the executive branch, but the five member team may not have more than 3 members of either party to minimize any political influence. Perhaps most important, all findings and analyses are not admissible during litigation (11). There are also well defined limits on the power and influence of the NTSB (10). The organization makes recommenda-
TABLE 2 What is a National Transportation Safety Board (NTSB)? Characteristics of the United States NTSB National and international investigative agency Federally chartered and funded Functions independently in ‘‘apolitical’’ environment Staffed with high-qualified individuals capable of investigating all mechanical, environmental, and human factors which contributed to accidents No direct regulatory authority Investigates all aviation accidents Investigates all major accidents in other areas of transportation, pipelines, and hazardous chemicals Collects data from individual events and integrates them with data from other occurrences to make recommendations regarding equipment, training, or operations to enhance safety Creates a most-wanted list of which improvements are most important for the future Supports investigations of other national and international agencies in time of crisis or catastrophe Scott. A clinical safety board for reproductive medicine. Fertil Steril 2013.
1520
tions, but has no authority to create policy or set standards regarding any aspect of aviation or the aviation industry. Their recommendations go back to the Federal Aviation Agency (FAA) or to industry for consideration. As such, the NTSB works closely with industry and regulatory aspects of the government to help assure that implementation is optimal. Perhaps most important, the NTSB has been a major component of a safety system which in aggregate has attained unparalled success. During an interval when there was no meaningful reduction in the prevalence of significant medical errors, there was an 83% reduction in fatal aviation accidents (7).
AN NTSB FOR REPRODUCTIVE MEDICINE Reproductive Medicine provides an excellent opportunity for a medical NTSB, also termed a Clinical Safety Board (CSB) (12, 13). The opportunity to integrate data attained from all the programs in the field would allow accumulation of sufficient information to empower evidenced-based recommendations which might aid in the prevention of even the most uncommon hazards. The importance of an integrated national model cannot be over-stated, most complications are rare enough that it is difficult to accrue valuable insight by considering data from even the largest individual centers. The author recently learned of a case which illustrates the importance of sharing data. A patient with one of the inherited thrombophilia's was advised by a hematologist to be on low molecular weight heparin during their in vitro fertilization cycle. The hematologist insisted that the patient continue their anticoagulation throughout the cycle, including the day of retrieval. The reproductive endocrinologist spoke to the hematologist but ultimately relented and agreed to maintain the anticoagulation. The patient ultimately suffered a rare serious complication from internal bleeding. Anecdotally, other practitioners have reported ‘‘severe hemorrhage’’ in patients receiving VOL. 100 NO. 6 / DECEMBER 2013
Fertility and Sterility® anticoagulation at the time of retrieval. A national CSB would allow accrual of information about these complications and the development of evidence-based recommendations which optimize management and improve safety. A CSB has great potential to work quite well because it would be able to evaluate problems in any clinical setting, not just those which occur in inpatients. Most critical and high-risk health care in other medical specialties occurs in the inpatient setting. As such, hospitals typically assume responsibility and provide direction for safety programs. The majority of reproductive medicine care is provided within the office facilities of the individual practices. These are generally not located within a hospitals inpatient or outpatient facilities. While there are myriad standards for the operation of these facilities, comprehensive safety programs for independent offices are less well-defined. Traditional safety programs rightfully focus on clinical care. Even CSB's modeled after those in the aviation industry (such as those implemented in anesthesia and radiology) are focused on actual clinical care. Implementing a CSB for reproductive medicine would also provide an opportunity to develop insight into safety issues in embryology. Embryology laboratories provide a truly unique and highly specialized clinical service, but are outside of the domain of many institutional clinical safety programs. Here again, a national safety board would allow accrual of actual data regarding repetitive errors which might lead to recommendations that improve the safety and precision of embryology practice. Prior efforts through more conventional regulatory mechanisms have led to burdensome and costly safety programs to prevent complications which had never even occurred. At a time of transition in the way health care is being provided and regulated, a CSB would provide means of focusing on all aspects of reproductive medicine.
BARRIERS TO IMPLEMENTATION While there are a number of similarities, aviation and medicine are also different in important ways. A number of factors would need to be addressed when designing a CSB for reproductive medicine.
THE COMPLEX PROVIDER MIX There are relative few airlines compared to the number of ART clinics. Additionally, each airline operates a limited number of aircraft types. This provides a greater opportunity for uniformity. Beyond that, many of the procedural aspects of aviation, including emergency procedures, are worked out by the manufacturers and not the airlines, providing even greater uniformity. If pilots are following similar procedures throughout the industry, it should be considerably easier to identify those procedures or issues which are prone to error or which may have contributed to accidents. It also makes it easier to implement enhanced safety procedures once a problem is identified—define a single procedural or mechanical change and simply allow all the carriers flying that type of aircraft to incorporate it in an essentially identical fashion. VOL. 100 NO. 6 / DECEMBER 2013
Medicine is much more ‘‘atomized’’. There are a much larger number of groups and each provides a relatively low percentage of the care being provided nationwide. While these programs are highly dependent on their equipment, manufacturers do not typically provide checklists for how they should be used when performing actual clinical care. In contrast, the manufacturers of commercial jets provide checklists that operators adopt for actual flight operations. Flight data recorders are provided by the manufacturers and the information collected in a near identical fashion for all flights of that type aircraft. This simplifies investigation and analyses of industry wide procedural issues. In contrast, individual ART programs design their own charts and collect and record data in their own unique ways. As electronic medical records become uniformly implemented, the limited number of commercial electronic medical records will make data collection more uniform. Unfortunately, procedural differences between programs will continue to make these types of investigations most challenging.
OPERATING IN HIGH-RISK ENVIRONMENTS Helicopter operations may be most analogous to the practice of reproductive medicine. There are a relatively large number of small independent operators, each with their own approach to safety, decision making, and operational tempo. Additionally, they commonly fly in ‘‘must-go’’ situations. While the major airlines delay or cancel flights in bad weather or if the full complement of coordination from the FAA is unavailable (as they should), helicopter operators frequently provide rescue and evacuation services. They are more likely to fly when the weather is extremely poor and thus operate in the most difficult and dangerous circumstances. Helicopters also fly intrinsically complicated and risky missions. Monitoring and repairing massive high-voltage power lines from a flying helicopter, with live power in those lines, is high risk no matter the weather. It might be analogous to dealing with a patient whose cervical ectopic pregnancy is rupturing. Medicine is also analogous to helicopter flying because high risk patients still need care. Only rarely is the risk so high that providers withhold care and refuse to attempt to ‘‘rescue’’ the patient. Accident rates for helicopter operations remain 10- to 15-fold higher than the commercial airlines. In spite of this, enhancements in the safety of helicopter operations have been achieved. Nothing of that order of magnitude has been realized in medicine.
EVENT REPORTING The NTSB model for enhancing safety is critically dependent on one thing, identification and reporting of accidents and near-accidents so they may be investigated. Major accidents in aviation are evident. If a plane runs off a runway or crashes enroute to its destination, an enormous number of people are immediately aware. Pilots, passengers, air traffic controllers, a variety of crew members, and emergency response personnel all know when an accident occurs. Activation of the NTSB accident response team is virtually assured. A great deal may also be learned by evaluation of 1521
VIEWS AND REVIEWS ‘‘near-miss’’ events. Situations where a deviation occurred but no accident or harm was done. These events are typically voluntarily reported by the pilots or flight controllers themselves. In contrast, not all accidents in medicine are so evident. In the case of ART, serious clinical complications such as abscess or embolus would be relatively easily identified and reported, but less serious complications, including those in the in vitro fertilization laboratory, may be less evident. Dropping a dish containing embryos is clearly a very serious event, but may not be witnessed. Self-reporting would be required to ‘‘flag’’ the adverse event. In medicine, voluntary reporting of deviations is uncommon. There are issues of liability, credentialing, reviews in very public venues such as departmental or hospital morbidity and mortality meetings which are commonly attended by business competitors, and a system which is focused on punitive measures once any event has occurred. So what motivates aviation professionals to ‘‘self-identify’’ deviations which occur? The FAA, the regulator of all air operations in the United States, commercial or private, has indeed enacted a non-punitive reporting system of errors and incidents, which is instrumental for enhancing safety. The system known as Aviation Safety Action Program (ASAP) allows pilots (and airline controllers) to report errors committed on the job. These reports guarantee no punitive consequences as long as no rules were blatantly and knowingly infringed (14). Practically, this has provided the FAA with a trove of information on errors in flying (and controlling flying). As might be expected, many errors committed while flying are menial in nature, sporadic in their occurrences, and display no particular pattern of recurrence. Other errors may be repetitive and represent a genuine and recurring risk to air safety. For example, a pilot turning left after departure when a right turn is expected might be at risk of flying into other aircraft in the congested airspace near an airport. Unfortunately, this may rarely occur just because it is in the human nature to make mistakes. Yet, if wrong side turns occur with any frequency, the situation becomes different. The root of the error is likely that something is unclear in the departure instructions and/or charts, which evidently needs prompt attention and remedy. Without implementing ASAP the regulator, the FAA, would never have known that it needed to amend the departure charts at the given airport, unless an accident ultimately occurred. Non-punitive reporting such as enacted by ASAP thus helps prevent accidents by identifying problems and implementing solutions before accidents occur. The incentive for pilots and controllers to file reports with the FAA within the ASAP format is to gain indemnification from possible sanctions for that error. ASAP works because in aviation the risk of sanction is high in case of errors. This stems from the fact that aviation is highly monitored including by automatic reporting of deviations in flight through the Flight Operational Quality Assurance program (15). The latter automatically investigates deviations in flight. Pilots may not use ASAP if they failed to notify in advance of the FAA advising them of an incident or investi1522
gation. It is safer to report the deviation and be protected from punitive measures unless a truly reckless violation occurred. The protection from sanctions is therefore the motivator which propels information into the ASAP system and in turn allowed the FAA, NTSB, and manufacturers to evaluate errors and enact meaningful safety solutions prior to catastrophe.
ENACTING NON-PUNITIVE REPORTING IN MEDICINE Systems enabling non-punitive reporting of errors are urgently needed in medicine. For such system to be effective however, a powerful incentive needs to exist (to be created) as doctors are not inclined to report their errors. The day to day routine practice of medicine is much less supervised than aviation. Every step of every procedure is not continuously monitored. As such, doctors don't fear sanctions for ‘‘near-miss’’ type errors simply because such sanctions rarely occur. If programs like ASAP were translated directly into medicine it is likely that utilization would still be low. A promise of no sanction has little effect when no one with the authority to levy a sanction typically even knows what happened. Selfreporting simply invites additional and unwanted scrutiny. One practical way for effectively creating an incentive for self-reporting errors in medicine in an ASAP-like way would be to make the self-reported data on errors inaccessible (and not usable) in court in case of litigation or disciplinary actions. A well-established mechanism for secluding information related to self-reporting of errors is the attorney-client privilege, which protects clients for all the sensitive information exchanged with their lawyers. The attorney-client privilege is the oldest privilege recognized by Anglo-American jurisprudence. There are precise definitions of its nature and strict definitions about circumstances which constitute exceptions when the privilege may be breached. This would provide protection for providers from regulatory issues, credentialing boards, and civil litigation. Without such protections, self-reporting would not make sense for the clinician and all the insight and opportunities to improve safety will continue to be lost.
FUNDING AND ADMINISTRATION A CSB for reproductive medicine would have real expenses and would likely require either federal or substantial professional society support. While the core staff might be fairly small, funding for travel, investigative tools, and administrative support might still be substantial. It might be possible for the project to be initiated with limited funds from professional societies, industry, or the ART programs themselves. It is likely that legislation would be needed to provide the privacy and indemnification which aviators enjoy at the current time and which is key to meaningful implementation. The NTSB grew out from the Department of Transportation, an organization with an excellent understanding of the issues faced by aviation. Similarly, a CSB for reproductive medicine will likely need affiliation with a professional organization with a deep understanding of this field of VOL. 100 NO. 6 / DECEMBER 2013
Fertility and Sterility® medicine, the safety issues, and the barriers to meaningful improvements. The American Society for Reproductive Medicine and the European Society for Human Reproduction and Embryology would be excellent choices for the United States and Europe, respectively. The focus of these groups need not be restricted to North America or Europe, but could also collaborate with programs around the world.
4.
5. 6. 7. 8.
SUMMARY As reproductive medicine and ART continue to evolve, enhanced safety programs should be integral to efforts to optimize outcomes. Creation of a CSB adapted from the aviation safety model provided by the NTSB offers a great opportunity to enhance patient safety while optimizing clinical practice.
REFERENCES 1. 2.
3.
Kohn LT, Corrigan JM, Donaldson MS. To err is human: building a safer health system. Washington, DC: National Academy Press; 2000. Phillips DP, Broeder CC. Morbidity and mortality from medical errors: an increasingly serious public health problem. Annu Rev Public Health 2002; 23:135–50. Land JA, Evers JLH. Risks and complications in assisted reproduction techniques: Report of an ESHRE consensus meeting. Humanit Rep 2003;18: 455–7.
VOL. 100 NO. 6 / DECEMBER 2013
9.
10.
11.
12.
13. 14.
15.
College of American Pathologists. Guidelines of embryology and andrology laboratories. Available at: http://www.cap.org/apps/docs/laboratory_accre ditation/build/pdf/standards_repro.pdf. Accessed August 10, 2013. Helmreich RL. On error management: lessons from aviation. BMJ 2000;18: 781–5. Ricci M, Panos AL, Lincoln J, Salerno TA, Warshauer L. Is aviation a good model to study human errors in health care? Am J Surg 2012;203:798–801. The Commercial Aviation Safety Team. CAST. Available at: http://www.castsafety.org/about_background.cfm. Accessed August 15, 2013. Dismukes K, Loukopoulos L. The limits of expertise: the misunderstood role of pilot error in airline accidents. Available at: http://humansystems.arc.na sa.gov/flightcognition/article2.htm. Accessed August 9, 2013. National Transportation Safety Board. History of the National Transportation Safety Board. Available at: http://www.ntsb.gov/about/history.html. Accessed August 9, 2013. National Transportation Safety Board. Structure of the National Transportation Safety Board. Available at: http://www.ntsb.gov/about/organization. html. Accessed August 9, 2013. National Transportation Safety Board. Legal issues regarding the National Transportation Safety Board. Available at: http://www.ntsb.gov/le gal/gc.html. Accessed August 13, 2013. Denham CR, Sullenberger CB, Quaid DW, Nance JJ. An NTSB for health care learning from innovation: debate and innovate or capitulate. Patient Saf 2012;8:3–14. Hall J. First, make no mistakes. The New York Times 2009:A23. Federal Aviation Administration. Federal Aviation Administration aviation safety action program (ASAP). Available at: http://www.faa.gov/about/ini tiatives/asap/. Accessed August 19, 2013. Federal Aviation Administration. Federal Aviation Administration flight operational quality assurance program (FOQA). Available at: http://www.faa.gov/ about/initiatives/atos/air_carrier/foqa/. Accessed August 19, 2013.
1523
