TRANSFUSION PRACTICE AABB validation study of the CDC’s National Healthcare Safety Network Hemovigilance Module adverse events definitions protocol James P. AuBuchon,1 Mark Fung,2 Barbee Whitaker,3 and Jacquelyn Malasky3
BACKGROUND: The utility of a hemovigilance system depends on appropriate, reproducible application of system definitions. This is even more important when submissions are not reviewed by an adjudicating body. We sought to determine how participants would code adverse reactions at institutions that had or had not received training on the application of definitions used in the CDC’s National Healthcare Safety Network Hemovigilance Module (HVM). STUDY DESIGN AND METHODS: Facilities that were (11) or were not (11) submitting adverse reaction data to the HVM reviewed 36 hypothetical cases containing elements of 37 case definitions from 12 different diagnostic groups. Respondents were required to determine the type of adverse event, if any, and assign a case definition (diagnostic probability), severity, and imputability using the January 2013 HVM Surveillance Protocol definitions. Those submitting HVM data had access to an instructional slide set prepared by CDC using similar hypothetical cases. Concordance with expert analysis was determined for the two groups of respondents. RESULTS: The frequencies of agreement with the expert assessment were not different according to prior training exposure in any of the diagnostic groups, and results were totaled across both groups. Response accuracy varied by type of categorization (adverse event type, 72.1%; match with case definition, 76.5%; severity, 69.6%; imputablity, 64.4%) and by type of adverse event. CONCLUSION: Despite delineated definitions, considerable variability in responses was seen, and this was not reduced by the available training. This degree of inconsistency in application of the surveillance definitions could degrade the utility of comparative reports.
H
emovigilance systems to capture, analyze, and report untoward outcomes of transfusion have been developed and implemented in many countries.1 These systems have led to process interventions that have improved transfusion recipient safety, such as identification of the role of plasma from female donors in causation of transfusionrelated acute lung injury (TRALI) and avoiding such donors to reduce the risk of this reaction.2 They have also provided some useful insights into the pathophysiology of certain reactions3 and highlighted the types, locations, circumstances, and impact of certain deviations from standard practice (often termed “occurrences” or “incidents”) that may contribute to adverse events in transfusion recipients and increased system cost.4 Successful use of any hemovigilance system depends on consistency of the information it captures and assurance that categorizations of observations are valid. These in turn depend on reporters’ ability to apply the system’s
ABBREVIATIONS: AHTR = acute hemolytic transfusion reaction; DHTR = delayed hemolytic transfusion reaction; HTR = hypotensive transfusion reaction; HVM = Hemovigilance Module; NHSN = National Healthcare Safety Network; PTP = posttransfusion purpura; TA-GVHD = transfusionassociated graft-versus-host disease; TACO = transfusionassociated circulatory overload; TAD = transfusion-associated dyspnea; TTI = transfusion-transmitted infection. From the 1Puget Sound Blood Center, Seattle, Washington; the 2 Department of Pathology and Laboratory Medicine, Fletcher Allen Health Care, Burlington, Vermont; and 3Research and Data Analysis, AABB, Bethesda, Maryland. Address reprint requests to: James P. AuBuchon, MD, Puget Sound Blood Center, 921 Terry Avenue, Seattle, WA 98104; e-mail: [email protected]. Received for publication October 29, 2013; revision received January 18, 2014, and accepted January 19, 2014. doi: 10.1111/trf.12620 © 2014 AABB TRANSFUSION 2014;54:2077-2083. Volume 54, August 2014 TRANSFUSION
2077
AUBUCHON ET AL.
TABLE 1. Adverse events defined in NHSN HVM and number of cases in validation trial* TACO [2] TRALI [3] TAD [4] Allergic reaction [4] HTR [4] Febrile nonhemolytic transfusion reaction [3] AHTR [1] DHTR [4] DSTR [2] TA-GVHD [1] PTP [2] TTI [1] * In addition, six of the 36 hypothetical cases did not meet all the criteria of any of the adverse events defined in the NHSN HVM. One of the cases had data meeting the criteria for reporting two adverse events (PTP, DHTR), and this was treated as two separate cases for the purpose of analysis of the results. DSTR = delayed serologic transfusion reaction.
definitions in an appropriate and reproducible manner. Some countries’ systems include a mechanism whereby initial reports are reviewed to ensure appropriate application of adverse event definitions, as well as imputability and severity scoring systems.5 Additional or clarifying information may also be sought to resolve unclear reports. Fidelity to and appropriate use of the definitions becomes even more important in systems where submissions are not reviewed by an adjudicating body. In such a circumstance, such as the Centers for Disease Control and Prevention’s (CDC) National Healthcare Safety Network (NHSN) Hemovigilance Module (HVM) in the United States, fidelity of the system to record the patient’s circumstances becomes dependent on staff in individual institutions gathering the necessary data elements and then utilizing them correctly to categorize the situation in an accurate and reproducible manner. The ability of such systems to ensure data fidelity and the role of staff training in meeting this goal are unknown. We sought to validate the applicability of the definitions developed for the HVM by challenging participants from transfusion services to apply them to fictional cases. We further attempted to define the role of prior didactic education of these participants in applying the definitions correctly.
MATERIALS AND METHODS Definitions used in the HVM were developed by a working group of transfusion medicine experts commissioned under the auspices of the AABB Interorganizational Biovigilance Network Task Force. Patterned after a similar set of definitions developed by the ISBT Hemovigilance Working Party, they defined 12 distinct adverse events (Table 1) and delineated specific clinical, temporal, labo2078
TRANSFUSION Volume 54, August 2014
ratory, and/or radiologic variables associated with each defined adverse event type classifying it into one of these categories as a definitive, probable, or possible example. (An “other/unknown” category of event was also created to allow reporting of events not meeting all the criteria of one of the defined event types.) Each type of adverse event has a set of definitions intended to be used to categorize the severity of the event (as nonsevere, severe, lifethreatening, or death). Similarly, rules for imputability (the likelihood of the transfusion to have caused the observed reaction) categorization are defined (definite, probable, possible, doubtful, not determined, or ruled out). These definition sets have subsequently undergone minor modifications by CDC staff and are published through the CDC Web site (http://www.cdc.gov/nhsn/ PDFs/Biovigilance/BV-HV-protocol-current.pdf). To test the applicability of case definition, severity, and imputability definitions, 36 fictional cases were developed (Appendix S1, available as supporting information in the online version of this paper). These clinical summaries ranged in length from 35 to 336 words and included most, but not necessarily all, elements of the definitions provided by the HVM. One case included two distinct types of adverse events, as defined by the HVM. All cases were patterned after ones that had been developed through the ISBT Hemovigilance Working Party for similar purposes. An earlier subset of these cases had been used in a set of slides developed by CDC to illustrate the proper application of the definitions and had been offered at seminars and on-line as an educational tool; the cases involved in this validation study were different from those used for the CDC’s training purposes, however, to ensure that participants did not have prior knowledge of or exposure to the cases. We contacted 12 hospitals that were participating in the HVM and 16 that were not and invited them to participate in this validation trial. All of those participating in the HVM had been exposed to, or had access to, the training material developed by the CDC and were actively contributing cases to the system. Of the 11 institutions that were contributing data to the HVM and participated in this trial, seven (64%) were academic institutions. (Nine of these 11 answered a question whether the respondent in this trial had personally participated in the available training, and all nine responded affirmatively.) Of the 11 that participated in the trial but were not contributing data to the HVM, eight (73%) were academic institutions (p = 0.647 by chi-square; see acknowledgments for listing of participants). In March 2013, participants received the 36 cases and the then-current (January 2013) NHSN HVM “manual” with instructions to apply the definitions of the manual in determining the appropriate adverse event diagnosis, degree of match with the case definition, severity, and imputability. Participants were reminded to apply the
VALIDATION OF ADVERSE EVENT DEFINITIONS
definitions from the manual rather than any internal criteria used in the management of transfusion reactions among their own patients. The individuals performing the assessment were those who regularly performed this task or, if the institution was not participating in the HVM, who regularly signed out transfusion reaction reports. Some institutions spread the responses among multiple individuals, but only one response was submitted from each institution. All responses were received within 6 weeks. The intended responses to the survey (“expert assessments”) were provided by the author of the cases (JPA) and another author with extensive knowledge of the HVM and the system’s definitions (MF). The frequency with which the type of adverse reaction (“diagnosis”), its degree of matching the case definition, the severity of the reaction, and the imputability (likelihood of transfusion as its cause) matched the expert assessment was totaled for both groups of respondents and all respondents. In those situations where the most frequent response did not match the expert assessment, the most frequent response was identified and included in the results table (Table 2). Missing responses reduced the denominator for that case and response type. To investigate whether a respondent being a part of the HVM network was associated with a higher probability of agreement with the expert assessment for matching with case definition, severity, or imputability, only those responses with a diagnosis matching the expert assessment were considered. A one-way analysis of variance with Tukey’s studentized range analysis was performed with a t test to determine if there was a significant (p < 0.05) difference in the number of correct answers between those contributing data to the HVM and those not contributing for diagnosis, case definition, severity, and imputability. A Fisher’s exact test was performed to determine the significance (p < 0.05) of the diagnosis, degree of match with the case definition, severity, and imputability matching the expert assessment among the two groups.6 There were six cases that did not contain all the elements of any of the adverse event types defined in the HVM. In these cases, designated as “no diagnosis” in Table 2, responses of “no reaction,” “other,” or “not related” were regarded as matching the expert assessment. In addition, a respondent providing the diagnosis that was most probable on medical grounds along with designation of the probability as “not meeting criteria” was regarded as matching the expert assessment. In these six cases, the severity and imputability responses were not assessed.
RESULTS The 36 clinical cases included 29 with a single definable adverse reaction according to HVM criteria, one with two definable reactions, and six with information that did not
fully meet any set of the HVM diagnostic criteria. Thus there were 31 definable reactions for respondents to identify and report as well as the six cases that described a posttransfusion event that did not meet any set of HVM criteria. The evaluation of these 36 cases by participants resulted in submission of 723 responses regarding the adverse events of which 521 (72.1%) were judged as matching the expert assessments (Table 2). The proportion of diagnoses judged as matching ranged from 4.5% (two cases) to 100% (three cases). After grouping the cases into 13 sets according to the type of adverse event, the allergic, febrile, transfusion-associated graft-versus-host disease (TA-GVHD), TRALI, acute hemolytic transfusion reaction (AHTR), delayed hemolytic transfusion reaction (DHTR), and posttransfusion purpura (PTP) groups had aggregate matching frequencies of higher than 70%. The transfusion-associated dyspnea (TAD) and transfusionassociated circulatory overload (TACO) groupings had the lowest aggregate matching response frequencies at 36% each. Across all cases presented to the two groups, there was no difference in the frequency with which the two groups’ responses matched the expert assessments. Respondents contributing data to the HVM matched the expert assessments in 660 of the 1011 responses (65.3%). For the group of respondents not participating in the HVM, 768 of 1097 responses (70.0%) matched the expert assessment (p = 0.06). Within the 13 diagnostic groups, there was no significant difference in the frequency of the diagnostic determination matching the expert assessment between those respondents who were participating in the HVM and those who were not (data not shown). However, there was one case where there was a difference in the frequency between the two groups of respondents. In Case 34, nine of 11 respondents not in the HVM categorized the adverse event correctly as a transfusion-transmitted infection (TTI), whereas only two of 11 HVM participants did so (p = 0.009). There were four cases where the most common diagnosis submitted did not match the expert assessment. Two of these were assessed by the experts as TAD cases. These cases contained two of the six criteria for a diagnosis of TACO according to the HVM system’s surveillance definition but lacked a third criterion that was necessary for this diagnosis. In one case of TAD (Case 6), the transfusion recipient’s record did not include at least three of the six defined signs of TACO, such as information indicating a positive fluid balance, required for a TACO diagnosis in the HVM system, although 95.5% of respondents assigned a diagnosis of TACO. Another case of TAD (Case 12) included events of an obvious mistransfusion of red blood cells (RBCs) but included no signs or symptoms attributable to hemolysis despite the transfusion being ABO incompatible; there were respiratory symptoms that could be categorized as TAD according to the HVM definitions. Volume 54, August 2014 TRANSFUSION
2079
2080
Allergic
Allergic
Allergic
Allergic
2
3
27
36
TRANSFUSION Volume 54, August 2014
Febrile
17
22
TRALI
TRALI
10
28
22
DHTR
DHTR
DHTR
DHTR
13
15
20
25B
22
22
22
22
Total for diagnostic group 23 AHTR
22
22
Total for diagnostic group 9 TRALI
TACO
22
22
22
TAD
30
22
29
TAD
12
22
22
22
21
22
22
22
21
22
Responses
Total for diagnostic group 8 TACO
TAD
6
Total for diagnostic group 5 TAD
Febrile
7
Total for diagnostic group 4 Febrile
Expert assessment
Case
22 100.0% 10 47.6% 21 95.5% 9 40.9% 71.3% 22 100.0% 4 19.0% 21 95.5% 72.3% 14 63.6% 2 9.1% 1 4.5% 15 68.2% 36.4% 10 45.5% 6 27.3% 36.4% 14 63.6% 18 81.8% 19 86.4% 77.3% 18 81.8% 21 63.6% 14 63.6% 17 77.3% 20 90.9%
Agreement with expert assessment
Diagnosis
TRALI
AHTR
TACO
NR; TRALI
Most common diagnosis if different than expert assessment
72.7%
81.8%
95.5%
28.6%; 28.6%
Agreement with most common diagnosis
PRO
DEF
PRO
DEF
PRO
DEF
DEF
DEF
DEF
DEF
DEF
DEF
DEF
DEF
DEF
DEF
DEF
DEF
DEF
DEF
DEF
Expert assessment
20
17
14
21
18
19
18
14
6
10
15
1
2
14
21
4
22
9
20
10
21
Responses with correct diagnosis 20 95.2% 8 80.0% 13 65.0% 8 88.9% 81.7% 20 90.9% 3 75.0% 17 81.0% 85.1% 8 57.1% 2 100.0% 1 100.0% 12 80.0% 71.9% 10 100.0% 3 50.0% 81.3% 14 100.0% 14 77.8% 13 68.4% 80.4% 4 22.2% 20 63.6% 7 50.0% 15 88.2% 10 50.0%
Agreement with expert assessment
Match with case definition
SEV
SEV
NS
LT
LT
LT
SEV
S
D
NS
NS
NS
NS
NS
D
NS
S
NS
S
NS
Expert Assessment
TABLE 2. Results of studies, by case
20
17
14
21
17
18
18
14
6
10
14
1
2
14
21
4
22
9
20
10
21
Responses
Severity
17 81.0% 3 30.0% 19 95.0% 7 77.8% 76.7% 22 100.0% 2 50.0% 20 95.2% 93.6% 11 78.6% 1 50.0% 0 0.0% 14 100.0% 83.9% 9 90.0% 0 0.0% 56.3% 13 92.9% 10 55.6% 13 55.0% 72.0% 15 88.2% 21 63.6% 11 78.6% 1 5.9% 12 40.0%
Agreement with expert assessment
DEF
DEF
DEF
DEF
POS
DEF
DEF
DEF
DEF
POS
DEF
DEF
DEF
DEF
POS
DEF
POS
DEF
DEF
DEF
Expert Assessment
20
17
14
21
17
18
18
14
6
10
14
1
2
14
21
4
21
9
20
10
21
Responses
Imputability
17 81.0% 0 0.0% 14 70.0% 3 33.3% 56.7% 14 66.7% 2 50.0% 14 66.7% 65.2% 6 42.9% 1 50.0% 1 100.0% 4 28.6% 38.7% 6 60.0% 3 50.0% 56.3% 12 85.7% 2 11.1% 11 70.0% 50.0% 17 100.0% 20 63.6% 11 78.6% 15 88.2% 17 85.0%
Agreement with expert assessment
AUBUCHON ET AL.
22
21
22
HTR
HTR
HTR
19
24
33
PTP
31
No diagnosis
No diagnosis
No diagnosis
No diagnosis
16
22
26
35
723
22
22
22
20
22
22
22
521 72.1%
81.8% 13 59.1% 15 68.2% 63.6% 10 45.5% 12 54.5% 1 4.5% 22 100.0% 51.1% 11* 50.0% 14 63.6% 20 90.9% 77.3% 20 90.9% 21 95.5% 4 18.2% 13 65.0% 21 95.5% 6 27.3% 20 90.9% 66.2%
Agreement with expert assessment
AHTR
TTI
Most common diagnosis if different than expert assessment
40.9%
95.5%
Agreement with most common diagnosis
PRO
POS
DEF
DEF
DEF
DEF
DEF
DEF
DEF
DEF
Expert assessment
434
20
20
14
11
22
1
12
10
15
13
Responses with correct diagnosis
332 76.5%
1 100.0% 21 95.5% 88.9% 10 90.9% 13 92.9% 10 50.0% 67.6% 12 60.0%
72.2% 13 100.0% 12 80.0% 89.3% 8 80.0% 10
Agreement with expert assessment
Match with case definition
TABLE 2. Continued
D
NS
SEV
D
D
D
SEV
LT
DEF
ND
Expert Assessment
425
20
20
14
11
22
1
11
10
12
11
Responses
Severity
308 72.5%
1 100.0% 21 95.5% 70.5% 9 81.8% 11 78.6% 11 55.0% 64.7% 19 95.0%
62.5% 6 54.5% 0 0.0% 26.1% 9 90.0% 0
Agreement with expert assessment
DEF
DEF
PRO
DOU
PRO
PRO
POS
PRO
ND
DEF
Expert Assessment
D = death; DEF = definite; DOU = doubtful; DSTR = delayed serologic transfusion reaction; LT = life-threatening; ND = not determined; NS = nonsevere; POS = possible; PRO = probable; S = severe.
Total for diagnostic group Across all diagnostic groups:
No diagnosis
No diagnosis
11
1
Total for diagnostic group 32 TA-GVHD
22
PTP
25A
22
22
Total for diagnostic group 34 TTI
22
22
22
Total for diagnostic group 18 HTR
DSTR
22
Responses
Total for diagnostic group 14 DSTR
Case
Expert assessment
Diagnosis
427
20
20
14
11
22
1
12
10
12
13
Responses
Imputability
275 64.4%
87.5% 12 92.3% 4 33.3% 64.0% 4 40.0% 5 41.6% 1 100.0% 9 40.9% 42.2% 8 72.7% 8 57.1% 17 85.0% 73.5% 17 85.0%
Agreement with expert assessment
VALIDATION OF ADVERSE EVENT DEFINITIONS
Volume 54, August 2014 TRANSFUSION
2081
AUBUCHON ET AL.
However, 81.8% of respondents submitted an AHTR diagnosis even though the criteria for HVM system’s surveillance definition AHTR were not met. (The event could have been reported as an “incident” in the HVM system, but there was no adverse outcome for the patient related to the mistransfusion.) In a case of TACO (Case 29), where the patient had received a large volume of plasma and developed pulmonary distress, the TRALI criteria of hypoxemia were not met, although 72.7% of respondents picked this diagnosis. In one complicated case (Case 7), 28.6% of respondents believed that the TRALI criteria had been met despite the lack of a posttransfusion chest X-ray, and the same proportion felt that no condition’s criteria had been met; a smaller proportion (19%) correctly identified that the criteria of a febrile reaction had been met. There were no cases where the most common response for the diagnosis matching with the case definition, the severity grading, or imputability assessment did not match the expert assessment among those cases where the type of adverse event submitted by the respondent was in agreement with the expert assessment. Case 24 illustrated what appeared to be a common difficulty in comparing clinical circumstances against the surveillance definitions of the HVM. Case 24 was created to illustrate an easily recognizable case of sepsis after transfusion of a RBC unit that was later confirmed as being bacterially contaminated. All but one of the respondents categorized this as a TTI; however, the HVM criteria for this diagnosis require documentation of the presence of the pathogen in the recipient, and no blood culture results were provided in the simulated case. Thus, no categorization of this event as a TTI could be made in the HVM despite the obvious clinical diagnosis. Because of the hypotension, a diagnosis of hypotensive transfusion reaction (HTR) could be made, and an assessment could be provided by one of the respondents; this case is listed in Table 2 as a case of HTR. The two experts had also initially misapplied the HVM criteria for TTI in this case until reminded of the specifics of the criteria by one of the respondents. Most of the cases met the criteria for “definite” assignment of the adverse event type, and 76.5% of responses matched the expert assessment in this category. Aggregate scores for severity (69.6%) and imputability (64.4%) were not different by participant group when analyzed in each case separately.
DISCUSSION Assignment of diagnoses, degrees of matching with the case definition, severities, and imputabilities according to the criteria of the HVM system was achieved, overall, approximately two-thirds of the time and fell far below that mark in many cases. Even more troubling was the observation that accurate application of the definitions 2082
TRANSFUSION Volume 54, August 2014
did not improve with prior participation in or access to training in their application. Despite admonitions to participants to carefully apply the specific criteria in the HVM manual in making the assessments in this validation trial rather than applying their medical knowledge and experience, the requirements of the HVM system’s surveillance definitions were not always applied appropriately by respondents. In some of these situations, one could assume that the tendency to apply “medical rationality” overcame the request to apply the system’s criteria strictly. Furthermore, in real-life situations, ambiguities or incomplete data in the fictional cases could have been investigated further to clarify the issues necessary to match the facts with the criteria in the HVM’s definitions. However, a diagnosis consistent with the HVM manual’s definitions was applied by at least half of the respondents in only three-quarters of the cases. There was one case where the adverse event type designated by the two groups of respondents differed. Those not participating in the HVM were more likely to have assigned the same diagnosis as the experts. We do not believe that this should be taken as evidence that the training provided to HVM participants was confusing or counterproductive given that this represented only one of 126 comparisons made between the groups, the remainder not showing a difference in accurate application of the definitions between them. Instead, we take this as a further indication of the complexity involved in applying detailed surveillance definitions. The fact that this case and another that involved apparent TTI (but that failed to satisfy the HVM TTI criteria and caused even the initial “expert assessment” to be incorrect) underlines the importance of ensuring accurate assignment of categories given the attention that TTI reports are likely to garner when results from the HVM are published. This failure to precisely apply the system’s definitions poses significant difficulties in the analysis of data that this system will generate. Some of this problem has been managed in other systems by lumping certain severities or imputabilities (e.g., definite and probable) when aggregating data. However, if participants are not able to assign diagnoses according to the system’s definitions, interpretation of the data generated by the system will be impaired. Different steps could be taken to solve this dilemma. Additional training and examples could be offered to HVM participants, but resources would have to be dedicated not only to provide these but also to take advantage of them. Such training should reinforce that the system requires the application of surveillance—not clinical— definitions, and the “diagnosis” assigned in a transfusion reaction report may differ from that applicable in this system. However, when such differences arise, it is important to explore the basis of these differences and to ensure whether the clinical or the surveillance report should be
VALIDATION OF ADVERSE EVENT DEFINITIONS
revised. Although the HVM definitions and criteria are intended to facilitate uniformity of data reported into the HVM, these are ultimately clinically derived categories and definitions and have merit in being used clinically where appropriate, as they promote a standardization of our reporting and communications clinically. Given the desired size of the reporting network in the United States, using a central—or even a regional—office to recheck the facts surrounding even the most severe or important adverse events that were reported, for example, TTI and TRALI, would probably strain even the most optimistic predictions of funding availability. An alternative to the latter approach might be to reprogram the Web-based reporting system to require entry of key observations that make up the diagnostic criteria so that, in essence, the system assigns the diagnosis, severity, and imputability score rather than depending on the reporter to apply the manual’s details correctly. A separate system to perform such assignments could also be created. Modifying or clarifying the elements of the definitions may also improve the precision of their application, particularly when the criteria for more than one type of adverse reaction appear to be satisfied, and this along with additional training for nonphysician personnel as reporters might allow reporting without “medical insight” confounding application of the definitions. (Recent revision of the manual to allow inclusion of “possible” AHTR and TTI situations would have allowed the most frequent diagnosis made in two cases [Case 12 and Case 24] to be reported in alignment with clinical thinking, and further, such modifications may be helpful.) Finally, proficiency testing should be considered for those who assign the categorizations. One or more of these will need to be implemented— and validated for effectiveness—before the HVM results can be relied on. ACKNOWLEDGMENTS This study would not have been possible without the many participants who took the time to review the cases and submit their responses: Sharlene Billiet, Genesis Medical Center; Robertson Davenport, University of Michigan; Meghan Delaney, Seattle Children’s Hospital–Puget Sound Blood Center; Sunny Dzik, Massachusetts General Hospital; Brenda Grossman, Washington University St Louis; Andrew Heaton, North Shore Long Island Jewish Hospital; Michele Herman, Beth Israel Deaconess Medical Center; Christi Marshall, Lisa Shifflet, Monica Pagano, and Paul Ness, The Johns Hopkins Hospital; Scott Kirkley, University of
Rochester; Jeff McCullough and Claudia Cohn, University of Minnesota; Robert Ranlett, Inland Northwest Blood Center Transfusion Service; Tom Lane, University of California, San Diego; Kashmira Patel and Grace Tenorio, Robert Wood Johnson University Hospital; Ursula Pedersen and Rachel Elder, Crouse Hospital; Beth Polstra, Children’s Healthcare of Atlanta; John Roback, Emory University; S. Gerald Sandler, MedStar Georgetown University Hospital; Sarah Sewall and Mary Kickbush, Aspirus Wausau Hospital; Marilyn Starzewski, Holy Family Memorial Hospital; Joseph Sweeney, Lifespan; Lynne Thompson, Memorial Health University Medical Center; and Ziggy Szczepiorkowski, Dartmouth-Hitchcock Medical Center.
CONFLICT OF INTEREST The authors report no conflicts of interest or funding sources.
REFERENCES 1. Engelfriet CP, Reesink HW. Haemovigilance. Vox Sang 2006;90:207-41. 2. Serious Hazards of Transfusion Steering Committee. Annual reports from the United Kingdom’s Serious Hazards of Transfusion hemovigilance system. [cited 2013 Oct 26]. Available from: http://www.shotuk.org 3. Williamson LM, Stainsby D, Jones H, et al. The impact of universal leukodepletion of the blood supply on hemovigilance reports of posttransfusion purpura and transfusion-associated graft-versus-host disease. Transfusion 2007;47:1455-67. 4. Callum JL, Robillard P, Hyson C, et al. The Canadian transfusion error surveillance system (TESS): results from the first two years of a national pilot project. Transfusion 2007; 47:1A. 5. Stainsby D, Jones H, Asher D, et al.; on behalf of the SHOT Steering Group. Serious hazards of transfusion: a decade of hemovigilance in the UK. Transfus Med Rev 2006;20:27382. 6. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159-74.
SUPPORTING INFORMATION Additional Supporting Information may be found in the online version of this article at the publisher’s Web site: Appendix S1. Scenarios for validation
Volume 54, August 2014 TRANSFUSION
2083
BACKGROUND: The utility of a hemovigilance system depends on appropriate, reproducible application of system definitions. This is even more important when submissions are not reviewed by an adjudicating body. We sought to determine how participants would code adverse reactions at institutions that had or had not received training on the application of definitions used in the CDC’s National Healthcare Safety Network Hemovigilance Module (HVM). STUDY DESIGN AND METHODS: Facilities that were (11) or were not (11) submitting adverse reaction data to the HVM reviewed 36 hypothetical cases containing elements of 37 case definitions from 12 different diagnostic groups. Respondents were required to determine the type of adverse event, if any, and assign a case definition (diagnostic probability), severity, and imputability using the January 2013 HVM Surveillance Protocol definitions. Those submitting HVM data had access to an instructional slide set prepared by CDC using similar hypothetical cases. Concordance with expert analysis was determined for the two groups of respondents. RESULTS: The frequencies of agreement with the expert assessment were not different according to prior training exposure in any of the diagnostic groups, and results were totaled across both groups. Response accuracy varied by type of categorization (adverse event type, 72.1%; match with case definition, 76.5%; severity, 69.6%; imputablity, 64.4%) and by type of adverse event. CONCLUSION: Despite delineated definitions, considerable variability in responses was seen, and this was not reduced by the available training. This degree of inconsistency in application of the surveillance definitions could degrade the utility of comparative reports.
H
emovigilance systems to capture, analyze, and report untoward outcomes of transfusion have been developed and implemented in many countries.1 These systems have led to process interventions that have improved transfusion recipient safety, such as identification of the role of plasma from female donors in causation of transfusionrelated acute lung injury (TRALI) and avoiding such donors to reduce the risk of this reaction.2 They have also provided some useful insights into the pathophysiology of certain reactions3 and highlighted the types, locations, circumstances, and impact of certain deviations from standard practice (often termed “occurrences” or “incidents”) that may contribute to adverse events in transfusion recipients and increased system cost.4 Successful use of any hemovigilance system depends on consistency of the information it captures and assurance that categorizations of observations are valid. These in turn depend on reporters’ ability to apply the system’s
ABBREVIATIONS: AHTR = acute hemolytic transfusion reaction; DHTR = delayed hemolytic transfusion reaction; HTR = hypotensive transfusion reaction; HVM = Hemovigilance Module; NHSN = National Healthcare Safety Network; PTP = posttransfusion purpura; TA-GVHD = transfusionassociated graft-versus-host disease; TACO = transfusionassociated circulatory overload; TAD = transfusion-associated dyspnea; TTI = transfusion-transmitted infection. From the 1Puget Sound Blood Center, Seattle, Washington; the 2 Department of Pathology and Laboratory Medicine, Fletcher Allen Health Care, Burlington, Vermont; and 3Research and Data Analysis, AABB, Bethesda, Maryland. Address reprint requests to: James P. AuBuchon, MD, Puget Sound Blood Center, 921 Terry Avenue, Seattle, WA 98104; e-mail: [email protected]. Received for publication October 29, 2013; revision received January 18, 2014, and accepted January 19, 2014. doi: 10.1111/trf.12620 © 2014 AABB TRANSFUSION 2014;54:2077-2083. Volume 54, August 2014 TRANSFUSION
2077
AUBUCHON ET AL.
TABLE 1. Adverse events defined in NHSN HVM and number of cases in validation trial* TACO [2] TRALI [3] TAD [4] Allergic reaction [4] HTR [4] Febrile nonhemolytic transfusion reaction [3] AHTR [1] DHTR [4] DSTR [2] TA-GVHD [1] PTP [2] TTI [1] * In addition, six of the 36 hypothetical cases did not meet all the criteria of any of the adverse events defined in the NHSN HVM. One of the cases had data meeting the criteria for reporting two adverse events (PTP, DHTR), and this was treated as two separate cases for the purpose of analysis of the results. DSTR = delayed serologic transfusion reaction.
definitions in an appropriate and reproducible manner. Some countries’ systems include a mechanism whereby initial reports are reviewed to ensure appropriate application of adverse event definitions, as well as imputability and severity scoring systems.5 Additional or clarifying information may also be sought to resolve unclear reports. Fidelity to and appropriate use of the definitions becomes even more important in systems where submissions are not reviewed by an adjudicating body. In such a circumstance, such as the Centers for Disease Control and Prevention’s (CDC) National Healthcare Safety Network (NHSN) Hemovigilance Module (HVM) in the United States, fidelity of the system to record the patient’s circumstances becomes dependent on staff in individual institutions gathering the necessary data elements and then utilizing them correctly to categorize the situation in an accurate and reproducible manner. The ability of such systems to ensure data fidelity and the role of staff training in meeting this goal are unknown. We sought to validate the applicability of the definitions developed for the HVM by challenging participants from transfusion services to apply them to fictional cases. We further attempted to define the role of prior didactic education of these participants in applying the definitions correctly.
MATERIALS AND METHODS Definitions used in the HVM were developed by a working group of transfusion medicine experts commissioned under the auspices of the AABB Interorganizational Biovigilance Network Task Force. Patterned after a similar set of definitions developed by the ISBT Hemovigilance Working Party, they defined 12 distinct adverse events (Table 1) and delineated specific clinical, temporal, labo2078
TRANSFUSION Volume 54, August 2014
ratory, and/or radiologic variables associated with each defined adverse event type classifying it into one of these categories as a definitive, probable, or possible example. (An “other/unknown” category of event was also created to allow reporting of events not meeting all the criteria of one of the defined event types.) Each type of adverse event has a set of definitions intended to be used to categorize the severity of the event (as nonsevere, severe, lifethreatening, or death). Similarly, rules for imputability (the likelihood of the transfusion to have caused the observed reaction) categorization are defined (definite, probable, possible, doubtful, not determined, or ruled out). These definition sets have subsequently undergone minor modifications by CDC staff and are published through the CDC Web site (http://www.cdc.gov/nhsn/ PDFs/Biovigilance/BV-HV-protocol-current.pdf). To test the applicability of case definition, severity, and imputability definitions, 36 fictional cases were developed (Appendix S1, available as supporting information in the online version of this paper). These clinical summaries ranged in length from 35 to 336 words and included most, but not necessarily all, elements of the definitions provided by the HVM. One case included two distinct types of adverse events, as defined by the HVM. All cases were patterned after ones that had been developed through the ISBT Hemovigilance Working Party for similar purposes. An earlier subset of these cases had been used in a set of slides developed by CDC to illustrate the proper application of the definitions and had been offered at seminars and on-line as an educational tool; the cases involved in this validation study were different from those used for the CDC’s training purposes, however, to ensure that participants did not have prior knowledge of or exposure to the cases. We contacted 12 hospitals that were participating in the HVM and 16 that were not and invited them to participate in this validation trial. All of those participating in the HVM had been exposed to, or had access to, the training material developed by the CDC and were actively contributing cases to the system. Of the 11 institutions that were contributing data to the HVM and participated in this trial, seven (64%) were academic institutions. (Nine of these 11 answered a question whether the respondent in this trial had personally participated in the available training, and all nine responded affirmatively.) Of the 11 that participated in the trial but were not contributing data to the HVM, eight (73%) were academic institutions (p = 0.647 by chi-square; see acknowledgments for listing of participants). In March 2013, participants received the 36 cases and the then-current (January 2013) NHSN HVM “manual” with instructions to apply the definitions of the manual in determining the appropriate adverse event diagnosis, degree of match with the case definition, severity, and imputability. Participants were reminded to apply the
VALIDATION OF ADVERSE EVENT DEFINITIONS
definitions from the manual rather than any internal criteria used in the management of transfusion reactions among their own patients. The individuals performing the assessment were those who regularly performed this task or, if the institution was not participating in the HVM, who regularly signed out transfusion reaction reports. Some institutions spread the responses among multiple individuals, but only one response was submitted from each institution. All responses were received within 6 weeks. The intended responses to the survey (“expert assessments”) were provided by the author of the cases (JPA) and another author with extensive knowledge of the HVM and the system’s definitions (MF). The frequency with which the type of adverse reaction (“diagnosis”), its degree of matching the case definition, the severity of the reaction, and the imputability (likelihood of transfusion as its cause) matched the expert assessment was totaled for both groups of respondents and all respondents. In those situations where the most frequent response did not match the expert assessment, the most frequent response was identified and included in the results table (Table 2). Missing responses reduced the denominator for that case and response type. To investigate whether a respondent being a part of the HVM network was associated with a higher probability of agreement with the expert assessment for matching with case definition, severity, or imputability, only those responses with a diagnosis matching the expert assessment were considered. A one-way analysis of variance with Tukey’s studentized range analysis was performed with a t test to determine if there was a significant (p < 0.05) difference in the number of correct answers between those contributing data to the HVM and those not contributing for diagnosis, case definition, severity, and imputability. A Fisher’s exact test was performed to determine the significance (p < 0.05) of the diagnosis, degree of match with the case definition, severity, and imputability matching the expert assessment among the two groups.6 There were six cases that did not contain all the elements of any of the adverse event types defined in the HVM. In these cases, designated as “no diagnosis” in Table 2, responses of “no reaction,” “other,” or “not related” were regarded as matching the expert assessment. In addition, a respondent providing the diagnosis that was most probable on medical grounds along with designation of the probability as “not meeting criteria” was regarded as matching the expert assessment. In these six cases, the severity and imputability responses were not assessed.
RESULTS The 36 clinical cases included 29 with a single definable adverse reaction according to HVM criteria, one with two definable reactions, and six with information that did not
fully meet any set of the HVM diagnostic criteria. Thus there were 31 definable reactions for respondents to identify and report as well as the six cases that described a posttransfusion event that did not meet any set of HVM criteria. The evaluation of these 36 cases by participants resulted in submission of 723 responses regarding the adverse events of which 521 (72.1%) were judged as matching the expert assessments (Table 2). The proportion of diagnoses judged as matching ranged from 4.5% (two cases) to 100% (three cases). After grouping the cases into 13 sets according to the type of adverse event, the allergic, febrile, transfusion-associated graft-versus-host disease (TA-GVHD), TRALI, acute hemolytic transfusion reaction (AHTR), delayed hemolytic transfusion reaction (DHTR), and posttransfusion purpura (PTP) groups had aggregate matching frequencies of higher than 70%. The transfusion-associated dyspnea (TAD) and transfusionassociated circulatory overload (TACO) groupings had the lowest aggregate matching response frequencies at 36% each. Across all cases presented to the two groups, there was no difference in the frequency with which the two groups’ responses matched the expert assessments. Respondents contributing data to the HVM matched the expert assessments in 660 of the 1011 responses (65.3%). For the group of respondents not participating in the HVM, 768 of 1097 responses (70.0%) matched the expert assessment (p = 0.06). Within the 13 diagnostic groups, there was no significant difference in the frequency of the diagnostic determination matching the expert assessment between those respondents who were participating in the HVM and those who were not (data not shown). However, there was one case where there was a difference in the frequency between the two groups of respondents. In Case 34, nine of 11 respondents not in the HVM categorized the adverse event correctly as a transfusion-transmitted infection (TTI), whereas only two of 11 HVM participants did so (p = 0.009). There were four cases where the most common diagnosis submitted did not match the expert assessment. Two of these were assessed by the experts as TAD cases. These cases contained two of the six criteria for a diagnosis of TACO according to the HVM system’s surveillance definition but lacked a third criterion that was necessary for this diagnosis. In one case of TAD (Case 6), the transfusion recipient’s record did not include at least three of the six defined signs of TACO, such as information indicating a positive fluid balance, required for a TACO diagnosis in the HVM system, although 95.5% of respondents assigned a diagnosis of TACO. Another case of TAD (Case 12) included events of an obvious mistransfusion of red blood cells (RBCs) but included no signs or symptoms attributable to hemolysis despite the transfusion being ABO incompatible; there were respiratory symptoms that could be categorized as TAD according to the HVM definitions. Volume 54, August 2014 TRANSFUSION
2079
2080
Allergic
Allergic
Allergic
Allergic
2
3
27
36
TRANSFUSION Volume 54, August 2014
Febrile
17
22
TRALI
TRALI
10
28
22
DHTR
DHTR
DHTR
DHTR
13
15
20
25B
22
22
22
22
Total for diagnostic group 23 AHTR
22
22
Total for diagnostic group 9 TRALI
TACO
22
22
22
TAD
30
22
29
TAD
12
22
22
22
21
22
22
22
21
22
Responses
Total for diagnostic group 8 TACO
TAD
6
Total for diagnostic group 5 TAD
Febrile
7
Total for diagnostic group 4 Febrile
Expert assessment
Case
22 100.0% 10 47.6% 21 95.5% 9 40.9% 71.3% 22 100.0% 4 19.0% 21 95.5% 72.3% 14 63.6% 2 9.1% 1 4.5% 15 68.2% 36.4% 10 45.5% 6 27.3% 36.4% 14 63.6% 18 81.8% 19 86.4% 77.3% 18 81.8% 21 63.6% 14 63.6% 17 77.3% 20 90.9%
Agreement with expert assessment
Diagnosis
TRALI
AHTR
TACO
NR; TRALI
Most common diagnosis if different than expert assessment
72.7%
81.8%
95.5%
28.6%; 28.6%
Agreement with most common diagnosis
PRO
DEF
PRO
DEF
PRO
DEF
DEF
DEF
DEF
DEF
DEF
DEF
DEF
DEF
DEF
DEF
DEF
DEF
DEF
DEF
DEF
Expert assessment
20
17
14
21
18
19
18
14
6
10
15
1
2
14
21
4
22
9
20
10
21
Responses with correct diagnosis 20 95.2% 8 80.0% 13 65.0% 8 88.9% 81.7% 20 90.9% 3 75.0% 17 81.0% 85.1% 8 57.1% 2 100.0% 1 100.0% 12 80.0% 71.9% 10 100.0% 3 50.0% 81.3% 14 100.0% 14 77.8% 13 68.4% 80.4% 4 22.2% 20 63.6% 7 50.0% 15 88.2% 10 50.0%
Agreement with expert assessment
Match with case definition
SEV
SEV
NS
LT
LT
LT
SEV
S
D
NS
NS
NS
NS
NS
D
NS
S
NS
S
NS
Expert Assessment
TABLE 2. Results of studies, by case
20
17
14
21
17
18
18
14
6
10
14
1
2
14
21
4
22
9
20
10
21
Responses
Severity
17 81.0% 3 30.0% 19 95.0% 7 77.8% 76.7% 22 100.0% 2 50.0% 20 95.2% 93.6% 11 78.6% 1 50.0% 0 0.0% 14 100.0% 83.9% 9 90.0% 0 0.0% 56.3% 13 92.9% 10 55.6% 13 55.0% 72.0% 15 88.2% 21 63.6% 11 78.6% 1 5.9% 12 40.0%
Agreement with expert assessment
DEF
DEF
DEF
DEF
POS
DEF
DEF
DEF
DEF
POS
DEF
DEF
DEF
DEF
POS
DEF
POS
DEF
DEF
DEF
Expert Assessment
20
17
14
21
17
18
18
14
6
10
14
1
2
14
21
4
21
9
20
10
21
Responses
Imputability
17 81.0% 0 0.0% 14 70.0% 3 33.3% 56.7% 14 66.7% 2 50.0% 14 66.7% 65.2% 6 42.9% 1 50.0% 1 100.0% 4 28.6% 38.7% 6 60.0% 3 50.0% 56.3% 12 85.7% 2 11.1% 11 70.0% 50.0% 17 100.0% 20 63.6% 11 78.6% 15 88.2% 17 85.0%
Agreement with expert assessment
AUBUCHON ET AL.
22
21
22
HTR
HTR
HTR
19
24
33
PTP
31
No diagnosis
No diagnosis
No diagnosis
No diagnosis
16
22
26
35
723
22
22
22
20
22
22
22
521 72.1%
81.8% 13 59.1% 15 68.2% 63.6% 10 45.5% 12 54.5% 1 4.5% 22 100.0% 51.1% 11* 50.0% 14 63.6% 20 90.9% 77.3% 20 90.9% 21 95.5% 4 18.2% 13 65.0% 21 95.5% 6 27.3% 20 90.9% 66.2%
Agreement with expert assessment
AHTR
TTI
Most common diagnosis if different than expert assessment
40.9%
95.5%
Agreement with most common diagnosis
PRO
POS
DEF
DEF
DEF
DEF
DEF
DEF
DEF
DEF
Expert assessment
434
20
20
14
11
22
1
12
10
15
13
Responses with correct diagnosis
332 76.5%
1 100.0% 21 95.5% 88.9% 10 90.9% 13 92.9% 10 50.0% 67.6% 12 60.0%
72.2% 13 100.0% 12 80.0% 89.3% 8 80.0% 10
Agreement with expert assessment
Match with case definition
TABLE 2. Continued
D
NS
SEV
D
D
D
SEV
LT
DEF
ND
Expert Assessment
425
20
20
14
11
22
1
11
10
12
11
Responses
Severity
308 72.5%
1 100.0% 21 95.5% 70.5% 9 81.8% 11 78.6% 11 55.0% 64.7% 19 95.0%
62.5% 6 54.5% 0 0.0% 26.1% 9 90.0% 0
Agreement with expert assessment
DEF
DEF
PRO
DOU
PRO
PRO
POS
PRO
ND
DEF
Expert Assessment
D = death; DEF = definite; DOU = doubtful; DSTR = delayed serologic transfusion reaction; LT = life-threatening; ND = not determined; NS = nonsevere; POS = possible; PRO = probable; S = severe.
Total for diagnostic group Across all diagnostic groups:
No diagnosis
No diagnosis
11
1
Total for diagnostic group 32 TA-GVHD
22
PTP
25A
22
22
Total for diagnostic group 34 TTI
22
22
22
Total for diagnostic group 18 HTR
DSTR
22
Responses
Total for diagnostic group 14 DSTR
Case
Expert assessment
Diagnosis
427
20
20
14
11
22
1
12
10
12
13
Responses
Imputability
275 64.4%
87.5% 12 92.3% 4 33.3% 64.0% 4 40.0% 5 41.6% 1 100.0% 9 40.9% 42.2% 8 72.7% 8 57.1% 17 85.0% 73.5% 17 85.0%
Agreement with expert assessment
VALIDATION OF ADVERSE EVENT DEFINITIONS
Volume 54, August 2014 TRANSFUSION
2081
AUBUCHON ET AL.
However, 81.8% of respondents submitted an AHTR diagnosis even though the criteria for HVM system’s surveillance definition AHTR were not met. (The event could have been reported as an “incident” in the HVM system, but there was no adverse outcome for the patient related to the mistransfusion.) In a case of TACO (Case 29), where the patient had received a large volume of plasma and developed pulmonary distress, the TRALI criteria of hypoxemia were not met, although 72.7% of respondents picked this diagnosis. In one complicated case (Case 7), 28.6% of respondents believed that the TRALI criteria had been met despite the lack of a posttransfusion chest X-ray, and the same proportion felt that no condition’s criteria had been met; a smaller proportion (19%) correctly identified that the criteria of a febrile reaction had been met. There were no cases where the most common response for the diagnosis matching with the case definition, the severity grading, or imputability assessment did not match the expert assessment among those cases where the type of adverse event submitted by the respondent was in agreement with the expert assessment. Case 24 illustrated what appeared to be a common difficulty in comparing clinical circumstances against the surveillance definitions of the HVM. Case 24 was created to illustrate an easily recognizable case of sepsis after transfusion of a RBC unit that was later confirmed as being bacterially contaminated. All but one of the respondents categorized this as a TTI; however, the HVM criteria for this diagnosis require documentation of the presence of the pathogen in the recipient, and no blood culture results were provided in the simulated case. Thus, no categorization of this event as a TTI could be made in the HVM despite the obvious clinical diagnosis. Because of the hypotension, a diagnosis of hypotensive transfusion reaction (HTR) could be made, and an assessment could be provided by one of the respondents; this case is listed in Table 2 as a case of HTR. The two experts had also initially misapplied the HVM criteria for TTI in this case until reminded of the specifics of the criteria by one of the respondents. Most of the cases met the criteria for “definite” assignment of the adverse event type, and 76.5% of responses matched the expert assessment in this category. Aggregate scores for severity (69.6%) and imputability (64.4%) were not different by participant group when analyzed in each case separately.
DISCUSSION Assignment of diagnoses, degrees of matching with the case definition, severities, and imputabilities according to the criteria of the HVM system was achieved, overall, approximately two-thirds of the time and fell far below that mark in many cases. Even more troubling was the observation that accurate application of the definitions 2082
TRANSFUSION Volume 54, August 2014
did not improve with prior participation in or access to training in their application. Despite admonitions to participants to carefully apply the specific criteria in the HVM manual in making the assessments in this validation trial rather than applying their medical knowledge and experience, the requirements of the HVM system’s surveillance definitions were not always applied appropriately by respondents. In some of these situations, one could assume that the tendency to apply “medical rationality” overcame the request to apply the system’s criteria strictly. Furthermore, in real-life situations, ambiguities or incomplete data in the fictional cases could have been investigated further to clarify the issues necessary to match the facts with the criteria in the HVM’s definitions. However, a diagnosis consistent with the HVM manual’s definitions was applied by at least half of the respondents in only three-quarters of the cases. There was one case where the adverse event type designated by the two groups of respondents differed. Those not participating in the HVM were more likely to have assigned the same diagnosis as the experts. We do not believe that this should be taken as evidence that the training provided to HVM participants was confusing or counterproductive given that this represented only one of 126 comparisons made between the groups, the remainder not showing a difference in accurate application of the definitions between them. Instead, we take this as a further indication of the complexity involved in applying detailed surveillance definitions. The fact that this case and another that involved apparent TTI (but that failed to satisfy the HVM TTI criteria and caused even the initial “expert assessment” to be incorrect) underlines the importance of ensuring accurate assignment of categories given the attention that TTI reports are likely to garner when results from the HVM are published. This failure to precisely apply the system’s definitions poses significant difficulties in the analysis of data that this system will generate. Some of this problem has been managed in other systems by lumping certain severities or imputabilities (e.g., definite and probable) when aggregating data. However, if participants are not able to assign diagnoses according to the system’s definitions, interpretation of the data generated by the system will be impaired. Different steps could be taken to solve this dilemma. Additional training and examples could be offered to HVM participants, but resources would have to be dedicated not only to provide these but also to take advantage of them. Such training should reinforce that the system requires the application of surveillance—not clinical— definitions, and the “diagnosis” assigned in a transfusion reaction report may differ from that applicable in this system. However, when such differences arise, it is important to explore the basis of these differences and to ensure whether the clinical or the surveillance report should be
VALIDATION OF ADVERSE EVENT DEFINITIONS
revised. Although the HVM definitions and criteria are intended to facilitate uniformity of data reported into the HVM, these are ultimately clinically derived categories and definitions and have merit in being used clinically where appropriate, as they promote a standardization of our reporting and communications clinically. Given the desired size of the reporting network in the United States, using a central—or even a regional—office to recheck the facts surrounding even the most severe or important adverse events that were reported, for example, TTI and TRALI, would probably strain even the most optimistic predictions of funding availability. An alternative to the latter approach might be to reprogram the Web-based reporting system to require entry of key observations that make up the diagnostic criteria so that, in essence, the system assigns the diagnosis, severity, and imputability score rather than depending on the reporter to apply the manual’s details correctly. A separate system to perform such assignments could also be created. Modifying or clarifying the elements of the definitions may also improve the precision of their application, particularly when the criteria for more than one type of adverse reaction appear to be satisfied, and this along with additional training for nonphysician personnel as reporters might allow reporting without “medical insight” confounding application of the definitions. (Recent revision of the manual to allow inclusion of “possible” AHTR and TTI situations would have allowed the most frequent diagnosis made in two cases [Case 12 and Case 24] to be reported in alignment with clinical thinking, and further, such modifications may be helpful.) Finally, proficiency testing should be considered for those who assign the categorizations. One or more of these will need to be implemented— and validated for effectiveness—before the HVM results can be relied on. ACKNOWLEDGMENTS This study would not have been possible without the many participants who took the time to review the cases and submit their responses: Sharlene Billiet, Genesis Medical Center; Robertson Davenport, University of Michigan; Meghan Delaney, Seattle Children’s Hospital–Puget Sound Blood Center; Sunny Dzik, Massachusetts General Hospital; Brenda Grossman, Washington University St Louis; Andrew Heaton, North Shore Long Island Jewish Hospital; Michele Herman, Beth Israel Deaconess Medical Center; Christi Marshall, Lisa Shifflet, Monica Pagano, and Paul Ness, The Johns Hopkins Hospital; Scott Kirkley, University of
Rochester; Jeff McCullough and Claudia Cohn, University of Minnesota; Robert Ranlett, Inland Northwest Blood Center Transfusion Service; Tom Lane, University of California, San Diego; Kashmira Patel and Grace Tenorio, Robert Wood Johnson University Hospital; Ursula Pedersen and Rachel Elder, Crouse Hospital; Beth Polstra, Children’s Healthcare of Atlanta; John Roback, Emory University; S. Gerald Sandler, MedStar Georgetown University Hospital; Sarah Sewall and Mary Kickbush, Aspirus Wausau Hospital; Marilyn Starzewski, Holy Family Memorial Hospital; Joseph Sweeney, Lifespan; Lynne Thompson, Memorial Health University Medical Center; and Ziggy Szczepiorkowski, Dartmouth-Hitchcock Medical Center.
CONFLICT OF INTEREST The authors report no conflicts of interest or funding sources.
REFERENCES 1. Engelfriet CP, Reesink HW. Haemovigilance. Vox Sang 2006;90:207-41. 2. Serious Hazards of Transfusion Steering Committee. Annual reports from the United Kingdom’s Serious Hazards of Transfusion hemovigilance system. [cited 2013 Oct 26]. Available from: http://www.shotuk.org 3. Williamson LM, Stainsby D, Jones H, et al. The impact of universal leukodepletion of the blood supply on hemovigilance reports of posttransfusion purpura and transfusion-associated graft-versus-host disease. Transfusion 2007;47:1455-67. 4. Callum JL, Robillard P, Hyson C, et al. The Canadian transfusion error surveillance system (TESS): results from the first two years of a national pilot project. Transfusion 2007; 47:1A. 5. Stainsby D, Jones H, Asher D, et al.; on behalf of the SHOT Steering Group. Serious hazards of transfusion: a decade of hemovigilance in the UK. Transfus Med Rev 2006;20:27382. 6. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159-74.
SUPPORTING INFORMATION Additional Supporting Information may be found in the online version of this article at the publisher’s Web site: Appendix S1. Scenarios for validation
Volume 54, August 2014 TRANSFUSION
2083
