TRANSFUSION PRACTICE A benchmarking program to reduce red blood cell outdating: implementation, evaluation, and a conceptual framework Rebecca L. Barty,1 Kathleen Gagliardi,2 Wendy Owens,2 Deborah Lauzon,2 Sheena Scheuermann,2 Yang Liu,1 Grace Wang,1 Menaka Pai,1,3 and Nancy M. Heddle1,4
BACKGROUND: Benchmarking is a quality improvement tool that compares an organization’s performance to that of its peers for selected indicators, to improve practice. STUDY DESIGN AND METHODS: Processes to develop evidence-based benchmarks for red blood cell (RBC) outdating in Ontario hospitals, based on RBC hospital disposition data from Canadian Blood Services, have been previously reported. These benchmarks were implemented in 160 hospitals provincewide with a multifaceted approach, which included hospital education, inventory management tools and resources, summaries of best practice recommendations, recognition of high-performing sites, and audit tools on the Transfusion Ontario website (http://transfusionontario.org). In this study we describe the implementation process and the impact of the benchmarking program on RBC outdating. A conceptual framework for continuous quality improvement of a benchmarking program was also developed. RESULTS: The RBC outdating rate for all hospitals trended downward continuously from April 2006 to February 2012, irrespective of hospitals’ transfusion rates or their distance from the blood supplier. The highest annual outdating rate was 2.82%, at the beginning of the observation period. Each year brought further reductions, with a nadir outdating rate of 1.02% achieved in 2011. The key elements of the successful benchmarking strategy included dynamic targets, a comprehensive and evidence-based implementation strategy, ongoing information sharing, and a robust data system to track information. CONCLUSION: The Ontario benchmarking program for RBC outdating resulted in continuous and sustained quality improvement. Our conceptual iterative framework for benchmarking provides a guide for institutions implementing a benchmarking program.
B
enchmarking is defined as a structured continuous collaborative process, which uses comparisons for selected indicators to identify factors that will improve practice.1 It is a useful tool to identify best practices and facilitate continuous quality improvement, as it provides an opportunity to compare an organization’s performance to that of its peers.1 The concept of benchmarking was developed by Rank Xerox in the late 1970s, in response to challenges imposed by an exceedingly competitive market.2 As benchmarking expanded to the health care sector, it underwent an evolution, focusing less on financial targets and more on patient outcomes; however, due to resource constraints in
ABBREVIATIONS: CBS 5 Canadian Blood Services; MTRP 5 McMaster Transfusion Research Program; ORBCoN 5 Ontario Regional Blood Coordinating Network; RBC 5 red blood cell. From the 1Department of Medicine and the 3Department of Pathology and Molecular Medicine, McMaster University, Hamilton, Ontario; the 2Ontario Regional Blood Coordinating Network (ORBCoN), Ontario; and 4Research and Development, Canadian Blood Services, Hamilton, Ontario, Canada Address correspondence to: Rebecca Barty, MLT BA MSc (c), Clinical Research Coordinator, McMaster Transfusion Research Program, Department of Medicine/Faculty of Health Sciences, McMaster University HSC-3H50, 1280 Main Street W., Hamilton, ON L8S 4K1, Canada; e-mail: [email protected] MP was supported by the E.J. Moran Campbell AFP Internal Career Research Award from McMaster University’s Department of Medicine. ORBCoN program funding from the Ministry of Health and Long-Term Care and infrastructure funding for the McMaster Transfusion Research Program is provided in part by Canadian Blood Services and Health Canada. Received for publication April 23, 2014; revision received December 5, 2014; and accepted January 6, 2015. doi:10.1111/trf.13055 C 2015 AABB V
TRANSFUSION 2015;55;1621–1627 Volume 55, July 2015 TRANSFUSION 1621
BARTY ET AL.
the health care system, its use to drive cost-effectiveness makes benchmarking extremely attractive to health care administrators and policy makers.3 Benchmarking activities reported in the transfusion medicine literature have been summarized in a systematic review. Only seven relevant articles were identified emphasizing opportunity for future research in this area. Major barriers to benchmarking included difficulty identifying best practices due to a lack of evidence and developing cost-effective methods of data collection.4 In 2006, Canadian Blood Services (CBS) implemented a voluntary standardized monthly hospital reporting system that involved centralized electronic reporting of blood inventory disposition data from hospitals in the province of Ontario. Information on a comprehensive range of variables was captured for all cellular blood products including units transfused, units outdated, and units transferred. The availability of these data being shared from CBS provided an opportunity for the Ontario Regional Blood Coordinating Network (ORBCoN) and the McMaster Transfusion Research Program (MTRP) to develop benchmarks for red blood cell (RBC) outdating in Ontario hospitals. In 2008, we described an approach to develop RBC outdate benchmarks.5 This included an analysis of disposition data from CBS to identify factors that affect RBC outdating and a statistical approach to develop the benchmark targets for Ontario hospitals. ORBCoN then developed a comprehensive awareness strategy to educate hospitals about the benchmark targets. This report describes the implementation of the benchmarking program in Ontario including hospital education, development and distribution of benchmarking tools, ongoing assessment of the effect of the benchmarking program on the frequency of RBC outdates, revisions to the benchmark targets to facilitate continuous quality improvement, and development of a conceptual framework as a model for a successful benchmarking program.
MATERIALS AND METHODS Participants At baseline, data from 156 hospitals in the province of Ontario were incorporated into the benchmarking program. As a result of increased willingness to report data, mergers, and distribution of services over time, the coded data set captured a total of 160 different hospitals; however, some hospitals only reported for a short period. Due to the dynamic nature and the consolidation of hospital transfusion services reporting the benchmarking target analysis for the second checkpoint captured 157 hospitals, and 155 hospitals contributed to the analysis at the third checkpoint. Based on our previously reported approach to developing benchmark targets for outdated RBCs, hospitals were grouped into three categories each with a differ1622 TRANSFUSION Volume 55, July 2015
ent target.5 These categories were based on two of the three factors that were found to significantly affect RBC outdates: distance from the blood supplier’s distribution center and mean number of transfused units of RBCs per month.5 The third significant factor was month of the year. However, it was not incorporated in the categorization of hospitals as it was not a hospital-related factor, and no conclusive monthly trends were observed in the data set. The optimal benchmark target for RBC outdating in each category was defined by the first quartile of data. The 73 Category 1 hospitals (64 km from the blood supplier or 200 RBC units transfused/month) had an outdate target of 0.4% RBC outdated per month. The 59 Category 2 hospitals (65-484 km from the blood supplier and 199 RBC units transfused/month) had a target of 1.1%, and the 24 Category 3 hospitals (>485 km from the blood supplier and 199 RBC units transfused/month) had a target of 20.4%.5 The targets were reviewed and approved by the three ORBCoN regional advisory committees before implementation.
Implementation of benchmarking targets The implementation of the RBC outdate targets in hospitals was performed in phases. Beginning in April 2008, ORBCoN staff provided Ontario hospitals with information on the benchmarking project during their annual site visits. At each visit, hospital representatives (typically charge technologists, general duty medical laboratory technologists, manager, and laboratory medical director) were provided with information on the concept of benchmarking, the meaning and implication of the targets, and how their hospital was performing. Also starting in April 2008, an inventory management tool kit developed by ORBCoN was introduced to hospitals at annual hospital site visits and at the annual CBS/ ORBCoN spring symposium. This tool kit provided hospitals with resources to optimize inventory management, including a calculator to determine appropriate inventory stock levels, resources to redistribute outdating RBCs (including provision of validated shipping containers and standard training and procedures), algorithms and clinical practice recommendations summarizing best practices, and Web-based audit tools available on the Transfusion Ontario website (http://transfusionontario.org).6,7 In February 2010, a benchmarking website developed and maintained by MTRP was introduced to hospitals (https://fhsmtrp.mcmaster.ca/benchmarking). This website allowed hospitals to access bubble plots that could be used to compare their RBC outdating rates with the rates of other hospitals of similar size, geographic location, and transfusion activity.5 Each hospital also had access to their trends in RBC outdating over time, using statistical process control charts.8 Training webinars ran from February
RBC OUTDATING BENCHMARKS
Fig. 1. Overall percentage of RBCs outdated every month, over time, for all participating hospitals.
to June 2010, and a user instruction manual was provided to participants. Inventory management webinars were held for six hospitals in the central region in October and November of 2011. These sites were among the lowest users of RBCs in the region. However, they were chosen for additional knowledge transfer efforts because of their relatively high outdating rates and a lack of improvement in RBC outdating rates since April 2008 (at the time benchmarks were introduced), compared to peer hospitals. These webinars reviewed the provincial landscape for utilization and outdating of RBCs, the importance of adhering to provincial benchmark targets, data from high-performing peer hospitals, the Ontario/National Contingency Plan for management of blood shortages (green, amber, red phases), and how inventory levels affect outdating rates.9,10 The webinars also reinforced how to use ORBCoN’s inventory management strategies and tools. In the northern and eastern regions, focused teleconferences with similar objectives were held in January 2012 for those sites that continued to show high outdating rates compared to their peers. After the webinars and teleconferences, most of the participating sites showed a reduction in RBC outdating. ORBCoN also provided positive feedback to recognize high-performing centers. In 2010, certificates of recognition were provided to hospitals that had already achieved benchmarking targets in the first quartile. Certificates were also provided between the release of the first and second benchmark targets.
Recalculation of RBC outdate benchmark targets After the benchmark targets were initially implemented in 2008, they were revised approximately every 18 months
using data collected since the last benchmarks were set. The second benchmarking model analysis was done in late 2009 with data from 157 hospitals over the time period of January 2008 to June 2009, using the same model described in our 2008 article.5 Benchmark targets were updated a third time using data from 155 hospitals (July 2009 to February 2012). Although targets were calculated using only the data since the last benchmarking target was set, the models used were checked each time using all available accumulated data. This ensured the validity of the original factors associated with RBC outdating and investigated if any other factors were potentially significant based on accumulated data. At each11 checkpoint, the significant factors associated with RBC outdating remained the same in models based on term data and accumulated data. All analyses were performed using computer software (SAS 9.2, SAS Institute, Inc., Cary, NC).11 Each time a new set of benchmark targets was released to participating hospitals, they also received a personalized report summarizing utilization and outdating at their site.
RESULTS The overall monthly RBC outdating rates for all hospitals showed a continuous downward trend from April 2006 to February 2012 (Fig. 1). The highest annual outdating rate of 2.82% was observed at the beginning of the observation period, in 2006. Each year, a progressive reduction in the provincial annual outdating was achieved, and in 2011 a nadir of 1.02% was attained. At baseline (Checkpoint 1), 25% of hospitals in each category were meeting the initial benchmark targets. Within the first 18 months of implementing the benchmarking Volume 55, July 2015 TRANSFUSION 1623
BARTY ET AL.
Fig. 2. RBC outdating targets and performance summaries at three checkpoints during the observation period, by hospital category. Category 1 includes 75 hospitals (64 km from the blood supplier or 200 RBC units transfused/month). Category 2 includes 60 hospitals (65-484 km from the blood supplier and 199 RBC units transfused/month). Category 3 includes 25 hospitals (>485 km from the blood supplier and 199 RBC units transfused/month). Box plots represent distribution of hospitals’ RBC outdating rates. Benchmarking targets (which represent the 25th percentile) are bolded on the box plots.
program, more than 40% of hospitals in each category were meeting the targets (Fig. 2). As hospitals were challenged with more rigorous targets after the second round of analysis, outdating rates in Category 3 (mainly small hospitals in more remote parts of the province) continued to decrease, while the performance of hospitals in Category 1 and 2 (larger hospitals closer to the blood supplier) leveled off and did not show further improvements. This suggests that for hospitals in Categories 1 and 2, very little additional improvement occurs beyond threshold outdate rates of 0.2 and 0.5% respectively. Logistic regression showed that three factors significantly predicted outdating: distance from the blood supplier, average monthly transfusion activity, and month of the year. These three predictive factors remained the same at the second and third checkpoints, even as the benchmarking model was tested on cumulative data. At the third checkpoint, the benchmark targets were reviewed with ORBCoN’s three regional advisory committees. Due to the blood supplier’s consolidation of product distribution (2012) from three sites to one central site (supplying products to all central and southwest Ontario hospitals), it was decided that new RBC outdating targets would not be released to hospitals. This decision was made primarily to 1624 TRANSFUSION Volume 55, July 2015
provide an opportunity for the hospitals to adapt and adjust to process changes resulting from the new supplier distribution center, projected differences to the delivery schedule, and encompass any other unknown factors that might arise from the consolidation. Instead, benchmark targets from the second checkpoint continued to be promoted. Figure 2 shows that at each of the checkpoints, the overall variation of the magnitude of outdating decreased in all three categories and benchmark targets were successfully lowered. Compared to baseline, by the third checkpoint the most improvement was seen in Category 3 hospitals with 63% reaching the baseline RBC outdate target. In Categories 1 and 2, a total of 38 and 46% of hospitals, respectively, reached the baseline target. From a provincial perspective the percentage of RBC outdates decreased considerably. Despite the fact that the total number of transfused RBC units increased from 379,995 in 2007 to 404,658 in 2011, the total annual number of outdated RBC units decreased from 7439 units in 2007 to 4152 units in 2011. Figure 3 shows the timeline of the introduction of benchmarks and knowledge translation strategies in relation to monthly outdating rates by category. As ORBCoN facilitated further benchmarking and educational interventions, each category continued to achieve decreasing outdating
RBC OUTDATING BENCHMARKS
Fig. 3. Overall percentage of RBCs outdated every month in each benchmarking category, over time, alongside timeline of ORBCoN’s implementation and knowledge translation activities.
rates. Decreases have been sustained. Despite slight monthto-month variations, none of the categories have regressed to their initial overall outdating rates (1.5, 8.3, and 33.5% in each of Categories 1, 2, and 3, respectively).
DISCUSSION ORBCoN’s benchmarking program for RBC outdating, which included the development of dynamic benchmarking targets and a comprehensive knowledge translation strategy to educate participants, resulted in continuous and sustained quality improvement. Monthly RBC outdating rates dropped over the observation period, irrespective of hospital’s transfusion rates or their distance from the blood supplier. The benchmarking program discussed in this article has several strengths. First, it uses a multifactorial and tailored implementation strategy, which directly addresses the barriers and facilitators to optimal RBC outdating; this is based on best evidence from the knowledge translation literature.12 ORBCoN used a variety of educational tools, both high-tech (webinars, audit and feedback, online resources) and low-tech (site visits, annual meetings) to publicize its benchmarking targets, and raise awareness of
issues surrounding inventory management. This strategy not only helped participants understand why the targets existed, but empowered them to move their performance closer to the targets. Metrics were shared nonpunitively and good performance was recognized. This encouraged participants to focus on improvement (as opposed to failure) and share best practices on inventory management with each other. Another major strength of the program was its robust data management system. Performance data (in a standard format) flowed directly from transfusion medicine programs at each participating hospital to CBS, then without any data manipulation, were transferred to MTRP for analysis. This process provided data accuracy and minimized the time spent on data validation. The result was that benchmarking targets could be updated rapidly and at regular intervals. These dynamic targets encouraged hospitals to continually strive for better performance. An additional strength of the program was the development of the Transfusion Ontario website (funded by the Ontario Ministry of Health and Long-Term Care and ORBCoN), which linked to the benchmarking website. It not only provided participants with up-to-date metrics for their site, but allowed them to compare their performance to peer hospitals. The Transfusion Ontario website also gave hospitals access to several tools to make Volume 55, July 2015 TRANSFUSION 1625
BARTY ET AL.
Fig. 4. Conceptual framework for a benchmarking program for transfusion medicine inventory management: two embedded iterative cycles to promote continuous information flow and continuous quality improvement.
daily blood management operations easier and helped them meet targets for RBC outdating. A limitation of ORBCoN’s benchmarking program is that it is not set up to identify the individual contribution rates of each strategy that was implemented as part of the benchmarking program on reductions in RBC outdating. All features of the benchmarking program, including target setting, measurement of hospital performance, continuous feedback, and knowledge translation strategies, may have contributed to the decrease in outdating that was observed. However, other factors could have contributed to this change, including changes in patient populations and/or clinician behavior. CBS’s data collection for the current hospital disposition data initiative began in April 2006, providing hospitals with newly collected discard data to improve component disposition performance. In April 2007, CBS launched a national program to target inventory management and discard reduction. CBS also provided utilization graphs with peer comparisons in the joint site visits with ORBCoN. These initiatives likely impacted RBC outdating as well, as showing data in different formats served to provide a united message resulting in a positive outcome. ORBCoN did not collect specific information or data where hospitals found their inventories to be too low resulting from pressures to reduce their outdating rates. However, annual site visits were conducted during this period where this information would likely have been exchanged. Strategies suggested were done so to encour1626 TRANSFUSION Volume 55, July 2015
age and provide the tools to gradually reduce outdating to prevent inventory shortages or drastic changes that could jeopardize patient care. Inventory management strategies provided resources in an attempt to provide a means to improving transfusion medicine care with final decision making on best practices to be done by the laboratory service staff at each site in conjunction with their medical director to ensure patient safety is not at risk. Some smaller sites that were geographically furthest from the blood supplier were noted to be reluctant to reduce their stock; this may have signaled a concern that understocking might result in shortages. ORBCoN attempted to counter this by providing resources to improve and expand existing redistribution programs, to ensure that smaller sites were linked with larger sites that could assist with redistributing blood resources in a dynamic way. The benchmarking program in Ontario also provides a unique opportunity to assess the impact of changes to the blood system in the province. For example, two of the three regions in Ontario now receive blood from a centralized distribution center and it is possible that this change may impact the outdating rate. With the current benchmarking program we have the opportunity to test this hypothesis, comparing outdating rates before and after centralization of product distribution. Our systematic review of benchmarking identified a clear paucity of such programs in transfusion medicine. ORBCoN’s program is a true benchmarking system, as it not only studies data trends over time, but collaborates
RBC OUTDATING BENCHMARKS
ACKNOWLEDGMENTS The authors acknowledge the commitment to quality of Ontario Hospital Transfusion Medicine Services, who monitor and report their data to Canadian Blood Services on an ongoing basis. The authors also extend their appreciation to the staff of ORBCoN, for their diligent work promoting benchmarking and inventory management. Denise Evanovitch and Anushka Jaffer’s assistance with the review and formatting of the manuscript was instrumental.
CONFLICT OF INTEREST The authors have disclosed no conflicts of interest.
REFERENCES Fig. 5. Components of a multipronged benchmarking strategy.
1. Kearns DT. Leadership through quality. Executive 1990;4: 86-89. 2. Walker R. Rank Xerox—management revolution. Long Range Plann 1992;25:9-21.
with participants to address the variability observed and continuously reevaluates performance leading to practice changes. Apelseth and colleagues4 underscored that benchmarking is a cyclic and dynamic process that evolves constantly. Adoption of this cyclic and dynamic model sets our program apart from those that simply observe trends (without communicating findings to participants) and those that do one-time audits (ignoring change over time). Our experience has informed the creation of a conceptual framework that can be used to develop other programs (Fig. 4). The conceptual model illustrates the iterative nature of benchmarking. Success of the program depends on continuous reevaluation of performance and continuous reinforcement of education through knowledge translation strategies. A robust data system represents a second iterative process that is embedded in the conceptual framework, encouraging constant performance reporting (as data flows from the participating hospitals to the data center) and constant knowledge translation (as data flows from the management center to the participating hospitals). All components of a multipronged strategy (Fig. 5) to decrease RBC outdating are featured in the program described in this article. Each factor is essential to a successful benchmarking strategy: dynamic targets, an implementation program based on evidence-based knowledge translation principles and practical inventory management tools, audit and recognition programs, ongoing information sharing between participating sites, the data center and the management center, and a robust data system to track information. Our multipronged strategy is a conceptual iterative framework for benchmarking in transfusion medicine, which can be adopted by other regions wishing to improve their transfusion practices.
3. Bisognano MA. Leadership strategies for breakthrough change in health care. In: Neuorientierung im Gesundheitswesen. New York: Springer; 1998. p. 89-100. 4. Apelseth TO, Molnar L, Arnold E, et al. Benchmarking: applications to transfusion medicine. Transfus Med Rev 2012;26: 321-32. 5. Heddle NM, Liu Y, Barty R, et al. Factors affecting the frequency of red blood cell outdates: an approach to establish benchmarking targets. Transfusion 2009;49:219-26. 6. Collins MA. Validation of a shipping container for inter hospital transfer of red blood cells. Can J Med Lab Sci 2010; 72:62-9. 7. Ontario Regional Blood Coordinating Network. (ORBCoN). Homepage. [cited 2014 April 1]. Available from: http://transfusionontario.org/ 8. Dzik WS, Beckman N, Selleng K, et al. Errors in patient specimen collection: application of statistical process control. Transfusion 2008;48:2143-51. 9. Ontario Contingency Plan for the Management of Blood Shortages Version 2 October 30, 2012. Ontario. Ministry of Health and Long-Term Care; 2012. [cited 2014 April 1] Available from: http://transfusionontario.org/en/cmdownloads/ categories/emergency_blood. 10. National Plan for the Management of Shortages of Labile Blood Components. National Advisory Committee on Blood and Blood Components and Canadian Blood Services. 201201-18. [cited 2014 April 1] Available from: http://www.nacblood.ca/resources/shortages-plan/National-shortagesplan-Jan-2012-final.pdf. 11. Heddle N, Barty R. Benchmarking: applications to transfusion medicine. ISBT Sci Ser 2013;8:93-99. 12. Grimshaw JM, Eccles MP, Lavis JN, et al. Knowledge translation of research findings. Implementation Sci 2012; 7:50. Volume 55, July 2015 TRANSFUSION 1627
BACKGROUND: Benchmarking is a quality improvement tool that compares an organization’s performance to that of its peers for selected indicators, to improve practice. STUDY DESIGN AND METHODS: Processes to develop evidence-based benchmarks for red blood cell (RBC) outdating in Ontario hospitals, based on RBC hospital disposition data from Canadian Blood Services, have been previously reported. These benchmarks were implemented in 160 hospitals provincewide with a multifaceted approach, which included hospital education, inventory management tools and resources, summaries of best practice recommendations, recognition of high-performing sites, and audit tools on the Transfusion Ontario website (http://transfusionontario.org). In this study we describe the implementation process and the impact of the benchmarking program on RBC outdating. A conceptual framework for continuous quality improvement of a benchmarking program was also developed. RESULTS: The RBC outdating rate for all hospitals trended downward continuously from April 2006 to February 2012, irrespective of hospitals’ transfusion rates or their distance from the blood supplier. The highest annual outdating rate was 2.82%, at the beginning of the observation period. Each year brought further reductions, with a nadir outdating rate of 1.02% achieved in 2011. The key elements of the successful benchmarking strategy included dynamic targets, a comprehensive and evidence-based implementation strategy, ongoing information sharing, and a robust data system to track information. CONCLUSION: The Ontario benchmarking program for RBC outdating resulted in continuous and sustained quality improvement. Our conceptual iterative framework for benchmarking provides a guide for institutions implementing a benchmarking program.
B
enchmarking is defined as a structured continuous collaborative process, which uses comparisons for selected indicators to identify factors that will improve practice.1 It is a useful tool to identify best practices and facilitate continuous quality improvement, as it provides an opportunity to compare an organization’s performance to that of its peers.1 The concept of benchmarking was developed by Rank Xerox in the late 1970s, in response to challenges imposed by an exceedingly competitive market.2 As benchmarking expanded to the health care sector, it underwent an evolution, focusing less on financial targets and more on patient outcomes; however, due to resource constraints in
ABBREVIATIONS: CBS 5 Canadian Blood Services; MTRP 5 McMaster Transfusion Research Program; ORBCoN 5 Ontario Regional Blood Coordinating Network; RBC 5 red blood cell. From the 1Department of Medicine and the 3Department of Pathology and Molecular Medicine, McMaster University, Hamilton, Ontario; the 2Ontario Regional Blood Coordinating Network (ORBCoN), Ontario; and 4Research and Development, Canadian Blood Services, Hamilton, Ontario, Canada Address correspondence to: Rebecca Barty, MLT BA MSc (c), Clinical Research Coordinator, McMaster Transfusion Research Program, Department of Medicine/Faculty of Health Sciences, McMaster University HSC-3H50, 1280 Main Street W., Hamilton, ON L8S 4K1, Canada; e-mail: [email protected] MP was supported by the E.J. Moran Campbell AFP Internal Career Research Award from McMaster University’s Department of Medicine. ORBCoN program funding from the Ministry of Health and Long-Term Care and infrastructure funding for the McMaster Transfusion Research Program is provided in part by Canadian Blood Services and Health Canada. Received for publication April 23, 2014; revision received December 5, 2014; and accepted January 6, 2015. doi:10.1111/trf.13055 C 2015 AABB V
TRANSFUSION 2015;55;1621–1627 Volume 55, July 2015 TRANSFUSION 1621
BARTY ET AL.
the health care system, its use to drive cost-effectiveness makes benchmarking extremely attractive to health care administrators and policy makers.3 Benchmarking activities reported in the transfusion medicine literature have been summarized in a systematic review. Only seven relevant articles were identified emphasizing opportunity for future research in this area. Major barriers to benchmarking included difficulty identifying best practices due to a lack of evidence and developing cost-effective methods of data collection.4 In 2006, Canadian Blood Services (CBS) implemented a voluntary standardized monthly hospital reporting system that involved centralized electronic reporting of blood inventory disposition data from hospitals in the province of Ontario. Information on a comprehensive range of variables was captured for all cellular blood products including units transfused, units outdated, and units transferred. The availability of these data being shared from CBS provided an opportunity for the Ontario Regional Blood Coordinating Network (ORBCoN) and the McMaster Transfusion Research Program (MTRP) to develop benchmarks for red blood cell (RBC) outdating in Ontario hospitals. In 2008, we described an approach to develop RBC outdate benchmarks.5 This included an analysis of disposition data from CBS to identify factors that affect RBC outdating and a statistical approach to develop the benchmark targets for Ontario hospitals. ORBCoN then developed a comprehensive awareness strategy to educate hospitals about the benchmark targets. This report describes the implementation of the benchmarking program in Ontario including hospital education, development and distribution of benchmarking tools, ongoing assessment of the effect of the benchmarking program on the frequency of RBC outdates, revisions to the benchmark targets to facilitate continuous quality improvement, and development of a conceptual framework as a model for a successful benchmarking program.
MATERIALS AND METHODS Participants At baseline, data from 156 hospitals in the province of Ontario were incorporated into the benchmarking program. As a result of increased willingness to report data, mergers, and distribution of services over time, the coded data set captured a total of 160 different hospitals; however, some hospitals only reported for a short period. Due to the dynamic nature and the consolidation of hospital transfusion services reporting the benchmarking target analysis for the second checkpoint captured 157 hospitals, and 155 hospitals contributed to the analysis at the third checkpoint. Based on our previously reported approach to developing benchmark targets for outdated RBCs, hospitals were grouped into three categories each with a differ1622 TRANSFUSION Volume 55, July 2015
ent target.5 These categories were based on two of the three factors that were found to significantly affect RBC outdates: distance from the blood supplier’s distribution center and mean number of transfused units of RBCs per month.5 The third significant factor was month of the year. However, it was not incorporated in the categorization of hospitals as it was not a hospital-related factor, and no conclusive monthly trends were observed in the data set. The optimal benchmark target for RBC outdating in each category was defined by the first quartile of data. The 73 Category 1 hospitals (64 km from the blood supplier or 200 RBC units transfused/month) had an outdate target of 0.4% RBC outdated per month. The 59 Category 2 hospitals (65-484 km from the blood supplier and 199 RBC units transfused/month) had a target of 1.1%, and the 24 Category 3 hospitals (>485 km from the blood supplier and 199 RBC units transfused/month) had a target of 20.4%.5 The targets were reviewed and approved by the three ORBCoN regional advisory committees before implementation.
Implementation of benchmarking targets The implementation of the RBC outdate targets in hospitals was performed in phases. Beginning in April 2008, ORBCoN staff provided Ontario hospitals with information on the benchmarking project during their annual site visits. At each visit, hospital representatives (typically charge technologists, general duty medical laboratory technologists, manager, and laboratory medical director) were provided with information on the concept of benchmarking, the meaning and implication of the targets, and how their hospital was performing. Also starting in April 2008, an inventory management tool kit developed by ORBCoN was introduced to hospitals at annual hospital site visits and at the annual CBS/ ORBCoN spring symposium. This tool kit provided hospitals with resources to optimize inventory management, including a calculator to determine appropriate inventory stock levels, resources to redistribute outdating RBCs (including provision of validated shipping containers and standard training and procedures), algorithms and clinical practice recommendations summarizing best practices, and Web-based audit tools available on the Transfusion Ontario website (http://transfusionontario.org).6,7 In February 2010, a benchmarking website developed and maintained by MTRP was introduced to hospitals (https://fhsmtrp.mcmaster.ca/benchmarking). This website allowed hospitals to access bubble plots that could be used to compare their RBC outdating rates with the rates of other hospitals of similar size, geographic location, and transfusion activity.5 Each hospital also had access to their trends in RBC outdating over time, using statistical process control charts.8 Training webinars ran from February
RBC OUTDATING BENCHMARKS
Fig. 1. Overall percentage of RBCs outdated every month, over time, for all participating hospitals.
to June 2010, and a user instruction manual was provided to participants. Inventory management webinars were held for six hospitals in the central region in October and November of 2011. These sites were among the lowest users of RBCs in the region. However, they were chosen for additional knowledge transfer efforts because of their relatively high outdating rates and a lack of improvement in RBC outdating rates since April 2008 (at the time benchmarks were introduced), compared to peer hospitals. These webinars reviewed the provincial landscape for utilization and outdating of RBCs, the importance of adhering to provincial benchmark targets, data from high-performing peer hospitals, the Ontario/National Contingency Plan for management of blood shortages (green, amber, red phases), and how inventory levels affect outdating rates.9,10 The webinars also reinforced how to use ORBCoN’s inventory management strategies and tools. In the northern and eastern regions, focused teleconferences with similar objectives were held in January 2012 for those sites that continued to show high outdating rates compared to their peers. After the webinars and teleconferences, most of the participating sites showed a reduction in RBC outdating. ORBCoN also provided positive feedback to recognize high-performing centers. In 2010, certificates of recognition were provided to hospitals that had already achieved benchmarking targets in the first quartile. Certificates were also provided between the release of the first and second benchmark targets.
Recalculation of RBC outdate benchmark targets After the benchmark targets were initially implemented in 2008, they were revised approximately every 18 months
using data collected since the last benchmarks were set. The second benchmarking model analysis was done in late 2009 with data from 157 hospitals over the time period of January 2008 to June 2009, using the same model described in our 2008 article.5 Benchmark targets were updated a third time using data from 155 hospitals (July 2009 to February 2012). Although targets were calculated using only the data since the last benchmarking target was set, the models used were checked each time using all available accumulated data. This ensured the validity of the original factors associated with RBC outdating and investigated if any other factors were potentially significant based on accumulated data. At each11 checkpoint, the significant factors associated with RBC outdating remained the same in models based on term data and accumulated data. All analyses were performed using computer software (SAS 9.2, SAS Institute, Inc., Cary, NC).11 Each time a new set of benchmark targets was released to participating hospitals, they also received a personalized report summarizing utilization and outdating at their site.
RESULTS The overall monthly RBC outdating rates for all hospitals showed a continuous downward trend from April 2006 to February 2012 (Fig. 1). The highest annual outdating rate of 2.82% was observed at the beginning of the observation period, in 2006. Each year, a progressive reduction in the provincial annual outdating was achieved, and in 2011 a nadir of 1.02% was attained. At baseline (Checkpoint 1), 25% of hospitals in each category were meeting the initial benchmark targets. Within the first 18 months of implementing the benchmarking Volume 55, July 2015 TRANSFUSION 1623
BARTY ET AL.
Fig. 2. RBC outdating targets and performance summaries at three checkpoints during the observation period, by hospital category. Category 1 includes 75 hospitals (64 km from the blood supplier or 200 RBC units transfused/month). Category 2 includes 60 hospitals (65-484 km from the blood supplier and 199 RBC units transfused/month). Category 3 includes 25 hospitals (>485 km from the blood supplier and 199 RBC units transfused/month). Box plots represent distribution of hospitals’ RBC outdating rates. Benchmarking targets (which represent the 25th percentile) are bolded on the box plots.
program, more than 40% of hospitals in each category were meeting the targets (Fig. 2). As hospitals were challenged with more rigorous targets after the second round of analysis, outdating rates in Category 3 (mainly small hospitals in more remote parts of the province) continued to decrease, while the performance of hospitals in Category 1 and 2 (larger hospitals closer to the blood supplier) leveled off and did not show further improvements. This suggests that for hospitals in Categories 1 and 2, very little additional improvement occurs beyond threshold outdate rates of 0.2 and 0.5% respectively. Logistic regression showed that three factors significantly predicted outdating: distance from the blood supplier, average monthly transfusion activity, and month of the year. These three predictive factors remained the same at the second and third checkpoints, even as the benchmarking model was tested on cumulative data. At the third checkpoint, the benchmark targets were reviewed with ORBCoN’s three regional advisory committees. Due to the blood supplier’s consolidation of product distribution (2012) from three sites to one central site (supplying products to all central and southwest Ontario hospitals), it was decided that new RBC outdating targets would not be released to hospitals. This decision was made primarily to 1624 TRANSFUSION Volume 55, July 2015
provide an opportunity for the hospitals to adapt and adjust to process changes resulting from the new supplier distribution center, projected differences to the delivery schedule, and encompass any other unknown factors that might arise from the consolidation. Instead, benchmark targets from the second checkpoint continued to be promoted. Figure 2 shows that at each of the checkpoints, the overall variation of the magnitude of outdating decreased in all three categories and benchmark targets were successfully lowered. Compared to baseline, by the third checkpoint the most improvement was seen in Category 3 hospitals with 63% reaching the baseline RBC outdate target. In Categories 1 and 2, a total of 38 and 46% of hospitals, respectively, reached the baseline target. From a provincial perspective the percentage of RBC outdates decreased considerably. Despite the fact that the total number of transfused RBC units increased from 379,995 in 2007 to 404,658 in 2011, the total annual number of outdated RBC units decreased from 7439 units in 2007 to 4152 units in 2011. Figure 3 shows the timeline of the introduction of benchmarks and knowledge translation strategies in relation to monthly outdating rates by category. As ORBCoN facilitated further benchmarking and educational interventions, each category continued to achieve decreasing outdating
RBC OUTDATING BENCHMARKS
Fig. 3. Overall percentage of RBCs outdated every month in each benchmarking category, over time, alongside timeline of ORBCoN’s implementation and knowledge translation activities.
rates. Decreases have been sustained. Despite slight monthto-month variations, none of the categories have regressed to their initial overall outdating rates (1.5, 8.3, and 33.5% in each of Categories 1, 2, and 3, respectively).
DISCUSSION ORBCoN’s benchmarking program for RBC outdating, which included the development of dynamic benchmarking targets and a comprehensive knowledge translation strategy to educate participants, resulted in continuous and sustained quality improvement. Monthly RBC outdating rates dropped over the observation period, irrespective of hospital’s transfusion rates or their distance from the blood supplier. The benchmarking program discussed in this article has several strengths. First, it uses a multifactorial and tailored implementation strategy, which directly addresses the barriers and facilitators to optimal RBC outdating; this is based on best evidence from the knowledge translation literature.12 ORBCoN used a variety of educational tools, both high-tech (webinars, audit and feedback, online resources) and low-tech (site visits, annual meetings) to publicize its benchmarking targets, and raise awareness of
issues surrounding inventory management. This strategy not only helped participants understand why the targets existed, but empowered them to move their performance closer to the targets. Metrics were shared nonpunitively and good performance was recognized. This encouraged participants to focus on improvement (as opposed to failure) and share best practices on inventory management with each other. Another major strength of the program was its robust data management system. Performance data (in a standard format) flowed directly from transfusion medicine programs at each participating hospital to CBS, then without any data manipulation, were transferred to MTRP for analysis. This process provided data accuracy and minimized the time spent on data validation. The result was that benchmarking targets could be updated rapidly and at regular intervals. These dynamic targets encouraged hospitals to continually strive for better performance. An additional strength of the program was the development of the Transfusion Ontario website (funded by the Ontario Ministry of Health and Long-Term Care and ORBCoN), which linked to the benchmarking website. It not only provided participants with up-to-date metrics for their site, but allowed them to compare their performance to peer hospitals. The Transfusion Ontario website also gave hospitals access to several tools to make Volume 55, July 2015 TRANSFUSION 1625
BARTY ET AL.
Fig. 4. Conceptual framework for a benchmarking program for transfusion medicine inventory management: two embedded iterative cycles to promote continuous information flow and continuous quality improvement.
daily blood management operations easier and helped them meet targets for RBC outdating. A limitation of ORBCoN’s benchmarking program is that it is not set up to identify the individual contribution rates of each strategy that was implemented as part of the benchmarking program on reductions in RBC outdating. All features of the benchmarking program, including target setting, measurement of hospital performance, continuous feedback, and knowledge translation strategies, may have contributed to the decrease in outdating that was observed. However, other factors could have contributed to this change, including changes in patient populations and/or clinician behavior. CBS’s data collection for the current hospital disposition data initiative began in April 2006, providing hospitals with newly collected discard data to improve component disposition performance. In April 2007, CBS launched a national program to target inventory management and discard reduction. CBS also provided utilization graphs with peer comparisons in the joint site visits with ORBCoN. These initiatives likely impacted RBC outdating as well, as showing data in different formats served to provide a united message resulting in a positive outcome. ORBCoN did not collect specific information or data where hospitals found their inventories to be too low resulting from pressures to reduce their outdating rates. However, annual site visits were conducted during this period where this information would likely have been exchanged. Strategies suggested were done so to encour1626 TRANSFUSION Volume 55, July 2015
age and provide the tools to gradually reduce outdating to prevent inventory shortages or drastic changes that could jeopardize patient care. Inventory management strategies provided resources in an attempt to provide a means to improving transfusion medicine care with final decision making on best practices to be done by the laboratory service staff at each site in conjunction with their medical director to ensure patient safety is not at risk. Some smaller sites that were geographically furthest from the blood supplier were noted to be reluctant to reduce their stock; this may have signaled a concern that understocking might result in shortages. ORBCoN attempted to counter this by providing resources to improve and expand existing redistribution programs, to ensure that smaller sites were linked with larger sites that could assist with redistributing blood resources in a dynamic way. The benchmarking program in Ontario also provides a unique opportunity to assess the impact of changes to the blood system in the province. For example, two of the three regions in Ontario now receive blood from a centralized distribution center and it is possible that this change may impact the outdating rate. With the current benchmarking program we have the opportunity to test this hypothesis, comparing outdating rates before and after centralization of product distribution. Our systematic review of benchmarking identified a clear paucity of such programs in transfusion medicine. ORBCoN’s program is a true benchmarking system, as it not only studies data trends over time, but collaborates
RBC OUTDATING BENCHMARKS
ACKNOWLEDGMENTS The authors acknowledge the commitment to quality of Ontario Hospital Transfusion Medicine Services, who monitor and report their data to Canadian Blood Services on an ongoing basis. The authors also extend their appreciation to the staff of ORBCoN, for their diligent work promoting benchmarking and inventory management. Denise Evanovitch and Anushka Jaffer’s assistance with the review and formatting of the manuscript was instrumental.
CONFLICT OF INTEREST The authors have disclosed no conflicts of interest.
REFERENCES Fig. 5. Components of a multipronged benchmarking strategy.
1. Kearns DT. Leadership through quality. Executive 1990;4: 86-89. 2. Walker R. Rank Xerox—management revolution. Long Range Plann 1992;25:9-21.
with participants to address the variability observed and continuously reevaluates performance leading to practice changes. Apelseth and colleagues4 underscored that benchmarking is a cyclic and dynamic process that evolves constantly. Adoption of this cyclic and dynamic model sets our program apart from those that simply observe trends (without communicating findings to participants) and those that do one-time audits (ignoring change over time). Our experience has informed the creation of a conceptual framework that can be used to develop other programs (Fig. 4). The conceptual model illustrates the iterative nature of benchmarking. Success of the program depends on continuous reevaluation of performance and continuous reinforcement of education through knowledge translation strategies. A robust data system represents a second iterative process that is embedded in the conceptual framework, encouraging constant performance reporting (as data flows from the participating hospitals to the data center) and constant knowledge translation (as data flows from the management center to the participating hospitals). All components of a multipronged strategy (Fig. 5) to decrease RBC outdating are featured in the program described in this article. Each factor is essential to a successful benchmarking strategy: dynamic targets, an implementation program based on evidence-based knowledge translation principles and practical inventory management tools, audit and recognition programs, ongoing information sharing between participating sites, the data center and the management center, and a robust data system to track information. Our multipronged strategy is a conceptual iterative framework for benchmarking in transfusion medicine, which can be adopted by other regions wishing to improve their transfusion practices.
3. Bisognano MA. Leadership strategies for breakthrough change in health care. In: Neuorientierung im Gesundheitswesen. New York: Springer; 1998. p. 89-100. 4. Apelseth TO, Molnar L, Arnold E, et al. Benchmarking: applications to transfusion medicine. Transfus Med Rev 2012;26: 321-32. 5. Heddle NM, Liu Y, Barty R, et al. Factors affecting the frequency of red blood cell outdates: an approach to establish benchmarking targets. Transfusion 2009;49:219-26. 6. Collins MA. Validation of a shipping container for inter hospital transfer of red blood cells. Can J Med Lab Sci 2010; 72:62-9. 7. Ontario Regional Blood Coordinating Network. (ORBCoN). Homepage. [cited 2014 April 1]. Available from: http://transfusionontario.org/ 8. Dzik WS, Beckman N, Selleng K, et al. Errors in patient specimen collection: application of statistical process control. Transfusion 2008;48:2143-51. 9. Ontario Contingency Plan for the Management of Blood Shortages Version 2 October 30, 2012. Ontario. Ministry of Health and Long-Term Care; 2012. [cited 2014 April 1] Available from: http://transfusionontario.org/en/cmdownloads/ categories/emergency_blood. 10. National Plan for the Management of Shortages of Labile Blood Components. National Advisory Committee on Blood and Blood Components and Canadian Blood Services. 201201-18. [cited 2014 April 1] Available from: http://www.nacblood.ca/resources/shortages-plan/National-shortagesplan-Jan-2012-final.pdf. 11. Heddle N, Barty R. Benchmarking: applications to transfusion medicine. ISBT Sci Ser 2013;8:93-99. 12. Grimshaw JM, Eccles MP, Lavis JN, et al. Knowledge translation of research findings. Implementation Sci 2012; 7:50. Volume 55, July 2015 TRANSFUSION 1627
