Focus on Quality
Original Contribution
Implementing a User-Driven Online Quality Improvement Toolkit for Cancer Care By Jeff Luck, MBA, PhD, Laura S. York, PhD, Candice Bowman, RN, PhD, Randall C. Gale, DrPH, Nina Smith, MPH, and Steven M. Asch, MD, MPH Oregon State University, Corvallis, OR; Veterans Administration (VA) Greater Los Angeles Healthcare System, Los Angeles; VA Palo Alto Health Care System; and Stanford University, Palo Alto, CA
Abstract Purpose: Peer-to-peer collaboration within integrated health systems requires a mechanism for sharing quality improvement lessons. The Veterans Health Administration (VA) developed online compendia of tools linked to specific cancer quality indicators. We evaluated awareness and use of the toolkits, variation across facilities, impact of social marketing, and factors influencing toolkit use.
site, of whom 24% downloaded at least one tool. Respondents’ awareness of the lung cancer quality performance of their facility, and facility participation in quality improvement collaboratives, were positively associated with Toolkit Series site use. Facilitylevel lung cancer tool implementation varied widely across tool types.
Conclusion: The VA Toolkit Series achieved widespread use Methods: A diffusion of innovations conceptual framework guided the collection of user activity data from the Toolkit Series SharePoint site and an online survey of potential Lung Cancer Care Toolkit users.
Results: The VA Toolkit Series site had 5,088 unique visitors in its first 22 months; 5% of users accounted for 40% of page views. Social marketing communications were correlated with site usage. Of survey respondents (n ⫽ 355), 54% had visited the
and a high degree of user engagement, although use varied widely across facilities. The most active users were aware of and active in cancer care quality improvement. Toolkit use seemed to be reinforced by other quality improvement activities. A combination of user-driven tool creation and centralized toolkit development seemed to be effective for leveraging health information technology to spread disease-specific quality improvement tools within an integrated health care system.
Introduction
Methods
The growth of integrated health systems is being encouraged by the Affordable Care Act and other market forces,1,2 but taking advantage of integration to reduce variation in the quality of care remains a challenge. The Veterans Health Administration (VA), a national integrated health system, has achieved high performance on many quality measures.3 Nevertheless, opportunities for improvement remain, including the provision of cancer care. For example, timeliness of lung cancer care was observed to vary widely across VA medical centers (hereafter, facilities).4-6 One promising approach to reducing clinical practice variation is online collaboration. Peer-to-peer sharing platforms can spread innovations developed by clinicians and quality improvement (QI) specialists across sites.7 Innovations may be utilities based in the electronic health record (EHR) or organizational or educational in nature. However, little is known about how widely online collaborative QI resources are used or which organizational factors promote or hinder their use. The VA has developed a series of online toolkits containing resources to improve performance on specific measures of cancer care. Our evaluation analyzed use of the Toolkit Series site and data from an online survey of potential Lung Cancer Care Toolkit users. Here we describe patterns of Toolkit Series use across VA facilities, facility and user characteristics that predict toolkit use, and implementation of lung cancer tools.
Cancer QI Tools and Toolkits
Copyright © 2015 by American Society of Clinical Oncology
M A Y 2015
In 2009, the VA Office of Analytics and Business Intelligence (OABI) and the Quality Enhancement Research Initiative began collaborating to develop online compendia of QI tools. On the basis of an in-depth OABI study of lung cancer care in the VA,8 the Lung Cancer Care Toolkit was chosen as the first in the VA Toolkit Series. The Office of Systems Redesign9 supported toolkit development and dissemination and provided health information technology (HIT) support. Additional toolkits were subsequently developed for colorectal, prostate, and head and neck cancers as well as for palliative care. A toolkit is an online resource package of ready-to-download tools that can help facilities better manage the care of patients with a specific type of cancer. Each tool is a tangible clinical resource that is not part of system-wide VA routine clinical practice. Although a tool may be clinical, technologic, organizational, or educational, it must comprise an electronic file users can download, such as a database, electronic document, spreadsheet, video, slideshow, or EHR note template. The Data Supplement provides a sample Lung Cancer Care Toolkit page. The Toolkit Series development team solicited user-developed tools from VA facilities nationwide. Tools in use at one or more facilities and that could be linked to VA cancer performance measures were eligible for inclusion and were vetted by a •
jop.ascopubs.org
e421
Luck et al
panel of cancer and QI experts. The VA Toolkit Series is hosted on a VA intranet SharePoint (Microsoft, Redmond, WA) site accessible to all staff. The Lung Cancer Care Toolkit was launched nationwide in November 2010, concurrent with the OABI release of facilitylevel lung cancer performance data. The other toolkits followed through 2012. A social marketing approach10 aimed to accelerate dissemination to target users, especially oncology clinicians and facility quality managers.11 The Toolkit Series development team demonstrated the toolkits and distributed brochures at three phases of national VA Cancer Care Collaboratives (hereafter, collaboratives), where teams gathered to learn and share cancer QI techniques. Collaborative participants were invited to help spread the word among their colleagues. The development team compiled e-mail distribution lists of potential users and used them to distribute e-brochures. The team also made presentations to VA opinion leaders, who were asked to forward marketing messages to their departments and teams. Periodic marketing messages (every 6 to 8 weeks) were sent to the toolkit users’ listserv to highlight new tools and Web site features.
Facilities Target users of the VA Toolkit Series were staff and clinicians involved in cancer care or QI at VA facilities nationwide. Each of 142 facilities was classified as high, medium, or low complexity, using a VA algorithm based on patient volume, clinical resources, and research funding. Cancer care or QI personnel at central and regional VA administrative offices were also potential users. An online survey (described under Online Survey) targeted potential Lung Cancer Care Toolkit users at the 108 facilities that provided care to ⱖ 10 veterans with lung cancer in 2007. The average annual number of lung cancer cases per facility was 72, with a maximum of 199. Most (61%) of these facilities were of high complexity; 8% were of low complexity. Between three and 10 of these facilities were located in each of the 21 regional VA networks, known as Veterans Integrated Service Networks. Twenty-eight facilities sent lung cancer teams to a collaborative, and 24 sent teams to a collaborative for one or more other cancer types.
Toolkit Series Site Use Using SharePoint 2007 (subsequently SharePoint 2010) analytics, we collected data on all Toolkit Series visits and counted unique users. Because of some data gaps during software upgrades, we used conservative methods to count visits and unique users. Toolkit Series development and evaluation team members were excluded. SharePoint provided data only for combined user traffic to the Toolkit Series site, not for individual toolkits. We present data for the first 22 months of Toolkit Series use, through September 2012.
Online Survey We conducted a targeted online survey of potential Lung Cancer Care Toolkit users to assess: (1) their awareness and use of e422
JOURNAL
OF
ONCOLOGY PRACTICE
•
this toolkit, and (2) facility-level use of specific lung cancer QI tools. To identify potential users, we collected e-mail addresses from the Toolkit Series listserv as well as for VA oncology physicians and nurses, quality managers, cancer registrars, and collaborative lung cancer care teams. Lists were de-duplicated and concatenated; e-mail solicitations to participate in the survey were endorsed by senior VA leaders. Starting approximately 36 weeks after the Lung Cancer Care Toolkit launch, we solicited 1,897 potential respondents, with three waves of follow-up, over 7 weeks from August to October 2011. Respondents identified their facility at the beginning of the survey; those not from one of the 108 target facilities were screened out of further questioning. We received 355 complete surveys, for a response rate of 18.7%. For tool implementation questions, we required at least two complete responses per facility, from at least two different clinical disciplines, to better capture the diversity of user experiences. Responses from 98 facilities met those criteria, for a facility response rate of 91%. We used Survey Monkey to present close-ended questions in three areas: user characteristics, such as clinical credentials and years working in VA; individual awareness, access, and use of the toolkit; and use of specific types of lung cancer tools at the respondent’s facility. To count adoption of specific tool types, we assumed that a “yes” response from any user at a facility trumped “no” or “don’t know” responses from that facility; absent any “yes” response for a specific tool type at a facility, any “no” trumped any “don’t know” response. The VA Greater Los Angeles Healthcare System Institutional Review Board approved the survey.
Statistical Analyses We completed descriptive analyses of site use and survey responses and tested the correlation between social marketing communications and site use changes using linear regression. Multivariable logistic regression was used to estimate the significance of potential predictors of toolkit use. Analyses were conducted using STATA (version 12; STATA, College Station, TX).
Results The VA Toolkit Series site was visited on a sustained basis from first availability in November 2010 through September 2012. The weekly average number of users was 100.7, with returning users accounting for 49.6% (on average) of users each week. Spikes in toolkit use were correlated (P ⬍ .01) with social marketing messages and broad communication about evaluation. There were 5,088 total unique visitors to the Toolkit Series site over the first 22 months of availability. The average cumulative number of clicks per user was 13; the median was five. Some so-called super-users accessed the toolkits frequently and accounted for a significant portion of visits; the top 1% of users accounted for 16% of total clicks on the site, and the top 5% of users for 40% of clicks. Each of the 21 VA regional networks and all 142 facilities were represented among Toolkit Series users. The largest num-
V O L . 11, I S S U E 3
Copyright © 2015 by American Society of Clinical Oncology
Online Toolkit for Cancer Care
ber of users per network was 512; the smallest was 63. There was also wide variation in use across VA facilities. The average number of users per facility was 34.8, with a maximum of 167 and a minimum of two. Survey responses provided additional detail about target users of the Lung Cancer Care Toolkit. As summarized in Table 1, nearly half (47%) of respondents were physicians, and almost one quarter (23%) were nurses or physician assistants. Also represented were case managers, cancer registrars, and quality managers. Fewer than half (44%) of respondents had worked in cancer care or cancer care QI for ⱖ 5 years. A similar proportion (46%) was aware of the performance of their facility on the study of lung cancer care quality. Respondents had mixed views of how easy it was to implement change at their facility, with 29% feeling it was very or somewhat easy, and 37% feeling it was somewhat or very difficult. More than half of survey respondents (54%) had visited the Toolkit Series site. Of those who did, 30% spent a few minutes there, 46% spent from a half hour to 1 hour, and 24% spent more than 1 hour. The vast majority of respondents who visited the site (91%) viewed at least one tool, and 24% downloaded at least one tool. Fewer than 10% either requested a tool that could not be immediately downloaded, uploaded or suggested a tool, or joined the online discussion. Of respondents who had not visited the Toolkit Series site, 61% stated the main reason was because they had not heard of it and 18% because they had not had time. Multivariable regression models (Table 2) showed that survey respondents who were aware of the performance of their facility on the OABI lung cancer study were much more likely to have visited the Toolkit Series site (P ⬍ .01). Nonphysicians were more likely than physicians to have visited the site (P ⫽ .02). Perceived ease of implementation at their facility was not associated with likelihood of visiting the site, nor was the length of time working in lung cancer care or QI. Respondents from facilities that had sent a lung cancer team to a VA Cancer Care Collaborative (P ⫽ .05) or had a larger number of lung cancer cases (P ⫽ .04) were more likely to have visited the site. Facility complexity was not associated with likelihood of visiting the site. The survey also measured whether different types of lung cancer QI tools had been implemented at respondents’ facilities (Table 3). Some EHR-based tools were reported as being widely available: pain assessment reminders at 91% of facilities; templates for specialty or imaging referrals at 79%; and templates for cancer diagnosis or staging and for chemotherapy ordering available at 66% and 62% of facilities, respectively. Other types of EHR tools, including pathology and nursing templates and links to guidelines, were available at fewer than half of facilities. Databases for patient tracking or performance measurement were implemented at 73% of facilities. Flowcharts or other decision guides for lung cancer were available in 62% of facilities, but flowcharts or tools for palliative care were available at only 48%. Some organizational tools were widely implemented, including service agreements at 86% of facilities and cancer patient coordinators at 79%. Interdisciplinary tools for Copyright © 2015 by American Society of Clinical Oncology
M A Y 2015
Table 1. Characteristics and Selected Responses of Respondents to Online Survey of Lung Cancer Care Toolkit Use (n ⫽ 355) Characteristic
%
Respondent specialty and role Physician Oncology
14.9
Primary care/internal medicine
10.7
Pulmonology Other
8.7 13.0
Advanced practice nurse or physician assistant
11.0
Nurse
12.1
Case manager
5.1
Cancer registrar
5.6
Quality manager Other
5.6 13.2
Experience in cancer care or cancer care QI, years ⱖ5
43.9
⬍ 5 or do not work in cancer care or QI
56.1
Aware of facility performance on lung cancer care study? Yes
45.6
No
54.4
Ease of implementing change at facility Very or somewhat easy
28.7
Neither easy nor difficult
34.1
Somewhat or very difficult
37.2
Visited Toolkit Series site? Yes
54.1
No
45.9
Abbreviation: QI, quality improvement.
cancer care were somewhat less widely available, including tumor board presentation standards or templates at 65% of facilities and multidisciplinary cancer care clinics at 62%. More than 40% of facilities allowed videoconference participation in tumor boards. Patient education tools were widely available (88% of facilities), and patient or caregiver support resources somewhat less widely (76%).
Discussion We evaluated the scope, patterns, and predictors of use of online toolkits designed to share cancer QI tools across sites within the national VA integrated health system. The Toolkit Series attracted more than 5,000 unique users within the first 2 years of availability. It was used in all 21 VA regional networks and all 108 facilities that had a significant volume of lung cancer cases. Several different types of lung cancer QI tools, such as service agreements, cancer patient coordinator position descriptions, and patient education materials, were implemented at a substantial majority of these facilities. Because cancer care encompasses screening, diagnosis, treatment, and end-of-life care, cancer care QI must be assessed within the context of a multilevel care delivery environment.12 A wide range of cancer care QI approaches have been studied, •
jop.ascopubs.org
e423
Luck et al
Table 2. Predictors of Toolkit Series Site Use Characteristic
OR
95% CI
P
Team sent to VA Cancer Care Collaborative (reference: none)
Yes (%)
Tool
No (%)
Don’t Know (%)
EHR*
Lung only
1.73
1.00 to 2.98
.05
Other only
1.09
0.46 to 2.57
.85
Both
Table 3. Implementation of Lung Cancer Quality Improvement Tools at VA Facilities (n ⫽ 98)
Reminders for pain assessment or patient education for pain
91
7
2
Templates for referrals to specialty clinics or imaging
79
15
6
Templates for diagnosis and staging (eg, lung nodule assessment)
66
24
9
1.01
0.55 to 1.85
.98
1.01
1.00 to 1.01
.04
Medium
1.71
0.89 to 3.29
.11
Templates for chemotherapy ordering
62
30
8
Low
1.72
0.77 to 3.82
.18
46
38
16
1.69
1.07 to 2.67
.02
Pathology templates or work aids specific to lung cancer Nursing templates for cancer (eg, symptom documentation)
46
37
17
Link from EHR to NCCN or other cancer care guidelines
44
44
12
Databases for local patient tracking or performance measurement
73
11
15
Lung cancer care flowcharts, pocket guides, or other paper-based decision guides
62
26
12
Palliative care flowchart or screening tools
48
28
24
No. of lung cancer cases in 2007 Facility complexity (reference: high)
Role (reference: physician) Perceived ease of implementation (reference: very difficult) Very easy
0.91
0.22 to 3.69
.89
Somewhat easy
1.21
0.44 to 3.35
.72
Neither easy nor difficult
1.34
0.51 to 3.57
.55
Somewhat difficult
1.67
0.61 to 4.53
.32
Length of work, years (reference: does not work in cancer care or QI) 1 to 4
1.87
ⱖ5
0.99 to 3.55
.05
Non-EHR
Organizational
0.85
0.44 to 1.63
.62
Aware of facility performance on lung cancer care study
3.10
1.91 to 5.01
.00
Service agreements (eg, with specialty clinics, imaging, or palliative care)
86
9
5
Constant
0.17
0.05 to 0.63
.01
Cancer patient coordinator (eg, care coordinator, navigator, case manager)
79
17
4
Standards or templates for presenting lung cancer cases at tumor board
65
24
10
Multidisciplinary cancer care clinic
62
32
6
Dedicated clinic for lung nodule assessment and surveillance
57
38
5
Videoconference participation in tumor board
42
51
7
Patient education materials related to diagnosis, treatment, or palliative care
88
7
5
Support groups for patients with cancer or caregiver resources
76
16
8
Abbreviations: OR, odds ratio; QI, quality improvement; VA, Veterans Administration.
including: team-based care or communities of practice13-15; clinical report tools16; common assessment tools, treatment plans, and symptom management guidelines17,18; peer comparison or audit and feedback19-24; training or education23,25,26; and tumor registries.27 The best way to share these approaches has been less studied. HIT is one potential mechanism; it has been shown to support and reinforce many QI approaches.28,29 Online toolkits are an HIT-enabled technique allowing integrated health systems to share QI approaches across sites. To date, few studies have evaluated the effectiveness of toolkits.30-34 The VA Toolkit Series achieved widespread use and a high degree of user engagement. Users visited the Toolkit Series multiple times, with return users consistently accounting for half of visitors. More than two of three survey respondents who visited the site spent at least half an hour there. Approximately one in four downloaded ⱖ one tool, although fewer than one in 10 uploaded or suggested a tool or participated in an online discussion forum. Nurses, case managers, and other nonphysician VA cancer care staff were more likely than physicians to have visited the Toolkit Series site, perhaps because of their broader care planning and coordination responsibilities. Although we were unable to comprehensively estimate the total number of VA cancer care and QI professionals who were potential Toolkit Series users, the actual number of users is more than an order of magnitude larger than the approximately 428 full-time equivalent hematology/oncology physicians and residents in the VA. e424
JOURNAL
OF
ONCOLOGY PRACTICE
•
Other
Abbreviations: EHR, electronic health record; NCCN, National Comprehensive Cancer Network; VA, Veterans Administration. * VA Computerized Patient Record System.
A core group of super-users visited the Toolkit Series site most frequently, consistent with the anecdotal observation of the development team that a subset of users is active in cancer QI and catalyzes those activities within their facilities. Online toolkits can enable such change agents to draw on ideas generated at other facilities as well as to spread their own ideas to other facilities. The strongest predictor of Toolkit Series use by survey respondents was awareness of the performance of their facility in a national VA study of lung cancer care. Interestingly, respondents who perceived tool implementation as being relatively more difficult at their facility were no less likely to visit the Toolkit Series site. Respondents whose facility had sent a lung cancer team to a Cancer Care Collaborative were more likely to have visited the site. Although performance improvement collaboratives have been examined in cancer care,35-37 clear and
V O L . 11, I S S U E 3
Copyright © 2015 by American Society of Clinical Oncology
Online Toolkit for Cancer Care
sustained benefits have not been uniformly observed.38-42 Our results suggest that toolkits and collaboratives may be mutually reinforcing techniques for sharing QI lessons across facilities. Systematic social marketing of the Toolkit Series also seemed to be effective in stimulating use. The cumulative number of users grew steadily, and use remained high nearly 2 years after initial availability. Significant correlation between social marketing communications and spikes in Toolkit Series site use suggests that ongoing marketing helped to maintaining user engagement. The number of Toolkit Series users varied widely across facilities. As expected, survey respondents from facilities with more lung cancer cases were more likely to use the toolkits. However, facility complexity was not a significant predictor of use, suggesting that less complex facilities with fewer internal QI resources can benefit from shared innovation within the context of an integrated health system. Reported use rates for specific types of lung cancer QI tools also varied across facilities. Although some EHR templates, such as for referrals and pain assessment, were implemented at most facilities, other templates, such as for nursing, were less widely used, even though all VA facilities have a comprehensive EHR system. Patient education tools, which require only modest implementation efforts, were widely used. Implementation rates for organizational tools, such as service agreements and care coordination, were also high, even though implementing such tools requires significant organizational commitment. Patient flowcharts were widely used, but fewer than half of facilities used such tools for palliative care. Variation in use across facilities and tool types suggests the market for toolkit use within the VA is not yet saturated. More than four in 10 survey respondents had not yet visited the Toolkit Series site, most commonly because they had not previously heard of it or had not had time to visit. This suggests that additional social marketing efforts may be effective. For example, in-person or telephone promotion efforts could target potential users at facilities with sizeable oncology caseloads but few toolkit users. Toolkit demonstrations could be performed at regional or national meetings of VA oncologists. Outreach could also target members of other disciplines in the cancer care team, including advanced practice nurses, social workers, and palliative care clinicians. This study has several limitations. The VA comprehensive EHR may make implementation of QI interventions and computerized tools easier than in other health systems. Reported rates of tool implementation may include tools that had been implemented before availability in the online toolkits. Survey
respondents may not be as representative of potential toolkit users as of facilities, although the response rate was consistent with expectations for online surveys. Future research on clinical outcomes will be needed to determine whether toolkit use is associated with higher-quality cancer care. In conclusion, the VA developed online compendia of cancer care QI tools targeted to specific measures of quality and timeliness. All tools were contributed by VA facilities where they were in use. Systematic marketing of these toolkits was amplified by VA performance improvement collaboratives and by toolkit super-users. Toolkits were widely accessed by target audiences of clinicians and QI staff, at facilities of all complexity levels. This model of user-driven tool creation and centralized toolkit development seems to be an effective mechanism for leveraging HIT and facility-level innovation to spread diseasespecific QI tools within a nationwide integrated health care system. Acknowledgment Supported by the Veterans Administration (VA) Office of Analytics and Business Intelligence and VA Office of Systems Redesign, with additional in-kind support from VA Center for Applied Systems Engineering. The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs or the US Government. We thank the following toolkit development and evaluation staff: Jenny Barnard, BA, Marlin Elenes, MPH, Carla Alvarado, MPH, Tonya Reznor, BA, Deborah Griffith, EdD, and Gail Edwards, RN. Authors’ Disclosures of Potential Conflicts of Interest Disclosures provided by the authors are available with this article at jop.ascopubs.org.
Author Contributions Conception and design: Jeff Luck, Candice Bowman, Steven M. Asch Administrative support: Randall C. Gale Collection and assembly of data: Jeff Luck, Laura S. York, Candice Bowman, Nina Smith, Steven M. Asch Data analysis and interpretation: Jeff Luck, Candice Bowman, Randall C. Gale, Steven M. Asch Manuscript writing: All authors Final approval of manuscript: All authors Corresponding author: Jeff Luck, MBA, PhD, College of Public Health and Human Sciences, Oregon State University, 401 Waldo Hall, Corvallis, OR 97331-6406; e-mail: [email protected].
DOI: 10.1200/JOP.2014.003012; published online ahead of print at jop.ascopubs.org on April 7, 2015.
References 1. Flood AB, Fennell ML, Devers KJ: Health reforms as examples of multilevel interventions in cancer care. J Natl Cancer Inst Monogr 2012:80-85, 2012 2. Korda H, Eldridge GN: Payment incentives and integrated care delivery: Levers for health system reform and cost containment. Inquiry 48:277-287, 2011-2012 3. Asch SM, McGlynn EA, Hogan MM, et al: Comparison of quality of care for patients in the Veterans Health Administration and patients in a national sample. Ann Intern Med 141:938-945, 2004
Copyright © 2015 by American Society of Clinical Oncology
M A Y 2015
4. Schultz EM, Powell AA, McMillan A, et al: Hospital characteristics associated with timeliness of care in veterans with lung cancer. Am J Respir Crit Care Med 179:595-600, 2009 5. Powell AA, Schultz EM, Ordin DL, et al: Timeliness across the continuum of care in veterans with lung cancer. J Thorac Oncol 3:951-957, 2008 6. Gould MK, Ghaus SJ, Olsson JK, et al: Timeliness of care in veterans with non-small cell lung cancer. Chest 133:1167-1173, 2008
•
jop.ascopubs.org
e425
Luck et al
7. Yano EM, Green LW, Glanz K, et al: Implementation and spread of interventions into the multilevel context of routine practice and policy: Implications for the cancer care continuum. J Natl Cancer Inst Monogr 2012:86-99, 2012
25. Coe SG, Crook JE, Diehl NN, et al: An endoscopic quality improvement program improves detection of colorectal adenomas. Am J Gastroenterol 108: 219-226, 2013; quiz 227
8. Veterans Health Administration Office of Analytics and Business Intelligence: The Quality of VHA Lung Cancer Care: Special Study. Washington, DC, Veterans Health Administration, 2010
26. Wright FC, Law CH, Last LD, et al: A blended knowledge translation initiative to improve colorectal cancer staging [ISRCTN56824239]. BMC Health Serv Res 6:4, 2006
9. Davies MV: VHA systems redesign: A plan for organizational transformation. www. hsrd.research.va.gov/for_researchers/cyber_seminars/archives/qip-052609.pdf
27. Redondo M, Rivas-Ruiz F, Guzman-Soler MC, et al: Monitoring indicators of health care quality by means of a hospital register of tumours. J Eval Clin Pract 14:1026-1030, 2008
10. Luck J, Hagigi F, Parker LE, et al: A social marketing approach to implementing evidence-based practice in VHA QUERI: The TIDES depression collaborative care model. Implement Sci 4:64, 2009
28. Au M, Felt-Lisk S, Anglin G, et al: Using Health IT: Eight Quality Improvement Stories. Rockville, MD, Agency for Healthcare Research and Quality, 2010
11. York L, Bruce B, Luck J, et al: Online toolkits for metric-driven quality improvement: The Veterans Health Administration managed grassroots approach. Jt Comm J Qual Patient Saf 39:561-569, 2013
29. Keyser DJ, Dembosky JW, Kmetik K, et al: Using health information technology-related performance measures and tools to improve chronic care. Jt Comm J Qual Patient Saf 35:248-255, 2009
12. Taplin SH, Anhang Price R, Edwards HM, et al: Introduction: Understanding and influencing multilevel factors across the cancer care continuum. J Natl Cancer Inst Monogr 2012:2-10, 2012
30. DeWalt DA, Broucksou KA, Hawk V, et al: Developing and testing the health literacy universal precautions toolkit. Nurs Outlook 59:85-94, 2011
13. Elward KS, Penberthy LT, Bear HD, et al: Collaborative initiatives in the care of early stage breast cancer: Reduction in variation of breast-conserving therapy. Breast J 6:108-114, 2000 14. Farber S, Andersen W, Branden C, et al: Improving cancer pain management through a system wide commitment. J Palliat Med 1:377-385, 1998 15. Fung-Kee-Fung M, Goubanova E, Sequeira K, et al: Development of communities of practice to facilitate quality improvement initiatives in surgical oncology. Qual Manag Health Care 17:174-185, 2008 16. Dubenske LL, Chih MY, Dinauer S, et al: Development and implementation of a clinician reporting system for advanced stage cancer: Initial lessons learned. J Am Med Inform Assoc 15:679-686, 2008 17. Dudgeon DJ, Knott C, Eichholz M, et al: Palliative care integration project (PCIP) quality improvement strategy evaluation. J Pain Symptom Manage 35:573582, 2008 18. Hains IM, Ward RL, Pearson SA: Implementing a Web-based oncology protocol system in Australia: Evaluation of the first 3 years of operation. Intern Med J 42:57-64, 2012 19. Aletti GD, Dowdy SC, Gostout BS, et al: Quality improvement in the surgical approach to advanced ovarian cancer: The Mayo Clinic experience. J Am Coll Surg 208:614-620, 2009 20. Dulko D: Audit and feedback as a clinical practice guideline implementation strategy: A model for acute care nurse practitioners. Worldviews Evid Based Nurs 4:200-209, 2007 21. Mansell G, Shapley M, Jordan JL, et al: Interventions to reduce primary care delay in cancer referral: A systematic review. Br J Gen Pract 61:e821-e835, 2011
31. Eagle KA, Gallogly M, Mehta RH, et al: Taking the national guideline for care of acute myocardial infarction to the bedside: Developing the guideline applied in practice (GAP) initiative in Southeast Michigan. Jt Comm J Qual Improv 28:5-19, 2002 32. Nowalk MP, Nutini J, Raymund M, et al: Evaluation of a toolkit to introduce standing orders for influenza and pneumococcal vaccination in adults: A multimodal pilot project. Vaccine 30:5978-5982, 2012 33. Speroff T, Ely EW, Greevy R, et al: Quality improvement projects targeting health care-associated infections: Comparing virtual collaborative and toolkit approaches. J Hosp Med 6:271-278, 2011 34. Wiechula R, Kitson A, Marcoionni D, et al: Improving the fundamentals of care for older people in the acute hospital setting: Facilitating practice improvement using a knowledge translation toolkit. Int J Evid Based Healthc 7:283-295, 2009 35. Miller DC, Murtagh DS, Suh RS, et al: Regional collaboration to improve radiographic staging practices among men with early stage prostate cancer. J Urol 186:844-849, 2011 36. Powell AA, Nugent S, Ordin DL, et al: Evaluation of a VHA collaborative to improve follow-up after a positive colorectal cancer screening test. Med Care 49:897-903, 2011 37. Shaw EK, Ohman-Strickland PA, Piasecki A, et al: Effects of facilitated team meetings and learning collaboratives on colorectal cancer screening rates in primary care practices: A cluster randomized trial. Ann Fam Med 11:220-228, 2013 38. Landon BE, Wilson IB, McInnes K, et al: Effects of a quality improvement collaborative on the outcome of care of patients with HIV infection: The EQHIV study. Ann Intern Med 140:887-896, 2004 39. Mittman BS: Creating the evidence base for quality improvement collaboratives. Ann Intern Med 140:897-901, 2004
22. O’Grady MA, Gitelson E, Swaby RF, et al: Development and implementation of a medical oncology quality improvement tool for a regional community oncology network: The Fox Chase Cancer Center partners initiative. J Natl Compr Canc Netw 5:875-882, 2007
40. Nembhard IM, Northrup V, Shaller D, et al: Improving organizational climate for quality and quality of care: Does membership in a collaborative help? Med Care 50:S74-S82, 2012 (suppl)
23. Simunovic M, Coates A, Goldsmith CH, et al: The cluster-randomized quality initiative in rectal cancer trial: Evaluating a quality-improvement strategy in surgery. CMAJ 182:1301-1306, 2010
41. Schouten LM, Hulscher ME, van Everdingen JJ, et al: Evidence for the impact of quality improvement collaboratives: Systematic review. BMJ 336:1491-1494, 2008
24. Winickoff RN, Coltin KL, Morgan MM, et al: Improving physician performance through peer comparison feedback. Med Care 22:527-534, 1984
42. ØVretveit J, Bate P, Cleary P, et al: Quality collaboratives: Lessons from research. Qual Saf Health Care 11:345-351, 2002
e426
JOURNAL
OF
ONCOLOGY PRACTICE
•
V O L . 11, I S S U E 3
Copyright © 2015 by American Society of Clinical Oncology
Online Toolkit for Cancer Care
AUTHORS’ DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST Implementing a User-Driven Online Quality Improvement Toolkit for Cancer Care The following represents disclosure information provided by authors of this manuscript. All relationships are considered compensated. Relationships are self-held unless noted. I ⫽ Immediate Family Member, Inst ⫽ My Institution. Relationships may not relate to the subject matter of this manuscript. For more information about ASCO’s conflict of interest policy, please refer to www.asco.org/rwc or jop.ascopubs.org/site/misc/ifc.xhtml. Jeff Luck Leadership: Biome Analytics Stock or Other Ownership: Biome Analytics Consulting or Advisory Role: Biome Analytics Patents, Royalties, Other Intellectual Property: Biome Analytics
Randall C. Gale No relationship to disclose Nina Smith No relationship to disclose Steven M. Asch No relationship to disclose
Laura S. York No relationship to disclose Candice Bowman No relationship to disclose
Copyright © 2015 by American Society of Clinical Oncology
M A Y 2015
•
jop.ascopubs.org
e427
Copyright of Journal of Oncology Practice is the property of American Society of Clinical Oncology and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
Original Contribution
Implementing a User-Driven Online Quality Improvement Toolkit for Cancer Care By Jeff Luck, MBA, PhD, Laura S. York, PhD, Candice Bowman, RN, PhD, Randall C. Gale, DrPH, Nina Smith, MPH, and Steven M. Asch, MD, MPH Oregon State University, Corvallis, OR; Veterans Administration (VA) Greater Los Angeles Healthcare System, Los Angeles; VA Palo Alto Health Care System; and Stanford University, Palo Alto, CA
Abstract Purpose: Peer-to-peer collaboration within integrated health systems requires a mechanism for sharing quality improvement lessons. The Veterans Health Administration (VA) developed online compendia of tools linked to specific cancer quality indicators. We evaluated awareness and use of the toolkits, variation across facilities, impact of social marketing, and factors influencing toolkit use.
site, of whom 24% downloaded at least one tool. Respondents’ awareness of the lung cancer quality performance of their facility, and facility participation in quality improvement collaboratives, were positively associated with Toolkit Series site use. Facilitylevel lung cancer tool implementation varied widely across tool types.
Conclusion: The VA Toolkit Series achieved widespread use Methods: A diffusion of innovations conceptual framework guided the collection of user activity data from the Toolkit Series SharePoint site and an online survey of potential Lung Cancer Care Toolkit users.
Results: The VA Toolkit Series site had 5,088 unique visitors in its first 22 months; 5% of users accounted for 40% of page views. Social marketing communications were correlated with site usage. Of survey respondents (n ⫽ 355), 54% had visited the
and a high degree of user engagement, although use varied widely across facilities. The most active users were aware of and active in cancer care quality improvement. Toolkit use seemed to be reinforced by other quality improvement activities. A combination of user-driven tool creation and centralized toolkit development seemed to be effective for leveraging health information technology to spread disease-specific quality improvement tools within an integrated health care system.
Introduction
Methods
The growth of integrated health systems is being encouraged by the Affordable Care Act and other market forces,1,2 but taking advantage of integration to reduce variation in the quality of care remains a challenge. The Veterans Health Administration (VA), a national integrated health system, has achieved high performance on many quality measures.3 Nevertheless, opportunities for improvement remain, including the provision of cancer care. For example, timeliness of lung cancer care was observed to vary widely across VA medical centers (hereafter, facilities).4-6 One promising approach to reducing clinical practice variation is online collaboration. Peer-to-peer sharing platforms can spread innovations developed by clinicians and quality improvement (QI) specialists across sites.7 Innovations may be utilities based in the electronic health record (EHR) or organizational or educational in nature. However, little is known about how widely online collaborative QI resources are used or which organizational factors promote or hinder their use. The VA has developed a series of online toolkits containing resources to improve performance on specific measures of cancer care. Our evaluation analyzed use of the Toolkit Series site and data from an online survey of potential Lung Cancer Care Toolkit users. Here we describe patterns of Toolkit Series use across VA facilities, facility and user characteristics that predict toolkit use, and implementation of lung cancer tools.
Cancer QI Tools and Toolkits
Copyright © 2015 by American Society of Clinical Oncology
M A Y 2015
In 2009, the VA Office of Analytics and Business Intelligence (OABI) and the Quality Enhancement Research Initiative began collaborating to develop online compendia of QI tools. On the basis of an in-depth OABI study of lung cancer care in the VA,8 the Lung Cancer Care Toolkit was chosen as the first in the VA Toolkit Series. The Office of Systems Redesign9 supported toolkit development and dissemination and provided health information technology (HIT) support. Additional toolkits were subsequently developed for colorectal, prostate, and head and neck cancers as well as for palliative care. A toolkit is an online resource package of ready-to-download tools that can help facilities better manage the care of patients with a specific type of cancer. Each tool is a tangible clinical resource that is not part of system-wide VA routine clinical practice. Although a tool may be clinical, technologic, organizational, or educational, it must comprise an electronic file users can download, such as a database, electronic document, spreadsheet, video, slideshow, or EHR note template. The Data Supplement provides a sample Lung Cancer Care Toolkit page. The Toolkit Series development team solicited user-developed tools from VA facilities nationwide. Tools in use at one or more facilities and that could be linked to VA cancer performance measures were eligible for inclusion and were vetted by a •
jop.ascopubs.org
e421
Luck et al
panel of cancer and QI experts. The VA Toolkit Series is hosted on a VA intranet SharePoint (Microsoft, Redmond, WA) site accessible to all staff. The Lung Cancer Care Toolkit was launched nationwide in November 2010, concurrent with the OABI release of facilitylevel lung cancer performance data. The other toolkits followed through 2012. A social marketing approach10 aimed to accelerate dissemination to target users, especially oncology clinicians and facility quality managers.11 The Toolkit Series development team demonstrated the toolkits and distributed brochures at three phases of national VA Cancer Care Collaboratives (hereafter, collaboratives), where teams gathered to learn and share cancer QI techniques. Collaborative participants were invited to help spread the word among their colleagues. The development team compiled e-mail distribution lists of potential users and used them to distribute e-brochures. The team also made presentations to VA opinion leaders, who were asked to forward marketing messages to their departments and teams. Periodic marketing messages (every 6 to 8 weeks) were sent to the toolkit users’ listserv to highlight new tools and Web site features.
Facilities Target users of the VA Toolkit Series were staff and clinicians involved in cancer care or QI at VA facilities nationwide. Each of 142 facilities was classified as high, medium, or low complexity, using a VA algorithm based on patient volume, clinical resources, and research funding. Cancer care or QI personnel at central and regional VA administrative offices were also potential users. An online survey (described under Online Survey) targeted potential Lung Cancer Care Toolkit users at the 108 facilities that provided care to ⱖ 10 veterans with lung cancer in 2007. The average annual number of lung cancer cases per facility was 72, with a maximum of 199. Most (61%) of these facilities were of high complexity; 8% were of low complexity. Between three and 10 of these facilities were located in each of the 21 regional VA networks, known as Veterans Integrated Service Networks. Twenty-eight facilities sent lung cancer teams to a collaborative, and 24 sent teams to a collaborative for one or more other cancer types.
Toolkit Series Site Use Using SharePoint 2007 (subsequently SharePoint 2010) analytics, we collected data on all Toolkit Series visits and counted unique users. Because of some data gaps during software upgrades, we used conservative methods to count visits and unique users. Toolkit Series development and evaluation team members were excluded. SharePoint provided data only for combined user traffic to the Toolkit Series site, not for individual toolkits. We present data for the first 22 months of Toolkit Series use, through September 2012.
Online Survey We conducted a targeted online survey of potential Lung Cancer Care Toolkit users to assess: (1) their awareness and use of e422
JOURNAL
OF
ONCOLOGY PRACTICE
•
this toolkit, and (2) facility-level use of specific lung cancer QI tools. To identify potential users, we collected e-mail addresses from the Toolkit Series listserv as well as for VA oncology physicians and nurses, quality managers, cancer registrars, and collaborative lung cancer care teams. Lists were de-duplicated and concatenated; e-mail solicitations to participate in the survey were endorsed by senior VA leaders. Starting approximately 36 weeks after the Lung Cancer Care Toolkit launch, we solicited 1,897 potential respondents, with three waves of follow-up, over 7 weeks from August to October 2011. Respondents identified their facility at the beginning of the survey; those not from one of the 108 target facilities were screened out of further questioning. We received 355 complete surveys, for a response rate of 18.7%. For tool implementation questions, we required at least two complete responses per facility, from at least two different clinical disciplines, to better capture the diversity of user experiences. Responses from 98 facilities met those criteria, for a facility response rate of 91%. We used Survey Monkey to present close-ended questions in three areas: user characteristics, such as clinical credentials and years working in VA; individual awareness, access, and use of the toolkit; and use of specific types of lung cancer tools at the respondent’s facility. To count adoption of specific tool types, we assumed that a “yes” response from any user at a facility trumped “no” or “don’t know” responses from that facility; absent any “yes” response for a specific tool type at a facility, any “no” trumped any “don’t know” response. The VA Greater Los Angeles Healthcare System Institutional Review Board approved the survey.
Statistical Analyses We completed descriptive analyses of site use and survey responses and tested the correlation between social marketing communications and site use changes using linear regression. Multivariable logistic regression was used to estimate the significance of potential predictors of toolkit use. Analyses were conducted using STATA (version 12; STATA, College Station, TX).
Results The VA Toolkit Series site was visited on a sustained basis from first availability in November 2010 through September 2012. The weekly average number of users was 100.7, with returning users accounting for 49.6% (on average) of users each week. Spikes in toolkit use were correlated (P ⬍ .01) with social marketing messages and broad communication about evaluation. There were 5,088 total unique visitors to the Toolkit Series site over the first 22 months of availability. The average cumulative number of clicks per user was 13; the median was five. Some so-called super-users accessed the toolkits frequently and accounted for a significant portion of visits; the top 1% of users accounted for 16% of total clicks on the site, and the top 5% of users for 40% of clicks. Each of the 21 VA regional networks and all 142 facilities were represented among Toolkit Series users. The largest num-
V O L . 11, I S S U E 3
Copyright © 2015 by American Society of Clinical Oncology
Online Toolkit for Cancer Care
ber of users per network was 512; the smallest was 63. There was also wide variation in use across VA facilities. The average number of users per facility was 34.8, with a maximum of 167 and a minimum of two. Survey responses provided additional detail about target users of the Lung Cancer Care Toolkit. As summarized in Table 1, nearly half (47%) of respondents were physicians, and almost one quarter (23%) were nurses or physician assistants. Also represented were case managers, cancer registrars, and quality managers. Fewer than half (44%) of respondents had worked in cancer care or cancer care QI for ⱖ 5 years. A similar proportion (46%) was aware of the performance of their facility on the study of lung cancer care quality. Respondents had mixed views of how easy it was to implement change at their facility, with 29% feeling it was very or somewhat easy, and 37% feeling it was somewhat or very difficult. More than half of survey respondents (54%) had visited the Toolkit Series site. Of those who did, 30% spent a few minutes there, 46% spent from a half hour to 1 hour, and 24% spent more than 1 hour. The vast majority of respondents who visited the site (91%) viewed at least one tool, and 24% downloaded at least one tool. Fewer than 10% either requested a tool that could not be immediately downloaded, uploaded or suggested a tool, or joined the online discussion. Of respondents who had not visited the Toolkit Series site, 61% stated the main reason was because they had not heard of it and 18% because they had not had time. Multivariable regression models (Table 2) showed that survey respondents who were aware of the performance of their facility on the OABI lung cancer study were much more likely to have visited the Toolkit Series site (P ⬍ .01). Nonphysicians were more likely than physicians to have visited the site (P ⫽ .02). Perceived ease of implementation at their facility was not associated with likelihood of visiting the site, nor was the length of time working in lung cancer care or QI. Respondents from facilities that had sent a lung cancer team to a VA Cancer Care Collaborative (P ⫽ .05) or had a larger number of lung cancer cases (P ⫽ .04) were more likely to have visited the site. Facility complexity was not associated with likelihood of visiting the site. The survey also measured whether different types of lung cancer QI tools had been implemented at respondents’ facilities (Table 3). Some EHR-based tools were reported as being widely available: pain assessment reminders at 91% of facilities; templates for specialty or imaging referrals at 79%; and templates for cancer diagnosis or staging and for chemotherapy ordering available at 66% and 62% of facilities, respectively. Other types of EHR tools, including pathology and nursing templates and links to guidelines, were available at fewer than half of facilities. Databases for patient tracking or performance measurement were implemented at 73% of facilities. Flowcharts or other decision guides for lung cancer were available in 62% of facilities, but flowcharts or tools for palliative care were available at only 48%. Some organizational tools were widely implemented, including service agreements at 86% of facilities and cancer patient coordinators at 79%. Interdisciplinary tools for Copyright © 2015 by American Society of Clinical Oncology
M A Y 2015
Table 1. Characteristics and Selected Responses of Respondents to Online Survey of Lung Cancer Care Toolkit Use (n ⫽ 355) Characteristic
%
Respondent specialty and role Physician Oncology
14.9
Primary care/internal medicine
10.7
Pulmonology Other
8.7 13.0
Advanced practice nurse or physician assistant
11.0
Nurse
12.1
Case manager
5.1
Cancer registrar
5.6
Quality manager Other
5.6 13.2
Experience in cancer care or cancer care QI, years ⱖ5
43.9
⬍ 5 or do not work in cancer care or QI
56.1
Aware of facility performance on lung cancer care study? Yes
45.6
No
54.4
Ease of implementing change at facility Very or somewhat easy
28.7
Neither easy nor difficult
34.1
Somewhat or very difficult
37.2
Visited Toolkit Series site? Yes
54.1
No
45.9
Abbreviation: QI, quality improvement.
cancer care were somewhat less widely available, including tumor board presentation standards or templates at 65% of facilities and multidisciplinary cancer care clinics at 62%. More than 40% of facilities allowed videoconference participation in tumor boards. Patient education tools were widely available (88% of facilities), and patient or caregiver support resources somewhat less widely (76%).
Discussion We evaluated the scope, patterns, and predictors of use of online toolkits designed to share cancer QI tools across sites within the national VA integrated health system. The Toolkit Series attracted more than 5,000 unique users within the first 2 years of availability. It was used in all 21 VA regional networks and all 108 facilities that had a significant volume of lung cancer cases. Several different types of lung cancer QI tools, such as service agreements, cancer patient coordinator position descriptions, and patient education materials, were implemented at a substantial majority of these facilities. Because cancer care encompasses screening, diagnosis, treatment, and end-of-life care, cancer care QI must be assessed within the context of a multilevel care delivery environment.12 A wide range of cancer care QI approaches have been studied, •
jop.ascopubs.org
e423
Luck et al
Table 2. Predictors of Toolkit Series Site Use Characteristic
OR
95% CI
P
Team sent to VA Cancer Care Collaborative (reference: none)
Yes (%)
Tool
No (%)
Don’t Know (%)
EHR*
Lung only
1.73
1.00 to 2.98
.05
Other only
1.09
0.46 to 2.57
.85
Both
Table 3. Implementation of Lung Cancer Quality Improvement Tools at VA Facilities (n ⫽ 98)
Reminders for pain assessment or patient education for pain
91
7
2
Templates for referrals to specialty clinics or imaging
79
15
6
Templates for diagnosis and staging (eg, lung nodule assessment)
66
24
9
1.01
0.55 to 1.85
.98
1.01
1.00 to 1.01
.04
Medium
1.71
0.89 to 3.29
.11
Templates for chemotherapy ordering
62
30
8
Low
1.72
0.77 to 3.82
.18
46
38
16
1.69
1.07 to 2.67
.02
Pathology templates or work aids specific to lung cancer Nursing templates for cancer (eg, symptom documentation)
46
37
17
Link from EHR to NCCN or other cancer care guidelines
44
44
12
Databases for local patient tracking or performance measurement
73
11
15
Lung cancer care flowcharts, pocket guides, or other paper-based decision guides
62
26
12
Palliative care flowchart or screening tools
48
28
24
No. of lung cancer cases in 2007 Facility complexity (reference: high)
Role (reference: physician) Perceived ease of implementation (reference: very difficult) Very easy
0.91
0.22 to 3.69
.89
Somewhat easy
1.21
0.44 to 3.35
.72
Neither easy nor difficult
1.34
0.51 to 3.57
.55
Somewhat difficult
1.67
0.61 to 4.53
.32
Length of work, years (reference: does not work in cancer care or QI) 1 to 4
1.87
ⱖ5
0.99 to 3.55
.05
Non-EHR
Organizational
0.85
0.44 to 1.63
.62
Aware of facility performance on lung cancer care study
3.10
1.91 to 5.01
.00
Service agreements (eg, with specialty clinics, imaging, or palliative care)
86
9
5
Constant
0.17
0.05 to 0.63
.01
Cancer patient coordinator (eg, care coordinator, navigator, case manager)
79
17
4
Standards or templates for presenting lung cancer cases at tumor board
65
24
10
Multidisciplinary cancer care clinic
62
32
6
Dedicated clinic for lung nodule assessment and surveillance
57
38
5
Videoconference participation in tumor board
42
51
7
Patient education materials related to diagnosis, treatment, or palliative care
88
7
5
Support groups for patients with cancer or caregiver resources
76
16
8
Abbreviations: OR, odds ratio; QI, quality improvement; VA, Veterans Administration.
including: team-based care or communities of practice13-15; clinical report tools16; common assessment tools, treatment plans, and symptom management guidelines17,18; peer comparison or audit and feedback19-24; training or education23,25,26; and tumor registries.27 The best way to share these approaches has been less studied. HIT is one potential mechanism; it has been shown to support and reinforce many QI approaches.28,29 Online toolkits are an HIT-enabled technique allowing integrated health systems to share QI approaches across sites. To date, few studies have evaluated the effectiveness of toolkits.30-34 The VA Toolkit Series achieved widespread use and a high degree of user engagement. Users visited the Toolkit Series multiple times, with return users consistently accounting for half of visitors. More than two of three survey respondents who visited the site spent at least half an hour there. Approximately one in four downloaded ⱖ one tool, although fewer than one in 10 uploaded or suggested a tool or participated in an online discussion forum. Nurses, case managers, and other nonphysician VA cancer care staff were more likely than physicians to have visited the Toolkit Series site, perhaps because of their broader care planning and coordination responsibilities. Although we were unable to comprehensively estimate the total number of VA cancer care and QI professionals who were potential Toolkit Series users, the actual number of users is more than an order of magnitude larger than the approximately 428 full-time equivalent hematology/oncology physicians and residents in the VA. e424
JOURNAL
OF
ONCOLOGY PRACTICE
•
Other
Abbreviations: EHR, electronic health record; NCCN, National Comprehensive Cancer Network; VA, Veterans Administration. * VA Computerized Patient Record System.
A core group of super-users visited the Toolkit Series site most frequently, consistent with the anecdotal observation of the development team that a subset of users is active in cancer QI and catalyzes those activities within their facilities. Online toolkits can enable such change agents to draw on ideas generated at other facilities as well as to spread their own ideas to other facilities. The strongest predictor of Toolkit Series use by survey respondents was awareness of the performance of their facility in a national VA study of lung cancer care. Interestingly, respondents who perceived tool implementation as being relatively more difficult at their facility were no less likely to visit the Toolkit Series site. Respondents whose facility had sent a lung cancer team to a Cancer Care Collaborative were more likely to have visited the site. Although performance improvement collaboratives have been examined in cancer care,35-37 clear and
V O L . 11, I S S U E 3
Copyright © 2015 by American Society of Clinical Oncology
Online Toolkit for Cancer Care
sustained benefits have not been uniformly observed.38-42 Our results suggest that toolkits and collaboratives may be mutually reinforcing techniques for sharing QI lessons across facilities. Systematic social marketing of the Toolkit Series also seemed to be effective in stimulating use. The cumulative number of users grew steadily, and use remained high nearly 2 years after initial availability. Significant correlation between social marketing communications and spikes in Toolkit Series site use suggests that ongoing marketing helped to maintaining user engagement. The number of Toolkit Series users varied widely across facilities. As expected, survey respondents from facilities with more lung cancer cases were more likely to use the toolkits. However, facility complexity was not a significant predictor of use, suggesting that less complex facilities with fewer internal QI resources can benefit from shared innovation within the context of an integrated health system. Reported use rates for specific types of lung cancer QI tools also varied across facilities. Although some EHR templates, such as for referrals and pain assessment, were implemented at most facilities, other templates, such as for nursing, were less widely used, even though all VA facilities have a comprehensive EHR system. Patient education tools, which require only modest implementation efforts, were widely used. Implementation rates for organizational tools, such as service agreements and care coordination, were also high, even though implementing such tools requires significant organizational commitment. Patient flowcharts were widely used, but fewer than half of facilities used such tools for palliative care. Variation in use across facilities and tool types suggests the market for toolkit use within the VA is not yet saturated. More than four in 10 survey respondents had not yet visited the Toolkit Series site, most commonly because they had not previously heard of it or had not had time to visit. This suggests that additional social marketing efforts may be effective. For example, in-person or telephone promotion efforts could target potential users at facilities with sizeable oncology caseloads but few toolkit users. Toolkit demonstrations could be performed at regional or national meetings of VA oncologists. Outreach could also target members of other disciplines in the cancer care team, including advanced practice nurses, social workers, and palliative care clinicians. This study has several limitations. The VA comprehensive EHR may make implementation of QI interventions and computerized tools easier than in other health systems. Reported rates of tool implementation may include tools that had been implemented before availability in the online toolkits. Survey
respondents may not be as representative of potential toolkit users as of facilities, although the response rate was consistent with expectations for online surveys. Future research on clinical outcomes will be needed to determine whether toolkit use is associated with higher-quality cancer care. In conclusion, the VA developed online compendia of cancer care QI tools targeted to specific measures of quality and timeliness. All tools were contributed by VA facilities where they were in use. Systematic marketing of these toolkits was amplified by VA performance improvement collaboratives and by toolkit super-users. Toolkits were widely accessed by target audiences of clinicians and QI staff, at facilities of all complexity levels. This model of user-driven tool creation and centralized toolkit development seems to be an effective mechanism for leveraging HIT and facility-level innovation to spread diseasespecific QI tools within a nationwide integrated health care system. Acknowledgment Supported by the Veterans Administration (VA) Office of Analytics and Business Intelligence and VA Office of Systems Redesign, with additional in-kind support from VA Center for Applied Systems Engineering. The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs or the US Government. We thank the following toolkit development and evaluation staff: Jenny Barnard, BA, Marlin Elenes, MPH, Carla Alvarado, MPH, Tonya Reznor, BA, Deborah Griffith, EdD, and Gail Edwards, RN. Authors’ Disclosures of Potential Conflicts of Interest Disclosures provided by the authors are available with this article at jop.ascopubs.org.
Author Contributions Conception and design: Jeff Luck, Candice Bowman, Steven M. Asch Administrative support: Randall C. Gale Collection and assembly of data: Jeff Luck, Laura S. York, Candice Bowman, Nina Smith, Steven M. Asch Data analysis and interpretation: Jeff Luck, Candice Bowman, Randall C. Gale, Steven M. Asch Manuscript writing: All authors Final approval of manuscript: All authors Corresponding author: Jeff Luck, MBA, PhD, College of Public Health and Human Sciences, Oregon State University, 401 Waldo Hall, Corvallis, OR 97331-6406; e-mail: [email protected].
DOI: 10.1200/JOP.2014.003012; published online ahead of print at jop.ascopubs.org on April 7, 2015.
References 1. Flood AB, Fennell ML, Devers KJ: Health reforms as examples of multilevel interventions in cancer care. J Natl Cancer Inst Monogr 2012:80-85, 2012 2. Korda H, Eldridge GN: Payment incentives and integrated care delivery: Levers for health system reform and cost containment. Inquiry 48:277-287, 2011-2012 3. Asch SM, McGlynn EA, Hogan MM, et al: Comparison of quality of care for patients in the Veterans Health Administration and patients in a national sample. Ann Intern Med 141:938-945, 2004
Copyright © 2015 by American Society of Clinical Oncology
M A Y 2015
4. Schultz EM, Powell AA, McMillan A, et al: Hospital characteristics associated with timeliness of care in veterans with lung cancer. Am J Respir Crit Care Med 179:595-600, 2009 5. Powell AA, Schultz EM, Ordin DL, et al: Timeliness across the continuum of care in veterans with lung cancer. J Thorac Oncol 3:951-957, 2008 6. Gould MK, Ghaus SJ, Olsson JK, et al: Timeliness of care in veterans with non-small cell lung cancer. Chest 133:1167-1173, 2008
•
jop.ascopubs.org
e425
Luck et al
7. Yano EM, Green LW, Glanz K, et al: Implementation and spread of interventions into the multilevel context of routine practice and policy: Implications for the cancer care continuum. J Natl Cancer Inst Monogr 2012:86-99, 2012
25. Coe SG, Crook JE, Diehl NN, et al: An endoscopic quality improvement program improves detection of colorectal adenomas. Am J Gastroenterol 108: 219-226, 2013; quiz 227
8. Veterans Health Administration Office of Analytics and Business Intelligence: The Quality of VHA Lung Cancer Care: Special Study. Washington, DC, Veterans Health Administration, 2010
26. Wright FC, Law CH, Last LD, et al: A blended knowledge translation initiative to improve colorectal cancer staging [ISRCTN56824239]. BMC Health Serv Res 6:4, 2006
9. Davies MV: VHA systems redesign: A plan for organizational transformation. www. hsrd.research.va.gov/for_researchers/cyber_seminars/archives/qip-052609.pdf
27. Redondo M, Rivas-Ruiz F, Guzman-Soler MC, et al: Monitoring indicators of health care quality by means of a hospital register of tumours. J Eval Clin Pract 14:1026-1030, 2008
10. Luck J, Hagigi F, Parker LE, et al: A social marketing approach to implementing evidence-based practice in VHA QUERI: The TIDES depression collaborative care model. Implement Sci 4:64, 2009
28. Au M, Felt-Lisk S, Anglin G, et al: Using Health IT: Eight Quality Improvement Stories. Rockville, MD, Agency for Healthcare Research and Quality, 2010
11. York L, Bruce B, Luck J, et al: Online toolkits for metric-driven quality improvement: The Veterans Health Administration managed grassroots approach. Jt Comm J Qual Patient Saf 39:561-569, 2013
29. Keyser DJ, Dembosky JW, Kmetik K, et al: Using health information technology-related performance measures and tools to improve chronic care. Jt Comm J Qual Patient Saf 35:248-255, 2009
12. Taplin SH, Anhang Price R, Edwards HM, et al: Introduction: Understanding and influencing multilevel factors across the cancer care continuum. J Natl Cancer Inst Monogr 2012:2-10, 2012
30. DeWalt DA, Broucksou KA, Hawk V, et al: Developing and testing the health literacy universal precautions toolkit. Nurs Outlook 59:85-94, 2011
13. Elward KS, Penberthy LT, Bear HD, et al: Collaborative initiatives in the care of early stage breast cancer: Reduction in variation of breast-conserving therapy. Breast J 6:108-114, 2000 14. Farber S, Andersen W, Branden C, et al: Improving cancer pain management through a system wide commitment. J Palliat Med 1:377-385, 1998 15. Fung-Kee-Fung M, Goubanova E, Sequeira K, et al: Development of communities of practice to facilitate quality improvement initiatives in surgical oncology. Qual Manag Health Care 17:174-185, 2008 16. Dubenske LL, Chih MY, Dinauer S, et al: Development and implementation of a clinician reporting system for advanced stage cancer: Initial lessons learned. J Am Med Inform Assoc 15:679-686, 2008 17. Dudgeon DJ, Knott C, Eichholz M, et al: Palliative care integration project (PCIP) quality improvement strategy evaluation. J Pain Symptom Manage 35:573582, 2008 18. Hains IM, Ward RL, Pearson SA: Implementing a Web-based oncology protocol system in Australia: Evaluation of the first 3 years of operation. Intern Med J 42:57-64, 2012 19. Aletti GD, Dowdy SC, Gostout BS, et al: Quality improvement in the surgical approach to advanced ovarian cancer: The Mayo Clinic experience. J Am Coll Surg 208:614-620, 2009 20. Dulko D: Audit and feedback as a clinical practice guideline implementation strategy: A model for acute care nurse practitioners. Worldviews Evid Based Nurs 4:200-209, 2007 21. Mansell G, Shapley M, Jordan JL, et al: Interventions to reduce primary care delay in cancer referral: A systematic review. Br J Gen Pract 61:e821-e835, 2011
31. Eagle KA, Gallogly M, Mehta RH, et al: Taking the national guideline for care of acute myocardial infarction to the bedside: Developing the guideline applied in practice (GAP) initiative in Southeast Michigan. Jt Comm J Qual Improv 28:5-19, 2002 32. Nowalk MP, Nutini J, Raymund M, et al: Evaluation of a toolkit to introduce standing orders for influenza and pneumococcal vaccination in adults: A multimodal pilot project. Vaccine 30:5978-5982, 2012 33. Speroff T, Ely EW, Greevy R, et al: Quality improvement projects targeting health care-associated infections: Comparing virtual collaborative and toolkit approaches. J Hosp Med 6:271-278, 2011 34. Wiechula R, Kitson A, Marcoionni D, et al: Improving the fundamentals of care for older people in the acute hospital setting: Facilitating practice improvement using a knowledge translation toolkit. Int J Evid Based Healthc 7:283-295, 2009 35. Miller DC, Murtagh DS, Suh RS, et al: Regional collaboration to improve radiographic staging practices among men with early stage prostate cancer. J Urol 186:844-849, 2011 36. Powell AA, Nugent S, Ordin DL, et al: Evaluation of a VHA collaborative to improve follow-up after a positive colorectal cancer screening test. Med Care 49:897-903, 2011 37. Shaw EK, Ohman-Strickland PA, Piasecki A, et al: Effects of facilitated team meetings and learning collaboratives on colorectal cancer screening rates in primary care practices: A cluster randomized trial. Ann Fam Med 11:220-228, 2013 38. Landon BE, Wilson IB, McInnes K, et al: Effects of a quality improvement collaborative on the outcome of care of patients with HIV infection: The EQHIV study. Ann Intern Med 140:887-896, 2004 39. Mittman BS: Creating the evidence base for quality improvement collaboratives. Ann Intern Med 140:897-901, 2004
22. O’Grady MA, Gitelson E, Swaby RF, et al: Development and implementation of a medical oncology quality improvement tool for a regional community oncology network: The Fox Chase Cancer Center partners initiative. J Natl Compr Canc Netw 5:875-882, 2007
40. Nembhard IM, Northrup V, Shaller D, et al: Improving organizational climate for quality and quality of care: Does membership in a collaborative help? Med Care 50:S74-S82, 2012 (suppl)
23. Simunovic M, Coates A, Goldsmith CH, et al: The cluster-randomized quality initiative in rectal cancer trial: Evaluating a quality-improvement strategy in surgery. CMAJ 182:1301-1306, 2010
41. Schouten LM, Hulscher ME, van Everdingen JJ, et al: Evidence for the impact of quality improvement collaboratives: Systematic review. BMJ 336:1491-1494, 2008
24. Winickoff RN, Coltin KL, Morgan MM, et al: Improving physician performance through peer comparison feedback. Med Care 22:527-534, 1984
42. ØVretveit J, Bate P, Cleary P, et al: Quality collaboratives: Lessons from research. Qual Saf Health Care 11:345-351, 2002
e426
JOURNAL
OF
ONCOLOGY PRACTICE
•
V O L . 11, I S S U E 3
Copyright © 2015 by American Society of Clinical Oncology
Online Toolkit for Cancer Care
AUTHORS’ DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST Implementing a User-Driven Online Quality Improvement Toolkit for Cancer Care The following represents disclosure information provided by authors of this manuscript. All relationships are considered compensated. Relationships are self-held unless noted. I ⫽ Immediate Family Member, Inst ⫽ My Institution. Relationships may not relate to the subject matter of this manuscript. For more information about ASCO’s conflict of interest policy, please refer to www.asco.org/rwc or jop.ascopubs.org/site/misc/ifc.xhtml. Jeff Luck Leadership: Biome Analytics Stock or Other Ownership: Biome Analytics Consulting or Advisory Role: Biome Analytics Patents, Royalties, Other Intellectual Property: Biome Analytics
Randall C. Gale No relationship to disclose Nina Smith No relationship to disclose Steven M. Asch No relationship to disclose
Laura S. York No relationship to disclose Candice Bowman No relationship to disclose
Copyright © 2015 by American Society of Clinical Oncology
M A Y 2015
•
jop.ascopubs.org
e427
Copyright of Journal of Oncology Practice is the property of American Society of Clinical Oncology and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
