ORIGINAL ARTICLE
Raising the bar: Using program evaluation for quality improvement Rebecca Anas, BSc, MBA; Jennifer Stiff, BAH, MRes; Brittany Speller, BSc (candidate); Nicoda Foster, BPAPM, MPH; Robert Bell, MSc, MD, FACS, CM, FRCSC; Virginia McLaughlin, BA; William K. Evans, MD, FRCPC
Abstract—The Cancer Quality Council of Ontario has undertaken 3 programmatic reviews, in partnership with Cancer Care Ontario, to evaluate an emerging or existing program and to provide future directions. The reviews are a quality improvement tool consisting of an assessment of the program’s current state, a critical appraisal, and an environmental scan in advance of an event where attendees hear best practices from jurisdictions and participate in a discussion leading to a set of recommendations for the program.
T
he Cancer Quality Council of Ontario (the Council) was established in 2002 by the Ministry of Health and Long-Term Care (the Ministry) as a quasiindependent advisory group to Cancer Care Ontario (CCO).1 CCO is a government agency, which aims to drive quality and continuous improvement in disease prevention and screening, and the delivery of cancer care and more recently, CCO has expanded its mandate to include chronic kidney disease, as well as access to care for key health services.2 The Council provides CCO and the Ministry with advice to improve the quality of cancer care in Ontario and reports directly to CCO’s Board of Directors (the Board). The mandate of the Council is to monitor and publicly report on the quality and performance of Ontario’s cancer system and encourage improvement through national and international benchmarking.3 The Council is composed of a multidisciplinary group of healthcare providers, cancer survivors, family members and experts in the areas of oncology, health system policy and administration, governance, performance measurement, and health services research. The Council also has a Secretariat team housed within CCO that benefits from economies of scale of the CCO infrastructure and has access to CCO’s data sources. This “inside-outside” relationship provides the Council with access to CCO’s sophisticated expertise and extensive data holdings while maintaining an independent oversight role with respect to CCO performance for the purposes of public reporting.3 To achieve its mandate and drive quality improvement, the Council uses 4 tools annually which include the Cancer From the Cancer Quality Council of Ontario Secretariat, Toronto, Ontario, Canada. Corresponding author: Jennifer Stiff, BAH, MRes, Cancer Quality Council of Ontario Secretariat, 620 University Avenue Toronto, Ontario, Canada, M5G 2L7. (e-mail: [email protected]) Healthcare Management Forum 2013 26:191–195 0840-4704/$ - see front matter & 2013 Canadian College of Health Leaders. Published by Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.hcmf.2013.09.007
System Quality Index (CSQI),4 which measures and reports publicly on the quality and performance of the cancer system; the signature event, which brings experts together to address system quality gaps and propose practical solutions to improve the quality of cancer care; the quality and innovation awards, which recognize, reward, and encourage quality improvement and innovation from the frontline; and the programmatic review, which critically evaluates a program within CCO. The programmatic review contributes to CCO’s performance improvement cycle by translating expertise into clear directions for the program and organization.5 The programmatic review process and lessons learned are described in detail. These reviews are a unique product that provides CCO with an external view of its emerging or existing programs, alongside best practice examples from international jurisdictions to assist in driving quality and performance improvement in the cancer system.
PROGRAMMATIC REVIEW Purpose The programmatic review is a collaborative initiative between the Council and CCO, which brings international experts to Ontario to share their best practices, review progress, and analyze the effectiveness of existing programs or to inform an emerging program. Depending on the CCO program selected, the review process can be a summative evaluation to review an existing program that has been in place for a number of years or a formative evaluation informing an emerging program to determine the best approach to proceed with its development.6 The review culminates with a set of recommendations for the future directions of the CCO program. The first review conducted by the Council in 2010 was a formative evaluation of CCO’s disease pathway management program. A summative evaluation of CCO’s palliative care program was undertaken in 2012, followed by a formative evaluation of CCO’s patient experience program in 2013.
Anas, Stiff, Speller, Foster, Bell, McLaughin and Evans
Timeline The programmatic review process undertaken by the Council is approximately 10 months in duration (Fig. 1), commencing with the identification of the program topic followed by a determination of the scope of the review to focus the resources, efforts, and recommendations. The majority of work is completed within the first 6 months, leading to the programmatic review event and the month following the review. All steps are collaboratively determined by an ad hoc working group, steering committee, the Council, and the CCO program being evaluated, which take into consideration the goals and priorities of the program within the context of the role of CCO and other external organizations or stakeholders in the province.
Structures Ad hoc working group and steering committee The ad hoc working group is comprised of individuals from the Council Secretariat, the CCO program being evaluated, and other appropriate CCO organizational staff whose work is related to the program being evaluated. Individuals on the working group are responsible for proposing the scope of the review and bringing insight and input to biweekly meetings on topics, such as work progress and milestone accomplishments. The steering committee validates the work undertaken by the ad hoc working group and also provides guidance and direction on experts to interview, background documents to be created for the programmatic review and the details of the event. The committee is used as a touchstone to ensure the right
people are engaged and the discussion focuses on the right content. The steering committee has representation from selected Council members (external to CCO); patients; families; and other appropriate clinical, scientific, and administrative experts, as well as partner organizations external to CCO.
Process Content work streams Content work steams include a current state assessment and critical appraisal of the CCO program, environmental scan, and the engagement of international experts.
Current state assessment or critical appraisal The current state assessment and critical appraisal of the program is conducted through interviews with key individuals internal and external to the organization. A current state assessment may also include interviews with other organizations or experts that provide care external to CCO. For example, in the palliative care summative programmatic review, providers outside of the cancer system were interviewed to identify opportunities for collaboration with the existing CCO program. Interview guides are constructed dependent on the individual’s affiliation with the program or based on their own organization’s priorities. Interviewees identify a number of opportunities to maximize the CCO program’s effect on improving the quality of care for patients within the context of Ontario’s health system. The interviews are conducted by the Council Secretariat, as well as members of the program who have
Figure 1. Programmatic review timeline. (Colour version of the figure is available on-line.)
192
Healthcare Management Forum Forum Gestion des soins de sante – Winter/Hiver 2013
RAISING THE BAR: USING PROGRAM EVALUATION FOR QUALITY IMPROVEMENT
specific content expertise, which helps deepen the conversation with interviewees. The working group then collectively analyzes the interviews for themes and organizes them in a Strengths, Weaknesses, Opportunities and Threats (SWOT) framework.7
Environmental scan Based on the opportunities identified for CCO, the Council Secretariat then focuses the environmental scan on jurisdictions or experts or both that are able to provide insights to help solve pressing challenges in Ontario. The Secretariat follows a multimethod research approach, scanning both literature and publicly accessible material from jurisdictions and augmenting the information with interviews of internal staff and participants in the program as well as international, national, regional, and local experts with knowledge of best practices. The literature review consists of a set of search terms relevant to the program being evaluated and areas determined to be in scope for the review. However, if formal search techniques yield poor results, then “snowball” methods are employed (ie, pursuing references for additional sources by reaching out to experts). From the international reviews and based on existing gaps or opportunities or both, the Council Secretariat recommends to the working group and steering committee individuals who should be invited to Ontario as speakers for the programmatic review event and who can potentially provide solutions to the challenges identified in the SWOT. In general, 3-4 experts are invited to give a presentation at the event to provide advice on best practices and to describe and critically appraise their own initiatives. They are also invited to analyze the effectiveness of the program at CCO and to contribute to the discussion and development of recommendations on the strategic directions and improvements that the CCO program should undertake. It is important that the speakers are informed about the Ontario context before the event so they can actively participate in the discussion.
Programmatic review event The event is held over 1 day with 30-40 attendees including Council members, the senior executive staff at CCO (including clinical and administrative leaders), partner organizations, patients, and family members. The event includes presentations from international experts, debates surrounding the presentations and a group discussion, facilitated by an assigned individual, to determine recommendations for the program moving forward. Attendees of the event are provided with the current state, critical appraisal and environmental scan documents, as well as additional relevant literature to review in advance of the event. The documents are circulated in advance to provide attendees with a context for the day, allowing them to effectively engage in the discussion and
determine relevant and actionable recommendations for the CCO program. A helpful tool to guide the day’s discussion includes focused questions that articulate the priorities for the day. These questions are built around the challenges and themes identified in the SWOT analysis, critical appraisal, and current state assessment. These key discussion questions are created by the working group, approved by the steering committee and then used by the assigned facilitator during the event.
After the event Following the event, the Council Secretariat prepares a proceedings document summarizing the event’s presentations and discussions, including the proposed final recommendations. The recommendations include suggestions of specific improvements, strategic directions, and future activities for the CCO program to enhance the quality of care received by patients. These recommendations are reviewed by all attendees and final recommendations are sent directly from the Council Chair to the Chair of CCO’s Board and to the CEO. The program then determines how to proceed with prioritizing and implementing the recommendations. A year following the programmatic review, the CCO program shares with the Council the actions that resulted from the review and proposed recommendations.
KEY LESSONS LEARNED The following 6 key lessons have been learned through the completion of 3 Council programmatic reviews: 1. The program chosen for review and scope of the review should be aligned with emerging priorities within CCO. For example, the Council determines which program to review based on gaps identified through the Council’s CSQI and CCO’s strategic priorities. For example, the DPM program was reviewed in advance of CCO’s release of the Ontario Cancer Plan (OCP) III to assist in shaping its future direction. 2. The current state assessment and critical appraisal document should include a detailed overview of the program being evaluated, as well as initiatives that span across the organizational structure and include the external context that the program operates in. This broadened scope helps break down silos and provides a detailed view of the work occurring within the organization. For example, the current state assessment and critical appraisal for palliative care explored how programs across the organization adopted the concept of palliative care in their work and how external partner organizations were prioritizing palliative and end-of-life care initiatives. International trends were also analyzed and incorporated. 3. The research methodology for the environmental scan primarily relies on peer-reviewed journal articles.
Healthcare Management Forum Forum Gestion des soins de sante – Winter/Hiver 2013
193
Anas, Stiff, Speller, Foster, Bell, McLaughin and Evans
However, as journals may not have the most timely information, evidence is also sought out through referrals from the steering committee and grey literature.8 The array of sources allows for a detailed overview of the topic to ensure that a comprehensive report is compiled for attendees of the event. 4. Interviews should be semistructured in design to avoid the use of prescriptive questions and to allow for open discussion.9 As interviewees will provide different lenses from their own expertise, the interviews should be tailored to each individual. The analysis should cascade and build on each interview in a way that allows previous comments to be validated and tested with each prospective expert. Incorporating analysis from interviews into the current state, critical appraisal and environmental scan must be tailored into a logical synthesis so that the understanding of the gaps of the programs can be matched to opportunities or solutions available from other jurisdictions or both. 5. There should be more than 1 international expert invited to speak at the event, allowing attendees to view best practices from different jurisdictions to improve the quality of care. For example, the patient experience review had experts from national and international jurisdictions10 who focused on their patient experience quality improvement work on the Lean methodology11 for quality improvement, patient experience measures, and models to develop education tools to promote patient experience in all aspects of care. 6. The programmatic review process involves bringing together groups of individuals for the working group, steering committee, and event. The size of each group is taken into consideration as research has indicated that a smaller group size is preferable to allow for adequate discussion from each of the participants.12,13 The working group is comprised of a maximum of 15 individuals, the steering committee consists of approximately 10 experts and the programmatic review event has a maximum of 40 attendees that participate in a large group discussion with no breakout groups. Although these groups are large, the Council recognizes that more in-depth discussion occurs owing to the different skills and perceptions. Additionally, a skilled facilitator is present at the event to guide the group discussion and ensure that all participants have an opportunity to share their insights and expertise to achieve consensus on the day14 and endorsement of recommendations.
CONCLUSION Programmatic reviews are one of the Council’s key tools for quality improvement. The process of the review can be adapted to any organizational structure and is scalable 194
dependent on the scope and timing of the review. The lessons learned section of the review highlights the reasons for the process, and these lessons ultimately drive change. Although the product of the Council’s programmatic review is a set of recommendations, the responsibility rests with the host program and organization to implement the recommendations and adopt them as part of their priorities and future work plans. In order for this to occur, supportive leadership within the program’s organization is critical to the success of the programmatic review. Having senior leadership present at the event and engaged during the process is essential. A comprehensive environmental scan focused on the SWOT analyses and which examines areas of importance to leadership, such as value for money, improved efficiency, and patient care can assist in engaging the key players to make the changes that result in quality improvement.
ACKNOWLEDGMENTS The authors would like to extend their gratitude to Dr. Carol Sawka, Ms. Katya Duvalko, Mr. Brendon Lalonde, and Ms. Natalie Assouad for their involvement in the creation of the first Council programmatic review. Dr. Sawka, past Executive Vice President of Clinical Programs and Quality Initiatives at CCO, was also the Executive Sponsor for the 3 events held between 2010 and 2013 and provided significant guidance leading to the success of the work and the ability for the host programs to ultimately use the Council’s programmatic review recommendations to target improvements.
REFERENCES 1. Cancer Quality Council of Ontario. Home. Available at: http:// www.cqco.ca;2013. 2. Cancer Care Ontario. Who we are. Available at: http://www. cancercare.on.ca/about/who; 2013. 3. Anas R, Bell R, Brown A, Evans W, Sawka CA. Ten-year history: the Cancer Quality Council of Ontario. Healthc Q 2012; 15:24–27. [doi:10.12927/hcq.2012.23162; Special issue]. 4. Cancer Quality Council of Ontario. Cancer system quality index. Available at: www.csqi.on.ca; 2013. 5. Duvalko KM, Sherar M, Sawka C. Creating a system for performance improvement in cancer care: Cancer Care Ontario’s clinical governance framework. Cancer Control 2009;16(4)293–302. 6. Hogan RL. The historical development of program evaluation: exploring the past and present. J Workforce Educ Dev 2007;2(4). 7. Helms MM, Nixon J. Exploring SWOT analysis—where are we now? A review of academic research from the last decade. J Strategy Manage 2010;3(3)215–225. http://dx.doi.org/10.1108/ 17554251011064837. 8. Benzies KM, Premji S, Hayden A, Serrett K. State-of-theevidence reviews: advantages and challenges of including grey literature. Worldviews Evid Based Nurs 2006;3(2)55–61. http://dx.doi.org/10.1111/j.1741-6787.2006.00051.x.
Healthcare Management Forum Forum Gestion des soins de sante – Winter/Hiver 2013
RAISING THE BAR: USING PROGRAM EVALUATION FOR QUALITY IMPROVEMENT
9. Al-Busaidi ZQ. Qualitative research and its uses in healthcare. Sultan Qaboos Univ Med J 2008;8(1)11–19. Available at http:// www.ncbi.nlm.nih.gov/pmc/articles/PMC3087733/. 10. Cancer Quality Council of Ontario. Programmatic review: patient and family experience. Available at: http://www. cqco.ca/cms/one.aspx?portalId=89613&pageId=282366; 2013. 11. Kimsey DB. Lean methodology in health care. AORN Journal 2010;92(1)53–60. http://dx.doi.org/10.1016/j.aorn.2010.01.015.
12. Hare AP. Group size. Am Behav Sci 1981;24(5)695–708. http: //dx.doi.org/10.1177/000276428102400507. 13. Forsyth DR. Group Dynamics, 5th ed, Belmont, CA: Wadsworth, Cengage Learning; 2006. 14. James S, Eggers M, Hughes-Rease M, Loup R, Seiford B. Facilitating large group meetings that get results every time. In: Schuman S, editor. The IAF Handbook of Group Facilitation: Best Practices from the Leading Organizations in Facilitation. San Francisco, CA: Jossey-Bass; 2005.
Healthcare Management Forum Forum Gestion des soins de sante – Winter/Hiver 2013
195
Raising the bar: Using program evaluation for quality improvement Rebecca Anas, BSc, MBA; Jennifer Stiff, BAH, MRes; Brittany Speller, BSc (candidate); Nicoda Foster, BPAPM, MPH; Robert Bell, MSc, MD, FACS, CM, FRCSC; Virginia McLaughlin, BA; William K. Evans, MD, FRCPC
Abstract—The Cancer Quality Council of Ontario has undertaken 3 programmatic reviews, in partnership with Cancer Care Ontario, to evaluate an emerging or existing program and to provide future directions. The reviews are a quality improvement tool consisting of an assessment of the program’s current state, a critical appraisal, and an environmental scan in advance of an event where attendees hear best practices from jurisdictions and participate in a discussion leading to a set of recommendations for the program.
T
he Cancer Quality Council of Ontario (the Council) was established in 2002 by the Ministry of Health and Long-Term Care (the Ministry) as a quasiindependent advisory group to Cancer Care Ontario (CCO).1 CCO is a government agency, which aims to drive quality and continuous improvement in disease prevention and screening, and the delivery of cancer care and more recently, CCO has expanded its mandate to include chronic kidney disease, as well as access to care for key health services.2 The Council provides CCO and the Ministry with advice to improve the quality of cancer care in Ontario and reports directly to CCO’s Board of Directors (the Board). The mandate of the Council is to monitor and publicly report on the quality and performance of Ontario’s cancer system and encourage improvement through national and international benchmarking.3 The Council is composed of a multidisciplinary group of healthcare providers, cancer survivors, family members and experts in the areas of oncology, health system policy and administration, governance, performance measurement, and health services research. The Council also has a Secretariat team housed within CCO that benefits from economies of scale of the CCO infrastructure and has access to CCO’s data sources. This “inside-outside” relationship provides the Council with access to CCO’s sophisticated expertise and extensive data holdings while maintaining an independent oversight role with respect to CCO performance for the purposes of public reporting.3 To achieve its mandate and drive quality improvement, the Council uses 4 tools annually which include the Cancer From the Cancer Quality Council of Ontario Secretariat, Toronto, Ontario, Canada. Corresponding author: Jennifer Stiff, BAH, MRes, Cancer Quality Council of Ontario Secretariat, 620 University Avenue Toronto, Ontario, Canada, M5G 2L7. (e-mail: [email protected]) Healthcare Management Forum 2013 26:191–195 0840-4704/$ - see front matter & 2013 Canadian College of Health Leaders. Published by Elsevier Inc. All rights reserved. http://dx.doi.org/10.1016/j.hcmf.2013.09.007
System Quality Index (CSQI),4 which measures and reports publicly on the quality and performance of the cancer system; the signature event, which brings experts together to address system quality gaps and propose practical solutions to improve the quality of cancer care; the quality and innovation awards, which recognize, reward, and encourage quality improvement and innovation from the frontline; and the programmatic review, which critically evaluates a program within CCO. The programmatic review contributes to CCO’s performance improvement cycle by translating expertise into clear directions for the program and organization.5 The programmatic review process and lessons learned are described in detail. These reviews are a unique product that provides CCO with an external view of its emerging or existing programs, alongside best practice examples from international jurisdictions to assist in driving quality and performance improvement in the cancer system.
PROGRAMMATIC REVIEW Purpose The programmatic review is a collaborative initiative between the Council and CCO, which brings international experts to Ontario to share their best practices, review progress, and analyze the effectiveness of existing programs or to inform an emerging program. Depending on the CCO program selected, the review process can be a summative evaluation to review an existing program that has been in place for a number of years or a formative evaluation informing an emerging program to determine the best approach to proceed with its development.6 The review culminates with a set of recommendations for the future directions of the CCO program. The first review conducted by the Council in 2010 was a formative evaluation of CCO’s disease pathway management program. A summative evaluation of CCO’s palliative care program was undertaken in 2012, followed by a formative evaluation of CCO’s patient experience program in 2013.
Anas, Stiff, Speller, Foster, Bell, McLaughin and Evans
Timeline The programmatic review process undertaken by the Council is approximately 10 months in duration (Fig. 1), commencing with the identification of the program topic followed by a determination of the scope of the review to focus the resources, efforts, and recommendations. The majority of work is completed within the first 6 months, leading to the programmatic review event and the month following the review. All steps are collaboratively determined by an ad hoc working group, steering committee, the Council, and the CCO program being evaluated, which take into consideration the goals and priorities of the program within the context of the role of CCO and other external organizations or stakeholders in the province.
Structures Ad hoc working group and steering committee The ad hoc working group is comprised of individuals from the Council Secretariat, the CCO program being evaluated, and other appropriate CCO organizational staff whose work is related to the program being evaluated. Individuals on the working group are responsible for proposing the scope of the review and bringing insight and input to biweekly meetings on topics, such as work progress and milestone accomplishments. The steering committee validates the work undertaken by the ad hoc working group and also provides guidance and direction on experts to interview, background documents to be created for the programmatic review and the details of the event. The committee is used as a touchstone to ensure the right
people are engaged and the discussion focuses on the right content. The steering committee has representation from selected Council members (external to CCO); patients; families; and other appropriate clinical, scientific, and administrative experts, as well as partner organizations external to CCO.
Process Content work streams Content work steams include a current state assessment and critical appraisal of the CCO program, environmental scan, and the engagement of international experts.
Current state assessment or critical appraisal The current state assessment and critical appraisal of the program is conducted through interviews with key individuals internal and external to the organization. A current state assessment may also include interviews with other organizations or experts that provide care external to CCO. For example, in the palliative care summative programmatic review, providers outside of the cancer system were interviewed to identify opportunities for collaboration with the existing CCO program. Interview guides are constructed dependent on the individual’s affiliation with the program or based on their own organization’s priorities. Interviewees identify a number of opportunities to maximize the CCO program’s effect on improving the quality of care for patients within the context of Ontario’s health system. The interviews are conducted by the Council Secretariat, as well as members of the program who have
Figure 1. Programmatic review timeline. (Colour version of the figure is available on-line.)
192
Healthcare Management Forum Forum Gestion des soins de sante – Winter/Hiver 2013
RAISING THE BAR: USING PROGRAM EVALUATION FOR QUALITY IMPROVEMENT
specific content expertise, which helps deepen the conversation with interviewees. The working group then collectively analyzes the interviews for themes and organizes them in a Strengths, Weaknesses, Opportunities and Threats (SWOT) framework.7
Environmental scan Based on the opportunities identified for CCO, the Council Secretariat then focuses the environmental scan on jurisdictions or experts or both that are able to provide insights to help solve pressing challenges in Ontario. The Secretariat follows a multimethod research approach, scanning both literature and publicly accessible material from jurisdictions and augmenting the information with interviews of internal staff and participants in the program as well as international, national, regional, and local experts with knowledge of best practices. The literature review consists of a set of search terms relevant to the program being evaluated and areas determined to be in scope for the review. However, if formal search techniques yield poor results, then “snowball” methods are employed (ie, pursuing references for additional sources by reaching out to experts). From the international reviews and based on existing gaps or opportunities or both, the Council Secretariat recommends to the working group and steering committee individuals who should be invited to Ontario as speakers for the programmatic review event and who can potentially provide solutions to the challenges identified in the SWOT. In general, 3-4 experts are invited to give a presentation at the event to provide advice on best practices and to describe and critically appraise their own initiatives. They are also invited to analyze the effectiveness of the program at CCO and to contribute to the discussion and development of recommendations on the strategic directions and improvements that the CCO program should undertake. It is important that the speakers are informed about the Ontario context before the event so they can actively participate in the discussion.
Programmatic review event The event is held over 1 day with 30-40 attendees including Council members, the senior executive staff at CCO (including clinical and administrative leaders), partner organizations, patients, and family members. The event includes presentations from international experts, debates surrounding the presentations and a group discussion, facilitated by an assigned individual, to determine recommendations for the program moving forward. Attendees of the event are provided with the current state, critical appraisal and environmental scan documents, as well as additional relevant literature to review in advance of the event. The documents are circulated in advance to provide attendees with a context for the day, allowing them to effectively engage in the discussion and
determine relevant and actionable recommendations for the CCO program. A helpful tool to guide the day’s discussion includes focused questions that articulate the priorities for the day. These questions are built around the challenges and themes identified in the SWOT analysis, critical appraisal, and current state assessment. These key discussion questions are created by the working group, approved by the steering committee and then used by the assigned facilitator during the event.
After the event Following the event, the Council Secretariat prepares a proceedings document summarizing the event’s presentations and discussions, including the proposed final recommendations. The recommendations include suggestions of specific improvements, strategic directions, and future activities for the CCO program to enhance the quality of care received by patients. These recommendations are reviewed by all attendees and final recommendations are sent directly from the Council Chair to the Chair of CCO’s Board and to the CEO. The program then determines how to proceed with prioritizing and implementing the recommendations. A year following the programmatic review, the CCO program shares with the Council the actions that resulted from the review and proposed recommendations.
KEY LESSONS LEARNED The following 6 key lessons have been learned through the completion of 3 Council programmatic reviews: 1. The program chosen for review and scope of the review should be aligned with emerging priorities within CCO. For example, the Council determines which program to review based on gaps identified through the Council’s CSQI and CCO’s strategic priorities. For example, the DPM program was reviewed in advance of CCO’s release of the Ontario Cancer Plan (OCP) III to assist in shaping its future direction. 2. The current state assessment and critical appraisal document should include a detailed overview of the program being evaluated, as well as initiatives that span across the organizational structure and include the external context that the program operates in. This broadened scope helps break down silos and provides a detailed view of the work occurring within the organization. For example, the current state assessment and critical appraisal for palliative care explored how programs across the organization adopted the concept of palliative care in their work and how external partner organizations were prioritizing palliative and end-of-life care initiatives. International trends were also analyzed and incorporated. 3. The research methodology for the environmental scan primarily relies on peer-reviewed journal articles.
Healthcare Management Forum Forum Gestion des soins de sante – Winter/Hiver 2013
193
Anas, Stiff, Speller, Foster, Bell, McLaughin and Evans
However, as journals may not have the most timely information, evidence is also sought out through referrals from the steering committee and grey literature.8 The array of sources allows for a detailed overview of the topic to ensure that a comprehensive report is compiled for attendees of the event. 4. Interviews should be semistructured in design to avoid the use of prescriptive questions and to allow for open discussion.9 As interviewees will provide different lenses from their own expertise, the interviews should be tailored to each individual. The analysis should cascade and build on each interview in a way that allows previous comments to be validated and tested with each prospective expert. Incorporating analysis from interviews into the current state, critical appraisal and environmental scan must be tailored into a logical synthesis so that the understanding of the gaps of the programs can be matched to opportunities or solutions available from other jurisdictions or both. 5. There should be more than 1 international expert invited to speak at the event, allowing attendees to view best practices from different jurisdictions to improve the quality of care. For example, the patient experience review had experts from national and international jurisdictions10 who focused on their patient experience quality improvement work on the Lean methodology11 for quality improvement, patient experience measures, and models to develop education tools to promote patient experience in all aspects of care. 6. The programmatic review process involves bringing together groups of individuals for the working group, steering committee, and event. The size of each group is taken into consideration as research has indicated that a smaller group size is preferable to allow for adequate discussion from each of the participants.12,13 The working group is comprised of a maximum of 15 individuals, the steering committee consists of approximately 10 experts and the programmatic review event has a maximum of 40 attendees that participate in a large group discussion with no breakout groups. Although these groups are large, the Council recognizes that more in-depth discussion occurs owing to the different skills and perceptions. Additionally, a skilled facilitator is present at the event to guide the group discussion and ensure that all participants have an opportunity to share their insights and expertise to achieve consensus on the day14 and endorsement of recommendations.
CONCLUSION Programmatic reviews are one of the Council’s key tools for quality improvement. The process of the review can be adapted to any organizational structure and is scalable 194
dependent on the scope and timing of the review. The lessons learned section of the review highlights the reasons for the process, and these lessons ultimately drive change. Although the product of the Council’s programmatic review is a set of recommendations, the responsibility rests with the host program and organization to implement the recommendations and adopt them as part of their priorities and future work plans. In order for this to occur, supportive leadership within the program’s organization is critical to the success of the programmatic review. Having senior leadership present at the event and engaged during the process is essential. A comprehensive environmental scan focused on the SWOT analyses and which examines areas of importance to leadership, such as value for money, improved efficiency, and patient care can assist in engaging the key players to make the changes that result in quality improvement.
ACKNOWLEDGMENTS The authors would like to extend their gratitude to Dr. Carol Sawka, Ms. Katya Duvalko, Mr. Brendon Lalonde, and Ms. Natalie Assouad for their involvement in the creation of the first Council programmatic review. Dr. Sawka, past Executive Vice President of Clinical Programs and Quality Initiatives at CCO, was also the Executive Sponsor for the 3 events held between 2010 and 2013 and provided significant guidance leading to the success of the work and the ability for the host programs to ultimately use the Council’s programmatic review recommendations to target improvements.
REFERENCES 1. Cancer Quality Council of Ontario. Home. Available at: http:// www.cqco.ca;2013. 2. Cancer Care Ontario. Who we are. Available at: http://www. cancercare.on.ca/about/who; 2013. 3. Anas R, Bell R, Brown A, Evans W, Sawka CA. Ten-year history: the Cancer Quality Council of Ontario. Healthc Q 2012; 15:24–27. [doi:10.12927/hcq.2012.23162; Special issue]. 4. Cancer Quality Council of Ontario. Cancer system quality index. Available at: www.csqi.on.ca; 2013. 5. Duvalko KM, Sherar M, Sawka C. Creating a system for performance improvement in cancer care: Cancer Care Ontario’s clinical governance framework. Cancer Control 2009;16(4)293–302. 6. Hogan RL. The historical development of program evaluation: exploring the past and present. J Workforce Educ Dev 2007;2(4). 7. Helms MM, Nixon J. Exploring SWOT analysis—where are we now? A review of academic research from the last decade. J Strategy Manage 2010;3(3)215–225. http://dx.doi.org/10.1108/ 17554251011064837. 8. Benzies KM, Premji S, Hayden A, Serrett K. State-of-theevidence reviews: advantages and challenges of including grey literature. Worldviews Evid Based Nurs 2006;3(2)55–61. http://dx.doi.org/10.1111/j.1741-6787.2006.00051.x.
Healthcare Management Forum Forum Gestion des soins de sante – Winter/Hiver 2013
RAISING THE BAR: USING PROGRAM EVALUATION FOR QUALITY IMPROVEMENT
9. Al-Busaidi ZQ. Qualitative research and its uses in healthcare. Sultan Qaboos Univ Med J 2008;8(1)11–19. Available at http:// www.ncbi.nlm.nih.gov/pmc/articles/PMC3087733/. 10. Cancer Quality Council of Ontario. Programmatic review: patient and family experience. Available at: http://www. cqco.ca/cms/one.aspx?portalId=89613&pageId=282366; 2013. 11. Kimsey DB. Lean methodology in health care. AORN Journal 2010;92(1)53–60. http://dx.doi.org/10.1016/j.aorn.2010.01.015.
12. Hare AP. Group size. Am Behav Sci 1981;24(5)695–708. http: //dx.doi.org/10.1177/000276428102400507. 13. Forsyth DR. Group Dynamics, 5th ed, Belmont, CA: Wadsworth, Cengage Learning; 2006. 14. James S, Eggers M, Hughes-Rease M, Loup R, Seiford B. Facilitating large group meetings that get results every time. In: Schuman S, editor. The IAF Handbook of Group Facilitation: Best Practices from the Leading Organizations in Facilitation. San Francisco, CA: Jossey-Bass; 2005.
Healthcare Management Forum Forum Gestion des soins de sante – Winter/Hiver 2013
195
