Expert Review of Pharmacoeconomics & Outcomes Research Downloaded from informahealthcare.com by UMEA University Library on 10/06/14 For personal use only.
Special Report
Multi-criteria decision analysis for health technology assessment in Canada: insights from an expert panel discussion Expert Rev. Pharmacoecon. Outcomes Res. Early online, 1–7 (2014)
Vakaramoko Diaby*1, Ron Goeree2, Jeffrey Hoch3,4 and Uwe Siebert5 1 Division of Economic, Social and Administrative Pharmacy, College of Pharmacy and Pharmaceutical Sciences, Florida A&M University, Tallahassee, FL, USA 2 McMaster University/Programs for Assessment of Technology in Health, Hamilton, ON, Canada 3 University of Toronto, Toronto, ON, Canada 4 Department of Health Policy, Management and Evaluation, Centre for Research on Inner City Health (CRICH), St. Michael’s Hospital, Toronto, ON, Canada 5 Department of Public Health and Health Technology Assessment, UMIT – University for Health Sciences, Medical Informatics and Technology, Hall i.T., Austria *Author for correspondence: [email protected]
informahealthcare.com
Multi-criteria decision analysis (MCDA), a decision-making tool, has received increasing attention in recent years, notably in the healthcare field. For Canada, it is unclear whether and how MCDA should be incorporated into the existing health technology assessment (HTA) decision-making process. To facilitate debate on improving HTA decision-making in Canada, a workshop was held in conjunction with the 8th World Congress on Health Economics of the International Health Economics Association in Toronto, Canada in July 2011. The objective of the workshop was to discuss the potential benefits and challenges related to the use of MCDA for HTA decision-making in Canada. This paper summarizes and discusses the recommendations of an expert panel convened at the workshop to discuss opportunities and concerns with reference to the implementation of MCDA in Canada. KEYWORDS: Canada • decision-making • healthcare • health technology assessment • multi-criteria decision analysis • workshop
Health technology assessment (HTA) is an important tool used in reimbursement decisionmaking in healthcare systems. Developed in North America in the mid-70s [1], HTA has been adopted worldwide. HTA bridges the gap between evidence and healthcare policy and reimbursement decisions [1]. In Canada, provincial HTA decisions are based on a deliberative process using four decisionmaking criteria, referred to as ‘traditional’ criteria. These criteria are efficacy, safety, cost–effectiveness and affordability. The same criteria, except for ‘affordability,’ are used by federal HTA bodies, the Common Drug Review [2] and the panCanadian Oncology Drug Review [3], to make HTA recommendations. In recent years, HTA bodies across the world, Canada included, have been challenged to expand the decision-making criteria and to make the decision-making process more transparent. Advocates for these changes argue that decisions about the adoption of new health technologies (HTs) are inherently complex and multifaceted and that these decisions 10.1586/14737167.2015.965155
inconsistently apply criteria and involve implicit value judgments. Since HTA decision-making involves multiple factors that may conflict and a variety of stakeholders with different values, a new rational, reproducible and transparent framework is needed. Multi-criteria decision analysis (MCDA), a decision-making framework that uses a large set of methods that simultaneously and explicitly take into account multiple and conflicting factors [4], has the potential to strengthen and complement the traditional decision-making process in HTA. These methods can be classified into four main families: elementary methods [5], value-based measurement models, goal programming models and outranking models [6]. Elementary methods include basic MCDA techniques that do not require preference elicitation as part of the decision-making process. These methods include the maxmin and maxmax approaches, the Hurwicz procedure and the even swap method [7]. Value-based measurement models require the construction and
Ó 2014 Informa UK Ltd
ISSN 1473-7167
1
Expert Review of Pharmacoeconomics & Outcomes Research Downloaded from informahealthcare.com by UMEA University Library on 10/06/14 For personal use only.
Special Report
Diaby, Goeree, Hoch & Siebert
comparison of numerical scores to represent how one alternative is preferred over another. The aggregation rule for these models usually uses a weighted sum approach. Examples of these models include the simple linear additive model, the analytical hierarchy process and the multiattribute utility theory. In goal programming and aspiration models, the aim is to identify the alternative that best satisfies predefined levels of achievement [6]. An example of a goal programming and aspiration model is the linear goal programming model. In outranking models, competing options are compared pairwise by criterion. These comparisons allow for the development of outranking relations (partial dominance), which are used to prioritize alternatives based on an exploitation algorithm. ELECTRE and PROMETHEE are the most prominent methods of this family. In terms of structure, the MCDA process consists of three steps [6]. The first step, referred to as problem identification and structuring, deals with identifying the decision-makers and identifying their goals. At this step, the relevant competing options and their evaluation criteria are defined. The second step, called multi-criteria evaluation model development and use, is dedicated to the selection of the most appropriate model that will be used to address the decision-making problem at stake. The last step, called the development of action plans, consists of making recommendations for implementation to the decision-makers. At this step, sensitivity analyses are presented to strengthen the decision-makers’ confidence in action plans they envision carrying out. Lately, many healthcare jurisdictions have been looking into the use of MCDA to support HTA decision-making. In the UK, the National Institute for Health and Clinical Excellence has been considering MCDA as one of the approaches to implement its new value-based pricing scheme [8]. In Germany, the German Institute for Quality and Efficiency in Health Care [9] has considered using MCDA to support HTA decision-making. Until now, there are very few attempts to develop and implement an MCDA-based framework for HTA decision-making in Canada. The evidence and value: impact on decision-making (EVIDEM) collaboration, an independent nonprofit organization, has developed and proposed a framework that brings together HTA and MCDA. EVIDEM has a strong Canadian representation and the framework has been field tested in the Canadian healthcare setting. In British Columbia (Canada), MCDA has long been used through program budgeting and marginal analysis [10–12], a tool to priority setting and reallocation of scarce healthcare resources (but not HTA). Even though MCDA holds promise for more rational and transparent decision-making, it is unclear how MCDA should be incorporated into the existing Canadian HTA decision-making. To facilitate the debate on improving HTA decision-making in Canada, a workshop involving an expert panel was organized around the potential benefits and challenges related to the use of MCDA for HTA decision-making. The current paper reports on the results of this expert panel discussion and touches on methodological concerns that arose doi: 10.1586/14737167.2015.965155
during the workshop with reference to the potential implementation of MCDA in Canada. Organization of the workshop
An open workshop was held in conjunction with the International Health Economics Association World Congress meeting held in Toronto, Ontario, Canada in July 2011. Five panel participants (from the UK, Brazil, Sweden, Germany and Canada) with expertise spanning key areas of concern for HTA were convened by Amgen Canada to obtain guidance and stimulate the debate on the use of MCDA to support HTA decision-making in Canada. The experts had experience in government, academia, research and nonprofit sectors, which brought a diversity of perspectives for the panel discussion. Potential conflicts of interest were assessed by enquiring about the panelists’ past and current relationships with Amgen Canada. The experts served as individual scientists, meaning they conveyed their own scientific opinions, not the views of the institutions they were affiliated with. The workshop audience represented a cross-section of ten Canadian stakeholders, such as payers, policy-makers, academia and industry. SUPPLEMENTARY APPENDIX A (Supplementary material can be found online at www.informahealthcare.com/ suppl/14737167.2015.965155) contains a list of the expert panelists and discussants. The key questions around which the debate was structured are presented below. • What are the potential benefits of using MCDA to support HTA decision-making in Canada? • What are the potential barriers to using MCDA to support HTA decision-making in Canada? • How should methodological concerns raised by the implementation of MCDA in Canada be dealt with? a. What additional criteria should be used for decisionmaking in Canada, using MCDA? b. How should decision-making criteria be measured? c. Whose perspective should be taken into consideration in preference elicitation? d. How should the most appropriate MCDA method for Canadian HTA be selected? e. How would cost–effectiveness fit into the new MCDAbased framework? More specifically, how would opportunity cost be explicitly dealt with? Before tackling these questions, the panelists were asked to make a short presentation on factors that influence decisions regarding the adoption of HTs in their respective countries. Then, panelists were asked to provide their scientific views on each question presented earlier. They were expected to reach a consensus on their responses and recommendations. When a consensus could not be reached among panelists, this was reported in the panel session debrief notes. One important feature of an expert panel discussion is that it offers an opportunity for panelists to both share their opinions on a topic of interest and then to discuss these views Expert Rev. Pharmacoecon. Outcomes Res.
MCDA for health technology assessment in Canada
Special Report
Expert Review of Pharmacoeconomics & Outcomes Research Downloaded from informahealthcare.com by UMEA University Library on 10/06/14 For personal use only.
Table 1. Factors involved in health technology assessment in five countries. Country
Publicly stated factors for decisions
Other factors applied in practice
UK
• • • • •
• Cost–effectiveness versus a threshold of £20,000–30,000; the threshold for cancer is more permissive and allows an additional £10,000–£20,000
Brazil
• Scientific evidence of safety, efficacy and effectiveness • Economic evaluation • Budget impact
• Budget constraints • Various ‘interests’
Sweden
• Severity of illness • Expected benefit of healthcare intervention • Cost–effectiveness – no explicit cut-off but implicit cut-off of about US$75,000
• Physicians set priorities within disease areas according to: Severity of condition Effect of intervention Evidence for effect Cost per gained effect
Germany
• Assessment of benefit versus cost of new versus existing treatments Points are plotted on an efficiency frontier The line on the cost–effectiveness plane is extrapolated; if the price of the new technology exceeds the line, the price must be reduced Framework for evaluation is within each disease not across diseases • Ethical, legal, social weights
• Demonstrated innovation or breakthrough can result in negotiated higher price • Attempt to value interventions across disease areas based on: Unmet medical need Clinical significance, severity of disease Quality/level of evidence Cost–effectiveness (disease-specific assessment) Vulnerable populations, equity fairness Previous decisions Factors ‘behind closed doors’
Canada (pCODR)
• • • •
• Not applicable
Canada (CDR)
• Clinical benefit – quality of evidence, magnitude of benefit • Economics – cost–effectiveness, price • Layperson/patient stakeholder values
Severity of illness End of life Stakeholder persuasion Significant innovation Disadvantaged populations, children
Overall clinical benefit Alignment with patient values Cost–effectiveness Feasibility of adoption into health systems
• • • • • •
Type of outcome (e.g., surrogate, intermediate, final) Comparative safety Historical precedent Trial size Trial duration Generalizability of results
Canada (excluding Quebec) has two national drug recommendation processes; pCODR is for cancer drugs and CDR is for all other drugs. CDR: Common drug review; N/A: Not applicable; pCODR: pan-Canadian Oncology Drug Review.
through interaction with the audience (questions and answers period). As part of the panel session, audience members were asked to rate the importance of different criteria in decisionmaking with respect to a hypothetical cancer scenario. They were also asked to rank barriers for identifying, weighing and establishing criteria to be considered in Canadian HTA decisions. The next section presents the insights obtained from the panel discussion. Insights from the expert panel session Summary from expert panel presentations
The panel members presented different decision-making rules where different factors influence decisions regarding the adoption of new HTs. Each of the national HTA bodies represented by the expert panel members has publicly stated the factors they consider in their decision-making. However, according to the panelists, additional factors influence decisions in practice. informahealthcare.com
A summary of the factors involved in HTA decision-making in the five countries is shown in TABLE 1. What are the potential benefits of using MCDA to support HTA decision-making in Canada?
The panel members and the audience concurred that MCDA provides an opportunity for healthcare authorities, Canadian provinces in particular, to make decisions that reflect the unique priorities of the residents within their jurisdictions. By securing clinician input regarding clinical criteria for drug/ intervention use and patient and caregiver input regarding the limitations of available alternatives (and the need for new alternatives), payers can be more confident that the decisions they make reflect local needs and priorities. Indeed, MCDA offers a range of methods that attempt to decompose and reflect decision-maker’ values as part of their thinking process. Another benefit of a transparent and comprehensive MCDA doi: 10.1586/14737167.2015.965155
Special Report
Diaby, Goeree, Hoch & Siebert
Expert Review of Pharmacoeconomics & Outcomes Research Downloaded from informahealthcare.com by UMEA University Library on 10/06/14 For personal use only.
Table 2. Common barriers to the implementation of multi-criteria decision analysis. Barriers
Rationale
Confidential pricing
Affects true cost–effectiveness
Unmet needs and severity
Conditions are similar and difficult to separate
Budget impact (low or high)
Will discriminate against people due to the size of the affected group
Double counting of dimensions measured by a criterion
May occur with overlapping factors, for example, degree of innovation and significant improvement
Meaningful and fair weighting of multiple outcomes
Decision-makers do not want this part of the process to be publicized
Complexity of comparing across health sectors and across wider sectors
It is often not clear what part of the health budget should be used for domestic policy rather than population’s health improvement
Political will
These factors are always at play
process lies in the public support that can be garnered for the decision itself and for the decision-makers’ difficult role in allocating scarce resources. With the movement toward increasing the number and type of decision criteria (quantitative and qualitative) for HTA decision-making, MCDA methods can play an important role in supporting HTA decisions by being able to adequately handle both quantitative and qualitative information. Finally, some MCDA methods allow for capturing uncertainty through sensitivity and robustness analyses, with reducing uncertainty representing a dimension of paramount importance in decision-making. What are the potential barriers to using MCDA to support HTA decision-making in Canada?
A review of decision-making rules for HTA, based on the presentations on implicit (used in practice) and explicit criteria (publically stated) in the five countries presented in TABLE 1, suggests a need for a systematic and transparent approach such as MCDA to ensure consistency in HTA decision-making across jurisdictions. However, methodological challenges raised by the rigorous incorporation of MCDA in HTA decision-making remains a hurdle in all countries, including Canada. The panelists reported a number of potential barriers that should be taken into consideration for the effective implementation of MCDA in Canada, which are summarized in TABLE 2. They also provided explanations as to why these dimensions should be considered as barriers (TABLE 2). Audience members representing a variety of stakeholder sectors were polled regarding their opinions about the barriers to MCDA in Canada. The respondents’ opinions regarding the biggest obstacles to the use of MCDA in Canada are shown in FIGURE 1. The latter suggests that the largest barrier both to identifying and weighing criteria for MCDA is the fact that decisions are fragmented and are made according to varying perspectives. Given that healthcare budgets are controlled by provincial governments and each province has varying social priorities, the implementation of MCDA in Canadian HTA doi: 10.1586/14737167.2015.965155
decision-making may result in further disparities in healthcare funding across the provinces if they develop their own MCDA process in ‘silos.’ In addition, ‘complexity’ was found to be the second largest barrier for the implementation of MCDA in Canada. ‘Complexity’ relates to the methodological concerns associated with the incorporation of MCDA to the existing Canadian HTA. These concerns are discussed in section How should methodological concerns raised by the implementation of MCDA in Canada be dealt with? On another note, compared with a rigid focus on clinical and economic evidence, MCDA is a more demanding process for making healthcare funding decisions, which can lead to increased time between regulatory approval and public funding of new technologies. That said, the panel members suggested that a formal MCDA framework in conjunction with traditional HTA may help address the concern about timely access to effective HTs. How should methodological concerns raised by the implementation of MCDA in Canada be dealt with? What additional criteria should be used for decision-making in Canada, using MCDA?
The panelists could not come to a consensus on what additional criteria should be considered. Some panelists noticed overlap between the criteria identified by the audience and the criteria used in practice in the five countries. These criteria were unmet medical needs, disease severity and drug innovation. They can be used as a starting point for further discussion regarding the establishment of additional criteria. Two other recommendations emerged from the panelist discussion. The first emphasized that as criteria are compiled, all the stakeholders of the HTA decision-making process should be engaged to ensure that recommendations and decisions are made consistently. The second recommendation stipulated that a few core criteria should be selected based on ethical considerations. The latter requires engaging philosophers and ethicists in the process. Canada can learn from the Swedish experience in developing an HTA-based decision-making process [13]. Sweden Expert Rev. Pharmacoecon. Outcomes Res.
tried to identify core principles and values for their HTA through an open discussion. It took Sweden 4 years of committee deliberation and then parliamentary debate to achieve its goals. Sweden kept only three basic principles as a start. Decisions were defended in public debate or in court. Audience members were asked to provide their views about additional criteria that should be incorporated into Canadian decision-making. The input from Canadian stakeholders revealed that, in order of preference, the additional areas that should be considered in decision-making were: • • • • •
Unmet medical needs Disease severity Societal benefits Enhanced patient experience Drug innovation
How should decision-making criteria be measured?
The measurement of criteria relates to the types of scales used to compare HTs. The panelists mentioned that this issue is critical especially when comparing different types of HTs with different therapeutic indications. Two types of scales can be used, namely, measurement scales and constructed scales [7]. Measurement scales are inherently linked to the data being evaluated. However, these scales have limited value when doing comparisons across different therapeutic areas. As a result, the panel recommended the use of constructed scales. These scales measure items beyond the specific characteristics of a given intervention, thereby solving the issue of full comparability across different therapeutic areas. Whose perspective should be taken into consideration in preference elicitation?
The panelists argued that preference elicitation should reflect the perspective of experts in HTA based on a democratic process. The rationale is that, as citizens, we elect candidates to represent our interests. We give them responsibilities and trust their decisions. The same reasoning should prevail in defining the best perspective to use. Specifically, a committee of Canadian experts in HTA should be selected to provide their preferences in regard to pre-established decision-making criteria. The appointment of experts to the committee should be based on their experience and contribution to Canadian HTA, according to democratic values. Preference elicitation would adopt the perspective of the committee members. Even though the audience members concurred with the panelists in regard to whose perspective to use in preference elicitation, they were more concerned with the preference elicitation per se. They raised the concern that no matter who the decision-makers are, their values are dynamic, meaning they change over time. It is therefore challenging to factor decisionmakers’ values in their choice when they make repeated decisions, as it is the case in HTA. A potential solution to this problem would be to treat decision-making criteria as variable interdependent parameters (VIP) subject to a set of informahealthcare.com
Special Report
70%
60%
59%
50% 45% Percentage
Expert Review of Pharmacoeconomics & Outcomes Research Downloaded from informahealthcare.com by UMEA University Library on 10/06/14 For personal use only.
MCDA for health technology assessment in Canada
40% 36% 30%
20%
18% 14%
10%
9%
9%
9%
0% Political will Fragmented Budget decisions (blue) constraints or differing perspectives (red) Barriers
Complexity
Figure 1. Barriers to identifying (blue vertical lines) and ranking (red dots) criteria for multi-criteria decision analysis (MCDA). This figure illustrates the audience members’ opinions regarding the biggest obstacles to the implementation of MCDA in Canada (identification and ranking). As illustrated, the largest barrier both to identifying and weighing criteria for MCDA is the fact that decisions are fragmented and are made according to varying perspectives. In addition, ‘complexity’ was found to be the second largest barrier for the implementation of MCDA in Canada. ‘Complexity’ relates to the methodological concerns associated with the incorporation of MCDA to the existing Canadian health technology assessment.
constraints [14]. The constraints are set based on the ranking of decision-making criteria and the inequalities obtained from this ranking. It is therefore possible to randomly and implicitly assign different weights to the criteria on the condition that the constraints are not violated. A robustness analysis can be conducted by calculating the payoffs generated by competing HTs, under the preset constraints. For more details on this approach, see Dias and Climaco [14] as well as Diaby and Goeree [7]. How should the most appropriate MCDA method for Canadian HTA be selected?
The panel members reported that it was an arduous task given the large number of MCDA methods available. They emphasized that even though no standard approach exists to choose the most appropriate MCDA method for HTA, the selection can be guided by the type of problem being tackled, whether the decision-making process allows for compensation or not, the type of the data to be analyzed and the level of sophistication of the model. MCDA methods are designed to assist in doi: 10.1586/14737167.2015.965155
Expert Review of Pharmacoeconomics & Outcomes Research Downloaded from informahealthcare.com by UMEA University Library on 10/06/14 For personal use only.
Special Report
Diaby, Goeree, Hoch & Siebert
dealing with three types of problems, namely, choice, ranking and sorting [6]. A choice problem deals with the identification of the ‘best alternative’ or a reduced set of ‘best alternatives.’ A ranking problem requires the rank ordering of alternatives from top to bottom. A sorting problem deals with the assignment of alternatives to predefined ordered categories of merit. For example, value-based measurement models and goal programming and aspiration models are appropriate for both choice and ranking problems, while outranking models can be used for all types of decision-making problems. Another criterion that can be used in addition to the decision-making problem is whether or not the decision-making process allows for compensation. As an illustration, a choice problem such as selecting the best device for eye surgery from a hospital perspective can be dealt with using a value-based measurement model when complete compensatory effects are allowed in the decision-making process. Compensatory effects may occur when the decision-making process allows alternatives to offset a poor performance on one criterion by a good performance on another criterion. The use of value-based measurement models is relevant especially when data to analyze are measured on a quantitative scale. When qualitative data are part of the body of evidence, the use of outranking models are more appropriate. It should be noted that the criteria, proposed to guide the selection of the most appropriate model to address the decision-making problem, are not mutually exclusive. The expert panel recommends that an algorithm for the selection of the appropriate MCDA methods to deal with HTA decisionmaking problems be developed based on the criteria previously mentioned. How would cost–effectiveness fit into the new MCDA-based framework and how would opportunity cost be explicitly dealt with?
This question was treated in two stages. The first stage relates to whether or not cost–effectiveness should be considered as an explicit criterion in MCDA modeling, while the second deals with the way opportunity cost is accounted for when recommendations are made from an MCDA standpoint. Criteria included in MCDA modeling have to satisfy a number of properties including independence [6] (avoiding double counting); meaning criteria must be mutually exclusive to allow decision-makers to trade-off if needed. Cost–effectiveness, as a criterion, violates this property when other criteria in the decision-making process capture dimensions related to the cost and effect of an HT. The panel recommended that cost and effectiveness be two separate criteria. The second aspect of ‘Question (e)’ deals with the way to deal with opportunity cost when using MCDA modeling. Some MDCA tools can be used to explicitly account for the opportunity cost of selecting an HT over its comparators. With the VIP analysis described earlier, it is possible to estimate the maximum loss of opportunity associated with selecting an HT, commonly referred to as ‘maximum regret’ [7,14]. In an MCDA evaluation, the alternative exhibiting the lowest maximum doi: 10.1586/14737167.2015.965155
regret is the best alternative. Maximum regret is perfectly grounded in economic theory, which makes it relevant for resource allocation under budget constraints. The ‘maximum regret’ and opportunity cost share the same roots and definition and therefore maximum regret can be used, at least, as a proxy for opportunity cost. With methodological advances in MCDA, it will be possible to refine the ability of MCDA to fully capture the opportunity cost concept. Expert commentary
To our knowledge, this paper is the first to report on expert panel and Canadian multistakeholder opinions about MCDA requirements and barriers in both oncology and nononcology HTA decision-making. This effort was preceded by the organization of a symposium by Amgen in conjunction with the Canadian Association for Population Therapeutics Conference in March 2010. The symposium aimed at discussing ‘pros’ and ‘cons’ of the use of MCDA in Canada. The information gathered through the symposium has not been published. The current paper provides insights from the expert panel discussion and contributions from Canadian stakeholders in HTA. There is a general desire to incorporate MCDA formal strategies into the existing HTA decision-making process in Canada. Fundamental and practical concerns that might hinder this progression include fragmented decision-making across provinces, political will and methodological challenges toward implementation of an MCDA process. The recommendations from the panel of experts to facilitate this implementation involve a collaborative approach involving all stakeholders to establish meaningful decision-making criteria based on ethical considerations. Potential avenues to overcome methodological concerns comprise the use of VIP analysis to account for the dynamic nature of decision-makers’ preferences and the use of maximum regret as a proxy for opportunity cost in MCDA modeling. The format used is appropriate for the purpose of the study. Expert panel discussions have the advantage of providing insights from a diverse group of knowledgeable and experienced individuals on a subject under evaluation. This format is both parsimonious and cost–effective as it does not require a significant amount of time to obtain credible results. This study design has been used in healthcare for similar purposes including the implementation of Coverage with Evidence Development in Canada [15,16]. Its main limitation is that expert panel members do not always come to a consensus in their recommendations. To address this issue, the paper reported any divergence that occurred among the panelists as part of their discussion. Another limitation of the study is the limited number of experts and discussants at the workshop. That said it can be imagined that this limited number of people allows for a better management of the discussion in terms of taking notes and achieving consensus in recommendations that are made. Five-year view
The journey regarding the debate on the generic use of MCDA in Canadian HTA has just begun. The normative choices raised Expert Rev. Pharmacoecon. Outcomes Res.
Expert Review of Pharmacoeconomics & Outcomes Research Downloaded from informahealthcare.com by UMEA University Library on 10/06/14 For personal use only.
MCDA for health technology assessment in Canada
by the implementation of an MCDA framework in Canada need to be made through a collaborative process that engages the stakeholders involved in HTA decision-making in Canada. The EVIDEM collaboration [17] is an example of a not-forprofit platform that brings together researchers, clinical experts, ethicists and decision-makers from all over the world. The aim of this platform is to facilitate collaboration and promote public health, taking advantage of diverse perspectives and expertise. The EVIDEM framework facilitates transparent and optimal healthcare decision-making through systematic and transparent appraisal of evidence. The results from the early applications of the EVIDEM MCDA-based tool by local and international decision-makers constitute a valuable resource to help reshape the face of Canadian HTA. In terms of future directions, it would be interesting to use the design of this study to conduct large-scale consultations with local experts and HTA stakeholders and compare the new findings to the results obtained in the current study. A step forward would be to secure the participation of politicians as they may be instrumental in moving from debate to action plans.
Special Report
Disclaimer
The ideas conveyed and conclusions drawn in this paper do not necessarily represent the position of Amgen Canada. Acknowledgements
The authors of the current paper are grateful to L Forte and B Bick for their contribution in the organization of the workshop. The authors would also like to thank J Barber and E Campbell, Associate Professors, Economic Social and Administrative Pharmacy, College of Pharmacy and Pharmaceutical Sciences, Florida A&M University, for their insightful comments on earlier versions of the paper. Financial & competing interests disclosure
Amgen Canada sponsored the IHEA session and proceedings as well as the literature search used by the expert panelists. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed. No writing assistance was utilized in the production of this manuscript.
Key issues • Multi-criteria decision analysis (MCDA) provides an opportunity for healthcare authorities, Canadian provinces in particular, to make decisions that reflect the unique priorities of the residents within their jurisdictions. • Potential barriers that oppose resistance to the successful implementation of MCDA in Canada include fragmented decision-making across provinces, political will and methodological challenges toward implementation of an MCDA process. • A collaborative approach involving all stakeholders will help establish meaningful decision-making criteria based on ethical considerations. • Potential avenues to overcome methodological concerns include the use of variable interdependent parameters analysis to account for the stochastic nature of decision-makers’ preferences and the use of maximum regret to approximate opportunity cost in MCDA modeling.
Pharmacoecon Outcomes Res 2014;14(1): 81-99
References 1.
Banta D. The development of health technology assessment. Health Policy 2003; 63(2):121-32
2.
CDR. Available from: http://cadth.ca/en/ products/cdr/cdr-overview
3.
pCODR. Available from: www.pcodr.ca/
4.
Baltussen R, Niessen L. Priority setting of health interventions: the need for multi-criteria decision analysis. Cost Effect Resour Alloc 2006;4(1):14
5.
Yoon KP, Hwang C. Multiple attribute decision making: an introduction. Sage; NY, USA: 1995
6.
Belton V, Stewart T. Multiple criteria decision analysis: an integrated approach. Springer; NY, USA: 2002
7.
Diaby V, Goeree R. How to use multi-criteria decision analysis methods for reimbursement decision-making in healthcare: a step-by-step guide. Expert Rev
informahealthcare.com
8.
9.
10.
11.
12.
Thokala P, Duenas A. Multiple criteria decision analysis for health technology assessment. Value Health 2012;15(8): 1172-81 Danner M, Hummel JM, Volz F, et al. Integrating patients’ views into health technology assessment: analytic hierarchy process (AHP) as a method to elicit patient preferences. Int J Technol Assess Health Care 2011;27(04):369-75 Mitton CR, Donaldson C. Setting priorities and allocating resources in health regions: lessons from a project evaluating program budgeting and marginal analysis (PBMA). Health Policy 2003;64(3):335-48 Ruta D, Mitton C, Bate A, Donaldson C. Programme budgeting and marginal analysis: bridging the divide between doctors and managers. BMJ 2005;330(7506):1501-3
using multi-attribute utility theory and programme budgeting and marginal analysis (PBMA). Soc Sci Med 2007;64(4):897-910 13.
Carlsson P. Health technology assessment and priority setting for health policy in Sweden. Int J Technol Assess Health Care 2004;20(01):44-54
14.
Dias L, Clı´maco J. Additive aggregation with variable interdependent parameters: the VIP analysis software. J Oper Res Soc 2000; 51(9):1070-82
15.
Menon D, McCabe CJ, Stafinski T, Edlin R. Principles of design of access with evidence development approaches. Pharmacoeconomics 2010;28(2):109-11
16.
Trueman P, Grainger DL, Downs KE. Coverage with evidence development: applications and issues. Int J Technol Assess Health Care 2010;26(01):79-85
17.
EVIDEM. Available from: www.evidem.org/
Peacock SJ, Richardson JR, Carter R, Edwards D. Priority setting in health care
doi: 10.1586/14737167.2015.965155
Special Report
Multi-criteria decision analysis for health technology assessment in Canada: insights from an expert panel discussion Expert Rev. Pharmacoecon. Outcomes Res. Early online, 1–7 (2014)
Vakaramoko Diaby*1, Ron Goeree2, Jeffrey Hoch3,4 and Uwe Siebert5 1 Division of Economic, Social and Administrative Pharmacy, College of Pharmacy and Pharmaceutical Sciences, Florida A&M University, Tallahassee, FL, USA 2 McMaster University/Programs for Assessment of Technology in Health, Hamilton, ON, Canada 3 University of Toronto, Toronto, ON, Canada 4 Department of Health Policy, Management and Evaluation, Centre for Research on Inner City Health (CRICH), St. Michael’s Hospital, Toronto, ON, Canada 5 Department of Public Health and Health Technology Assessment, UMIT – University for Health Sciences, Medical Informatics and Technology, Hall i.T., Austria *Author for correspondence: [email protected]
informahealthcare.com
Multi-criteria decision analysis (MCDA), a decision-making tool, has received increasing attention in recent years, notably in the healthcare field. For Canada, it is unclear whether and how MCDA should be incorporated into the existing health technology assessment (HTA) decision-making process. To facilitate debate on improving HTA decision-making in Canada, a workshop was held in conjunction with the 8th World Congress on Health Economics of the International Health Economics Association in Toronto, Canada in July 2011. The objective of the workshop was to discuss the potential benefits and challenges related to the use of MCDA for HTA decision-making in Canada. This paper summarizes and discusses the recommendations of an expert panel convened at the workshop to discuss opportunities and concerns with reference to the implementation of MCDA in Canada. KEYWORDS: Canada • decision-making • healthcare • health technology assessment • multi-criteria decision analysis • workshop
Health technology assessment (HTA) is an important tool used in reimbursement decisionmaking in healthcare systems. Developed in North America in the mid-70s [1], HTA has been adopted worldwide. HTA bridges the gap between evidence and healthcare policy and reimbursement decisions [1]. In Canada, provincial HTA decisions are based on a deliberative process using four decisionmaking criteria, referred to as ‘traditional’ criteria. These criteria are efficacy, safety, cost–effectiveness and affordability. The same criteria, except for ‘affordability,’ are used by federal HTA bodies, the Common Drug Review [2] and the panCanadian Oncology Drug Review [3], to make HTA recommendations. In recent years, HTA bodies across the world, Canada included, have been challenged to expand the decision-making criteria and to make the decision-making process more transparent. Advocates for these changes argue that decisions about the adoption of new health technologies (HTs) are inherently complex and multifaceted and that these decisions 10.1586/14737167.2015.965155
inconsistently apply criteria and involve implicit value judgments. Since HTA decision-making involves multiple factors that may conflict and a variety of stakeholders with different values, a new rational, reproducible and transparent framework is needed. Multi-criteria decision analysis (MCDA), a decision-making framework that uses a large set of methods that simultaneously and explicitly take into account multiple and conflicting factors [4], has the potential to strengthen and complement the traditional decision-making process in HTA. These methods can be classified into four main families: elementary methods [5], value-based measurement models, goal programming models and outranking models [6]. Elementary methods include basic MCDA techniques that do not require preference elicitation as part of the decision-making process. These methods include the maxmin and maxmax approaches, the Hurwicz procedure and the even swap method [7]. Value-based measurement models require the construction and
Ó 2014 Informa UK Ltd
ISSN 1473-7167
1
Expert Review of Pharmacoeconomics & Outcomes Research Downloaded from informahealthcare.com by UMEA University Library on 10/06/14 For personal use only.
Special Report
Diaby, Goeree, Hoch & Siebert
comparison of numerical scores to represent how one alternative is preferred over another. The aggregation rule for these models usually uses a weighted sum approach. Examples of these models include the simple linear additive model, the analytical hierarchy process and the multiattribute utility theory. In goal programming and aspiration models, the aim is to identify the alternative that best satisfies predefined levels of achievement [6]. An example of a goal programming and aspiration model is the linear goal programming model. In outranking models, competing options are compared pairwise by criterion. These comparisons allow for the development of outranking relations (partial dominance), which are used to prioritize alternatives based on an exploitation algorithm. ELECTRE and PROMETHEE are the most prominent methods of this family. In terms of structure, the MCDA process consists of three steps [6]. The first step, referred to as problem identification and structuring, deals with identifying the decision-makers and identifying their goals. At this step, the relevant competing options and their evaluation criteria are defined. The second step, called multi-criteria evaluation model development and use, is dedicated to the selection of the most appropriate model that will be used to address the decision-making problem at stake. The last step, called the development of action plans, consists of making recommendations for implementation to the decision-makers. At this step, sensitivity analyses are presented to strengthen the decision-makers’ confidence in action plans they envision carrying out. Lately, many healthcare jurisdictions have been looking into the use of MCDA to support HTA decision-making. In the UK, the National Institute for Health and Clinical Excellence has been considering MCDA as one of the approaches to implement its new value-based pricing scheme [8]. In Germany, the German Institute for Quality and Efficiency in Health Care [9] has considered using MCDA to support HTA decision-making. Until now, there are very few attempts to develop and implement an MCDA-based framework for HTA decision-making in Canada. The evidence and value: impact on decision-making (EVIDEM) collaboration, an independent nonprofit organization, has developed and proposed a framework that brings together HTA and MCDA. EVIDEM has a strong Canadian representation and the framework has been field tested in the Canadian healthcare setting. In British Columbia (Canada), MCDA has long been used through program budgeting and marginal analysis [10–12], a tool to priority setting and reallocation of scarce healthcare resources (but not HTA). Even though MCDA holds promise for more rational and transparent decision-making, it is unclear how MCDA should be incorporated into the existing Canadian HTA decision-making. To facilitate the debate on improving HTA decision-making in Canada, a workshop involving an expert panel was organized around the potential benefits and challenges related to the use of MCDA for HTA decision-making. The current paper reports on the results of this expert panel discussion and touches on methodological concerns that arose doi: 10.1586/14737167.2015.965155
during the workshop with reference to the potential implementation of MCDA in Canada. Organization of the workshop
An open workshop was held in conjunction with the International Health Economics Association World Congress meeting held in Toronto, Ontario, Canada in July 2011. Five panel participants (from the UK, Brazil, Sweden, Germany and Canada) with expertise spanning key areas of concern for HTA were convened by Amgen Canada to obtain guidance and stimulate the debate on the use of MCDA to support HTA decision-making in Canada. The experts had experience in government, academia, research and nonprofit sectors, which brought a diversity of perspectives for the panel discussion. Potential conflicts of interest were assessed by enquiring about the panelists’ past and current relationships with Amgen Canada. The experts served as individual scientists, meaning they conveyed their own scientific opinions, not the views of the institutions they were affiliated with. The workshop audience represented a cross-section of ten Canadian stakeholders, such as payers, policy-makers, academia and industry. SUPPLEMENTARY APPENDIX A (Supplementary material can be found online at www.informahealthcare.com/ suppl/14737167.2015.965155) contains a list of the expert panelists and discussants. The key questions around which the debate was structured are presented below. • What are the potential benefits of using MCDA to support HTA decision-making in Canada? • What are the potential barriers to using MCDA to support HTA decision-making in Canada? • How should methodological concerns raised by the implementation of MCDA in Canada be dealt with? a. What additional criteria should be used for decisionmaking in Canada, using MCDA? b. How should decision-making criteria be measured? c. Whose perspective should be taken into consideration in preference elicitation? d. How should the most appropriate MCDA method for Canadian HTA be selected? e. How would cost–effectiveness fit into the new MCDAbased framework? More specifically, how would opportunity cost be explicitly dealt with? Before tackling these questions, the panelists were asked to make a short presentation on factors that influence decisions regarding the adoption of HTs in their respective countries. Then, panelists were asked to provide their scientific views on each question presented earlier. They were expected to reach a consensus on their responses and recommendations. When a consensus could not be reached among panelists, this was reported in the panel session debrief notes. One important feature of an expert panel discussion is that it offers an opportunity for panelists to both share their opinions on a topic of interest and then to discuss these views Expert Rev. Pharmacoecon. Outcomes Res.
MCDA for health technology assessment in Canada
Special Report
Expert Review of Pharmacoeconomics & Outcomes Research Downloaded from informahealthcare.com by UMEA University Library on 10/06/14 For personal use only.
Table 1. Factors involved in health technology assessment in five countries. Country
Publicly stated factors for decisions
Other factors applied in practice
UK
• • • • •
• Cost–effectiveness versus a threshold of £20,000–30,000; the threshold for cancer is more permissive and allows an additional £10,000–£20,000
Brazil
• Scientific evidence of safety, efficacy and effectiveness • Economic evaluation • Budget impact
• Budget constraints • Various ‘interests’
Sweden
• Severity of illness • Expected benefit of healthcare intervention • Cost–effectiveness – no explicit cut-off but implicit cut-off of about US$75,000
• Physicians set priorities within disease areas according to: Severity of condition Effect of intervention Evidence for effect Cost per gained effect
Germany
• Assessment of benefit versus cost of new versus existing treatments Points are plotted on an efficiency frontier The line on the cost–effectiveness plane is extrapolated; if the price of the new technology exceeds the line, the price must be reduced Framework for evaluation is within each disease not across diseases • Ethical, legal, social weights
• Demonstrated innovation or breakthrough can result in negotiated higher price • Attempt to value interventions across disease areas based on: Unmet medical need Clinical significance, severity of disease Quality/level of evidence Cost–effectiveness (disease-specific assessment) Vulnerable populations, equity fairness Previous decisions Factors ‘behind closed doors’
Canada (pCODR)
• • • •
• Not applicable
Canada (CDR)
• Clinical benefit – quality of evidence, magnitude of benefit • Economics – cost–effectiveness, price • Layperson/patient stakeholder values
Severity of illness End of life Stakeholder persuasion Significant innovation Disadvantaged populations, children
Overall clinical benefit Alignment with patient values Cost–effectiveness Feasibility of adoption into health systems
• • • • • •
Type of outcome (e.g., surrogate, intermediate, final) Comparative safety Historical precedent Trial size Trial duration Generalizability of results
Canada (excluding Quebec) has two national drug recommendation processes; pCODR is for cancer drugs and CDR is for all other drugs. CDR: Common drug review; N/A: Not applicable; pCODR: pan-Canadian Oncology Drug Review.
through interaction with the audience (questions and answers period). As part of the panel session, audience members were asked to rate the importance of different criteria in decisionmaking with respect to a hypothetical cancer scenario. They were also asked to rank barriers for identifying, weighing and establishing criteria to be considered in Canadian HTA decisions. The next section presents the insights obtained from the panel discussion. Insights from the expert panel session Summary from expert panel presentations
The panel members presented different decision-making rules where different factors influence decisions regarding the adoption of new HTs. Each of the national HTA bodies represented by the expert panel members has publicly stated the factors they consider in their decision-making. However, according to the panelists, additional factors influence decisions in practice. informahealthcare.com
A summary of the factors involved in HTA decision-making in the five countries is shown in TABLE 1. What are the potential benefits of using MCDA to support HTA decision-making in Canada?
The panel members and the audience concurred that MCDA provides an opportunity for healthcare authorities, Canadian provinces in particular, to make decisions that reflect the unique priorities of the residents within their jurisdictions. By securing clinician input regarding clinical criteria for drug/ intervention use and patient and caregiver input regarding the limitations of available alternatives (and the need for new alternatives), payers can be more confident that the decisions they make reflect local needs and priorities. Indeed, MCDA offers a range of methods that attempt to decompose and reflect decision-maker’ values as part of their thinking process. Another benefit of a transparent and comprehensive MCDA doi: 10.1586/14737167.2015.965155
Special Report
Diaby, Goeree, Hoch & Siebert
Expert Review of Pharmacoeconomics & Outcomes Research Downloaded from informahealthcare.com by UMEA University Library on 10/06/14 For personal use only.
Table 2. Common barriers to the implementation of multi-criteria decision analysis. Barriers
Rationale
Confidential pricing
Affects true cost–effectiveness
Unmet needs and severity
Conditions are similar and difficult to separate
Budget impact (low or high)
Will discriminate against people due to the size of the affected group
Double counting of dimensions measured by a criterion
May occur with overlapping factors, for example, degree of innovation and significant improvement
Meaningful and fair weighting of multiple outcomes
Decision-makers do not want this part of the process to be publicized
Complexity of comparing across health sectors and across wider sectors
It is often not clear what part of the health budget should be used for domestic policy rather than population’s health improvement
Political will
These factors are always at play
process lies in the public support that can be garnered for the decision itself and for the decision-makers’ difficult role in allocating scarce resources. With the movement toward increasing the number and type of decision criteria (quantitative and qualitative) for HTA decision-making, MCDA methods can play an important role in supporting HTA decisions by being able to adequately handle both quantitative and qualitative information. Finally, some MCDA methods allow for capturing uncertainty through sensitivity and robustness analyses, with reducing uncertainty representing a dimension of paramount importance in decision-making. What are the potential barriers to using MCDA to support HTA decision-making in Canada?
A review of decision-making rules for HTA, based on the presentations on implicit (used in practice) and explicit criteria (publically stated) in the five countries presented in TABLE 1, suggests a need for a systematic and transparent approach such as MCDA to ensure consistency in HTA decision-making across jurisdictions. However, methodological challenges raised by the rigorous incorporation of MCDA in HTA decision-making remains a hurdle in all countries, including Canada. The panelists reported a number of potential barriers that should be taken into consideration for the effective implementation of MCDA in Canada, which are summarized in TABLE 2. They also provided explanations as to why these dimensions should be considered as barriers (TABLE 2). Audience members representing a variety of stakeholder sectors were polled regarding their opinions about the barriers to MCDA in Canada. The respondents’ opinions regarding the biggest obstacles to the use of MCDA in Canada are shown in FIGURE 1. The latter suggests that the largest barrier both to identifying and weighing criteria for MCDA is the fact that decisions are fragmented and are made according to varying perspectives. Given that healthcare budgets are controlled by provincial governments and each province has varying social priorities, the implementation of MCDA in Canadian HTA doi: 10.1586/14737167.2015.965155
decision-making may result in further disparities in healthcare funding across the provinces if they develop their own MCDA process in ‘silos.’ In addition, ‘complexity’ was found to be the second largest barrier for the implementation of MCDA in Canada. ‘Complexity’ relates to the methodological concerns associated with the incorporation of MCDA to the existing Canadian HTA. These concerns are discussed in section How should methodological concerns raised by the implementation of MCDA in Canada be dealt with? On another note, compared with a rigid focus on clinical and economic evidence, MCDA is a more demanding process for making healthcare funding decisions, which can lead to increased time between regulatory approval and public funding of new technologies. That said, the panel members suggested that a formal MCDA framework in conjunction with traditional HTA may help address the concern about timely access to effective HTs. How should methodological concerns raised by the implementation of MCDA in Canada be dealt with? What additional criteria should be used for decision-making in Canada, using MCDA?
The panelists could not come to a consensus on what additional criteria should be considered. Some panelists noticed overlap between the criteria identified by the audience and the criteria used in practice in the five countries. These criteria were unmet medical needs, disease severity and drug innovation. They can be used as a starting point for further discussion regarding the establishment of additional criteria. Two other recommendations emerged from the panelist discussion. The first emphasized that as criteria are compiled, all the stakeholders of the HTA decision-making process should be engaged to ensure that recommendations and decisions are made consistently. The second recommendation stipulated that a few core criteria should be selected based on ethical considerations. The latter requires engaging philosophers and ethicists in the process. Canada can learn from the Swedish experience in developing an HTA-based decision-making process [13]. Sweden Expert Rev. Pharmacoecon. Outcomes Res.
tried to identify core principles and values for their HTA through an open discussion. It took Sweden 4 years of committee deliberation and then parliamentary debate to achieve its goals. Sweden kept only three basic principles as a start. Decisions were defended in public debate or in court. Audience members were asked to provide their views about additional criteria that should be incorporated into Canadian decision-making. The input from Canadian stakeholders revealed that, in order of preference, the additional areas that should be considered in decision-making were: • • • • •
Unmet medical needs Disease severity Societal benefits Enhanced patient experience Drug innovation
How should decision-making criteria be measured?
The measurement of criteria relates to the types of scales used to compare HTs. The panelists mentioned that this issue is critical especially when comparing different types of HTs with different therapeutic indications. Two types of scales can be used, namely, measurement scales and constructed scales [7]. Measurement scales are inherently linked to the data being evaluated. However, these scales have limited value when doing comparisons across different therapeutic areas. As a result, the panel recommended the use of constructed scales. These scales measure items beyond the specific characteristics of a given intervention, thereby solving the issue of full comparability across different therapeutic areas. Whose perspective should be taken into consideration in preference elicitation?
The panelists argued that preference elicitation should reflect the perspective of experts in HTA based on a democratic process. The rationale is that, as citizens, we elect candidates to represent our interests. We give them responsibilities and trust their decisions. The same reasoning should prevail in defining the best perspective to use. Specifically, a committee of Canadian experts in HTA should be selected to provide their preferences in regard to pre-established decision-making criteria. The appointment of experts to the committee should be based on their experience and contribution to Canadian HTA, according to democratic values. Preference elicitation would adopt the perspective of the committee members. Even though the audience members concurred with the panelists in regard to whose perspective to use in preference elicitation, they were more concerned with the preference elicitation per se. They raised the concern that no matter who the decision-makers are, their values are dynamic, meaning they change over time. It is therefore challenging to factor decisionmakers’ values in their choice when they make repeated decisions, as it is the case in HTA. A potential solution to this problem would be to treat decision-making criteria as variable interdependent parameters (VIP) subject to a set of informahealthcare.com
Special Report
70%
60%
59%
50% 45% Percentage
Expert Review of Pharmacoeconomics & Outcomes Research Downloaded from informahealthcare.com by UMEA University Library on 10/06/14 For personal use only.
MCDA for health technology assessment in Canada
40% 36% 30%
20%
18% 14%
10%
9%
9%
9%
0% Political will Fragmented Budget decisions (blue) constraints or differing perspectives (red) Barriers
Complexity
Figure 1. Barriers to identifying (blue vertical lines) and ranking (red dots) criteria for multi-criteria decision analysis (MCDA). This figure illustrates the audience members’ opinions regarding the biggest obstacles to the implementation of MCDA in Canada (identification and ranking). As illustrated, the largest barrier both to identifying and weighing criteria for MCDA is the fact that decisions are fragmented and are made according to varying perspectives. In addition, ‘complexity’ was found to be the second largest barrier for the implementation of MCDA in Canada. ‘Complexity’ relates to the methodological concerns associated with the incorporation of MCDA to the existing Canadian health technology assessment.
constraints [14]. The constraints are set based on the ranking of decision-making criteria and the inequalities obtained from this ranking. It is therefore possible to randomly and implicitly assign different weights to the criteria on the condition that the constraints are not violated. A robustness analysis can be conducted by calculating the payoffs generated by competing HTs, under the preset constraints. For more details on this approach, see Dias and Climaco [14] as well as Diaby and Goeree [7]. How should the most appropriate MCDA method for Canadian HTA be selected?
The panel members reported that it was an arduous task given the large number of MCDA methods available. They emphasized that even though no standard approach exists to choose the most appropriate MCDA method for HTA, the selection can be guided by the type of problem being tackled, whether the decision-making process allows for compensation or not, the type of the data to be analyzed and the level of sophistication of the model. MCDA methods are designed to assist in doi: 10.1586/14737167.2015.965155
Expert Review of Pharmacoeconomics & Outcomes Research Downloaded from informahealthcare.com by UMEA University Library on 10/06/14 For personal use only.
Special Report
Diaby, Goeree, Hoch & Siebert
dealing with three types of problems, namely, choice, ranking and sorting [6]. A choice problem deals with the identification of the ‘best alternative’ or a reduced set of ‘best alternatives.’ A ranking problem requires the rank ordering of alternatives from top to bottom. A sorting problem deals with the assignment of alternatives to predefined ordered categories of merit. For example, value-based measurement models and goal programming and aspiration models are appropriate for both choice and ranking problems, while outranking models can be used for all types of decision-making problems. Another criterion that can be used in addition to the decision-making problem is whether or not the decision-making process allows for compensation. As an illustration, a choice problem such as selecting the best device for eye surgery from a hospital perspective can be dealt with using a value-based measurement model when complete compensatory effects are allowed in the decision-making process. Compensatory effects may occur when the decision-making process allows alternatives to offset a poor performance on one criterion by a good performance on another criterion. The use of value-based measurement models is relevant especially when data to analyze are measured on a quantitative scale. When qualitative data are part of the body of evidence, the use of outranking models are more appropriate. It should be noted that the criteria, proposed to guide the selection of the most appropriate model to address the decision-making problem, are not mutually exclusive. The expert panel recommends that an algorithm for the selection of the appropriate MCDA methods to deal with HTA decisionmaking problems be developed based on the criteria previously mentioned. How would cost–effectiveness fit into the new MCDA-based framework and how would opportunity cost be explicitly dealt with?
This question was treated in two stages. The first stage relates to whether or not cost–effectiveness should be considered as an explicit criterion in MCDA modeling, while the second deals with the way opportunity cost is accounted for when recommendations are made from an MCDA standpoint. Criteria included in MCDA modeling have to satisfy a number of properties including independence [6] (avoiding double counting); meaning criteria must be mutually exclusive to allow decision-makers to trade-off if needed. Cost–effectiveness, as a criterion, violates this property when other criteria in the decision-making process capture dimensions related to the cost and effect of an HT. The panel recommended that cost and effectiveness be two separate criteria. The second aspect of ‘Question (e)’ deals with the way to deal with opportunity cost when using MCDA modeling. Some MDCA tools can be used to explicitly account for the opportunity cost of selecting an HT over its comparators. With the VIP analysis described earlier, it is possible to estimate the maximum loss of opportunity associated with selecting an HT, commonly referred to as ‘maximum regret’ [7,14]. In an MCDA evaluation, the alternative exhibiting the lowest maximum doi: 10.1586/14737167.2015.965155
regret is the best alternative. Maximum regret is perfectly grounded in economic theory, which makes it relevant for resource allocation under budget constraints. The ‘maximum regret’ and opportunity cost share the same roots and definition and therefore maximum regret can be used, at least, as a proxy for opportunity cost. With methodological advances in MCDA, it will be possible to refine the ability of MCDA to fully capture the opportunity cost concept. Expert commentary
To our knowledge, this paper is the first to report on expert panel and Canadian multistakeholder opinions about MCDA requirements and barriers in both oncology and nononcology HTA decision-making. This effort was preceded by the organization of a symposium by Amgen in conjunction with the Canadian Association for Population Therapeutics Conference in March 2010. The symposium aimed at discussing ‘pros’ and ‘cons’ of the use of MCDA in Canada. The information gathered through the symposium has not been published. The current paper provides insights from the expert panel discussion and contributions from Canadian stakeholders in HTA. There is a general desire to incorporate MCDA formal strategies into the existing HTA decision-making process in Canada. Fundamental and practical concerns that might hinder this progression include fragmented decision-making across provinces, political will and methodological challenges toward implementation of an MCDA process. The recommendations from the panel of experts to facilitate this implementation involve a collaborative approach involving all stakeholders to establish meaningful decision-making criteria based on ethical considerations. Potential avenues to overcome methodological concerns comprise the use of VIP analysis to account for the dynamic nature of decision-makers’ preferences and the use of maximum regret as a proxy for opportunity cost in MCDA modeling. The format used is appropriate for the purpose of the study. Expert panel discussions have the advantage of providing insights from a diverse group of knowledgeable and experienced individuals on a subject under evaluation. This format is both parsimonious and cost–effective as it does not require a significant amount of time to obtain credible results. This study design has been used in healthcare for similar purposes including the implementation of Coverage with Evidence Development in Canada [15,16]. Its main limitation is that expert panel members do not always come to a consensus in their recommendations. To address this issue, the paper reported any divergence that occurred among the panelists as part of their discussion. Another limitation of the study is the limited number of experts and discussants at the workshop. That said it can be imagined that this limited number of people allows for a better management of the discussion in terms of taking notes and achieving consensus in recommendations that are made. Five-year view
The journey regarding the debate on the generic use of MCDA in Canadian HTA has just begun. The normative choices raised Expert Rev. Pharmacoecon. Outcomes Res.
Expert Review of Pharmacoeconomics & Outcomes Research Downloaded from informahealthcare.com by UMEA University Library on 10/06/14 For personal use only.
MCDA for health technology assessment in Canada
by the implementation of an MCDA framework in Canada need to be made through a collaborative process that engages the stakeholders involved in HTA decision-making in Canada. The EVIDEM collaboration [17] is an example of a not-forprofit platform that brings together researchers, clinical experts, ethicists and decision-makers from all over the world. The aim of this platform is to facilitate collaboration and promote public health, taking advantage of diverse perspectives and expertise. The EVIDEM framework facilitates transparent and optimal healthcare decision-making through systematic and transparent appraisal of evidence. The results from the early applications of the EVIDEM MCDA-based tool by local and international decision-makers constitute a valuable resource to help reshape the face of Canadian HTA. In terms of future directions, it would be interesting to use the design of this study to conduct large-scale consultations with local experts and HTA stakeholders and compare the new findings to the results obtained in the current study. A step forward would be to secure the participation of politicians as they may be instrumental in moving from debate to action plans.
Special Report
Disclaimer
The ideas conveyed and conclusions drawn in this paper do not necessarily represent the position of Amgen Canada. Acknowledgements
The authors of the current paper are grateful to L Forte and B Bick for their contribution in the organization of the workshop. The authors would also like to thank J Barber and E Campbell, Associate Professors, Economic Social and Administrative Pharmacy, College of Pharmacy and Pharmaceutical Sciences, Florida A&M University, for their insightful comments on earlier versions of the paper. Financial & competing interests disclosure
Amgen Canada sponsored the IHEA session and proceedings as well as the literature search used by the expert panelists. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed. No writing assistance was utilized in the production of this manuscript.
Key issues • Multi-criteria decision analysis (MCDA) provides an opportunity for healthcare authorities, Canadian provinces in particular, to make decisions that reflect the unique priorities of the residents within their jurisdictions. • Potential barriers that oppose resistance to the successful implementation of MCDA in Canada include fragmented decision-making across provinces, political will and methodological challenges toward implementation of an MCDA process. • A collaborative approach involving all stakeholders will help establish meaningful decision-making criteria based on ethical considerations. • Potential avenues to overcome methodological concerns include the use of variable interdependent parameters analysis to account for the stochastic nature of decision-makers’ preferences and the use of maximum regret to approximate opportunity cost in MCDA modeling.
Pharmacoecon Outcomes Res 2014;14(1): 81-99
References 1.
Banta D. The development of health technology assessment. Health Policy 2003; 63(2):121-32
2.
CDR. Available from: http://cadth.ca/en/ products/cdr/cdr-overview
3.
pCODR. Available from: www.pcodr.ca/
4.
Baltussen R, Niessen L. Priority setting of health interventions: the need for multi-criteria decision analysis. Cost Effect Resour Alloc 2006;4(1):14
5.
Yoon KP, Hwang C. Multiple attribute decision making: an introduction. Sage; NY, USA: 1995
6.
Belton V, Stewart T. Multiple criteria decision analysis: an integrated approach. Springer; NY, USA: 2002
7.
Diaby V, Goeree R. How to use multi-criteria decision analysis methods for reimbursement decision-making in healthcare: a step-by-step guide. Expert Rev
informahealthcare.com
8.
9.
10.
11.
12.
Thokala P, Duenas A. Multiple criteria decision analysis for health technology assessment. Value Health 2012;15(8): 1172-81 Danner M, Hummel JM, Volz F, et al. Integrating patients’ views into health technology assessment: analytic hierarchy process (AHP) as a method to elicit patient preferences. Int J Technol Assess Health Care 2011;27(04):369-75 Mitton CR, Donaldson C. Setting priorities and allocating resources in health regions: lessons from a project evaluating program budgeting and marginal analysis (PBMA). Health Policy 2003;64(3):335-48 Ruta D, Mitton C, Bate A, Donaldson C. Programme budgeting and marginal analysis: bridging the divide between doctors and managers. BMJ 2005;330(7506):1501-3
using multi-attribute utility theory and programme budgeting and marginal analysis (PBMA). Soc Sci Med 2007;64(4):897-910 13.
Carlsson P. Health technology assessment and priority setting for health policy in Sweden. Int J Technol Assess Health Care 2004;20(01):44-54
14.
Dias L, Clı´maco J. Additive aggregation with variable interdependent parameters: the VIP analysis software. J Oper Res Soc 2000; 51(9):1070-82
15.
Menon D, McCabe CJ, Stafinski T, Edlin R. Principles of design of access with evidence development approaches. Pharmacoeconomics 2010;28(2):109-11
16.
Trueman P, Grainger DL, Downs KE. Coverage with evidence development: applications and issues. Int J Technol Assess Health Care 2010;26(01):79-85
17.
EVIDEM. Available from: www.evidem.org/
Peacock SJ, Richardson JR, Carter R, Edwards D. Priority setting in health care
doi: 10.1586/14737167.2015.965155
