doi: 10.1111/hex.12178

Multicriteria decision analysis in oncology Georges Adunlin MA,* Vakaramoko Diaby PhD,† Alberto J. Montero MD MBA‡ and Hong Xiao PhD§ *PhD Candidate, Division of Economic, Social and Administrative Pharmacy, College of Pharmacy and Pharmaceutical Sciences, Florida A&M University, Tallahassee, FL, †Assistant Professor, Division of Economic, Social and Administrative Pharmacy, College of Pharmacy and Pharmaceutical Sciences, Florida A&M University, Tallahassee, FL, ‡Physician, Cleveland Clinic, Taussig Cancer Institute, Cleveland, OH and §Professor, Division of Economic, Social and Administrative Pharmacy, College of Pharmacy and Pharmaceutical Sciences, Florida A&M University, Tallahassee, FL, USA

Abstract Correspondence Georges Adunlin, MA PhD Candidate Division of Economic, Social and Administrative Pharmacy College of Pharmacy and Pharmaceutical Sciences Florida A & M University 200E Dyson Pharmacy Bldg, 1520 Martin Luther King Jr. Blvd Tallahassee FL 32307 USA E-mail: [email protected] Accepted for publication 3 February 2014 Keywords: decision making, multicriteria decision analysis, multidisciplinary team, oncology

Background There has been a growing interest in the development and application of alternative decision-making frameworks within health care, including multicriteria decision analysis (MCDA). Even though the literature includes several reviews on MCDA methods, applications of MCDA in oncology are lacking. Aim The aim of this paper is to discuss a rationale for the use of MCDA in oncology. In this context, the following research question emerged: How can MCDA be used to develop a clinical decision support tool in oncology? Methods In this paper, a brief background on decision making is presented, followed by an overview of MCDA methods and process. The paper discusses some applications of MCDA, proposes research opportunities in the context of oncology and presents an illustrative example of how MCDA can be applied to oncology. Findings Decisions in oncology involve trade-offs between possible benefits and harms. MCDA can help analyse trade-off preferences. A wide range of MCDA methods exist. Each method has its strengths and weaknesses. Choosing the appropriate method varies depending on the source and nature of information used to inform decision making. The literature review identified eight studies. The analytical hierarchy process (AHP) was the most often used method in the identified studies. Conclusion Overall, MCDA appears to be a promising tool that can be used to assist clinical decision making in oncology. Nonetheless, field testing is desirable before MCDA becomes an established decision-making tool in this field.

Introduction ‘Meta-analysis, decision analysis and costeffectiveness analysis are the cornerstones of evidence-based medicine. These related quanti-

ª 2014 John Wiley & Sons Ltd Health Expectations

tative methods have become essential tools in the formulation of clinical and public policy based on the synthesis of evidence’.1 Decision analysis can be defined as ‘a systematic procedure for transforming opaque decision

1

2

Multicriteria decision analysis, G Adunlin et al.

problems into transparent decision problems by a sequence of transparent steps’.2 Decision analysis can apply to conditions of certainty, risk or uncertainty. Several disciplines such as marketing, management, engineering and operations research have contributed to the growing field of decision analysis.3 Over the past decade, cancer treatments have become more sophisticated as a result of research and technological advances. Some treatment decisions in oncology are straightforward, with clear evidence to support one best approach, while other decisions are more complex and multifaceted.4,5 This owes to the fact that the evidence on outcomes may be uncertain or perhaps the benefit is clear, but there is a discrepancy among health care providers regarding the magnitude of risk between the different available options. Such decisions involve trade-offs between the possible benefits and harms of these treatments.6 Certain problems carry quantitative features which can be evaluated by means of numerical values. However, others carry qualitative features that are complex to evaluate by means of numerical values. Some health care providers when confronted with such problems solely rely on clinical practice guidelines, results of clinical trials or heuristic approaches to simplify the complexity of the problem.7–9 Relying on these evidences, a lot of information may be misplaced, conflicting positions may be thrust aside, and elements of uncertainty might be overlooked. As a perfect illustration, physicians have been faced with uncertainty in applying disease-specific guidelines for the care of older persons with multiple conditions. As individuals have cognitive biases that may impair the ability to accurately quantify risks and benefits, studies have suggested that teams, rather than individuals, should make such decisions.10 In cancer care, multidisciplinary teams (MDTs) have to work together to create a patient’s overall treatment plan that combines different types of treatments.11,12 An MDT brings together specialists from many fields, including physicians, namely medical oncologists, radiation oncologists, surgical oncologist,

pathologists and other health-care professionals such as pharmacists, nurses, physical therapists or social workers. The purpose of an MDT is to leverage the knowledge, skills and experience that each health care provider has to offer to deliver the best possible care.13,14 Providing good patient care requires high-quality decision making in the MDT meeting.11,13 Nonetheless, it is not clear that team decision making necessarily offsets cognitive biases, particularly if the decision process is carried out in an informal and non-transparent manner. The need for a more strategic decisionmaking process has never been greater in oncology. Hence, there has been a rapid proliferation of decision aids to help guide decisionmakers, providers and patients in deliberation and communication.15–18 However, there has also been a debate regarding the effectiveness of some of these decision aids in achieving a high-quality process and choice.19–21 A study highlighted problems with existing decision aids as they offer limited guidance on the integration and judgement of the importance of outcomes.21 These problems have further been compounded by the rapid evolution of clinical practice guidelines for most cancers. Clinical decision-making process has mostly focused on the patient side. There remains an important gap in knowledge about the physician decisionmaking process and decision-making factors in oncology that may be associated with observed variation in cancer treatment and patient outcomes. These challenges do not appear only in oncology. Indeed, other therapeutic areas are prone to challenges in clinical and reimbursement decision making. Nonetheless, there exist a broad set of techniques that can help make the decision-making process more transparent and are referred to as multicriteria decision analysis (MCDA). These methods are being investigated as a decision-making support tool in many different therapeutic areas and may represent a more effective alternative to traditional decision making in oncology. Even though the literature includes several reviews on MCDA methods, applications of MDCA in oncology are lacking. There is a need for clini-

ª 2014 John Wiley & Sons Ltd Health Expectations

Multicriteria decision analysis, G Adunlin et al.

cians and policymakers to be fully informed of what MCDA might offer to oncology. In this paper, a brief introduction on decision making is presented, followed by methodological background and process in MCDA. The paper also discusses some applications of MCDA, proposes research opportunities and presents an illustrative example of the application of MCDA in the context of oncology.

Multicriteria decision analysis (MCDA) What is multicriteria decision analysis? MCDA is both an approach and a set of techniques developed in the field of decision theory to aid in problem-solving.22 Many terminologies can be used to refer to MCDA such as multicriteria decision making (MCDM) and multiattribute decision analysis (MADA). However, our preference goes towards using MCDA terminology. The MCDA approach was initially used in the field of operations research, which is generally described as a discipline that deals with the application of advanced analytical methods to help make better decisions.22 MCDA has become a topic of great interest in decision sciences and has been applied in a broad spectrum of studies, from theoretical to applied studies. The use of MCDA in both the private and the public sectors is increasing rapidly. MCDA is used to address diverse issues in sustainable energy management,23,24 energy planning,25,26 transportation,27,28 geographical information systems,29,30 budgeting and resource allocation,31,32 and many other fields where multiple criteria are explicitly considered in the decision-making process. Multicriteria decision analysis methods and process Interest in MCDA has led to the development of a great variety of methods within the operations research field. These methods are broadly classified into three families: outranking models; value measurement models; and goal, aspiration or reference-level models.33,34 These

ª 2014 John Wiley & Sons Ltd Health Expectations

methods have been previously discussed in other publications: Hwang and Yoon,35 Dolan,36 Ehrgott et al.,37 Dodgson,38 Figueira et al.,39 Keeney,40 Belton and Stewart,33 Doumpos and Zopounidis,41 Triantaphyllou.42 In general, the MCDA process involves executing the same set of steps. Depending on the magnitude of the projects, the steps can be expanded or reduced. Furthermore, the steps are not always conducted in a set order. The steps are summarized in Fig. 1 and briefly explicated below. Choosing the best MCDA method varies depending on the source and nature of information used to inform decision making, but also on the model the decision-makers believe match their ability. A general feature of any multicriteria analysis is a performance matrix or consequence table, which are comprised of rows describing options and columns describing the performance of the options against each criterion. Diverse methods are used to aggregate judgements such as the weighted sum product and the weighed sum method. A simple weighted sum model is given by the following expression:

Step 1. Identification of the problem issue

Step 2. Problem structuring

Step 3. Model building

Step 4. Using to model to inform and challenge thinking

Step 5. Developing an action plan Figure 1 Multicriteria decision analysis process (Adapted from Belton and Stewart33).

3

4 Multicriteria decision analysis, G Adunlin et al.

WBSj ¼

n X

wi Sij ;

i¼1

where i = 1,. . ., n criteria; wi: criteria weights; j: alternatives; sij = scores for alternatives for different criteria; WBS: weighted benefit score. A number of software/programs are readily available to apply these methods. A nonexhaustive list of software/program can be found online.43,44 A prerequisite for researchers is the need to identify the most relevant software/program requirements. Nevertheless, the availability of software/program makes MCDA easier and less costly to implement, making its increased use in future much more likely. There are several challenges and limitations associated with MCDA. Our aim is not to provide a complete list of the challenges and limitations, but to mention the most prevalent ones. A major challenge is how to deal with conflicting decision criteria. As an example, a drug A might have a better performance than drug B in terms of effectiveness while having a worse performance than drug B on the safety side. As the most accurate information about the decision facts is essential, cautious consideration of the choice and presentation during the decision-making process constitutes an important challenge. In order for committee/ panel members to be efficient during the MCDA decision process, they need to be properly trained. The facilitator is also required to master the MCDA technique to avoid any pitfall.

Multicriteria decision analysis in oncology Literature review on applications Interest in MCDA has grown in health care over the last few years.45,46 A targeted literature search was conducted to identify published English language studies that used MCDA within the context of oncology. Electronic databases (Medline, PubMed, ProQuest, Ovid, Embase and Web of Knowledge) were searched between 1 January 1980 and 31 August 2013. The time period set for databases search

seemed appropriate to identify the large majority of the oncology-related MCDA studies published during the last three decades. Databases were searched using the free text terms: ‘multicriteria decision analysis’, ‘multicriteria decision analysis’, ‘multiple criteria decision aiding’, ‘multicriteria decision making’, ‘multicriteria decision making’, ‘multicriteria analysis’, ‘multi-attribute utility’, ‘multi-attribute utility’ and ‘multiple objective problems’. The above free text terms were combined with ‘oncology’ and ‘cancer’ using Boolean operators when necessary. This list of terms reflects the different terms used to refer to MCDA. Hand-searching for the specified period was undertaken for the following journals: International Journal of Multicriteria Decision Making (IJMCDM), Journal of Multi-Criteria Decision Analysis (JMCDA), European Journal of Operational Research (EJOR), Decision Support Systems (DSS), International Journal of Technology Assessment in Health Care, BMC Medical Informatics and Decision Making, Medical Decision Making (MDM) and Operations Research for Health Care. Reference lists from identified articles were scanned to find additional studies not identified by the electronic searches. Authors and experts in the field were contacted for help in identifying relevant studies. The literature search generated eight bibliographic references. Retrieved articles were reviewed by the first author (G.A) to identify both applied area and MCDA methods used. A study by Miot et al.,47 applied the Evidence and Value: Impact on Decision-Making (EVIDEM) framework to field test an MCDA framework as a support for coverage decision making on a cervical cancer screening test liquid-based cytology (LBC). The decision was to be made by a private health plan in South Africa. The study design included two major steps: (i) health technology assessment (HTA) report and (ii) field testing by a committee composed of medical doctors (specialists and general practitioners), pharmacists and nurses. During the first step, a literature review was conducted, and committee input was sought to

ª 2014 John Wiley & Sons Ltd Health Expectations

Multicriteria decision analysis, G Adunlin et al.

develop an HTA report on LBC. The HTA investigated 14 MCDA decision criteria of the framework organized in a matrix. During workshop sessions, committee members assigned: (i) weights to each criterion of the MCDA matrix to express their perspectives, (ii) scores for LBC for each criterion of the MCDA matrix based on data from the by-criterion HTA report and (iii) the qualitative impact of system-related criteria on the appraisal. In this study, the HTA report was entered into an interactive web prototype (Tikiwiki v2.2), while weights, scores and impact obtained from committee members were entered into Excel software. Post-testing survey on the adoptability and utility of the EVIDEM approach indicated that the committee felt the framework brought greater clarity to the decision-making process and was easily adaptable to different types of health interventions. Cunich and colleagues developed a grounded MCDA web-based decision support template, Annalisa© (AL).48 Annalisa was pilot tested as a decision support tool for prostate cancer screening (ALProst) in Australia. Being grounded in MCDA, AL ‘enables widely varying balances between intuition and analysis, rigour and relevance, and complexity and practicality’.48 The decision attributes specifically the benefit and potential harms of prostate cancer screening were included into the decision support tools. Attributes were identified by means of a literature review. Face-to-face interviews were conducted with a sample of primary care physicians referred to as general practitioners (GPs) in the study. During the course of the interviews, GPs nominated past decisions, stated prior preferences on PSA, watched an introductory video of ALProst, had hands-on experience with ALProst and rated statements on the template, AL and ALProst. Attributes weights were elicited during the interview. The authors believe that AL can be useful at all levels of the health care system. Vidal et al.,49 used the analytic hierarchy process (AHP) ‘to develop a decision support tool in order to assist pharmacists choosing the

ª 2014 John Wiley & Sons Ltd Health Expectations

anticancer drugs that can be produced in advance’.49 The setting of this study was the pharmacy department of a French hospital. Criteria and subcriteria for the AHP framework were identified by means of expert interviews and literature review. The authors noted the advantage of the AHP framework and suggested opportunities for future research. Liberatore et al.,50 discussed the development and implementation of an AHP-based decision-counselling protocol for prostate cancer screening. The decision-counselling protocol was designed by a multidisciplinary research team. The protocol was tested in four primary care practices in Philadelphia. The protocol was tested to assist men in deciding whether they would undergo both a digital rectal examination (DRE) and prostate-specific antigen (PSA) testing. The study indicated that the successful application of the decision-counselling protocol is appropriate within primary care setting only if it is well structured and coordinated by an experienced analyst. Richman et al.,51 proposed ‘a strategic, computer-based, prostate cancer decision-making model based on the AHP’.51 A group of physician expert and a group of patients were sought to validate the model. The validation process consisted of comparing and ranking management choices of prostate cancer. A particular characteristic of this study is the analysis on quality of life issues. The model was found to be a good fit for weighting priorities while being superior to traditional models. Dolan et al.,52 conducted a pilot test of a decision aid designed to help patients choose among recommended colorectal cancer screening programs at two internal medicine practices in Rochester, New York. The study population consisted of patients at average risk of colon cancer being seen for routine appointments. The study was designed as a randomized controlled trial comparing a patient decision aid based on multicriteria decision-making theory with a simple educational intervention. The study made use of the AHP defined by the authors as ‘a multicriteria decision-making method that was specifically

5

6 Multicriteria decision analysis, G Adunlin et al.

designed for decisions that require integration of quantitative data with less tangible, qualitative considerations such as values and preferences’.52 All ‘judgements’ were entered and tested by the collaboration decision software expert choice, so the total weight could be calculated. The study found that the decision support tool improved patient decision-making process. However, the decision support tool was found not to have an effect on decision implementation. Carter et al.,53 compared the analytic network process (ANP), the AHP and the Markov process all together in the evaluation of the optimal post-lumpectomy treatment strategy. The treatment alternatives considered in the study were as follows: observation, radiation, tamoxifen, radiation and tamoxifen combination, and simple mastectomy. The patient was a 74-year-old woman with a mammographically detected, non-palpable early-stage breast cancer. Combined radiation and tamoxifen was found to be the preferred treatment across all three methods. Of the three methods compared in the study, the AHP was the quickest to generate results. The Markov process and the two MCDA approaches (AHP, ANP) used in the study performed well. The study shows that the choice of a particular method depends on the context, as well as on the requirement set by the type analysis to be conducted. Dolan54 determined whether patients are capable and willing to use the AHP to help make clinical decisions. The author used 20 volunteers to perform an AHP analysis of the choice among five screening regimens for colon cancer. Dolan hypothesized that ‘a substantial proportion of patients, arbitrarily defined as 50%, would be capable of using and willing to use the AHP to help make cancer screening decisions’.54 The analysis was conducted using the standard AHP software package Expert Choice. The results of the study suggest that a lot of patients were competent to use AHP to assess difficult clinical problem. Overall, Dolan showed that the AHP can assist in health care decision making.

In this literature review, six47,48,50,52–54 of the eight studies focused on decision making for cancer screening. Cancer screening decisions can be complex, and decision aids have been developed to assist patients and providers in these decisions. The debate over cancer screening is interconnected to the issues of overdiagnosis and overtreatment. These issues are part of an even larger concern over delivery of unnecessary medical care. Four studies49,50,52,54 demonstrated applicability and acceptability of the AHP technique for patients within the health care setting. The AHP is one to the widely used MCDA tool in health care because of its extensive applicability and user-friendly characteristic. Several applications of AHP have been published in different areas of health care such as performance assessment,55 discharge planning,56 performance measurement,57 resource planning,58,59 shared decision making60 and many more. Comprehensive reviews of the application of AHP both in health care and other fields are covered elsewhere.61-66 Research opportunity In recent years, there has been increasing attention paid to the variations in the process of care for cancer patients.67–70 These variations have been attributed to a wide range of characteristics including: patient, provider and treatment facility type. To illustrate the use of MCDA in oncology, the examples selected in this paper focus on significant unanswered questions or issues where a consensus has not been reached among experts. Prostate cancer Prostate cancer is the most commonly diagnosed cancer after skin cancer and is the second leading cause of cancer death among men in the United States (U.S.).71 Recent changes to prostate cancer screening guidelines have generated controversies. Indeed, the American Cancer Society (ACS), American Urological Association (AUA) and the U.S. Preventive Services Task Force (USPSTF) have released guidelines over the years recommending against routine PSA

ª 2014 John Wiley & Sons Ltd Health Expectations

Multicriteria decision analysis, G Adunlin et al.

screening in men over age 70 or any man with less than a 10- to 15-year life expectancy. The USPSTF 2008 recommendation against screening men 75 years of age and older for prostate cancer, resulted in only an 8% decline in annual PSA testing rates over 2 years.72,73 Thus, performing PSA tests is still a common practice among the vast majority of men. There has been a call for informed decision making to help guide patient/providers decision regarding the use of PSA.74,75 Patients will often leave the screening decision up to their providers. However, providers’ decisions may be impacted by their own beliefs about screening. In the light of the controversies and complexities associated with prostate screening, a rigorous decision aid is needed to assist patients and providers in more optimal decision making. Prostate cancer screening decision entails taking into account multiple criteria simultaneously. The decision-making process also includes the need to elicit, clarify and integrate patients and/or providers values and preferences all into a single decision-making model. MCDA techniques are fit to deal with these requirements in a structured mode. Breast cancer Tamoxifen has been used for more than 30 years, and aromatase inhibitors (AIs) have been used since the early 2000s as adjuvant endocrine therapy in the treatment and prevention of primary breast cancer. AIs were found to reduce the risk of breast cancer recurrence when compared to tamoxifen in several large international randomized trials.76,77 Studies comparing AIs and tamoxifen during a 5-year period among post-menopausal women diagnosed with early-stage hormone receptor positive breast cancer used disease-free survival (DFS) as an efficacy outcome in lieu of overall survival (OS). Subsequent studies reported a significant improvement in DFS with AIs, relative to tamoxifen; however, this has not yet translated into significant improvements in OS.78,79 Both treatments have different side-effect profiles. AIs are generally considered by experts to have a more favourable side-effect profile than tamoxi-

ª 2014 John Wiley & Sons Ltd Health Expectations

fen, as the latter carries an approximate 1–2% annual risk of thromboembolic and cerebrovascular events (DVT and CVA), and an approximate 0.6% 10 year risk of endometrial cancer.80,81 AIs have a distinct toxicity profile to tamoxifen, which includes a higher risk of osteoporosis leading to fractures (approximately 3%), and myalgias/arthralgias (approximately 6%).80 Accurate benefit–risk assessment is of great importance as it can grant the opportunity for proactive interventions.82 Specifically, evaluating potential benefits and risks can help improve patient health outcomes and subsequently lead to decreasing the financial burden in oncology. MCDA is well suited to address issues related to benefit–risk trade-offs between treatment options. Benefit–risk model based on MCDA has been successfully proposed in other settings.83–86 Felli et al.,87 advocate for a multicriteria decision analytic approach to benefit– risk assessment and the use of an additive model. There are many applications of MCDA in several areas of health care, including eliciting preference in health technology assessment (HTA);88–90 evaluating treatment options;5 implementing formulary listing;91assisting in disease diagnosis process;92,93 managing disease;94 setting priorities in health policy;95–99 and planning for public health.100 MCDA can contribute to several areas of oncology, with the key areas being measurement of health outcomes, elicitation of preferences, quantification of trade-offs and inform policy analysis. To illustrate the potential benefit of applying MCDA methods to oncology, a case study is presented in the next section. Hypothetical scenario Imagine that the MDT from a cancer centre intends to select the best treatment among three potential alternatives (ai). The criteria (C) described below are those against which the decision has to be made. 1. Overall survival (C1): C1 is measured on a quantitative scale (time in months) and is expected to be maximized;

7

8

Multicriteria decision analysis, G Adunlin et al.

2. Safety profile (C2): C2 is measured on a qualitative scale (Low, Mild, Moderate, High) and is expected to be minimized; 3. Compliance to treatment (C3): C3 is measured on a qualitative scale (Low, Fair, and High) and is expected to be maximized. This criterion applies only to oral drugs. 4. Cost (C4): C4 is measured in dollars ($) on a quantitative scale. It is also expected to minimize this criterion. The MDT has compiled information on the criteria that allowed for the construction of the following performance matrix (see Table 1). There are many MCDA methods that can address choice problems. However, compensatory MCDA methods such as weighted sum models are not suitable for oncology decisionmaking problems. In fact, an MDT would be interested in treatments that are both effective (efficacy criterion) with and acceptable safety profile (safety criterion). A compensatory effect would arise if an alternative that exhibits a very good performance on efficacy and a poor performance on safety is preferred to another that has a good performance on efficacy criterion and a fair performance on safety, on the basis of a weighted sum score. Under these circumstances, it is worthwhile considering non-compensatory methods such as outranking models. The illustration of the application of outranking models is carried out using the ELECTRE I model. ELECTRE I belongs to a family of methods (outranking models) that has been developed as an alternative to compensatory models. This type of models uses outranking relations to compare different alternatives pairwise. The application of ELECTRE I involves two steps.

The first step consists in constructing a crisp (yes or no) outranking relation noted ‘S’, meaning ‘at least as good as’. As part of the comparison of a pair of alternatives (a,b), the following situation may arise: 1. a is at least as good as b, but the inverse is not true: a is therefore strictly preferred to b (commonly noted a S b) 2. b is at least as good as a, but the inverse is not true: b is therefore strictly preferred to a (commonly noted b S a) 3. a is at least as good as b, and the inverse is true: a is therefore indifferent to b (commonly noted a I b) 4. a is not at least as good as b, and the inverse is true: a is therefore incomparable to b (commonly noted a R b). For each pair of alternatives (a, b), the following questions need to be addressed to establish the outranking relations. 1. Does ax S ay? 2. Does ay S ax? Given (a, b), an alternative a is said to outrank another b if the following conditions are both true:101 1. Concordance condition: a outperforms b on enough criteria of sufficient importance, corresponding to the sum of the criteria weights (voting powers) kj. 2. Veto condition: a is not outperformed by b in the sense of recording a significantly inferior performance on any one criterion. In other words, a veto represents a maximum difference, in terms of performance of alternatives against a criterion that cannot be compensated.39

Table 1 Performance matrix*

Alternative

Overall survival (Months) C1 (max)

Safety profile (Low-High) C2 (min)

Compliance‡ (Low-High) C3 (max)

Cost ($) C4 (min)

a1 a2 a3

22 18 25

Low High Mild

High Fair Low

3189 2989 2727

*All data listed in Table 1 are only indicative; ai: i-th alternative; Cj: j-th criterion; $: dollars; min: To minimize; max: To maximize; Months: Time from randomization until death from any cause; Low–High: Low to high; ‡Compliance applies only to oral drugs.

ª 2014 John Wiley & Sons Ltd Health Expectations

Multicriteria decision analysis, G Adunlin et al.

The second step in the application of ELECTRE I deals with exploiting the outranking relations to find the kernel, meaning the set of non-outranked alternatives. This can be achieved based on the algorithm presented in Fig. 2. A detailed application of ELECTRE I is presented in the next paragraphs. To apply ELECTRE I, concordance index (CI), voting powers (kj) and veto values (vj) have to be determined by the decision-makers. For the purpose of this hypothetical scenario, let us consider that the MDT was able to set the values, respectively, for CI, kj and vj. These values are included in the modified performance matrix as shown in Table 2. Step 1. Constructing outranking relationships between competing alternatives Constructing outranking relationships between competing alternatives implies the pairwise

comparison of alternatives. These comparisons are made below. Does a1 outrank a2? C(a1,a2) = 0.35 + 0.15 = 0.5 < CI and no veto condition. Thus, a1 does not outrank a2 and is noted [~(a1 S a2]. Does a2 outrank a1? C(a2,a1) = 0.30 + 0.20 = 0.5 ≤ C and no veto condition applies. Thus, a1 does not outrank a2 and is noted [~(a2 S a1]. Does a1 outrank a3? C(a1,a3) = 0.15 < CI and no veto condition. Thus, a1 does not outrank a3 and is noted [~ (a1 S a3]. Does a3 outrank a1? C(a3,a1) = 0.35 + 0.30 + 0.20 = 0.85 > CI and no veto condition applies. Thus, a3 outranks a1 and is noted a3 S a1.

1

Reduce all cycles* to indifference classes

2

Place in the kernel (K) all actions that are not outranked

3

Remove all actions outranked by actions placed in K

4

If there are actions outside of K, reiterate from 1, otherwise

5

Use of other procedures in case of ex aequo ranking (Copeland’s score)

Figure 2 Algorithm for exploiting outranking relations to find the kernel. * A cycle contains alternatives that are considered indifferent.

ª 2014 John Wiley & Sons Ltd Health Expectations

9

10 Multicriteria decision analysis, G Adunlin et al. Table 2 Performance matrix* for ELECTRE I

CI= 0.55 Alternative

Overall survival (%) C1 (max)

Safety profile (Low-High) C2 (min)

Compliance‡ (Low-High) C3 (max)

Cost ($) C4 (min)

a1 a2 a3 kj vj

22 18 25 0.35 5

Low High Mild 0.30 –

High Fair Low 0.15 –

3189 2989 2727 0.20 500

*All data listed in Table 2 are only indicative; ai: i-th alternative; Cj: j-th criterion; $: dollars; min: To minimize; max: To maximize; %: Percentage; Low–High: Low to high; ‡Compliance applies only to oral drugs; CI: Concordance threshold; kj: Weight assigned to the criterion; vj: Veto. Bold values only represent the newly added values.

Does a2 outrank a3? C(a2,a3) = 0.30 + 0.15 = 0.45 < CI and the veto condition applies to criterion C1. Thus, a2 does not outrank a3 and is noted [~(a2 S a3]. Does a3 outrank a2? C(a3,a2) = 0.35 + 0.20 = 0.55 = CI and no veto condition applies. Thus, a3 outranks a2 and is noted a3 S a2. Step 2. Finding the kernel, the set of best alternatives The application of the algorithm presented in Fig. 2 results in the Fig. 3 shown below. Based on the ELECTRE I analysis, the alternative a3 should be selected by the MDT. Additionally, the committee can conduct sensitivity analyses to ensure that the findings are robust. This analysis would consist in testing the impact of changing the parameters of the ELECTRE I model (e.g. the concordance threshold, the voting powers and the vetos) on the results.

Discussion and conclusion This paper reviewed current practices and applications of MCDA in oncology. The paper also a1

touched on a number of important design and analysis issues for future research. MCDA is gaining importance in health care as a tool for decision making. MCDA methods appear to be well suited to elicit preferences to support existing clinical evidence such as clinical trials and guidelines. The care that patients receive does not always align with their preferences. In response to this concern, the U.S. Patient Protection and Affordable Care Act of 2010 (AKA ObamaCare, PPACA or just ACA)102 contains provisions to promote and utilize the shared decision-making (SDM) process. According to the Informed Medical Decisions Foundation, ‘SDM is a collaborative process that allows patients and their providers to make health care decisions together, taking into account the best scientific evidence available, as well as the patient’s values and preferences’.103 MCDA decision process can bring experts and patients advocate groups to the same table. In doing so, MCDA may reduce conflicts that are inherent to clinical decision making, improve knowledge and achieve outcomes that are in line with the values of the decision-makers. There is no single best approach or method in applying MCDA. Nonetheless, the decision-making context may help identify the appropriate method. It is also

a1 a2

a2 a3

a3

a3

Figure 3 Kernel obtained from the comparison of three hypothetical cancer treatments.

ª 2014 John Wiley & Sons Ltd Health Expectations

Multicriteria decision analysis, G Adunlin et al.

imperative to link the research question to issues that warrant multicriteria decisions. Cancer is a major health challenge and will continue to be in years to come. Hence, research in the field of oncology is mounting in absolute numbers. As the understanding of the complexity of cancer increases, the development of novel drug and technology is equally increasing. Some of these drugs and technologies, although effective, may negatively impact patients’ quality of life. This effect has led the consideration of evidence-based medicine in cancer care. MCDA can be a useful tool for decision making in oncology. It is designed to allow users to apply evidence-based medicine to make informed decisions in multifaceted and complex situations. However, methodological application is desirable before MCDA becomes an established decision-making tool in oncology. In this regard, it would be worthwhile exploring the development of a comprehensive MCDA model to assist MDTs in their decision making and then apply it to a real-world practice setting.

Acknowledgement This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors. Georges Adunlin would like to thank the Florida A&M University CRTCS for providing training and education on cancer research.

Conflict of interests No conflict of interests have been declared.

References 1 Petitti DB. Meta-Analysis, Decision Analysis, and Cost-Effectiveness Analysis: Methods for Quantitative Synthesis in Medicine. New York, NY: Oxford University Press, 1999. 2 Howard RA. Decision analysis: practice and promise. Management Science, 1988; 34: 679–695. 3 Pitz GF, Sachs NJ. Judgment and decision: theory and application. Annual Review of Psychology, 1984; 35: 139–164.

ª 2014 John Wiley & Sons Ltd Health Expectations

4 Katz SJ, Morrow M. The challenge of individualizing treatments for patients with breast cancer. JAMA: The Journal of the American Medical Association, 2012; 307: 1379–1380. 5 Dancey JE, Bedard PL, Onetto N, Hudson TJ. The genetic basis for cancer treatment decisions. Cell, 2012; 148: 409–420. 6 Blinman P, King M, Norman R, Viney R, Stockler MR. Preferences for cancer treatments: an overview of methods and applications in oncology. Annals of Oncology, 2012; 23: 1104–1110. 7 Bates DW, Kuperman GJ, Wang S et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. Journal of the American Medical Informatics Association, 2003; 10: 523–530. 8 Marewski JN, Gigerenzer G. Heuristic decision making in medicine. Dialogues in Clinical Neuroscience, 2012; 14: 77. 9 Sniderman AD, Furberg CD. Why guideline-making requires reform. JAMA: The Journal of the American Medical Association, 2009; 301: 429–431. 10 Haslam C, Alexander Haslam S, Knight C, Gleibs I, Ysseldyk R, McCloskey L. We can work it out: group decision making builds social identity and enhances the cognitive performance of care residents. British Journal of Psychology, 2012; 105: 17–34. 11 McKinlay E, Gray B, Pullon S. Interdisciplinary Collaboration: Working in Teams for Patient Care. Published by the Medical Council of New Zealand 2013:142. 12 Jalil R, Ahmed M, Green JS, Sevdalis N. Factors that can make an impact on decision-making and decision implementation in cancer multidisciplinary teams: an interview study of the provider perspective. International Journal of Surgery, 2013; 11: 389–394. 13 Lamb B, Sevdalis N, Taylor C, Vincent C, Green J. Multidisciplinary team working across different tumour types: analysis of a national survey. Annals of Oncology, 2012; 23: 1293–1300. 14 Sidhom MA, Poulsen MG. Multidisciplinary care in oncology: medicolegal implications of group decisions. The Lancet Oncology, 2006; 7: 951–954. 15 Reisman Y. Computer-based clinical decision aids. A review of methods and assessment of systems. Informatics for Health and Social Care, 1996; 21: 179–197. 16 Payne TH. Computer decision support systems. CHEST Journal, 2000; 118(Suppl. 2): 47S–52S. 17 Stacey D, Bennett CL, Barry MJ et al. Decision aids for people facing health treatment or screening decisions. Cochrane Database Systematic Review, 2011; 10(10).

11

12

Multicriteria decision analysis, G Adunlin et al. 18 Stacey D, Samant R, Bennett C. Decision making in oncology: a review of patient decision aids to support patient participation. CA: A Cancer Journal for Clinicians, 2008; 58: 293–304. 19 Jimbo M, Rana GK, Hawley S et al. What is lacking in current decision aids on cancer screening? CA: A Cancer Journal for Clinicians, 2013; 63: 193–214. 20 Lin GA, Aaronson DS, Knight SJ, Carroll PR, Dudley RA. Patient decision aids for prostate cancer treatment: a systematic review of the literature. CA: A Cancer Journal for Clinicians, 2009; 59: 379–390. 21 O’Brien MA, Whelan TJ, Villasis-Keever M et al. Are cancer-related decision aids effective? A systematic review and meta-analysis. Journal of Clinical Oncology, 2009; 27: 974–985. 22 Oliveira M, Fontes DB, Pereira T. Multicriteria Decision Making: A Case Study in the Automobile Industry 2013. 23 Pohekar S, Ramachandran M. Application of multi-criteria decision making to sustainable energy planning—a review. Renewable and Sustainable Energy Reviews, 2004; 8: 365–381. 24 Wang J, Jing Y, Zhang C, Zhao J. Review on multi-criteria decision analysis aid in sustainable energy decision-making. Renewable and Sustainable Energy Reviews, 2009; 13: 2263–2278. 25 Haralambopoulos D, Polatidis H. Renewable energy projects: structuring a multi-criteria group decision-making framework. Renewable Energy, 2003; 28: 961–973. 26 Beccali M, Cellura M, Mistretta M. Decision-making in energy planning. Application of the Electre method at regional level for the diffusion of renewable energy technology. Renewable Energy, 2003; 28: 2063–2087. 27 Tzeng G, Lin C, Opricovic S. Multi-criteria analysis of alternative-fuel buses for public transportation. Energy Policy, 2005; 33: 1373– 1383. 28 Bit A, Biswal M, Alam S. Fuzzy programming approach to multicriteria decision making transportation problem. Fuzzy Sets and Systems, 1992; 50: 135–141. 29 Ligmann-Zielinska A, Jankowski P. Impact of proximity-adjusted preferences on rank-order stability in geographical multicriteria decision analysis. Journal of Geographical Systems, 2012; 14: 167–187. 30 Gbanie SP, Tengbe PB, Momoh JS, Medo J, Kabba VTS. Modelling landfill location using Geographic Information Systems (GIS) and Multi-Criteria Decision Analysis (MCDA): case study Bo, Southern Sierra Leone. Applied Geography, 2012; 36: 3–12.

31 Lootsma F, Mensch T, Vos F. Multi-criteria analysis and budget reallocation in long-term research planning. European Journal of Operational Research, 1990; 47: 293–305. 32 Phillips LD, e Costa CAB. Transparent prioritisation, budgeting and resource allocation with multicriteria decision analysis and decision conferencing. Annals of Operations Research, 2007; 154 : 51–68. 33 Belton V, Stewart TJ. Multiple Criteria Decision Analysis: An Integrated Approach. Dordrecht, the Netherlands: Springer, 2002, p. 9. 34 Thokala P, Duenas A. Multiple Criteria Decision Analysis for Health Technology Assessment. Value in Health, 2012; 15: 1172–1181. 35 Hwang C, Yoon K. Multiple Attribute Decision Making Methods and Applications. London: Springer, 1981. 36 Dolan JG. Multi-Criteria clinical decision support. The Patient: Patient-Centered Outcomes Research, 2010; 3: 229–248. 37 Ehrgott M, Figueira JR, Greco S. Trends in Multiple Criteria Decision Analysis. New York: Springer, 2010. 38 Dodgson J, Spackman M, Pearman A, Phillips L. Multi-Criteria Analysis: A Manual. London: Department for Communities and Local Government, 2009. 39 Figueira J, Greco S, Ehrgott M. Multiple Criteria Decision Analysis: State of the art Surveys, Vol. 78. New York: Springer, 2005. 40 Keeney RL. Decisions With Multiple Objectives: Preferences and Value Trade-Offs. Cambridge: Cambridge University Press, 1993. 41 Doumpos M, Zopounidis C. Multicriteria Decision aid Classification Methods, Vol. 73. Dordrecht, the Netherlands: Springer, 2002. 42 Triantaphyllou E. Multi-Criteria Decision Making Methods. New York: Springer, 2000. 43 EURO. The Association of European Operational Research Societies. EURO Working Group. Multicriteria Decision Aiding (EWG-MCDA). MCDA Software. 2013; Available at: http://www. inescc.pt/  ewgmcda/Software.html, accessed 7 August 2013. 44 International Society on Multiple Criteria Decision Making. Software Related to MCDM. 2013; Available at: http://www.mcdmsociety.org/soft. html, accessed 7 August 2013. 45 Diaby V, Campbell K, Goeree R. Multi-criteria decision analysis (MCDA) in health care: a bibliometric analysis. Operations Research for Health Care, 2013; 2: 20–24. 46 Baltussen R, Youngkong S, Paolucci F, Niessen L. Multi-criteria decision analysis to prioritize health interventions: capitalizing on first experiences. Health Policy, 2010; 96: 262–264.

ª 2014 John Wiley & Sons Ltd Health Expectations

Multicriteria decision analysis, G Adunlin et al. 47 Miot J, Wagner M, Khoury H, Rindress D, Goetghebeur MM. Field testing of a multicriteria decision analysis (MCDA) framework for coverage of a screening test for cervical cancer in South Africa. Cost Effectiveness and Resource Allocation, 2012; 10: 1–12. 48 Cunich M, Salkeld G, Dowie J et al. Integrating evidence and individual preferences using a web-based multi-criteria decision analytic tool. The Patient: Patient-Centered Outcomes Research, 2011; 4: 153–162. 49 Vidal L, Sahin E, Martelli N, Berhoune M, Bonan B. Applying AHP to select drugs to be produced by anticipation in a chemotherapy compounding unit. Expert Systems With Applications, 2010; 37: 1528–1534. 50 Liberatore MJ, Myers RE, Nydick RL et al. Decision counseling for men considering prostate cancer screening. Computers & Operations Research, 2003; 30: 1421–1434. 51 Richman MB, Forman EH, Bayazit Y, Einstein DB, Resnick MI, Stovsky MD. A novel computer based expert decision making model for prostate cancer disease management. Journal of Urology, 2005; 174: 2310–2318. 52 Dolan JG, Frisina S. Randomized controlled trial of a patient decision aid for colorectal cancer screening. Medical Decision Making, 2002; 22: 125–139. 53 Carter KJ, Ritchey NP, Castro F, Caccamo LP, Kessler E, Erickson BA. Analysis of Three Decision-making Methods A Breast Cancer Patient as a Model. Medical Decision Making, 1999; 19: 49–57. 54 Dolan JG. Are patients capable of using the analytic hierarchy process and willing to use it to help make clinical decisions? Medical Decision Making, 1995; 15: 76–80. 55 Chan YL. An analytic hierarchy framework for evaluating balanced scorecards of healthcare organizations. Canadian Journal of Administrative Sciences/Revue Canadienne des Sciences de L’Administration, 2006; 23: 85–104. 56 Chang C, Cheng B, Su J. Using case-based reasoning to establish a continuing care information system of discharge planning. Expert Systems With Applications, 2004; 26: 601–613. 57 Hariharan S, Dey PK, Chen DR, Moseley HS, Kumar AY. Application of analytic hierarchy process for measuring and comparing the global performance of intensive care units. Journal of Critical Care, 2005; 20: 117–124. 58 Kwak N, McCarthy KJ, Parker GE. A human resource planning model for hospital/medical technologists: an analytic hierarchy process approach. Journal of Medical Systems, 1997; 21: 173–187.

ª 2014 John Wiley & Sons Ltd Health Expectations

59 Lee C, Kwak N. Information resource planning for a health-care system using an AHP-based goal programming method. Journal of the Operational Research Society, 1999; 50: 1191–1198. 60 Singpurwalla N, Forman E, Zalkind D. Promoting shared health care decision making using the analytic hierarchy process. Socio-Economic Planning Sciences, 1999; 33: 277–299. 61 Ho W, Xu X, Dey PK. Multi-criteria decision making approaches for supplier evaluation and selection: a literature review. European Journal of Operational Research, 2010; 202: 16–24. 62 Ho W. Integrated analytic hierarchy process and its applications–A literature review. European Journal of Operational Research, 2008; 186: 211– 228. 63 Liberatore MJ, Nydick RL. The analytic hierarchy process in medical and health care decision making: a literature review. European Journal of Operational Research, 2008; 189: 194–207. 64 Vaidya OS, Kumar S. Analytic hierarchy process: an overview of applications. European Journal of Operational Research, 2006; 169: 1–29. 65 Steuer RE, Na P. Multiple criteria decision making combined with finance: a categorized bibliographic study. European Journal of Operational Research, 2003; 150: 496–515. 66 Shim P. Bibliographical research on the analytic hierarchy process (AHP). Socio-Economic Planning Sciences, 1989; 23: 161–167. 67 Cooperberg MR, Broering JM, Carroll PR. Time trends and local variation in primary treatment of localized prostate cancer. Journal of Clinical Oncology, 2010; 28: 1117–1123. 68 Haymart MR, Banerjee M, Yang D et al. Variation in the Management of Thyroid Cancer. Journal of Clinical Endocrinology & Metabolism, 2013; 98: 2001–2008. 69 Schildmann J, Tan J, Salloch S, Vollmann J. “Well, i think there is great variation. . .”: a qualitative study of oncologists’ experiences and views regarding medical criteria and other factors relevant to treatment decisions in advanced cancer. Oncologist, 2013; 18: 90–96. 70 Wouters MW, Siesling S, Jansen-Landheer ML et al. Variation in treatment and outcome in patients with non-small cell lung cancer by region, hospital type and volume in the Netherlands. European Journal of Surgical Oncology (EJSO), 2010; 36: S83–S92. 71 American Cancer Society. Cancer Facts & Figures 2013. Atlanta, GA: American Cancer Society, 2013. 72 Howard DH, Tangka FK, Guy GP, Ekwueme DU, Lipscomb J. Prostate cancer screening in men ages 75 and older fell by 8 percentage points after

13

14 Multicriteria decision analysis, G Adunlin et al.

73

74

75

76

77

78

79

80

81

82

83

84

task force recommendation. Health Affairs, 2013; 32: 596–602. Howard DH. Declines in prostate cancer incidence after changes in screening recommendations. Archives of Internal Medicine, 2012; 172: 1267– 1268. Volk RJ, Linder SK, Kallen MA et al. Primary care physicians’ use of an informed decision-making process for prostate cancer screening. The Annals of Family Medicine, 2013; 11: 67–74. Rubel SK, Miller JW, Stephens RL et al. Testing the effects of a decision aid for prostate cancer screening. Journal of Health Communication, 2010; 15: 307–321. Forbes J, Cuzick J, Buzdar A, Howell A, Tobias J, Baum M. Effect of anastrozole and tamoxifen as adjuvant treatment for early-stage breast cancer: 100-month analysis of the ATAC trial. The Lancet Oncology, 2008; 9: 45–53. Zurrida S. Letrozole therapy alone or in sequence with tamoxifen in women with breast cancer. New England Journal of Medicine, 2009; 361: 766–776. Dowsett M, Cuzick J, Ingle J et al. Meta-analysis of breast cancer outcomes in adjuvant trials of aromatase inhibitors versus tamoxifen. Journal of Clinical Oncology, 2010; 28: 509–518. Amir E, Seruga B, Niraula S, Carlsson L, Oca~ na A. Toxicity of adjuvant endocrine therapy in postmenopausal breast cancer patients: a systematic review and meta-analysis. Journal of the National Cancer Institute, 2011; 103: 1299–1309. Howell A, Cuzick J, Baum M et al. Results of the ATAC (Arimidex, Tamoxifen, Alone or in Combination) trial after completion of 5 years’ adjuvant treatment for breast cancer. Lancet, 2005; 365: 60–62. Cuzick J, Sestak I, Baum M et al. Effect of anastrozole and tamoxifen as adjuvant treatment for early-stage breast cancer: 10-year analysis of the ATAC trial. The Lancet Oncology, 2010; 11: 1135–1141. Guo JJ, Pandey S, Doyle J, Bian B, Lis Y, Raisch DW. A review of quantitative risk–benefit methodologies for assessing drug safety and efficacy report of the ISPOR Risk-Benefit Management Working Group. Value in Health, 2010; 13: 657–666. Van Valkenhoef G, Tervonen T, Zhao J, de Brock B, Hillege HL, Postmus D. Multicriteria benefit– risk assessment using network meta-analysis. Journal of Clinical Epidemiology, 2012; 65: 394– 403. Tervonen T, van Valkenhoef G, Buskens E, Hillege HL, Postmus D. A stochastic multicriteria model for evidence-based decision making in drug

85

86

87

88

89

90

91

92

93

94

95

benefit-risk analysis. Statistics in Medicine, 2011; 30: 1419–1428. Tervonen T, Hillege H, Buskens E, Postmus D. A State-of-the-art Multi-Criteria Model for Drug Benefit-Risk Analysis. Groningen, the Netherlands: University of Groningen, Research Institute SOM, 2010. Mussen F, Salek S, Walker S. A quantitative approach to benefit-risk assessment of medicines– part 1: the development of a new model using multi-criteria decision analysis. Pharmacoepidemiology and Drug Safety, 2007; 16: S2–S15. Felli JC, Noel RA, Cavazzoni PA. A multiattribute model for evaluating the benefit-risk profiles of treatment alternatives. Medical Decision Making, 2009; 29: 104–115. Goetghebeur MM, Wagner M, Khoury H, Levitt RJ, Erickson LJ, Rindress D. Bridging Health Technology Assessment (HTA) and Efficient Health Care Decision Making with Multicriteria Decision Analysis (MCDA) applying the EVIDEM framework to medicines appraisal. Medical Decision Making, 2012; 32: 376–388. Danner M, Hummel JM, Volz F et al. Integrating patients’ views into health technology assessment: analytic hierarchy process (AHP) as a method to elicit patient preferences. International Journal of Technology Assessment in Health Care, 2011; 27: 369–375. Tony M, Wagner M, Khoury H et al. Bridging health technology assessment (HTA) with multicriteria decision analyses (MCDA): field testing of the EVIDEM framework for coverage decisions by a public payer in Canada. BMC Health Services Research, 2011; 11: 329. Diaby V, Lachaine J. An application of a proposed framework for formulary listing in low-income countries. Applied Health Economics and Health Policy 2011; 9: 389–402. Filho B, Pinheiro PR, Coelho AL. Towards the early diagnosis of Alzheimer’s Disease via a Multicriteria Classification Model. Evolutionary Multi-Criterion Optimization: Springer, 2009; 5467: 393–406. Towards an Applied Multicriteria Model to the Diagnosis of Alzheimer’s Disease: A Neuroimaging Study Case. Intelligent Computing and Intelligent Systems, 2009. ICIS 2009. IEEE International Conference on: IEEE; 2009. Singh S, Dolan JG, Centor RM. Optimal management of adults with pharyngitis–a multi-criteria decision analysis. BMC Medical Informatics and Decision Making, 2006; 6: 14. Youngkong S, Teerawattananon Y, Tantivess S, Baltussen R. Multi-criteria decision analysis for setting priorities on HIV/AIDS interventions in

ª 2014 John Wiley & Sons Ltd Health Expectations

Multicriteria decision analysis, G Adunlin et al.

96

97

98

99

Thailand. Health Research Policy and Systems, 2012; 10: 1–8. Bots PW, Hulshof JA. Designing multi-criteria decision analysis processes for priority setting in health policy. Journal of Multi-Criteria Decision Analysis, 2000; 9: 56–75. Baltussen R, Ten Asbroek A, Koolman X, Shrestha N, Bhattarai P, Niessen L. Priority setting using multiple criteria: should a lung health programme be implemented in Nepal? Health Policy Plan, 2007; 22: 178–185. Jehu-Appiah C, Baltussen R, Acquah C et al. Balancing equity and efficiency in health priorities in Ghana: the use of multicriteria decision analysis. Value in Health, 2008; 11 : 1081–1087. Nobre FF, Trotta LTF, Gomes LFAM. Multi-criteria decision making– an approach to setting priorities in health care. Statistics in Medicine, 1999; 18 : 3345–3354.

ª 2014 John Wiley & Sons Ltd Health Expectations

100 Van Gennip C, Hulshof J, Lootsma F. A multi-criteria evaluation of diseases in a study for public-health planning. European Journal of Operational Research, 1997; 99: 236–240. 101 Baltussen R, Niessen L. Priority setting of health interventions: the need for multi-criteria decision analysis. Cost Effectiveness and Resource Allocation, 2006; 4: 14. 102 U.S. Department of Health & Human Services. Affordable Care Act. 2013; Available at: http:// www.hhs.gov/opa/affordable-care-act/index.html, accessed 6 October 2013. 103 Informed Medical Decisions Foundation. What is Shared Decision Making? 2013; Available at: http://www.informedmedicaldecisions.org/ what-is-shared-decision-making/, accessed 4 October 2013.

15