Surg Endosc DOI 10.1007/s00464-014-4044-2
and Other Interventional Techniques
Incorporating patient-preference evidence into regulatory decision making Martin P. Ho • Juan Marcos Gonzalez • Herbert P. Lerner • Carolyn Y. Neuland • Joyce M. Whang • Michelle McMurry-Heath A. Brett Hauber • Telba Irony
•
Received: 5 September 2014 / Accepted: 9 November 2014 Springer Science+Business Media New York (outside the USA) 2015
Abstract Background Patients have a unique role in deciding what treatments should be available for them and regulatory agencies should take their preferences into account when making treatment approval decisions. This is the first study designed to obtain quantitative patient-preference evidence to inform regulatory approval decisions by the Food and Drug Administration Center for Devices and Radiological Health. Methods Five-hundred and forty United States adults with body mass index (BMI) C30 kg/m2 evaluated tradeoffs among effectiveness, safety, and other attributes of weight-loss devices in a scientific survey. Discrete-choice experiments were used to quantify the importance of safety, effectiveness, and other attributes of weight-loss devices to obese respondents. A tool based on these measures is being used to inform benefit-risk assessments for premarket approval of medical devices. Electronic supplementary material The online version of this article (doi:10.1007/s00464-014-4044-2) contains supplementary material, which is available to authorized users.
Results Respondent choices yielded preference scores indicating their relative value for attributes of weight-loss devices in this study. We developed a tool to estimate the minimum weight loss acceptable by a patient to receive a device with a given risk profile and the maximum mortality risk tolerable in exchange for a given weight loss. For example, to accept a device with 0.01 % mortality risk, a risk tolerant patient will require about 10 % total body weight loss lasting 5 years. Conclusions Patient preference evidence was used make regulatory decision making more patient-centered. In addition, we captured the heterogeneity of patient preferences allowing market approval of effective devices for risk tolerant patients. CDRH is using the study tool to define minimum clinical effectiveness to evaluate new weight-loss devices. The methods presented can be applied to a wide variety of medical products. This study supports the ongoing development of a guidance document on incorporating patient preferences into medical-device premarket approval decisions.
M. P. Ho H. P. Lerner C. Y. Neuland J. M. Whang M. McMurry-Heath T. Irony (&) Center for Devices and Radiological Health, U.S. Food and Drug Administration, 10903 New Hampshire Ave, Building 66, Room 2232, Silver Spring, MD 20993-0002, USA e-mail: [email protected]
M. McMurry-Heath e-mail: [email protected]
M. P. Ho e-mail: [email protected]
A. Brett Hauber e-mail: [email protected]
H. P. Lerner e-mail: [email protected]
M. McMurry-Heath FaegreBD Consulting, Washington, USA
J. M. Gonzalez A. Brett Hauber RTI Health Solutions, Durham, USA e-mail: [email protected]
C. Y. Neuland e-mail: [email protected] J. M. Whang e-mail: [email protected]
123
Surg Endosc
Keywords Patient preferences Weight-loss devices Obesity treatment FDA Benefit-risk assessment Regulatory-approval decisions In the past, the United States (US) Food and Drug Administration (FDA) did not have a scientific way to consider a representative opinion of patients when weighing the benefits and risks of a medical product to approve it to be marketed in the United States [1, 2]. Since the release of its guidance document on benefit-risk determinations in 2012, the FDA Center for Devices and Radiological Health (CDRH) has made clear that patient preference is an important factor to consider when evaluating medical devices for market approval. The guidance explicitly states that Risk tolerance will vary among patients, and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit. …FDA would consider evidence relating to patients’ perspective of what constitutes a meaningful benefit. For example, when evaluating weight-loss devices for market approval, CDRH seeks to identify appropriate levels of benefit that outweigh different levels of risk [3]. Patients have a unique role in deciding what treatments should be available for them. Their perspectives about the value of benefits and the impact of risks of medical treatments are essential because only patients live with their medical conditions and consequences of the choices they make for their own care. Moreover, their perspectives can be different from those of regulators and care providers. As the principle of patient-centered heath care has been widely accepted, regulatory authorities are very interested in obtaining input on patient perspectives on benefits and tolerance for risk for making regulatory evaluations, knowing that unless a treatment is approved, it is not an option for patients regardless of their risk tolerance [4–6]. This study is the first step in demonstrating how to implement this idea and offers a quantitative approach to bridging the gap between what CDRH reviewers and patients regard as acceptable benefit-risk tradeoffs for weight-loss devices. Although numerous efforts of regulatory agencies have been made to actively reach out and listen to the opinion of patients with various conditions, the response received has been qualitative in nature. While these contributions can be valuable for other purposes, they cannot adequately address the approval decisions because they lack measurability, representativeness, accuracy, and inclusiveness. In contrast, this study provides quantitative evidence of patient benefit-risk tradeoff preference that includes a wide spectrum of patients. It quantifies patients’
123
perspectives on benefits, providing the minimum benefit they expect from a weight-loss device to tolerate a given level of risk and other device attributes. Patient tolerance for risk is quantified as the maximum device-related mortality risk patients are willing to tolerate for a given weight loss and other device attributes. Because patients’ perspectives on benefits and risks may be very heterogeneous within a patient group, CDRH needs to consider the whole spectrum of patient preferences. Some patients in the top quartile of the risk-tolerance distribution could be ‘‘early adopters,’’ meaning they will tolerate higher risks to gain quick access to innovative treatments. CDRH will consider approving a medical device that demonstrates meaningful benefits even though its benefit-risk profile would be acceptable only to a subset of patients who are risk-tolerant. Such a device’s Indication for Use will explain that the benefit-risk profile may be suitable to only a subgroup of patients who should consult with their care providers in a context of shared medical decision making. The objectives of this study were to (1) apply wellestablished methods for quantifying the relative importance of effectiveness, safety, and other attributes of weight-loss devices to patients; (2) derive measures of patients’ preferences to inform benefit-risk assessments; and (3) incorporate such measures into regulatory decision making. This is the first patient-preference study designed by regulatory reviewers and used to inform regulatory decision making. The results are providing direction for designing clinical trials for premarket approval, informing benefit-risk assessments once the trial results are being evaluated, and guiding post-approval decisions. The conceptual framework and quantitative methods can be generalized to a wide variety of medical treatments and are particularly relevant when patients and regulators face difficult decisions when weighing potential treatment benefits against serious adverse-event risks.
Materials and methods Design overview Choice-experiment surveys ask respondents to choose the most-preferred alternative from a set of two or more constructed virtual alternatives in a series of questions. Statistical analysis of the pattern of these choices reveals the implicit relative importance of the attributes of the health intervention that influence respondents’ choices among the alternatives offered. The result is an estimate of the perceived value of an intervention as a weighted sum of the intervention attributes, where the weights reflect the mean
Surg Endosc
relative importance of each factor. In this study we call these weights ‘‘preference scores.’’ Setting and participants A Web-enabled, cross-sectional survey was administered between September and November 2012 to a stratified sample of English-speaking US residents with a current or previous BMI of at least 30 kg/m2 who were willing to lose weight. The sample was drawn from the online GfK KnowledgePanel, divided equally among three groups: 30 B BMI \ 35, 35 B BMI \ 40, and 40 B BMI. Respondents who did not meet the BMI inclusion criteria were screened out. Respondents who did not answer any question or who always picked the same alternative were excluded from the sample. Because we compared the available choices with a no-device alternative, data that did not include an answer to the follow-up device-acceptance question or for which the answer was ‘‘Don’t know/not sure’’ were excluded. The consent form, survey instrument, and data-collection protocol were approved by the RTI International Office of Research Protection and Ethics and the Office of Information and Regulatory Affairs in the US Office of Management and Budget. Survey instrument Survey content was developed based on published clinical research on obesity, regulatory knowledge of weight-loss device technology, and interviews with obese individuals. Eight weight-loss device attributes were selected based on knowledge about devices that are likely to be reviewed by CDRH in the near future. The attributes were of primary concern to physicians, regulators, and obese subjects consulted during survey development: (1) total body weight loss (TBWL) as percentage of current weight, (2) duration of weight loss, (3) duration of mild-to-moderate side effects, (4) mortality risk, (5) chance of a side effect requiring hospitalization, (6) recommended dietary restrictions, (7) reduction in chance of comorbidity or reduction in prescription dosage for existing comorbidity, and (8) type of surgery. Each attribute had three to five levels. The levels of the continuous attributes covered a wide range of values to allow for interpolation of estimates, whereas the levels of the categorical attributes included several plausible categories. Clinical-outcome attributes were described as average results for each device profile. Respondents were asked to assume that all costs were covered by insurance. All the device attributes and levels used in the study are listed in Appendix A in Supplementary material. The study design included several internal-validity tests, with particular attention to verifying that respondents
understood basic risk concepts. Before answering the choice questions, respondents received a risk tutorial and answered a question to verify their understanding. The choice questions employed a standard format for choice-experiment surveys [7–9]. Each respondent evaluated eight pairs of virtual weight-loss device profiles and were asked ‘‘Which weight-loss device do you think is better for people like you?’’ Figure 1 is an example choice question. Respondents were asked in a separate follow-up question whether they personally would accept the better device if it were available or they would prefer no device. Device profiles in the choice questions varied according to a predetermined experimental design with known statistical properties. An experimental-design algorithm yielded 120 choice questions [10–12]. Good-practice guidelines recommend 8–12 questions per respondent to limit measurement error due to fatigue. Because of the complexity of a choice task that includes a probabilistic outcome, the design was divided into 15 survey versions of 8 questions each, and respondents were randomized to one of the versions. More information on the final experimental design is included in Appendix B in Supplementary material. A draft of the survey instrument was pretested in 90-min, face-to-face, semistructured interviews with nine obese subjects in accordance with US Office of Management and Budget regulations. Following final ethics and regulatory review, the Web-enabled instrument was further field-tested by drawing 86 respondents from the GfK Web panel before its full release.
Results Data quality Of 1,057 panel members randomly drawn from GfK’s KnowledgePanel, a large Web panel that matches the demographics of the general US population, 710 responded to the invitation and 568 qualified for the survey: 1.2 % did not answer any tradeoff question, 1.9 % always picked the same alternative, and 14.3 % did not answer the follow-up device-acceptance question. After these exclusions, 540 respondents were available for the final analysis, yielding a qualification rate of 80.0 % and a final-stage completion rate of 67.2 %. Additional details on response rates are provided in Appendix C in Supplementary material. Results of internal validity tests indicated the data satisfied high quality standards. For example, only 34 respondents (6.3 %) failed a quiz question following the risk tutorial. Additional details on internal validity are provided in Appendices D and E in Supplementary material.
123
Surg Endosc Fig. 1 Example tradeoff question
The demographic characteristics of respondents who were included and excluded in the analysis were similar. Although the sample was stratified by BMI, the analysis sample on average was about 2 years older and about 8.5 %
123
heavier than the general US obese population. Additional details on comparisons among included respondents, excluded respondents, and the general obese population are provided in Appendix F in Supplementary material.
Surg Endosc
Fig. 2 Relative-importance estimates for attributes of weight-loss devices preference parameters were rescaled relative to 10, corresponding to the absolute value of the most important outcome—5 % mortality risk. Attributes with connecting lines were modelled as nonlinear Box–Cox transformed continuous variables. Other attributes were modelled as categorical variables. Vertical bars around preference scores denote 95 % confidence intervals. Benefits have positive scores and harms have negative scores. Differences between
the best and worst level in each attribute indicate relative attribute importance over the level ranges in the study design. Attributes are sorted from most important benefit to most important harm. Mortality risk is the most important outcome or feature, followed by weight loss, weight-loss duration, and side-effect duration. The relative importance of a 1 % increase in risk is -3.5, in comparison with the relative importance of a 30 % total body weight loss of ?4.3
Preference-score estimates
Table 1 shows the estimated preference scores for the attributes and their levels, indicating their relative importance. For example, the importance of an increase in mortality risk from 0 to 1 % can be quantified by the preference-score difference between these two levels; it was about -3.5 (6.5 - 10). In comparison, the importance of a 30 % TBWL was about 4.3 (4.3 - 0). Therefore, if other device attributes are kept the same, 30 % TBWL more than compensates for having to bear a 1 % risk of death from the device.
The analysis provides estimates of the preference scores that best explain the pattern of observed choices in the data. Additional details on parameter estimates are provided in Appendix G in Supplementary material. Figure 2 shows the preference-score estimates for all attribute levels. They are on a scale from -10 (least preferred), to 10 (most preferred), where -10 is the perceived value of 5 % mortality risk. The difference between the best and worst preference scores for each attribute indicates how influential that attribute is in explaining device choices. Mortality risk was the most important attribute, followed by weight loss, weight-loss duration, and side-effect duration. However, no side effect requiring hospitalization and a 5 % risk of a side effect requiring hospitalization with no surgery were considered much less important.
Benefit-risk tradeoff comparisons In this study, we used the estimated preference scores to calculate the minimum acceptable benefit (MinB), i.e., minimum weight loss respondents expect from a device to tolerate a specific level of risk and other device attributes.
123
Surg Endosc Table 1 Estimates of preference scores by attributes and levels Attribute Average amount of weight loss (TBWL)
Weight loss duration
Side effect duration
Chance of side effects requiring hospitalization
Dietary restrictions
Average reduction in dose of prescription drugs for comorbiditya
Type of operation
Chance of dying from getting the device
Level
Preference score (SE)
30 %
?4.3 (0.52)
20 %
?2.0 (0.11)
10 %
?0.6 (0.15)
5%
?0.2 (0.23)
0%
Reference level
60 months
?4.3 (0.47)
12 months
?2.0 (0.01)
6 months
?1.4 (0.1)
0 months
Reference level
0 months
Reference level
1 months 12 months
-1.0 (0.11) -2.0 (0.09)
60 months
-3.2 (0.31)
None
Reference level
5 % chance of hospitalization, no surgery
-0.2 (0.39)
20 % chance of hospitalization, no surgery
-0.5 (0.35)
5 % chance of hospitalization, with surgery
-0.6 (0.36)
Eat 1/4 cup of food at a time
Reference level
Wait 4 h between eating
-0.1 (0.29)
Can’t eat sweets or foods that are hard to digest
-2.2 (0.33)
Eliminate need/risk
?3.2 (0.37)
50 % dose/risk
?2.2 (0.29)
No change
Reference level
Laparoscopic surgery
Reference level
Endoscopic surgery
-0.5 (0.3)
Open surgery
-2.5 (0.31)
0% 1%
Reference level -3.5 (0.13)
3%
-7.1 (0.15)
5%
-10 (0.37)
Receiving a weight-loss device through open surgery versus endoscopic surgery had a preference-score difference of -2 (-2.5 minus -0.5). Maintaining weight loss for 12 months versus no months had a preference-score difference of ?2 (2 - 0). Holding everything else constant, receiving a device placed through open surgery instead of endoscopically requires at least 12 months of weight loss to compensate. The same metric can be used to measure patient tolerance for risks of adverse events TBWL total body weight loss a
Average reduction in dose of prescription drugs for the current primary comorbid condition or chance of getting the most feared comorbid condition at the lower weight
We also calculated the maximum acceptable risk (MaxR), i.e., maximum device-related mortality risk respondents are willing to tolerate for a given weight loss and other device attributes. The statistical analysis accounted for heterogeneity in patients’ preferences and allowed quantitative segmentation of obese respondents according to their risk tolerance. The MinB and MaxR estimates in the middle 50 % and the upper 25 % of the sample indicate the quantitative benefitrisk tradeoff preferences of average patients and risk-tolerant patients, respectively.
123
Table 2 compares the overall acceptability of three devices relative to the no-device option indicated by MinB and MaxR for the gastric-band and for Devices A and B, which are worse and better than the gastric band, respectively. Preference-score estimates predict that only 5.0 % of respondents would judge Device A better than the nodevice alternative, while 27.5 % of respondents would judge Device B better than the no-device alternative. Devices with less attractive attributes require either larger weight loss or smaller mortality risks in compensation. For example, the gastric band offers only 13 % TBWL, but the
Surg Endosc Table 2 Preference relative to a no-device alternative, minimum acceptable benefit, and maximum acceptable risk of a 243 pound respondent for three weight-loss device profiles Weight loss device profile
Virtual device A
% Respondents who judged better than a no-device alternative 5.0 %
Risk Tolerance
Minimum acceptable TBWLa
Maximum acceptable mortality riskb
Middle 50 %
[30 %c
0.1 % (0.04–0.26)
Upper 25 %
15.1 % (11.63–19.05)
3.8 % (1.85–7.77)
Middle 50 %
[30 %c
0.16 % (0.07–0.38)
Upper 25 %
13.0 % (9.76–16.90)
7.1 % (3.82–13.50)
Middle 50 %
[30 %c
0.6 % (0.29–1.12)
Upper 25 %
12.8 % (9.85–16.45)
10.9 % (6.54–18.78)
Benefit: 5 % TBWL Risk: 1 % chance of death Type of surgery: laparoscopic surgery Dietary restriction: eat 1/4 cup of food at a time Weight-loss duration: 36 months Minor side-effect duration: 36 months Chance of hospitalization with surgery: 5 % Comorbidity: no improvement Gastric band Benefit: 13 % TBWL
11.6 %
Risk: 1 % chance of death Type of surgery: laparoscopic surgery Dietary restriction: eat 1/4 cup of food at a time Weight-loss duration: 5 years Minor side-effect duration: 5 years Chance of hospitalization for side effects requiring surgery: 5% Comorbidity: no improvement Virtual device B
27.5 %
Benefit: 20 % TBWL Risk: 1 % chance of death Type of surgery: endoscopic surgery Dietary restriction: eat 1/4 cup of food at a time Weight-loss duration: 60 months Minor side-effect duration: 12 months Chance of hospitalization but no surgery: 5 % Comorbidity: 50 % improvement 95 % confidence intervals in parentheses TBWL total body weight loss a
For indicated mortality risk and other device attributes
b
For indicated weight-loss benefit and other device attributes
c
Exceeds upper limit of weight-loss levels included in the study design
middle 50 % of respondents require a mean TBWL of more than 30 % given the attributes of the gastric band. Similarly, the gastric band has a 1 % mortality risk while the middle 50 % of respondents would tolerate only a 0.16 % risk for the attributes of the gastric band. On the other hand, for risk-tolerant early adopters, the mean minimum acceptable TBWL is only 13.0 % and the mean maximum acceptable risk is 7.1 %, which compare favorably to the gastric band’s profile. Overall, estimates imply that only 11.6 % of respondents would judge the gastric band better than no device, which is consistent with the fact
that only a small percentage of obese patients have chosen gastric-band surgery [13, 14]. The tool One of the main issues faced by regulatory reviewers when designing clinical studies and analyzing their results is determination of the ‘‘minimum clinical effectiveness’’ that is sufficient to offset the risks and inconveniences posed by the treatments under review. This value is used to size the clinical studies and, when results are available, to decide
123
Surg Endosc
whether the benefits of the treatment outweigh the risks for market approval. There is an enormous amount of discussion among device makers, regulators and experts about this value. To include patient preference input in these discussions, we use the preference-score estimates to build a MaxR– MinB calculator for weight-loss devices (‘‘the tool’’). The MinB obtained from the tool is used to inform CDRH reviewers in establishing the minimum clinical effectiveness (weight loss) of devices under review to size clinical studies. In addition, once the results of the study are available, reviewers use the minimum clinical effectiveness as a reference to evaluate the benefit of a device that is being considered for premarket approval. Given the profile of the weight-loss device, the tool can answer three different but related questions: 1. 2. 3.
What percentage of subjects would prefer getting the device over a no-device alternative? What is the MaxR of an average subject or of an early adopter for a device that provides a given weight loss? What is the MinB of an average subject or of an early adopter for a device that poses a given mortality risk?
Development of the tool went through several evaluations and iterations with the regulatory reviewers to make it usable, understandable, and relevant to the review work. Examples of the tool’s appearance, input fields, and function are shown in Fig. 3.
Discussion This is the first study designed to provide regulators with quantitative data on patient preferences in support of regulatory benefit-risk tradeoff determinations. Previous information on patient preferences largely has consisted of qualitative, anecdotal testimony obtained in advisorycommittee proceedings and public meetings. In contrast, this study provides quantitative data obtained in a controlled experiment from a cross-sectional sample of US obese respondents. FDA regulators and the investigators jointly developed the survey instrument to ensure the study design provided relevant tradeoff information for devices likely to be reviewed by the agency in the foreseeable future. Moreover, these attributes were important to obese subjects, and their levels are clinically meaningful and relevant to regulatory decision making. Patients’ perspectives on benefits and tolerance for risks are likely to be diverse within a patient population and, as a whole, may also differ from clinicians’ perspectives. It is important that a regulatory agency takes into account variations in risk tolerance of the whole spectrum of possible users of a medical treatment because unless a
123
Fig. 3 The MaxR–MinB Tool. 3.1 Using the tool to estimate c minimum acceptable benefit (25 %TBWL). 3.2 Using the tool to estimate maximum acceptable risk (0.5 % mortality). As shown in 3.1 and 3.2, the tool provides an input area at the top where the device attributes, including mortality risk (0.05 %) for (3.1) and benefit (15 % TBWL) for (3.2) are entered. The green cells in the center display the estimated MinB (25 %) and MaxR (0.5 %) respectively. Results can be shown for average patients (middle quartile of the distribution) or for early adopters (upper quartile of the distribution) and for different base weights (Color figure online)
treatment is approved, it is not available to physicians and patients, regardless of their risk tolerance. Patients who are more tolerant of higher risks are likely to be ‘‘early adopters,’’ meaning they could be willing to accept higher risks to gain faster access to innovative treatments. Risktolerant early adopters also are likely to participate in clinical trials of a novel technology and play a critical role in postapproval evaluations by accepting the novel technology once it is on market. If only risk-tolerant patients would accept a certain device profile, the FDA might consider approving such a device only for risk-tolerant patients. Such indication for use will be explained in the device label. CDRH is using the MaxR–MinB calculator ‘‘the tool,’’ to inform reviewers in establishing the minimum clinical effectiveness of weight-loss devices under review. Similar decision-aid calculators can be developed for other medical products. For a given medical product profile, the tool can assist regulatory reviewers to determine the minimum clinical effectiveness to be used in trial design and interpretation of trial results. In addition, opinions among clinicians and patients may be mixed when evaluating a new medical product that offers an incremental improvement in effectiveness but a slightly worse safety profile than existing treatment options. Such medical products could be preferred by some but rejected by others, and the FDA should take this into account. This study may have been subject to some technical limitations. Choices among virtual weight-loss devices do not have the same clinical and emotional consequences as actual choices. However, clearly defined and validated definitions of device attributes helped increase the reliability of the study results by ensuring that respondents were well informed. The potential for hypothetical bias was further reduced by first eliciting judgments about which combination of plausible attributes was better and subsequently eliciting the stated choice between device and nodevice alternatives. The inclusion criteria relied on patient-reported height and weight, not measured BMI. Assurance of anonymity reduced incentives to understate weight and overstate height. In fact, the mean self-reported weight in the study
Surg Endosc
123
Surg Endosc
sample was greater than the national average for the same age cohort. Furthermore, we found no significant differences in preferences between respondents with BMI less than or greater than 40. Whereas the Web panel matches the demographics of the general U.S. population, our stratified sample of obese panel members does not need to match the demographics of the general obese population. Nevertheless, we found only a 2-year difference in mean age. Because we did not find significant differences in preferences between respondents by BMI, the 8.5 % difference in mean weight between the study and national sample has no effect on estimates. Despite these limitations, the estimated preference scores for treatment attributes and their levels provide a quantitative solution to the problem of comparing different outcomes in benefit-risk assessments. They also provide information for making patient-centered, evidence-based regulatory decisions. This study has demonstrated the practical feasibility of eliciting and using evidence on patient benefit-risk tradeoff preferences to inform regulatory decisions. The experience acquired from this study serves as a proof of principle to support the ongoing development of a CDRH guidance document on including evidence on patient preferences in medical-device submissions for premarket approval. We believe that the approach used here is superior to making regulatory decisions without an understanding of the patients’ values and their heterogeneity. This study is a significant step in the direction of incorporating patients’ benefit preferences and risk tolerance into the existing evidence-based regulatory process. Acknowledgments We acknowledge Jeffrey Shuren, MD, JD for his support of the study and insightful advice on the manuscript. We also acknowledge Priya Venkataraman-Rao, MD; Megan Shackelford, MS; Rebecca Nipper; Richard Kotz, MS; Kathleen Olvey; and Martin Golding, MD, for their regulatory input in the development of the survey instrument. We are grateful to the FDA CDRH Obesity Devices Working Group for their comments on interpretation of study results and their feedback on using the MinB–MaxR calculator in regulatory reviews of weight-loss device submissions. The members of that group include Jeffrey Cooper, DVM; Megan Shackelford, MS; Irene Bacalocostantis, PhD; Brandan Reid, PhD; Martha Betz, PhD; Elizabeth Katz, PhD; David Pudwill; Martin Golding, MD; Priya Venkataraman-Rao, MD; and Benjamin Fisher, PhD. The acknowledged persons above are of CDRH, FDA.
123
Disclosures Drs. Hauber and Gonzalez received compensation for their work through a contract with the Center for Devices and Radiological Health (CDRH) of the U.S. Food and Drug Administration (FDA). Ho, Lerner, Neuland, Whang, McMurry-Heath, and Irony have no conflicts of interest or financial ties to disclose.
References 1. http://www.fda.gov/downloads/MedicalDevices/DeviceR egulationandGuidance/GuidanceDocuments/UCM296379.pdf. Accessed 2 Sept 2014 2. Section 513(a) of the Federal Food, Drug, and Cosmetic Act [21 U.S.C. 360c(a)] 3. Lerner H, Whang J, Nipper R (2013) Benefit-risk paradigm for clinical trial design of obesity devices: FDA proposal. Surg Endosc 27(3):702–707 4. Shuren J, Woodcock J (2013) Reviewing FDA’s implementation of FDASIA. http://www.fda.gov/NewsEvents/Testimony/ucm374 544.htm. Accessed 2 Sept 2014 5. McMurry-Heath M (2013) FDA brings patients into the process. http://blogs.fda.gov/fdavoice/index.php/2013/09/fda-bringspatients-into-the-process/. Accessed 2 Sept 2014 6. http://www.ema.europa.eu/docs/en_GB/document_library/ Report/2013/10/WC500153276.pdf. Accessed 2 Sept 2014 7. Marshall D, Bridges JFP, Hauber B et al (2010) Conjoint analysis applications in health: how are studies being designed and reported? Patient 3(4):1178–1653 8. Bridges JFP, Hauber AB, Marshall D et al (2011) Conjoint analysis applications in health—a checklist: a report of the ISPOR good research practices for conjoint analysis task force. Value Health 14(4):403–413 9. Johnson FR, Lancsar E, Marshall D et al (2013) Constructing experimental designs for discrete-choice experiments: report of the ISPOR conjoint analysis experimental design good research practices task force. Value Health 16(1):3–13 10. Kuhfeld WF, Tobias RD, Garratt M (1994) Efficient experimental design with marketing research applications. J Mark Res 31:545–557 11. Kuhfeld WF (2010) Marketing research methods in SAS. http:// support.sas.com/techsup/technote/mr2010.pdf. Accessed 2 Sept 2014 12. Ogden CL, Carroll MD, Kit BK, Flegal KM (2012) Prevalence of obesity in the United States, 2009–2010. NCHS Data Brief No. 82 13. Nguyen NT, Masoomi H, Magno CP, Nguyen XMT, Laugenour K, Lane J (2011) Trends in use of bariatric surgery, 2003–2008. J Am Coll Surg 213(2):261–266 14. Callegaro M, DiSogra C (2008) Computing response metrics for online panels. Public Opin Q 72(5):1008–1032
and Other Interventional Techniques
Incorporating patient-preference evidence into regulatory decision making Martin P. Ho • Juan Marcos Gonzalez • Herbert P. Lerner • Carolyn Y. Neuland • Joyce M. Whang • Michelle McMurry-Heath A. Brett Hauber • Telba Irony
•
Received: 5 September 2014 / Accepted: 9 November 2014 Springer Science+Business Media New York (outside the USA) 2015
Abstract Background Patients have a unique role in deciding what treatments should be available for them and regulatory agencies should take their preferences into account when making treatment approval decisions. This is the first study designed to obtain quantitative patient-preference evidence to inform regulatory approval decisions by the Food and Drug Administration Center for Devices and Radiological Health. Methods Five-hundred and forty United States adults with body mass index (BMI) C30 kg/m2 evaluated tradeoffs among effectiveness, safety, and other attributes of weight-loss devices in a scientific survey. Discrete-choice experiments were used to quantify the importance of safety, effectiveness, and other attributes of weight-loss devices to obese respondents. A tool based on these measures is being used to inform benefit-risk assessments for premarket approval of medical devices. Electronic supplementary material The online version of this article (doi:10.1007/s00464-014-4044-2) contains supplementary material, which is available to authorized users.
Results Respondent choices yielded preference scores indicating their relative value for attributes of weight-loss devices in this study. We developed a tool to estimate the minimum weight loss acceptable by a patient to receive a device with a given risk profile and the maximum mortality risk tolerable in exchange for a given weight loss. For example, to accept a device with 0.01 % mortality risk, a risk tolerant patient will require about 10 % total body weight loss lasting 5 years. Conclusions Patient preference evidence was used make regulatory decision making more patient-centered. In addition, we captured the heterogeneity of patient preferences allowing market approval of effective devices for risk tolerant patients. CDRH is using the study tool to define minimum clinical effectiveness to evaluate new weight-loss devices. The methods presented can be applied to a wide variety of medical products. This study supports the ongoing development of a guidance document on incorporating patient preferences into medical-device premarket approval decisions.
M. P. Ho H. P. Lerner C. Y. Neuland J. M. Whang M. McMurry-Heath T. Irony (&) Center for Devices and Radiological Health, U.S. Food and Drug Administration, 10903 New Hampshire Ave, Building 66, Room 2232, Silver Spring, MD 20993-0002, USA e-mail: [email protected]
M. McMurry-Heath e-mail: [email protected]
M. P. Ho e-mail: [email protected]
A. Brett Hauber e-mail: [email protected]
H. P. Lerner e-mail: [email protected]
M. McMurry-Heath FaegreBD Consulting, Washington, USA
J. M. Gonzalez A. Brett Hauber RTI Health Solutions, Durham, USA e-mail: [email protected]
C. Y. Neuland e-mail: [email protected] J. M. Whang e-mail: [email protected]
123
Surg Endosc
Keywords Patient preferences Weight-loss devices Obesity treatment FDA Benefit-risk assessment Regulatory-approval decisions In the past, the United States (US) Food and Drug Administration (FDA) did not have a scientific way to consider a representative opinion of patients when weighing the benefits and risks of a medical product to approve it to be marketed in the United States [1, 2]. Since the release of its guidance document on benefit-risk determinations in 2012, the FDA Center for Devices and Radiological Health (CDRH) has made clear that patient preference is an important factor to consider when evaluating medical devices for market approval. The guidance explicitly states that Risk tolerance will vary among patients, and this will affect individual patient decisions as to whether the risks are acceptable in exchange for a probable benefit. …FDA would consider evidence relating to patients’ perspective of what constitutes a meaningful benefit. For example, when evaluating weight-loss devices for market approval, CDRH seeks to identify appropriate levels of benefit that outweigh different levels of risk [3]. Patients have a unique role in deciding what treatments should be available for them. Their perspectives about the value of benefits and the impact of risks of medical treatments are essential because only patients live with their medical conditions and consequences of the choices they make for their own care. Moreover, their perspectives can be different from those of regulators and care providers. As the principle of patient-centered heath care has been widely accepted, regulatory authorities are very interested in obtaining input on patient perspectives on benefits and tolerance for risk for making regulatory evaluations, knowing that unless a treatment is approved, it is not an option for patients regardless of their risk tolerance [4–6]. This study is the first step in demonstrating how to implement this idea and offers a quantitative approach to bridging the gap between what CDRH reviewers and patients regard as acceptable benefit-risk tradeoffs for weight-loss devices. Although numerous efforts of regulatory agencies have been made to actively reach out and listen to the opinion of patients with various conditions, the response received has been qualitative in nature. While these contributions can be valuable for other purposes, they cannot adequately address the approval decisions because they lack measurability, representativeness, accuracy, and inclusiveness. In contrast, this study provides quantitative evidence of patient benefit-risk tradeoff preference that includes a wide spectrum of patients. It quantifies patients’
123
perspectives on benefits, providing the minimum benefit they expect from a weight-loss device to tolerate a given level of risk and other device attributes. Patient tolerance for risk is quantified as the maximum device-related mortality risk patients are willing to tolerate for a given weight loss and other device attributes. Because patients’ perspectives on benefits and risks may be very heterogeneous within a patient group, CDRH needs to consider the whole spectrum of patient preferences. Some patients in the top quartile of the risk-tolerance distribution could be ‘‘early adopters,’’ meaning they will tolerate higher risks to gain quick access to innovative treatments. CDRH will consider approving a medical device that demonstrates meaningful benefits even though its benefit-risk profile would be acceptable only to a subset of patients who are risk-tolerant. Such a device’s Indication for Use will explain that the benefit-risk profile may be suitable to only a subgroup of patients who should consult with their care providers in a context of shared medical decision making. The objectives of this study were to (1) apply wellestablished methods for quantifying the relative importance of effectiveness, safety, and other attributes of weight-loss devices to patients; (2) derive measures of patients’ preferences to inform benefit-risk assessments; and (3) incorporate such measures into regulatory decision making. This is the first patient-preference study designed by regulatory reviewers and used to inform regulatory decision making. The results are providing direction for designing clinical trials for premarket approval, informing benefit-risk assessments once the trial results are being evaluated, and guiding post-approval decisions. The conceptual framework and quantitative methods can be generalized to a wide variety of medical treatments and are particularly relevant when patients and regulators face difficult decisions when weighing potential treatment benefits against serious adverse-event risks.
Materials and methods Design overview Choice-experiment surveys ask respondents to choose the most-preferred alternative from a set of two or more constructed virtual alternatives in a series of questions. Statistical analysis of the pattern of these choices reveals the implicit relative importance of the attributes of the health intervention that influence respondents’ choices among the alternatives offered. The result is an estimate of the perceived value of an intervention as a weighted sum of the intervention attributes, where the weights reflect the mean
Surg Endosc
relative importance of each factor. In this study we call these weights ‘‘preference scores.’’ Setting and participants A Web-enabled, cross-sectional survey was administered between September and November 2012 to a stratified sample of English-speaking US residents with a current or previous BMI of at least 30 kg/m2 who were willing to lose weight. The sample was drawn from the online GfK KnowledgePanel, divided equally among three groups: 30 B BMI \ 35, 35 B BMI \ 40, and 40 B BMI. Respondents who did not meet the BMI inclusion criteria were screened out. Respondents who did not answer any question or who always picked the same alternative were excluded from the sample. Because we compared the available choices with a no-device alternative, data that did not include an answer to the follow-up device-acceptance question or for which the answer was ‘‘Don’t know/not sure’’ were excluded. The consent form, survey instrument, and data-collection protocol were approved by the RTI International Office of Research Protection and Ethics and the Office of Information and Regulatory Affairs in the US Office of Management and Budget. Survey instrument Survey content was developed based on published clinical research on obesity, regulatory knowledge of weight-loss device technology, and interviews with obese individuals. Eight weight-loss device attributes were selected based on knowledge about devices that are likely to be reviewed by CDRH in the near future. The attributes were of primary concern to physicians, regulators, and obese subjects consulted during survey development: (1) total body weight loss (TBWL) as percentage of current weight, (2) duration of weight loss, (3) duration of mild-to-moderate side effects, (4) mortality risk, (5) chance of a side effect requiring hospitalization, (6) recommended dietary restrictions, (7) reduction in chance of comorbidity or reduction in prescription dosage for existing comorbidity, and (8) type of surgery. Each attribute had three to five levels. The levels of the continuous attributes covered a wide range of values to allow for interpolation of estimates, whereas the levels of the categorical attributes included several plausible categories. Clinical-outcome attributes were described as average results for each device profile. Respondents were asked to assume that all costs were covered by insurance. All the device attributes and levels used in the study are listed in Appendix A in Supplementary material. The study design included several internal-validity tests, with particular attention to verifying that respondents
understood basic risk concepts. Before answering the choice questions, respondents received a risk tutorial and answered a question to verify their understanding. The choice questions employed a standard format for choice-experiment surveys [7–9]. Each respondent evaluated eight pairs of virtual weight-loss device profiles and were asked ‘‘Which weight-loss device do you think is better for people like you?’’ Figure 1 is an example choice question. Respondents were asked in a separate follow-up question whether they personally would accept the better device if it were available or they would prefer no device. Device profiles in the choice questions varied according to a predetermined experimental design with known statistical properties. An experimental-design algorithm yielded 120 choice questions [10–12]. Good-practice guidelines recommend 8–12 questions per respondent to limit measurement error due to fatigue. Because of the complexity of a choice task that includes a probabilistic outcome, the design was divided into 15 survey versions of 8 questions each, and respondents were randomized to one of the versions. More information on the final experimental design is included in Appendix B in Supplementary material. A draft of the survey instrument was pretested in 90-min, face-to-face, semistructured interviews with nine obese subjects in accordance with US Office of Management and Budget regulations. Following final ethics and regulatory review, the Web-enabled instrument was further field-tested by drawing 86 respondents from the GfK Web panel before its full release.
Results Data quality Of 1,057 panel members randomly drawn from GfK’s KnowledgePanel, a large Web panel that matches the demographics of the general US population, 710 responded to the invitation and 568 qualified for the survey: 1.2 % did not answer any tradeoff question, 1.9 % always picked the same alternative, and 14.3 % did not answer the follow-up device-acceptance question. After these exclusions, 540 respondents were available for the final analysis, yielding a qualification rate of 80.0 % and a final-stage completion rate of 67.2 %. Additional details on response rates are provided in Appendix C in Supplementary material. Results of internal validity tests indicated the data satisfied high quality standards. For example, only 34 respondents (6.3 %) failed a quiz question following the risk tutorial. Additional details on internal validity are provided in Appendices D and E in Supplementary material.
123
Surg Endosc Fig. 1 Example tradeoff question
The demographic characteristics of respondents who were included and excluded in the analysis were similar. Although the sample was stratified by BMI, the analysis sample on average was about 2 years older and about 8.5 %
123
heavier than the general US obese population. Additional details on comparisons among included respondents, excluded respondents, and the general obese population are provided in Appendix F in Supplementary material.
Surg Endosc
Fig. 2 Relative-importance estimates for attributes of weight-loss devices preference parameters were rescaled relative to 10, corresponding to the absolute value of the most important outcome—5 % mortality risk. Attributes with connecting lines were modelled as nonlinear Box–Cox transformed continuous variables. Other attributes were modelled as categorical variables. Vertical bars around preference scores denote 95 % confidence intervals. Benefits have positive scores and harms have negative scores. Differences between
the best and worst level in each attribute indicate relative attribute importance over the level ranges in the study design. Attributes are sorted from most important benefit to most important harm. Mortality risk is the most important outcome or feature, followed by weight loss, weight-loss duration, and side-effect duration. The relative importance of a 1 % increase in risk is -3.5, in comparison with the relative importance of a 30 % total body weight loss of ?4.3
Preference-score estimates
Table 1 shows the estimated preference scores for the attributes and their levels, indicating their relative importance. For example, the importance of an increase in mortality risk from 0 to 1 % can be quantified by the preference-score difference between these two levels; it was about -3.5 (6.5 - 10). In comparison, the importance of a 30 % TBWL was about 4.3 (4.3 - 0). Therefore, if other device attributes are kept the same, 30 % TBWL more than compensates for having to bear a 1 % risk of death from the device.
The analysis provides estimates of the preference scores that best explain the pattern of observed choices in the data. Additional details on parameter estimates are provided in Appendix G in Supplementary material. Figure 2 shows the preference-score estimates for all attribute levels. They are on a scale from -10 (least preferred), to 10 (most preferred), where -10 is the perceived value of 5 % mortality risk. The difference between the best and worst preference scores for each attribute indicates how influential that attribute is in explaining device choices. Mortality risk was the most important attribute, followed by weight loss, weight-loss duration, and side-effect duration. However, no side effect requiring hospitalization and a 5 % risk of a side effect requiring hospitalization with no surgery were considered much less important.
Benefit-risk tradeoff comparisons In this study, we used the estimated preference scores to calculate the minimum acceptable benefit (MinB), i.e., minimum weight loss respondents expect from a device to tolerate a specific level of risk and other device attributes.
123
Surg Endosc Table 1 Estimates of preference scores by attributes and levels Attribute Average amount of weight loss (TBWL)
Weight loss duration
Side effect duration
Chance of side effects requiring hospitalization
Dietary restrictions
Average reduction in dose of prescription drugs for comorbiditya
Type of operation
Chance of dying from getting the device
Level
Preference score (SE)
30 %
?4.3 (0.52)
20 %
?2.0 (0.11)
10 %
?0.6 (0.15)
5%
?0.2 (0.23)
0%
Reference level
60 months
?4.3 (0.47)
12 months
?2.0 (0.01)
6 months
?1.4 (0.1)
0 months
Reference level
0 months
Reference level
1 months 12 months
-1.0 (0.11) -2.0 (0.09)
60 months
-3.2 (0.31)
None
Reference level
5 % chance of hospitalization, no surgery
-0.2 (0.39)
20 % chance of hospitalization, no surgery
-0.5 (0.35)
5 % chance of hospitalization, with surgery
-0.6 (0.36)
Eat 1/4 cup of food at a time
Reference level
Wait 4 h between eating
-0.1 (0.29)
Can’t eat sweets or foods that are hard to digest
-2.2 (0.33)
Eliminate need/risk
?3.2 (0.37)
50 % dose/risk
?2.2 (0.29)
No change
Reference level
Laparoscopic surgery
Reference level
Endoscopic surgery
-0.5 (0.3)
Open surgery
-2.5 (0.31)
0% 1%
Reference level -3.5 (0.13)
3%
-7.1 (0.15)
5%
-10 (0.37)
Receiving a weight-loss device through open surgery versus endoscopic surgery had a preference-score difference of -2 (-2.5 minus -0.5). Maintaining weight loss for 12 months versus no months had a preference-score difference of ?2 (2 - 0). Holding everything else constant, receiving a device placed through open surgery instead of endoscopically requires at least 12 months of weight loss to compensate. The same metric can be used to measure patient tolerance for risks of adverse events TBWL total body weight loss a
Average reduction in dose of prescription drugs for the current primary comorbid condition or chance of getting the most feared comorbid condition at the lower weight
We also calculated the maximum acceptable risk (MaxR), i.e., maximum device-related mortality risk respondents are willing to tolerate for a given weight loss and other device attributes. The statistical analysis accounted for heterogeneity in patients’ preferences and allowed quantitative segmentation of obese respondents according to their risk tolerance. The MinB and MaxR estimates in the middle 50 % and the upper 25 % of the sample indicate the quantitative benefitrisk tradeoff preferences of average patients and risk-tolerant patients, respectively.
123
Table 2 compares the overall acceptability of three devices relative to the no-device option indicated by MinB and MaxR for the gastric-band and for Devices A and B, which are worse and better than the gastric band, respectively. Preference-score estimates predict that only 5.0 % of respondents would judge Device A better than the nodevice alternative, while 27.5 % of respondents would judge Device B better than the no-device alternative. Devices with less attractive attributes require either larger weight loss or smaller mortality risks in compensation. For example, the gastric band offers only 13 % TBWL, but the
Surg Endosc Table 2 Preference relative to a no-device alternative, minimum acceptable benefit, and maximum acceptable risk of a 243 pound respondent for three weight-loss device profiles Weight loss device profile
Virtual device A
% Respondents who judged better than a no-device alternative 5.0 %
Risk Tolerance
Minimum acceptable TBWLa
Maximum acceptable mortality riskb
Middle 50 %
[30 %c
0.1 % (0.04–0.26)
Upper 25 %
15.1 % (11.63–19.05)
3.8 % (1.85–7.77)
Middle 50 %
[30 %c
0.16 % (0.07–0.38)
Upper 25 %
13.0 % (9.76–16.90)
7.1 % (3.82–13.50)
Middle 50 %
[30 %c
0.6 % (0.29–1.12)
Upper 25 %
12.8 % (9.85–16.45)
10.9 % (6.54–18.78)
Benefit: 5 % TBWL Risk: 1 % chance of death Type of surgery: laparoscopic surgery Dietary restriction: eat 1/4 cup of food at a time Weight-loss duration: 36 months Minor side-effect duration: 36 months Chance of hospitalization with surgery: 5 % Comorbidity: no improvement Gastric band Benefit: 13 % TBWL
11.6 %
Risk: 1 % chance of death Type of surgery: laparoscopic surgery Dietary restriction: eat 1/4 cup of food at a time Weight-loss duration: 5 years Minor side-effect duration: 5 years Chance of hospitalization for side effects requiring surgery: 5% Comorbidity: no improvement Virtual device B
27.5 %
Benefit: 20 % TBWL Risk: 1 % chance of death Type of surgery: endoscopic surgery Dietary restriction: eat 1/4 cup of food at a time Weight-loss duration: 60 months Minor side-effect duration: 12 months Chance of hospitalization but no surgery: 5 % Comorbidity: 50 % improvement 95 % confidence intervals in parentheses TBWL total body weight loss a
For indicated mortality risk and other device attributes
b
For indicated weight-loss benefit and other device attributes
c
Exceeds upper limit of weight-loss levels included in the study design
middle 50 % of respondents require a mean TBWL of more than 30 % given the attributes of the gastric band. Similarly, the gastric band has a 1 % mortality risk while the middle 50 % of respondents would tolerate only a 0.16 % risk for the attributes of the gastric band. On the other hand, for risk-tolerant early adopters, the mean minimum acceptable TBWL is only 13.0 % and the mean maximum acceptable risk is 7.1 %, which compare favorably to the gastric band’s profile. Overall, estimates imply that only 11.6 % of respondents would judge the gastric band better than no device, which is consistent with the fact
that only a small percentage of obese patients have chosen gastric-band surgery [13, 14]. The tool One of the main issues faced by regulatory reviewers when designing clinical studies and analyzing their results is determination of the ‘‘minimum clinical effectiveness’’ that is sufficient to offset the risks and inconveniences posed by the treatments under review. This value is used to size the clinical studies and, when results are available, to decide
123
Surg Endosc
whether the benefits of the treatment outweigh the risks for market approval. There is an enormous amount of discussion among device makers, regulators and experts about this value. To include patient preference input in these discussions, we use the preference-score estimates to build a MaxR– MinB calculator for weight-loss devices (‘‘the tool’’). The MinB obtained from the tool is used to inform CDRH reviewers in establishing the minimum clinical effectiveness (weight loss) of devices under review to size clinical studies. In addition, once the results of the study are available, reviewers use the minimum clinical effectiveness as a reference to evaluate the benefit of a device that is being considered for premarket approval. Given the profile of the weight-loss device, the tool can answer three different but related questions: 1. 2. 3.
What percentage of subjects would prefer getting the device over a no-device alternative? What is the MaxR of an average subject or of an early adopter for a device that provides a given weight loss? What is the MinB of an average subject or of an early adopter for a device that poses a given mortality risk?
Development of the tool went through several evaluations and iterations with the regulatory reviewers to make it usable, understandable, and relevant to the review work. Examples of the tool’s appearance, input fields, and function are shown in Fig. 3.
Discussion This is the first study designed to provide regulators with quantitative data on patient preferences in support of regulatory benefit-risk tradeoff determinations. Previous information on patient preferences largely has consisted of qualitative, anecdotal testimony obtained in advisorycommittee proceedings and public meetings. In contrast, this study provides quantitative data obtained in a controlled experiment from a cross-sectional sample of US obese respondents. FDA regulators and the investigators jointly developed the survey instrument to ensure the study design provided relevant tradeoff information for devices likely to be reviewed by the agency in the foreseeable future. Moreover, these attributes were important to obese subjects, and their levels are clinically meaningful and relevant to regulatory decision making. Patients’ perspectives on benefits and tolerance for risks are likely to be diverse within a patient population and, as a whole, may also differ from clinicians’ perspectives. It is important that a regulatory agency takes into account variations in risk tolerance of the whole spectrum of possible users of a medical treatment because unless a
123
Fig. 3 The MaxR–MinB Tool. 3.1 Using the tool to estimate c minimum acceptable benefit (25 %TBWL). 3.2 Using the tool to estimate maximum acceptable risk (0.5 % mortality). As shown in 3.1 and 3.2, the tool provides an input area at the top where the device attributes, including mortality risk (0.05 %) for (3.1) and benefit (15 % TBWL) for (3.2) are entered. The green cells in the center display the estimated MinB (25 %) and MaxR (0.5 %) respectively. Results can be shown for average patients (middle quartile of the distribution) or for early adopters (upper quartile of the distribution) and for different base weights (Color figure online)
treatment is approved, it is not available to physicians and patients, regardless of their risk tolerance. Patients who are more tolerant of higher risks are likely to be ‘‘early adopters,’’ meaning they could be willing to accept higher risks to gain faster access to innovative treatments. Risktolerant early adopters also are likely to participate in clinical trials of a novel technology and play a critical role in postapproval evaluations by accepting the novel technology once it is on market. If only risk-tolerant patients would accept a certain device profile, the FDA might consider approving such a device only for risk-tolerant patients. Such indication for use will be explained in the device label. CDRH is using the MaxR–MinB calculator ‘‘the tool,’’ to inform reviewers in establishing the minimum clinical effectiveness of weight-loss devices under review. Similar decision-aid calculators can be developed for other medical products. For a given medical product profile, the tool can assist regulatory reviewers to determine the minimum clinical effectiveness to be used in trial design and interpretation of trial results. In addition, opinions among clinicians and patients may be mixed when evaluating a new medical product that offers an incremental improvement in effectiveness but a slightly worse safety profile than existing treatment options. Such medical products could be preferred by some but rejected by others, and the FDA should take this into account. This study may have been subject to some technical limitations. Choices among virtual weight-loss devices do not have the same clinical and emotional consequences as actual choices. However, clearly defined and validated definitions of device attributes helped increase the reliability of the study results by ensuring that respondents were well informed. The potential for hypothetical bias was further reduced by first eliciting judgments about which combination of plausible attributes was better and subsequently eliciting the stated choice between device and nodevice alternatives. The inclusion criteria relied on patient-reported height and weight, not measured BMI. Assurance of anonymity reduced incentives to understate weight and overstate height. In fact, the mean self-reported weight in the study
Surg Endosc
123
Surg Endosc
sample was greater than the national average for the same age cohort. Furthermore, we found no significant differences in preferences between respondents with BMI less than or greater than 40. Whereas the Web panel matches the demographics of the general U.S. population, our stratified sample of obese panel members does not need to match the demographics of the general obese population. Nevertheless, we found only a 2-year difference in mean age. Because we did not find significant differences in preferences between respondents by BMI, the 8.5 % difference in mean weight between the study and national sample has no effect on estimates. Despite these limitations, the estimated preference scores for treatment attributes and their levels provide a quantitative solution to the problem of comparing different outcomes in benefit-risk assessments. They also provide information for making patient-centered, evidence-based regulatory decisions. This study has demonstrated the practical feasibility of eliciting and using evidence on patient benefit-risk tradeoff preferences to inform regulatory decisions. The experience acquired from this study serves as a proof of principle to support the ongoing development of a CDRH guidance document on including evidence on patient preferences in medical-device submissions for premarket approval. We believe that the approach used here is superior to making regulatory decisions without an understanding of the patients’ values and their heterogeneity. This study is a significant step in the direction of incorporating patients’ benefit preferences and risk tolerance into the existing evidence-based regulatory process. Acknowledgments We acknowledge Jeffrey Shuren, MD, JD for his support of the study and insightful advice on the manuscript. We also acknowledge Priya Venkataraman-Rao, MD; Megan Shackelford, MS; Rebecca Nipper; Richard Kotz, MS; Kathleen Olvey; and Martin Golding, MD, for their regulatory input in the development of the survey instrument. We are grateful to the FDA CDRH Obesity Devices Working Group for their comments on interpretation of study results and their feedback on using the MinB–MaxR calculator in regulatory reviews of weight-loss device submissions. The members of that group include Jeffrey Cooper, DVM; Megan Shackelford, MS; Irene Bacalocostantis, PhD; Brandan Reid, PhD; Martha Betz, PhD; Elizabeth Katz, PhD; David Pudwill; Martin Golding, MD; Priya Venkataraman-Rao, MD; and Benjamin Fisher, PhD. The acknowledged persons above are of CDRH, FDA.
123
Disclosures Drs. Hauber and Gonzalez received compensation for their work through a contract with the Center for Devices and Radiological Health (CDRH) of the U.S. Food and Drug Administration (FDA). Ho, Lerner, Neuland, Whang, McMurry-Heath, and Irony have no conflicts of interest or financial ties to disclose.
References 1. http://www.fda.gov/downloads/MedicalDevices/DeviceR egulationandGuidance/GuidanceDocuments/UCM296379.pdf. Accessed 2 Sept 2014 2. Section 513(a) of the Federal Food, Drug, and Cosmetic Act [21 U.S.C. 360c(a)] 3. Lerner H, Whang J, Nipper R (2013) Benefit-risk paradigm for clinical trial design of obesity devices: FDA proposal. Surg Endosc 27(3):702–707 4. Shuren J, Woodcock J (2013) Reviewing FDA’s implementation of FDASIA. http://www.fda.gov/NewsEvents/Testimony/ucm374 544.htm. Accessed 2 Sept 2014 5. McMurry-Heath M (2013) FDA brings patients into the process. http://blogs.fda.gov/fdavoice/index.php/2013/09/fda-bringspatients-into-the-process/. Accessed 2 Sept 2014 6. http://www.ema.europa.eu/docs/en_GB/document_library/ Report/2013/10/WC500153276.pdf. Accessed 2 Sept 2014 7. Marshall D, Bridges JFP, Hauber B et al (2010) Conjoint analysis applications in health: how are studies being designed and reported? Patient 3(4):1178–1653 8. Bridges JFP, Hauber AB, Marshall D et al (2011) Conjoint analysis applications in health—a checklist: a report of the ISPOR good research practices for conjoint analysis task force. Value Health 14(4):403–413 9. Johnson FR, Lancsar E, Marshall D et al (2013) Constructing experimental designs for discrete-choice experiments: report of the ISPOR conjoint analysis experimental design good research practices task force. Value Health 16(1):3–13 10. Kuhfeld WF, Tobias RD, Garratt M (1994) Efficient experimental design with marketing research applications. J Mark Res 31:545–557 11. Kuhfeld WF (2010) Marketing research methods in SAS. http:// support.sas.com/techsup/technote/mr2010.pdf. Accessed 2 Sept 2014 12. Ogden CL, Carroll MD, Kit BK, Flegal KM (2012) Prevalence of obesity in the United States, 2009–2010. NCHS Data Brief No. 82 13. Nguyen NT, Masoomi H, Magno CP, Nguyen XMT, Laugenour K, Lane J (2011) Trends in use of bariatric surgery, 2003–2008. J Am Coll Surg 213(2):261–266 14. Callegaro M, DiSogra C (2008) Computing response metrics for online panels. Public Opin Q 72(5):1008–1032
