At the Intersection of Health, Health Care and Policy Cite this article as: David O. Meltzer and Jeanette W. Chung The Population Value Of Quality Indicator Reporting: A Framework For Prioritizing Health Care Performance Measures Health Affairs, 33, no.1 (2014):132-139 doi: 10.1377/hlthaff.2011.1283
The online version of this article, along with updated information and services, is available at: http://content.healthaffairs.org/content/33/1/132.full.html
For Reprints, Links & Permissions: http://healthaffairs.org/1340_reprints.php E-mail Alerts : http://content.healthaffairs.org/subscriptions/etoc.dtl To Subscribe: http://content.healthaffairs.org/subscriptions/online.shtml
Health Affairs is published monthly by Project HOPE at 7500 Old Georgetown Road, Suite 600, Bethesda, MD 20814-6133. Copyright © 2014 by Project HOPE - The People-to-People Health Foundation. As provided by United States copyright law (Title 17, U.S. Code), no part of Health Affairs may be reproduced, displayed, or transmitted in any form or by any means, electronic or mechanical, including photocopying or by information storage or retrieval systems, without prior written permission from the Publisher. All rights reserved.
Not for commercial use or unauthorized distribution Downloaded from content.healthaffairs.org by Health Affairs on June 11, 2015 at UNIVERSITY OF BIRMINGHAM
Quality Measures By David O. Meltzer and Jeanette W. Chung 10.1377/hlthaff.2011.1283 HEALTH AFFAIRS 33, NO. 1 (2014): 132–139 ©2014 Project HOPE— The People-to-People Health Foundation, Inc.
doi:
David O. Meltzer (dmeltzer@ medicine.bsd.uchicago.edu) is an associate professor in the Department of General Internal Medicine, University of Chicago, in Illinois. Jeanette W. Chung is a research assistant professor in surgical oncology at Northwestern University, in Chicago.
The Population Value Of Quality Indicator Reporting: A Framework For Prioritizing Health Care Performance Measures ABSTRACT The Agency for Healthcare Research and Quality (AHRQ) National Healthcare Quality and Disparities Reports contain more than 250 quality indicators, such as whether a patient with a suspected heart attack received an aspirin. The Department of Health and Human Services National Quality Measures Clearinghouse identifies more than 2,100 such indicators. Because resources for making quality improvements are limited, there is a need to prioritize among these indicators. We propose an approach to assess how reporting specific quality indicators would change care to improve the length and quality of life of the US population. Using thirteen AHRQ quality indicators with readily available data on the benefits of indicator reporting, we found that seven of them account for 93 percent of total benefits, while the remaining six account for only 7 percent of total benefits. Use of a framework such as this could focus resources on indicators having the greatest expected impact on population health.
n “The Opportunity Costs of Haphazard Social Investments in Life-Saving,” Tammy Tengs and John Graham studied the costs and benefits of 185 interventions that reduce the risk of premature mortality. According to their estimates, the United States spent $21 billion on a variety of lifesaving interventions that prevented 56,700 premature deaths in 1996. However, if those same dollars had been spent on the most cost-effective set of interventions, an additional 60,200 premature deaths could have been avoided.1 Tengs and Graham’s analysis provides a cautionary tale for policy makers and others concerned with health care quality improvement, patient safety, and disparities. There are more than 2,100 measures in the 2013 U.S. Department of Health and Human Services National Quality Measures Clearinghouse2 and more than 250 measures in the 2012 AHRQ National Healthcare Quality and Disparities Reports.3 Examples of these measures are whether a patient
I
132
Health Affairs
J a n u a ry 201 4
3 3: 1
with a suspected heart attack received an aspirin upon arrival at the hospital, or whether adults with diagnosed diabetes have their blood pressure under control. High-quality care necessarily involves many such specific actions, and any specific action might not always be appropriate in a given clinical context. Thus, individual quality measures are best considered indicators of quality, and many such indicators are used in measuring quality. However, although quality improvement may sometimes reduce costs,4 the financial resources, time, and effort available for quality improvement are limited—whether within a single hospital, a payer network, a state, or a nation. Spending on quality improvement generally implies a trade-off against other activities, and focusing on specific quality indicators may divert resources away from other indicators or other components of health care. Thus, it is imperative to determine what measures will produce changes in care that result in the greatest im-
Downloaded from content.healthaffairs.org by Health Affairs on June 11, 2015 at UNIVERSITY OF BIRMINGHAM
provements in health care quality and reductions in disparities in health between groups of people. The Affordable Care Act further intensifies the need to make intelligent decisions about priorities for quality measurement by expanding requirements for public reporting of health outcomes, value-based purchasing, and tying hospital Medicare payments to performance. In this article we propose a conceptual and methodological framework to quantify the improvements in population health that may result from reporting health care quality indicators. Our objective is to prioritize indicators that may produce the largest improvements in population health. Our approach in this study was similar to those of recent studies that have sought to quantify the population health gains expected from research.5–7 Following this literature, we measured improvements in population health as the gains in life expectancy adjusted for quality of life, commonly known as quality-adjusted life expectancy. We also took into account the monetary costs of achieving these population health gains by subtracting the gains in health that could have been accomplished by using those funds for other cost-effective interventions. We call this estimate of net potential population health benefit the expected population value of quality indicator reporting. We report here an application of this approach we originally developed for an Institute of Medicine (IOM) report to guide the future development of the AHRQ quality and disparities indicators.8 Our primary goal in developing these estimates for the IOM was to illustrate potential issues that could arise in the application of the approach. This article summarizes the key issues in applying this approach and focuses on a striking finding of our analysis: Seven of the thirteen quality indictors we examined account for 93 percent of the net population health benefits, and the remaining six indicators accounted for only 7 percent of the net health benefits. This suggests that the use of a framework such as this to prioritize indicators could allow policy makers, providers, patients, and others concerned with health care quality to focus their efforts on a smaller number of indicators that have the greatest potential to improve population health. We first define fundamental concepts and develop our methodological framework. Next, we demonstrate the empirical implementation of our framework for selected measures in the AHRQ quality reports.We then discuss the scope of potential application for our proposed method, its limitations, and implementation issues. We conclude with a discussion of areas for future development.
Key Concepts And Framework The Concept Of ‘Net Health Benefit’ Which of the hundreds of existing quality indicators will result in changes in care that produce the greatest improvements in health and reductions in disparities? Answering this question requires an ability to compare how much improvement is produced by different measures, net of their costs.We use net health benefit as a common metric to measure population health gains net of costs that result from using specific quality measures. Net health benefit refers to the gains in health from an intervention compared to an alternative intervention, after subtracting improvements in health that may be forgone because of the costs of the intervention.9 The extra health benefits are calculated as the difference in effectiveness of the intervention and the alternative, often expressed in terms of quality-adjusted life-years (QALYs). The QALY is a measure of gains in life expectancy with years of life adjusted by a quality-of-life factor that ranges from 0 (equal to death) to 1 (equal to perfect health). Costs in this context include both the cost of the quality improvement effort and the cost of resulting changes in care. The potential health gains forgone due to the cost of the health intervention are calculated as the gains in QALYs that could have been produced if the resources used to improve care had been used to improve health in a way that just met society’s valuation of one additional QALY. Many cost-effectiveness studies in the United States have assumed this threshold to be $50,000–$100,000 per QALY. To illustrate this approach, if a threshold of $100,000 per QALY is used, an intervention producing ten QALYs but costing $100,000 would be calculated to have produced a net health benefit of nine QALYs once the forgone benefits of the $100,000 in spending was accounted for. Net health benefit estimates such as this can be used to compare the health benefits of diverse interventions net of their costs. Our framework adapts this concept of net health benefit to assess the expected net returns to reporting quality measures related to specific standards of care. The Benefit Of A Standard Of Care The first step in estimating the value of reporting quality measures is to develop measures of the value of improved quality. All quality measures are based explicitly or implicitly on some standard of care. This might be a structural measure of care, a specific process of care for a group of patients, or an outcome. All other quality-of-care measures, whether of structure, process, or outcome—including “usual care” and “doing nothing”—can be viewed as other alternatives within our framework. In many situations, a January 2014
33:1
Downloaded from content.healthaffairs.org by Health Affairs on June 11, 2015 at UNIVERSITY OF BIRMINGHAM
Health A ffairs
133
Quality Measures measure will have multiple alternatives. For simplicity, we developed our framework around the situation in which there is one alternative; we discuss below how to extend our approach to address multiple alternatives. For each instance in which a standard of care applies, the incremental benefit is the difference in health benefits (that is, change in outcomes) that can be obtained from applying the standard compared to the alternative, minus the difference in health benefits forgone because of the added costs of applying the standard compared to the alternative, including any costs of implementing the improvement in quality. These measures of the benefits of applying quality standards can then be used to calculate several different measures of the population value of quality improvement. In principle, any measure of health benefit can be used if it allows comparisons across multiple interventions. We use QALYs to measure health because the QALY is the most commonly reported measure of health benefits that incorporates the length and quality of life and is intended to be readily comparable across interventions. Using this measure also allows us to draw upon existing studies of the cost-effectiveness of medical interventions that may be the target for quality improvement. Perfect implementation of a standard occurs when it is applied to all patients in the “measure population” of patients for whom the expected benefits of the standard exceed its expected harms and when the standard is not applied to patients outside that measure population. The population value of perfect implementation of a standard is the total net health benefit achieved when the standard of care is applied to every patient in the population for whom the intervention produces a net health benefit and not to any patient for whom it would not produce a net health benefit. It is calculated by multiplying the average per capita net health benefit in the population with a positive net health benefit times the number of people in that population (see Appendix Note 1).10 The potential gains from quality improvement for any standard will be affected by the current rate of implementation of the standard. Therefore, it is useful to define the population value of current implementation, which is the total net health benefit achieved from the health intervention given current implementation (see Appendix Note 2).10 The maximum population value of quality improvement is the total net health benefit that can be gained from improving implementation from current rates to perfect implementation. It is equal to the difference between the population value of perfect implementation 134
H e a lt h A f fai r s
J a n u a ry 2 01 4
3 3: 1
The Affordable Care Act intensifies the need to make intelligent decisions about priorities for quality measurement.
and the population value of current implementation. Because quality improvement efforts are rarely 100 percent effective, gains in net health benefits from health care quality improvement typically fall short of the maximum population value of quality improvement.11 The expected population value of quality improvement is the net health benefit gained from the changes in the rate of implementation resulting from quality improvement (see Appendix Note 3).10 The value of reporting of quality indicators can be understood within this context. A crucial consideration is that reporting does not itself change quality; rather, it assumes an action model in which reporting is intended to produce quality improvement efforts that increase adherence to standards of care that improve outcomes (see Appendix Exhibit 1).10 Two important limitations, which we discuss further below, are that there is little research on the probability that reporting leads to efforts to improve quality, and that there is limited evidence for the effectiveness of quality improvement efforts in improving quality. However, both are clearly often far below the theoretical maximum of 100 percent.
Summary Of The Framework In our approach, we assumed that reporting a quality measure for a standard has some effect on the probability of quality improvement action (see Appendix Exhibit 2).10 The effectiveness of that quality improvement action is measured by the effect of that action on the provision of care consistent with that quality standard. The population value of perfect implementation is the net health benefit that can be achieved by improving quality on a measure to perfect (100 percent) performance for the standard. The expected population value of quality indicator reporting is estimated by multiplying the probability that quality reporting leads to quality
Downloaded from content.healthaffairs.org by Health Affairs on June 11, 2015 at UNIVERSITY OF BIRMINGHAM
improvement efforts times the improvement in the probability of implementation that comes from these quality improvement efforts and the population value of perfect implementation. The availability of data needed to estimate all of these elements is an obvious concern. We address this further below. However, even when estimates for some of these elements are not available, it may be possible to gain insight into the magnitude of potential gains from quality indicator reporting. For example, since the maximum population value of quality improvement reflects the largest potential gain in population net health benefits that can be achieved by closing the quality gap for that measure, it provides an upper bound on the expected value of quality indicator reporting.
Using The Framework To Prioritize Measures Net Health Benefits Estimates of the net health benefits for the standard of care can be calculated based on estimates of the cost and effectiveness of implementing the standard of care relative to the alternative(s) and valuations of the monetary value of health improvements. These items may often be obtained from published cost-effectiveness studies evaluating the standard of care against the alternative(s). Cost-effectiveness studies conducted in a population (for example, age group or country) similar to the population defined by the denominator of the measure in question may be preferred in estimating its net health benefits. Costeffectiveness studies must also report sufficient data to assess effects on both costs and effectiveness measured in QALYs. Cost-effectiveness studies that report only cost-effectiveness ratios (dollars per QALY) are not sufficient to calculate net health benefit because neither costs nor effectiveness is known. Measure Population It is necessary to have an estimate of the number of people eligible for the standard of care (the size of the denominator population). For some public health measures, the eligible population may be estimated from the US census. For measures denominated based on health care utilization, such as hospitalizations, weighted population estimates of utilization from national health care surveys such as the National Hospital Discharge Survey may be useful.12 For measures defined by specific clinical process of care, the size of the denominator population may be estimated by the condition’s prevalence. Current Implementation Rate Data on current implementation rates may be available for quality measures already in use. In addition to
the National Healthcare Quality Report published yearly by AHRQ, data sources for recent rates of implementation include the Behavioral Risk Factor Surveillance System13 and other quality reports. Probability Of Quality Improvement To what extent will providers respond to measurement? There is relatively little evidence supporting a strong direct link between public reporting and quality improvement activities.14 This weak link may be partly attributable to the finding that hospitals and physicians often discount the methodological validity and quality of report cards, so performance reporting may have little direct effect on providers’ quality improvement actions.15–17 A second issue complicating the link between public reporting of quality indicators and quality improvement action is that public reporting has often been studied in the context of pay-for-performance, which makes it difficult to identify the independent effect of public reporting on quality improvement activities and/ or outcomes.18,19 Finally, much of the literature has focused on responses to state-level or payerspecific reporting programs. The response to national reporting may differ. For these reasons, information about the potential effectiveness of quality reporting is often lacking. Effectiveness Of Quality Improvement Numerous studies have been undertaken of the effectiveness of quality improvement programs—for example, to improve cancer screening;20,21 to change practice/provider behavior through continuing medical education;22 and educational outreach.23,24 However, little is known about how the effectiveness of these interventions varies across care standards or settings. Because there is limited evidence on the effect of quality reporting on quality improvement activities and of quality improvement activities on care, efforts to quantify the expected population value of quality indicator reporting can often provide only upper bounds on the population value of quality improvement.
Calculations For Selected Measures In this section we present the results of our attempt to bound expected population value of quality indicator reporting calculations for thirteen National Health Care Quality Report measures for which we were able to obtain information on costs, effectiveness (in QALYs), denominator population, and current implementation rate. Sources of data elements and detailed measures for each measure are reported in the Appendix10 and in the IOM background paper by Meltzer and Chung.8 January 2014
33:1
Downloaded from content.healthaffairs.org by Health Affairs on June 11, 2015 at UNIVERSITY OF BIRMINGHAM
Health A ffairs
135
Quality Measures Exhibit 1 Expected Population Value Of Quality Indicator Reporting Calculations For Thirteen National Healthcare Quality Report (NHQR) Measures Population value of perfect implementation (PVPI) Denominator population
NHQR measure
QALYs
Population value of current implementation (PVCI)
Percent of total QALYs
QALYs
Population maximum value of quality improvement
Percent of total QALYs
QALYs
Percent of total QALYs
Percent of adults with diagnosed diabetes with most recent blood pressure < 140=80 mmHg
17,268,973
7,021,537
39
4,107,599
35
2,913,938
47
Percent of adults age 40+ with diagnosed diabetes with total cholesterol < 200 mg=dL
17,268,973
1,828,056
10
1,003,602
9
824,453
13
Percent of adults age 40+ with diagnosed diabetes with feet checked for sores in past year
17,268,973
2,326,165
13
1,644,599
14
681,566
11
17,268,973
1,474,394
8
805,019
7
669,375
11
126,006,034
529,704
3
241,545
2
288,159
5
Percent of adults with diabetes with HbA1c >9.5% (poor control);
The online version of this article, along with updated information and services, is available at: http://content.healthaffairs.org/content/33/1/132.full.html
For Reprints, Links & Permissions: http://healthaffairs.org/1340_reprints.php E-mail Alerts : http://content.healthaffairs.org/subscriptions/etoc.dtl To Subscribe: http://content.healthaffairs.org/subscriptions/online.shtml
Health Affairs is published monthly by Project HOPE at 7500 Old Georgetown Road, Suite 600, Bethesda, MD 20814-6133. Copyright © 2014 by Project HOPE - The People-to-People Health Foundation. As provided by United States copyright law (Title 17, U.S. Code), no part of Health Affairs may be reproduced, displayed, or transmitted in any form or by any means, electronic or mechanical, including photocopying or by information storage or retrieval systems, without prior written permission from the Publisher. All rights reserved.
Not for commercial use or unauthorized distribution Downloaded from content.healthaffairs.org by Health Affairs on June 11, 2015 at UNIVERSITY OF BIRMINGHAM
Quality Measures By David O. Meltzer and Jeanette W. Chung 10.1377/hlthaff.2011.1283 HEALTH AFFAIRS 33, NO. 1 (2014): 132–139 ©2014 Project HOPE— The People-to-People Health Foundation, Inc.
doi:
David O. Meltzer (dmeltzer@ medicine.bsd.uchicago.edu) is an associate professor in the Department of General Internal Medicine, University of Chicago, in Illinois. Jeanette W. Chung is a research assistant professor in surgical oncology at Northwestern University, in Chicago.
The Population Value Of Quality Indicator Reporting: A Framework For Prioritizing Health Care Performance Measures ABSTRACT The Agency for Healthcare Research and Quality (AHRQ) National Healthcare Quality and Disparities Reports contain more than 250 quality indicators, such as whether a patient with a suspected heart attack received an aspirin. The Department of Health and Human Services National Quality Measures Clearinghouse identifies more than 2,100 such indicators. Because resources for making quality improvements are limited, there is a need to prioritize among these indicators. We propose an approach to assess how reporting specific quality indicators would change care to improve the length and quality of life of the US population. Using thirteen AHRQ quality indicators with readily available data on the benefits of indicator reporting, we found that seven of them account for 93 percent of total benefits, while the remaining six account for only 7 percent of total benefits. Use of a framework such as this could focus resources on indicators having the greatest expected impact on population health.
n “The Opportunity Costs of Haphazard Social Investments in Life-Saving,” Tammy Tengs and John Graham studied the costs and benefits of 185 interventions that reduce the risk of premature mortality. According to their estimates, the United States spent $21 billion on a variety of lifesaving interventions that prevented 56,700 premature deaths in 1996. However, if those same dollars had been spent on the most cost-effective set of interventions, an additional 60,200 premature deaths could have been avoided.1 Tengs and Graham’s analysis provides a cautionary tale for policy makers and others concerned with health care quality improvement, patient safety, and disparities. There are more than 2,100 measures in the 2013 U.S. Department of Health and Human Services National Quality Measures Clearinghouse2 and more than 250 measures in the 2012 AHRQ National Healthcare Quality and Disparities Reports.3 Examples of these measures are whether a patient
I
132
Health Affairs
J a n u a ry 201 4
3 3: 1
with a suspected heart attack received an aspirin upon arrival at the hospital, or whether adults with diagnosed diabetes have their blood pressure under control. High-quality care necessarily involves many such specific actions, and any specific action might not always be appropriate in a given clinical context. Thus, individual quality measures are best considered indicators of quality, and many such indicators are used in measuring quality. However, although quality improvement may sometimes reduce costs,4 the financial resources, time, and effort available for quality improvement are limited—whether within a single hospital, a payer network, a state, or a nation. Spending on quality improvement generally implies a trade-off against other activities, and focusing on specific quality indicators may divert resources away from other indicators or other components of health care. Thus, it is imperative to determine what measures will produce changes in care that result in the greatest im-
Downloaded from content.healthaffairs.org by Health Affairs on June 11, 2015 at UNIVERSITY OF BIRMINGHAM
provements in health care quality and reductions in disparities in health between groups of people. The Affordable Care Act further intensifies the need to make intelligent decisions about priorities for quality measurement by expanding requirements for public reporting of health outcomes, value-based purchasing, and tying hospital Medicare payments to performance. In this article we propose a conceptual and methodological framework to quantify the improvements in population health that may result from reporting health care quality indicators. Our objective is to prioritize indicators that may produce the largest improvements in population health. Our approach in this study was similar to those of recent studies that have sought to quantify the population health gains expected from research.5–7 Following this literature, we measured improvements in population health as the gains in life expectancy adjusted for quality of life, commonly known as quality-adjusted life expectancy. We also took into account the monetary costs of achieving these population health gains by subtracting the gains in health that could have been accomplished by using those funds for other cost-effective interventions. We call this estimate of net potential population health benefit the expected population value of quality indicator reporting. We report here an application of this approach we originally developed for an Institute of Medicine (IOM) report to guide the future development of the AHRQ quality and disparities indicators.8 Our primary goal in developing these estimates for the IOM was to illustrate potential issues that could arise in the application of the approach. This article summarizes the key issues in applying this approach and focuses on a striking finding of our analysis: Seven of the thirteen quality indictors we examined account for 93 percent of the net population health benefits, and the remaining six indicators accounted for only 7 percent of the net health benefits. This suggests that the use of a framework such as this to prioritize indicators could allow policy makers, providers, patients, and others concerned with health care quality to focus their efforts on a smaller number of indicators that have the greatest potential to improve population health. We first define fundamental concepts and develop our methodological framework. Next, we demonstrate the empirical implementation of our framework for selected measures in the AHRQ quality reports.We then discuss the scope of potential application for our proposed method, its limitations, and implementation issues. We conclude with a discussion of areas for future development.
Key Concepts And Framework The Concept Of ‘Net Health Benefit’ Which of the hundreds of existing quality indicators will result in changes in care that produce the greatest improvements in health and reductions in disparities? Answering this question requires an ability to compare how much improvement is produced by different measures, net of their costs.We use net health benefit as a common metric to measure population health gains net of costs that result from using specific quality measures. Net health benefit refers to the gains in health from an intervention compared to an alternative intervention, after subtracting improvements in health that may be forgone because of the costs of the intervention.9 The extra health benefits are calculated as the difference in effectiveness of the intervention and the alternative, often expressed in terms of quality-adjusted life-years (QALYs). The QALY is a measure of gains in life expectancy with years of life adjusted by a quality-of-life factor that ranges from 0 (equal to death) to 1 (equal to perfect health). Costs in this context include both the cost of the quality improvement effort and the cost of resulting changes in care. The potential health gains forgone due to the cost of the health intervention are calculated as the gains in QALYs that could have been produced if the resources used to improve care had been used to improve health in a way that just met society’s valuation of one additional QALY. Many cost-effectiveness studies in the United States have assumed this threshold to be $50,000–$100,000 per QALY. To illustrate this approach, if a threshold of $100,000 per QALY is used, an intervention producing ten QALYs but costing $100,000 would be calculated to have produced a net health benefit of nine QALYs once the forgone benefits of the $100,000 in spending was accounted for. Net health benefit estimates such as this can be used to compare the health benefits of diverse interventions net of their costs. Our framework adapts this concept of net health benefit to assess the expected net returns to reporting quality measures related to specific standards of care. The Benefit Of A Standard Of Care The first step in estimating the value of reporting quality measures is to develop measures of the value of improved quality. All quality measures are based explicitly or implicitly on some standard of care. This might be a structural measure of care, a specific process of care for a group of patients, or an outcome. All other quality-of-care measures, whether of structure, process, or outcome—including “usual care” and “doing nothing”—can be viewed as other alternatives within our framework. In many situations, a January 2014
33:1
Downloaded from content.healthaffairs.org by Health Affairs on June 11, 2015 at UNIVERSITY OF BIRMINGHAM
Health A ffairs
133
Quality Measures measure will have multiple alternatives. For simplicity, we developed our framework around the situation in which there is one alternative; we discuss below how to extend our approach to address multiple alternatives. For each instance in which a standard of care applies, the incremental benefit is the difference in health benefits (that is, change in outcomes) that can be obtained from applying the standard compared to the alternative, minus the difference in health benefits forgone because of the added costs of applying the standard compared to the alternative, including any costs of implementing the improvement in quality. These measures of the benefits of applying quality standards can then be used to calculate several different measures of the population value of quality improvement. In principle, any measure of health benefit can be used if it allows comparisons across multiple interventions. We use QALYs to measure health because the QALY is the most commonly reported measure of health benefits that incorporates the length and quality of life and is intended to be readily comparable across interventions. Using this measure also allows us to draw upon existing studies of the cost-effectiveness of medical interventions that may be the target for quality improvement. Perfect implementation of a standard occurs when it is applied to all patients in the “measure population” of patients for whom the expected benefits of the standard exceed its expected harms and when the standard is not applied to patients outside that measure population. The population value of perfect implementation of a standard is the total net health benefit achieved when the standard of care is applied to every patient in the population for whom the intervention produces a net health benefit and not to any patient for whom it would not produce a net health benefit. It is calculated by multiplying the average per capita net health benefit in the population with a positive net health benefit times the number of people in that population (see Appendix Note 1).10 The potential gains from quality improvement for any standard will be affected by the current rate of implementation of the standard. Therefore, it is useful to define the population value of current implementation, which is the total net health benefit achieved from the health intervention given current implementation (see Appendix Note 2).10 The maximum population value of quality improvement is the total net health benefit that can be gained from improving implementation from current rates to perfect implementation. It is equal to the difference between the population value of perfect implementation 134
H e a lt h A f fai r s
J a n u a ry 2 01 4
3 3: 1
The Affordable Care Act intensifies the need to make intelligent decisions about priorities for quality measurement.
and the population value of current implementation. Because quality improvement efforts are rarely 100 percent effective, gains in net health benefits from health care quality improvement typically fall short of the maximum population value of quality improvement.11 The expected population value of quality improvement is the net health benefit gained from the changes in the rate of implementation resulting from quality improvement (see Appendix Note 3).10 The value of reporting of quality indicators can be understood within this context. A crucial consideration is that reporting does not itself change quality; rather, it assumes an action model in which reporting is intended to produce quality improvement efforts that increase adherence to standards of care that improve outcomes (see Appendix Exhibit 1).10 Two important limitations, which we discuss further below, are that there is little research on the probability that reporting leads to efforts to improve quality, and that there is limited evidence for the effectiveness of quality improvement efforts in improving quality. However, both are clearly often far below the theoretical maximum of 100 percent.
Summary Of The Framework In our approach, we assumed that reporting a quality measure for a standard has some effect on the probability of quality improvement action (see Appendix Exhibit 2).10 The effectiveness of that quality improvement action is measured by the effect of that action on the provision of care consistent with that quality standard. The population value of perfect implementation is the net health benefit that can be achieved by improving quality on a measure to perfect (100 percent) performance for the standard. The expected population value of quality indicator reporting is estimated by multiplying the probability that quality reporting leads to quality
Downloaded from content.healthaffairs.org by Health Affairs on June 11, 2015 at UNIVERSITY OF BIRMINGHAM
improvement efforts times the improvement in the probability of implementation that comes from these quality improvement efforts and the population value of perfect implementation. The availability of data needed to estimate all of these elements is an obvious concern. We address this further below. However, even when estimates for some of these elements are not available, it may be possible to gain insight into the magnitude of potential gains from quality indicator reporting. For example, since the maximum population value of quality improvement reflects the largest potential gain in population net health benefits that can be achieved by closing the quality gap for that measure, it provides an upper bound on the expected value of quality indicator reporting.
Using The Framework To Prioritize Measures Net Health Benefits Estimates of the net health benefits for the standard of care can be calculated based on estimates of the cost and effectiveness of implementing the standard of care relative to the alternative(s) and valuations of the monetary value of health improvements. These items may often be obtained from published cost-effectiveness studies evaluating the standard of care against the alternative(s). Cost-effectiveness studies conducted in a population (for example, age group or country) similar to the population defined by the denominator of the measure in question may be preferred in estimating its net health benefits. Costeffectiveness studies must also report sufficient data to assess effects on both costs and effectiveness measured in QALYs. Cost-effectiveness studies that report only cost-effectiveness ratios (dollars per QALY) are not sufficient to calculate net health benefit because neither costs nor effectiveness is known. Measure Population It is necessary to have an estimate of the number of people eligible for the standard of care (the size of the denominator population). For some public health measures, the eligible population may be estimated from the US census. For measures denominated based on health care utilization, such as hospitalizations, weighted population estimates of utilization from national health care surveys such as the National Hospital Discharge Survey may be useful.12 For measures defined by specific clinical process of care, the size of the denominator population may be estimated by the condition’s prevalence. Current Implementation Rate Data on current implementation rates may be available for quality measures already in use. In addition to
the National Healthcare Quality Report published yearly by AHRQ, data sources for recent rates of implementation include the Behavioral Risk Factor Surveillance System13 and other quality reports. Probability Of Quality Improvement To what extent will providers respond to measurement? There is relatively little evidence supporting a strong direct link between public reporting and quality improvement activities.14 This weak link may be partly attributable to the finding that hospitals and physicians often discount the methodological validity and quality of report cards, so performance reporting may have little direct effect on providers’ quality improvement actions.15–17 A second issue complicating the link between public reporting of quality indicators and quality improvement action is that public reporting has often been studied in the context of pay-for-performance, which makes it difficult to identify the independent effect of public reporting on quality improvement activities and/ or outcomes.18,19 Finally, much of the literature has focused on responses to state-level or payerspecific reporting programs. The response to national reporting may differ. For these reasons, information about the potential effectiveness of quality reporting is often lacking. Effectiveness Of Quality Improvement Numerous studies have been undertaken of the effectiveness of quality improvement programs—for example, to improve cancer screening;20,21 to change practice/provider behavior through continuing medical education;22 and educational outreach.23,24 However, little is known about how the effectiveness of these interventions varies across care standards or settings. Because there is limited evidence on the effect of quality reporting on quality improvement activities and of quality improvement activities on care, efforts to quantify the expected population value of quality indicator reporting can often provide only upper bounds on the population value of quality improvement.
Calculations For Selected Measures In this section we present the results of our attempt to bound expected population value of quality indicator reporting calculations for thirteen National Health Care Quality Report measures for which we were able to obtain information on costs, effectiveness (in QALYs), denominator population, and current implementation rate. Sources of data elements and detailed measures for each measure are reported in the Appendix10 and in the IOM background paper by Meltzer and Chung.8 January 2014
33:1
Downloaded from content.healthaffairs.org by Health Affairs on June 11, 2015 at UNIVERSITY OF BIRMINGHAM
Health A ffairs
135
Quality Measures Exhibit 1 Expected Population Value Of Quality Indicator Reporting Calculations For Thirteen National Healthcare Quality Report (NHQR) Measures Population value of perfect implementation (PVPI) Denominator population
NHQR measure
QALYs
Population value of current implementation (PVCI)
Percent of total QALYs
QALYs
Population maximum value of quality improvement
Percent of total QALYs
QALYs
Percent of total QALYs
Percent of adults with diagnosed diabetes with most recent blood pressure < 140=80 mmHg
17,268,973
7,021,537
39
4,107,599
35
2,913,938
47
Percent of adults age 40+ with diagnosed diabetes with total cholesterol < 200 mg=dL
17,268,973
1,828,056
10
1,003,602
9
824,453
13
Percent of adults age 40+ with diagnosed diabetes with feet checked for sores in past year
17,268,973
2,326,165
13
1,644,599
14
681,566
11
17,268,973
1,474,394
8
805,019
7
669,375
11
126,006,034
529,704
3
241,545
2
288,159
5
Percent of adults with diabetes with HbA1c >9.5% (poor control);
