J Health Serv Res Policy OnlineFirst, published on June 25, 2015 as doi:10.1177/1355819615593772
Perspective
Campbell’s Law: implications for health care
Journal of Health Services Research & Policy 0(0) 1–3 ! The Author(s) 2015 Reprints and permissions: sagepub.co.uk/journalsPermissions.nav DOI: 10.1177/1355819615593772 jhsrp.rsmjournals.com
Michael Poku
Abstract Campbell’s Law is a well-known social science dictum describing the potential ill-effects of using a quantitative measure both as a target and an indicator to drive decision making. Campbell’s Law states that using a measure in this way tends to distort its usefulness to achieve the desired goal while driving participants to adopt corrupt practices. There is evidence of Campbell’s Law at play in various industries, including health care. Fortunately, the deleterious effects of Campbell’s Law can be successfully mitigated. The two main mitigation strategies involve either keeping the goals of a functioning system separated from the metrics used to assess progress towards that goal, or ensuring the proper safeguards are in place to combat the corrupting pressures.
Keywords health care evaluation, health care quality metrics, health care delivery reform
Introduction Holding people accountable to a set of highly incentivized performance targets is plainly problematic. Such pressure can result in actors in a system omitting important processes that do not directly influence targets, engaging in selective external reporting and resorting to even more significant deception and corruption. This issue has been highlighted in the health care domain.1 Campbell’s Law states, ‘the more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor’.2 This axiom was developed in the 1970s by the celebrated American social scientist, Donald T Campbell. He argued that if a particular metric is used to measure success and to drive system changes, that metric’s ability to accurately measure success tends to be compromised. He noted that such metrics become unreliable as actors in the system are pressured to reach targets at the expense of any other aspect of the endeavour at hand, lie in the reporting of such metrics and otherwise adopt corrupt practices for purposes of achieving targets.
police departments held accountable for crime clearance rates (i.e. the percentage of crimes solved) failing to record every complaint or postponing recording complaints until and unless they are solved. Campbell also points to an early period in the Vietnam War when the use of ‘‘body count’’ (the number of the enemy dead left on the battlefield) was used to evaluate the effectiveness of US military units. He notes that how optimizing this quantitative metric began to compete with and even supplant more traditional goals of warfare to the detriment of the US war effort. He also highlights the infamous Texarkana educational performance experiment. Contractors were hired to provide supplementary education and tutoring to students. The contractors were paid based on student’s improvement on standardized tests. As it turns out, the hired tutors were teaching items taken directly from the final test to be administered, rather than tutoring robustly on math, science and reading.
Vanderbilt University Medical School, USA
Historical examples A few specific examples Campbell cites illustrate the law well. For instance, he points to examples of
Corresponding author: Michael Poku, Vanderbilt University Medical School, 2310 Elliott Ave #833, Nashville, TN 37204, USA. Email: [email protected]
Downloaded from hsr.sagepub.com at University of Otago Library on September 27, 2015
2
Journal of Health Services Research & Policy 0(0)
Health care analogues These specific examples ought to serve as a cautionary tale for the pay-for-performance/pay-for-quality faithful. For years, various health care delivery interest groups, experts and other stakeholders have decried the perverse incentives wrought by the fee-for-service system of provider payments. Reimbursement systems with the flavour of pay-for-performance and pay-forquality are gaining popularity. As we drive deeper into these novel payment arrangements, we ought to take stock of the examples of Campbell’s Law that have already been exhibited in health care in the United States (US). For instance, since 1992, the Pennsylvania Cost Containment Council has published a public report, ‘‘A Consumer Guide to Coronary Artery Bypass Graft Surgery’’, highlighting in-hospital mortality and readmission rates for certain cardiac procedures, broken down by facility and by individual surgeon. In a study published in 1996, a majority of surveyed cardiac surgeons in the state reported that this annual report made them less willing to operate on critically ill patients needing the procedures, though it is unclear if their practice patterns were actually altered as a result.3 To highlight another example, we can look at the impact of treatment guidelines for ST-elevation myocardial infarction (i.e. STEMI). The recommended door-to-balloon time (i.e. the time from when the patient arrives at the hospital to when that patient undergoes percutaneous coronary intervention) established by the American Heart Association and the American College of Cardiology is 90 minutes or less. ‘Time is muscle’, as the old adage goes; and the faster the artery blockage is dealt with, the better the patient will fare. However, researchers found that while national door-to-balloon times declined significantly in the years following the establishment of the treatment guidelines, in-hospital mortality remained unchanged.4 In light of these findings (which have been repeated in other studies), there is notable concern that while door-to-balloon time is indeed an important aspect of improving care for STEMI patients, an overemphasis of this metric may also be diverting health care providers’ attention and resources away from other drivers of effective treatment in these cases.5 A final (and more controversial) example of Campbell’s Law at work deals with the interplay of hospital readmissions and hospital observation stays. In 2012, per the Affordable Care Act, Medicare began issuing financial penalties to hospitals exhibiting ‘‘greater than expected’’ 30-day readmission rates. Since that time, the average 30-day hospital readmission rate for Medicare fee-for-service beneficiaries has declined. The Center for Medicare and Medicaid Services and other stakeholders attribute this
improvement to improved transitions of care efforts and enhanced care coordination initiatives led by hospitals. Others believe that the continued rise in hospital observation stays (i.e. stays where patients are treated for one or more days in the hospital, technically as outpatients), which do not count towards hospitals’ readmission tallies, are also partly responsible for this trend.
Mitigation strategies Fortunately, there are ways to mitigate the effects of Campbell’s Law. In general, there are two strategies available: (1) divorce the overarching goals of the health care delivery system (i.e. keeping populations healthy, delivering cost-effective and high-quality care, etc.) from the measures used to track progress towards that ideal or (2) implement necessary safeguards to minimize corruption and distortion of the metrics used for accountability. The first strategy entails measuring the relevant outcome, process, structural and quality metrics but not directly tying these to any form or system of reward or punishment. Health care providers would utilize these measures internally to make continuous improvements in the name of quality and patient safety. Campbell himself supported such a strategy for education—he asserted that standardized testing may be valuable ‘‘under conditions of normal teaching aimed at general competence’’, that is, essentially under appropriate and typical professional behaviour.2 Of course, the principal issue with this strategy is how to enforce accountability in an environment where objectives are completely separated from the measures used to track success towards those goals. Many may doubt that health care providers will do the right thing without a proper incentive system. In light of this shortcoming, the alternative strategy to mitigate Campbell’s Law may be more palatable. The idea behind the second strategy is that metrics can be valuable indicators and drivers of quality and performance if care is taken to minimize corruption pressures. One specific tactic includes utilizing multiple measures to evaluate care in order to decrease the pressure to manipulate any single metric. Medicare’s Hospital Value-Based Purchasing initiative in the US is a good example of this, as it incorporates multiple measures along different dimensions (e.g. efficiency, patient experience, clinical process, and outcomes). The diversification of measures used should paint a clearer picture of overall system performance. Such diversification becomes increasingly important as more consequences are attached to the metrics. Another useful safeguarding tactic is independent verification of reported data. In the health care context, this would entail regular auditing of relevant
Downloaded from hsr.sagepub.com at University of Otago Library on September 27, 2015
Poku
3
documentation and periodic observation visits from external inspectors. The Joint Commission, a United States-based nonprofit organization that accredits health care organizations in the United States, performs some aspects of this with its periodic surveys as part of its accreditation process, but a much more robust system would have to be implemented to ensure the veracity of reported data from health care providers, and this could be quite costly.
Conclusion Campbell’s Law states that anytime a measure of success becomes a target for players in a system that measure loses its value as an indicator of success and tends to distort processes within the system. The ramifications of Campbell’s Law have been described across industries including health care. As health care systems continue to move away from fee-for-service reimbursement towards so-called value-based payment models, health policy makers and other stakeholders ought to be aware of Campbell’s Law and its implications. The detrimental consequences of Campbell’s Law can be mitigated with a thoughtful approach to utilizing
metrics. Systems will have to decide how best to balance the need for undistorted data and uncorrupted behaviour with the need not to burden the health care system with extra bureaucracy and additional administrative costs. References 1. Bevan G and Hood C. What’s measured is what matters: targets and gaming in the English public health care system. Public Admin 2006; 84: 517–538. 2. Campbell DT. Assessing the impact of planned social change. Occasional Paper Series, #8, http://eric.ed.gov/ ?id¼ED303512 (1976, accessed 25 February 2015). 3. Schneider EC and Epstein AM. Influence of cardiac-surgery performance reports on referral practices and access to care—a survey of cardiovascular specialists. N Engl J Med 1996; 335: 251–256. 4. Menees DS, Peterson ED, Wang Y, et al. Door-to-balloon time and mortality among patients undergoing primary PCI. N Engl J Med 2013; 369: 901–909. 5. Flynn A, Moscucci M, Share D, et al. Trends in door-toballoon time and mortality in patients with ST-elevation myocardial infarction undergoing primary percutaneous coronary intervention. Arch Intern Med 2010; 170: 1842–1849.
Downloaded from hsr.sagepub.com at University of Otago Library on September 27, 2015
Perspective
Campbell’s Law: implications for health care
Journal of Health Services Research & Policy 0(0) 1–3 ! The Author(s) 2015 Reprints and permissions: sagepub.co.uk/journalsPermissions.nav DOI: 10.1177/1355819615593772 jhsrp.rsmjournals.com
Michael Poku
Abstract Campbell’s Law is a well-known social science dictum describing the potential ill-effects of using a quantitative measure both as a target and an indicator to drive decision making. Campbell’s Law states that using a measure in this way tends to distort its usefulness to achieve the desired goal while driving participants to adopt corrupt practices. There is evidence of Campbell’s Law at play in various industries, including health care. Fortunately, the deleterious effects of Campbell’s Law can be successfully mitigated. The two main mitigation strategies involve either keeping the goals of a functioning system separated from the metrics used to assess progress towards that goal, or ensuring the proper safeguards are in place to combat the corrupting pressures.
Keywords health care evaluation, health care quality metrics, health care delivery reform
Introduction Holding people accountable to a set of highly incentivized performance targets is plainly problematic. Such pressure can result in actors in a system omitting important processes that do not directly influence targets, engaging in selective external reporting and resorting to even more significant deception and corruption. This issue has been highlighted in the health care domain.1 Campbell’s Law states, ‘the more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor’.2 This axiom was developed in the 1970s by the celebrated American social scientist, Donald T Campbell. He argued that if a particular metric is used to measure success and to drive system changes, that metric’s ability to accurately measure success tends to be compromised. He noted that such metrics become unreliable as actors in the system are pressured to reach targets at the expense of any other aspect of the endeavour at hand, lie in the reporting of such metrics and otherwise adopt corrupt practices for purposes of achieving targets.
police departments held accountable for crime clearance rates (i.e. the percentage of crimes solved) failing to record every complaint or postponing recording complaints until and unless they are solved. Campbell also points to an early period in the Vietnam War when the use of ‘‘body count’’ (the number of the enemy dead left on the battlefield) was used to evaluate the effectiveness of US military units. He notes that how optimizing this quantitative metric began to compete with and even supplant more traditional goals of warfare to the detriment of the US war effort. He also highlights the infamous Texarkana educational performance experiment. Contractors were hired to provide supplementary education and tutoring to students. The contractors were paid based on student’s improvement on standardized tests. As it turns out, the hired tutors were teaching items taken directly from the final test to be administered, rather than tutoring robustly on math, science and reading.
Vanderbilt University Medical School, USA
Historical examples A few specific examples Campbell cites illustrate the law well. For instance, he points to examples of
Corresponding author: Michael Poku, Vanderbilt University Medical School, 2310 Elliott Ave #833, Nashville, TN 37204, USA. Email: [email protected]
Downloaded from hsr.sagepub.com at University of Otago Library on September 27, 2015
2
Journal of Health Services Research & Policy 0(0)
Health care analogues These specific examples ought to serve as a cautionary tale for the pay-for-performance/pay-for-quality faithful. For years, various health care delivery interest groups, experts and other stakeholders have decried the perverse incentives wrought by the fee-for-service system of provider payments. Reimbursement systems with the flavour of pay-for-performance and pay-forquality are gaining popularity. As we drive deeper into these novel payment arrangements, we ought to take stock of the examples of Campbell’s Law that have already been exhibited in health care in the United States (US). For instance, since 1992, the Pennsylvania Cost Containment Council has published a public report, ‘‘A Consumer Guide to Coronary Artery Bypass Graft Surgery’’, highlighting in-hospital mortality and readmission rates for certain cardiac procedures, broken down by facility and by individual surgeon. In a study published in 1996, a majority of surveyed cardiac surgeons in the state reported that this annual report made them less willing to operate on critically ill patients needing the procedures, though it is unclear if their practice patterns were actually altered as a result.3 To highlight another example, we can look at the impact of treatment guidelines for ST-elevation myocardial infarction (i.e. STEMI). The recommended door-to-balloon time (i.e. the time from when the patient arrives at the hospital to when that patient undergoes percutaneous coronary intervention) established by the American Heart Association and the American College of Cardiology is 90 minutes or less. ‘Time is muscle’, as the old adage goes; and the faster the artery blockage is dealt with, the better the patient will fare. However, researchers found that while national door-to-balloon times declined significantly in the years following the establishment of the treatment guidelines, in-hospital mortality remained unchanged.4 In light of these findings (which have been repeated in other studies), there is notable concern that while door-to-balloon time is indeed an important aspect of improving care for STEMI patients, an overemphasis of this metric may also be diverting health care providers’ attention and resources away from other drivers of effective treatment in these cases.5 A final (and more controversial) example of Campbell’s Law at work deals with the interplay of hospital readmissions and hospital observation stays. In 2012, per the Affordable Care Act, Medicare began issuing financial penalties to hospitals exhibiting ‘‘greater than expected’’ 30-day readmission rates. Since that time, the average 30-day hospital readmission rate for Medicare fee-for-service beneficiaries has declined. The Center for Medicare and Medicaid Services and other stakeholders attribute this
improvement to improved transitions of care efforts and enhanced care coordination initiatives led by hospitals. Others believe that the continued rise in hospital observation stays (i.e. stays where patients are treated for one or more days in the hospital, technically as outpatients), which do not count towards hospitals’ readmission tallies, are also partly responsible for this trend.
Mitigation strategies Fortunately, there are ways to mitigate the effects of Campbell’s Law. In general, there are two strategies available: (1) divorce the overarching goals of the health care delivery system (i.e. keeping populations healthy, delivering cost-effective and high-quality care, etc.) from the measures used to track progress towards that ideal or (2) implement necessary safeguards to minimize corruption and distortion of the metrics used for accountability. The first strategy entails measuring the relevant outcome, process, structural and quality metrics but not directly tying these to any form or system of reward or punishment. Health care providers would utilize these measures internally to make continuous improvements in the name of quality and patient safety. Campbell himself supported such a strategy for education—he asserted that standardized testing may be valuable ‘‘under conditions of normal teaching aimed at general competence’’, that is, essentially under appropriate and typical professional behaviour.2 Of course, the principal issue with this strategy is how to enforce accountability in an environment where objectives are completely separated from the measures used to track success towards those goals. Many may doubt that health care providers will do the right thing without a proper incentive system. In light of this shortcoming, the alternative strategy to mitigate Campbell’s Law may be more palatable. The idea behind the second strategy is that metrics can be valuable indicators and drivers of quality and performance if care is taken to minimize corruption pressures. One specific tactic includes utilizing multiple measures to evaluate care in order to decrease the pressure to manipulate any single metric. Medicare’s Hospital Value-Based Purchasing initiative in the US is a good example of this, as it incorporates multiple measures along different dimensions (e.g. efficiency, patient experience, clinical process, and outcomes). The diversification of measures used should paint a clearer picture of overall system performance. Such diversification becomes increasingly important as more consequences are attached to the metrics. Another useful safeguarding tactic is independent verification of reported data. In the health care context, this would entail regular auditing of relevant
Downloaded from hsr.sagepub.com at University of Otago Library on September 27, 2015
Poku
3
documentation and periodic observation visits from external inspectors. The Joint Commission, a United States-based nonprofit organization that accredits health care organizations in the United States, performs some aspects of this with its periodic surveys as part of its accreditation process, but a much more robust system would have to be implemented to ensure the veracity of reported data from health care providers, and this could be quite costly.
Conclusion Campbell’s Law states that anytime a measure of success becomes a target for players in a system that measure loses its value as an indicator of success and tends to distort processes within the system. The ramifications of Campbell’s Law have been described across industries including health care. As health care systems continue to move away from fee-for-service reimbursement towards so-called value-based payment models, health policy makers and other stakeholders ought to be aware of Campbell’s Law and its implications. The detrimental consequences of Campbell’s Law can be mitigated with a thoughtful approach to utilizing
metrics. Systems will have to decide how best to balance the need for undistorted data and uncorrupted behaviour with the need not to burden the health care system with extra bureaucracy and additional administrative costs. References 1. Bevan G and Hood C. What’s measured is what matters: targets and gaming in the English public health care system. Public Admin 2006; 84: 517–538. 2. Campbell DT. Assessing the impact of planned social change. Occasional Paper Series, #8, http://eric.ed.gov/ ?id¼ED303512 (1976, accessed 25 February 2015). 3. Schneider EC and Epstein AM. Influence of cardiac-surgery performance reports on referral practices and access to care—a survey of cardiovascular specialists. N Engl J Med 1996; 335: 251–256. 4. Menees DS, Peterson ED, Wang Y, et al. Door-to-balloon time and mortality among patients undergoing primary PCI. N Engl J Med 2013; 369: 901–909. 5. Flynn A, Moscucci M, Share D, et al. Trends in door-toballoon time and mortality in patients with ST-elevation myocardial infarction undergoing primary percutaneous coronary intervention. Arch Intern Med 2010; 170: 1842–1849.
Downloaded from hsr.sagepub.com at University of Otago Library on September 27, 2015
