RESEARCH AND PRACTICE
Relationship Between Quality Improvement Implementation and Accreditation Seeking in Local Health Departments Li-Wu Chen, PhD, MHSA, Anh Nguyen, MSPH, Janelle J. Jacobson, MPH, Niodita Gupta, MD, MPH, Sarbinaz Bekmuratova, MS, and David Palm, PhD
Since early in this century, national, state, and local organizations have sought to develop a process for accreditation of local health departments (LHDs). Several attempts have been made to identify public health accreditation criteria at the state and local levels. In 2002, the National Association of County and City Health Officials developed a set of 45 public health standards matched to the 10 essential public health services and associated metrics, which became known as an operational definition of a functional LHD.1 The operational definition served as a base for the development of other accreditation tools. In addition, the National Public Health Performance Standards Program was initiated in 1998 as a collaborative effort of 7 national public health organizations that were coordinated and funded by the Centers for Disease Control and Prevention.2 The National Public Health Performance Standards Program represented an attempt to define public health practice standards for state and local public health systems and governing bodies.2 In 2002, it released 3 assessment instruments framed around the 10 essential public health services.2 These tools provided the basis for a national voluntary accreditation program. Building on these previous efforts, in 2004 the Exploring Accreditation Project was developed as a joint initiative of the National Association of County and City Health Officials, the Association of State and Territorial Health Officials, the National Association of Local Boards of Health, and the American Public Health Association to determine the implications and feasibility of a national public health accreditation system.3 In accordance with the recommendations of the Exploring Accreditation Project Steering Committee, the Public Health Accreditation Board (PHAB) was formed to implement and oversee national
Objectives. We examined the relationship between quality improvement (QI) maturity and accreditation attributes of US local health departments (LHDs), specifically those in Nebraska. Methods. Using 2011 Nebraska LHD QI survey data, we conducted Spearman correlation analyses between QI maturity domains and accreditation attributes. Using the 2010 National Association of County and City Health Officials’ National Profile of LHDs, we conducted logistic regression analyses to examine the relationships between specific QI strategies and attitude toward seeking accreditation. Results. Leaders’ commitment to and length of time engaged in QI were positively associated with LHDs’ general attitude toward seeking accreditation. Use of QI strategies and integration of QI policies and practices were positively associated with LHDs’ confidence in their capacity to obtain accreditation. LHDs that had used at least 1 QI framework and at least 1 QI technique in the past year were more likely to agree that they would seek accreditation within 2 years of the national accreditation program. Conclusions. Experience with and expertise in QI implementation play an important role in LHDs’ decision to seek accreditation, and their accreditationseeking efforts may benefit from prior implementation of systematic QI strategies. (Am J Public Health. 2015;105:S295–S302. doi:10.2105/AJPH.2014. 302278)
public health department accreditation.4 The key focus of the PHAB accreditation program is to strengthen the public health infrastructure.5 Although the process of accreditation can be arduous, it has several benefits to the LHD, including identifying performance improvement areas and improving management, leadership, and relationships within the community. 6 Paralleling the PHAB’s efforts, the Multistate Learning Collaborative on Performance and Capacity Assessment or Accreditation of Public Health Departments worked with states with existing assessment or accreditation as natural laboratories to enhance existing initiatives and disseminate the findings of these mature programs.3 The collaborative’s efforts have yielded valuable information. For instance, some LHDs observed benefits of the accreditation process in the standardization of measures for accountability and the development
Supplement 2, 2015, Vol 105, No. S2 | American Journal of Public Health
of morale among LHD staff members.5 However, other LHDs experienced problems during the accreditation process, including the redundant documentation of standards and difficulty understanding how to meet the documentation standards.3 In addition, some LHDs found that the short period of onsite review and the lack of funding made accreditation difficult.6 Moreover, researchers have noted that marketing accreditation to LHDs without offering incentives could be challenging, and such processes may cause anxiety and reluctance to participate. These challenges are also likely to be exacerbated by the different types and sizes of LHDs.7 Much previous research on accreditation has focused on the role of financing and incentives in voluntary adoption of accreditation standards. Accreditation can be an expensive and time-consuming process for LHDs, but these efforts are rewarded during the accreditation
Chen et al. | Peer Reviewed | Research and Practice | S295
RESEARCH AND PRACTICE
process when the LHD’s quality efforts are documented.8 One of the key features that has been highlighted throughout the literature is the role of quality improvement (QI) in the accreditation process. QI, regardless of where a department stands on accreditation, is an essential activity for all health departments.9 QI has been incorporated as a central concept throughout PHAB’s Standards and Measures Version 1.5,10 with a strong focus on the QI processes used by the LHD. Domain 9 of the PHAB standards and measures focuses on the evaluation of all programs, interventions, and the key public health process, as well as the implementation of a formalized QI process.10 LHDs that have already implemented a continuous QI plan or performance management system can document this existing system and submit it to fulfill this requirement.10 Although QI implementation is essential in LHDs, a number of barriers to its implementation exist, including program requirements and lack of funding, incentives for implementation, leadership, and additional technical assistance resources.11 In addition, the language used to describe QI processes is not uniform across LHDs, making it difficult to study QI activities. More important, although QI activities are essential for accreditation, little has been written about the relationship between QI and accreditation and how LHDs can better integrate QI and accreditation strategies. In 2013, Shah et al.12 examined the relationship between PHAB accreditation prerequisites and LHDs’ intention to seek PHAB accreditation. Although their study provides an understanding of how conducting a community health assessment, community health improvement planning, and strategic planning are associated with LHDs’ intent to seek accreditation, the specific characteristics of QI implementation and the effect of implementation on LHDs’ intention to seek accreditation have not been examined. We examined the relationship between QI maturity and strategies and accreditation attributes in the United States, specifically Nebraska.
METHODS We used different methods and data sources for the analysis of US and Nebraska LHDs, respectively. They are described as follows.
Data Sources For the Nebraska sample, we used 2011 Nebraska LHD QI survey data. This survey was electronically administered by our research team to all 21 LHD directors in Nebraska from May 2011 to August 2011. A total of 19 LHD directors (90.5% of the sample) responded to the online survey. We designed the survey instrument using the QI taxonomy developed by the University of Minnesota, input from the Nebraska Public Health Practice---Based Research Network Steering Committee, and questions adapted from the Multistate Learning Collaborative on Performance and Capacity Assessment or Accreditation of Public Health Departments 2011 Annual Survey, designed by the University of Southern Maine.9,13 The questions covered QI maturity domains and accreditation attributes (attitudes, beliefs, and readiness).13 Respondents were asked to rate the level to which they agreed or disagreed with statements on a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). Measuring QI maturity within the public health department setting is a fairly new concept and is based on 4 domains, including organizational culture, capacity and competency, practice, and alignment and spread.13 Organizational culture refers to the QI values and norms that influence the internal and external interactions at the agency. The 2 organizational culture dimensions include commitment (a desire to adopt new ideas and improve services) and collaboration (a mutually supportive work environment in which QI can thrive). Capacity and competency refer to the skills, functions, and approaches used for QI. The 3 capacity and competency dimensions include skills (training in basic QI methods), methods (application of QI tools, approaches, and data), and investment (designated and established QI staff and priorities). Alignment and spread refer to the extent to which QI is supported and diffused within the organization. The 4 alignment and spread dimensions include integration (agency policies and practices that support QI), authority (staff ability to make and implement QI decisions), value (the perception that QI is an important investment), and implementation (the ease of implementing QI).13 For the US sample, we obtained data from National Association of County and City Health
S296 | Research and Practice | Peer Reviewed | Chen et al.
Officials’ 2010 National Profile of LHDs.14 The 2010 National Profile of LHDs included data from a set of core questions that was sent to all LHDs in the United States and from supplemental questions that were grouped into 2 modules and randomly sent to select LHDs. We used data on governance, funding, population size, and staffing from the core questions and data on QI strategies and accreditation attitudes from module 1. We used an indicator from module 1 to select only LHDs that reported having implemented formal or informal QI activities. We included 449 of 531 LHD respondents from module 1 in the analysis. Estimation weights were provided to account for sampling and to obtain national estimates for all LHDs in the United States.
Data Analysis Using 2011 Nebraska LHD QI survey data, we created measures for QI maturity domains and their respective dimensions by summing the scores for the items that fell within that domain or dimension. We used reverse coding as appropriate so that a higher score was better, and we treated “don’t know” responses as missing. We measured QI practice by whether an LHD had ever implemented a QI project, how many QI projects had been implemented in the past 12 months, how long an LHD had engaged in QI efforts, and the extent to which an LHD had used QI models, techniques, and metrics. We used Spearman correlation analyses, a nonparametric correlation technique often used when 1 or both variables are ordinal or not normally distributed or when the sample size is small, to analyze the relationship between QI maturity (organizational culture, capacity and competency, practice, and alignment and spread) and accreditation attitudes, beliefs, and readiness for Nebraska LHDs. Using the National Association of County and City Health Officials’ 2010 National Profile of LHDs, we created the following QI strategy-related measures: whether an LHD had implemented at least 1 formal QI project in the past year, whether an LHD had used at least 1 QI framework (e.g., Baldrige, Lean, Six Sigma) in the past year, whether an LHD had used at least 1 QI technique (e.g., process map, root-cause analysis) in the past year, whether an LHD had dedicated QI staff, and whether LHD staff had received any formal training in QI
American Journal of Public Health | Supplement 2, 2015, Vol 105, No. S2
RESEARCH AND PRACTICE
methods. We created 2 accreditation attitude measures to indicate whether an LHD would seek national accreditation and whether an LHD would seek national accreditation within the first 2 years of the program. We conducted multivariable logistic regression analyses to examine the association between QI strategies and accreditation attitudes among LHDs that had implemented QI activities in the United States. Specifically, we examined how implementing particular QI strategies affected an LHD’s likelihood of wanting to seek accreditation in general and wanting to seek accreditation within 2 years, controlling for an LHD’s previous involvement in performance-related activities (e.g., MAPP (Mobilizing for Action through Planning and Partnerships), Turning Point), the presence of an agency-wide strategic plan, the size of the total population served by an LHD, jurisdiction type, governance (i.e., the presence of local board of health), budget cuts, per capita expenditures, and staffing cuts. We analyzed all data using SAS version 9.3 (SAS Institute Inc., Cary, NC).
RESULTS Table 1 shows the levels of QI maturity and accreditation attributes in Nebraska’s LHDs in 2011. Median scores at the upper end or above the midpoint value of the possible score range are considered favorable. The median score for organizational culture was 32 (of 35), suggesting that Nebraska’s LHDs were highly mature in fostering an environment with QI embedded in organizational values and norms. The median score for capacity and competency was 37 (of 50), suggesting that Nebraska’s LHDs struggled with applying appropriate QI strategies and lacked the capacity to implement QI. The median score for alignment and spread was 36 (of 55), suggesting that Nebraska’s LHDs had many opportunities to improve policies and practices that support QI and perceptions about implementing QI. The majority (83.3%) of responding LHDs indicated that their agency had engaged in implementing a QI process. A median of 3 formal projects had been implemented by Nebraska LHDs within the past 12 months. About half (47.4%) of responding LHDs indicated that their QI efforts had been consistent for more than 5 years. More than half (53.3%) indicated that their LHD had
TABLE 1—Quality Improvement Maturity and Accreditation Attitudes, Beliefs, and Readiness in Local Health Departments (n = 19): Nebraska, 2011 QI Maturity Domain and Dimensions
No. of Items
Score Range
%
Mean
Median
7
7–35
...
31.4
32.0
Commitment
3
3–15
...
13.5
14.0
Collaboration
4
4–20
...
18.0
18.0
10
10–50
...
36.5
37.0
Skills Methods
2 6
2–10 6–30
... ...
7.7 22.6
8.0 22.0
Investment
2
2–10
...
6.1
6.0
Ever implemented QI: yes
...
...
83.3
...
...
Ever implemented QI: no
...
...
16.7
...
...
No. of projects, past 12 mo
...
...
...
3.8
3.0
No systematic QI efforts in place
...
...
15.8
...
...
Engaged in QI < 1 y Engaged in QI 1–2 y
... ...
... ...
5.3 21.1
... ...
... ...
Engaged in QI 3–4 y
...
...
10.5
...
...
Engaged in QI ‡ 5 y
...
...
47.4
...
...
No use of QI strategies
...
...
13.3
...
...
Use of QI model, technique, or metric
...
...
13.3
...
...
Use of combination of QI model and technique,
...
...
20.0
...
...
... 11
... 11–55
53.3 ...
... 37.3
... 36.0
Integration
8
8–40
...
27.2
26.5
Authority
1
1–5
...
4.0
4.0
Value
1
1–5
...
4.3
4.0
1
1–5
...
1.8
2.0
1–5 1–5
... ...
3.9 3.0
4.0 3.0
Organizational culture
Capacity and competency
Practice
QI model and metric, or QI technique and metric Use of QI model, technique, and metric Alignment and spread
Implementation
Accreditation measuresa Attitude toward accreditation Agency will seek accreditation Agency will seek accreditation within first 2 y
1 1
Belief in accreditation Believe that national standards are a good idea
1
1–5
...
4.1
4.0
Believe accreditation will strengthen agency
1
1–5
...
3.9
4.0
Confident in agency’s capacity to obtain accreditation
1
1–5
...
2.9
3.0
Agency has begun preparing for accreditation
1
1–5
...
3.8
4.0
Perceived readiness for accreditation
Note. QI = quality improvement. Source. 2011 Nebraska Local Health Department Quality Improvement Survey. a The scale for each item was based on a 5-point Likert scale (1 = strongly disagree to 5 = strongly agree). We calculated QI maturity domain and dimension scores by summing the scores for the items that fell within that domain or dimension.
used all 3 types of QI strategies (models, techniques, and metrics). Items related to accreditation (attitudes, beliefs, and readiness) were rated using a 5-point Likert scale (5 = strongly agree, 4 = agree, 3 = neither agree nor disagree, 2 = disagree, 1 = strongly disagree). Therefore, on the basis
Supplement 2, 2015, Vol 105, No. S2 | American Journal of Public Health
of this scale and using the median scores of all respondents, the results suggest that although LHD directors in Nebraska generally agreed that their agency would seek national accreditation (median = 4), they were unsure whether accreditation would be sought within the first 2 years of the program (median = 3). LHD
Chen et al. | Peer Reviewed | Research and Practice | S297
RESEARCH AND PRACTICE
(r = .60; P = .008). Integration of QI policies and practices (r = .58; P = .014) and the alignment and spread of QI (r = .53; P = .037) within an LHD were positively associated with the belief that accreditation would strengthen the agency. The ability to apply QI methods was also positively associated with the belief that accreditation would strengthen the agency; however, the association was marginally significant (r = .47; P = .049). LHD leaders’ commitment to QI was also positively associated with the agency’s having begun to prepare for accreditation (r = .57; P = .015). The use of QI strategies (r = .57; P = .033) and the integration of QI policies and practices (r = .50; P = .049) within the agency were positively associated with an LHD’s confidence in its capacity to obtain accreditation. Table 3 shows the characteristics of US LHDs that have implemented QI activities. In 2010, an estimated 84.4% of all US LHDs indicated that their LHD had implemented some formal or informal QI activities. More than one third (39.3%) of these LHDs served
directors in Nebraska generally believed that national standards were a good idea and would strengthen their agencies (median = 4). Although LHDs in Nebraska had generally begun preparation for accreditation (median = 4), respondents were unsure whether their agency had sufficient capacity to obtain accreditation (median = 3). Table 2 shows the results of the correlation analysis between QI maturity and accreditation attributes for Nebraska LHDs. The length of time engaged in QI (r = .58; P = .014) was positively associated with an LHD’s general attitude toward seeking accreditation. A commitment to QI among leaders was also positively associated with an LHD’s general attitude toward seeking accreditation; however, the association was marginally significant (r = .49; P = .049). The use of QI strategies was positively associated with an LHD’s attitude toward seeking accreditation within the first 2 years of the program (r = .76; P = .003). Valuing QI was positively associated with the belief that national standards were a good idea
a population of less than 25 000. The majority were also LHDs that served a single county or city (89.5%) and had at least 1 local board of health (78.2%). Almost one half (47.7%) of these LHDs experienced budget cuts, and the estimated average percentages of employees who were laid off, who were lost through attrition, whose work hours were reduced, and who were placed on mandatory furlough in the previous year were, respectively, 1.9%, 3.2%, 2.8%, and 9.2%. In 2010, 73.1% of LHDs nationwide had implemented at least 1 formal QI project, 38.7% had used at least 1 QI framework, and 78.3% had used at least 1 QI technique in the past year. Also, 31.6% of the LHDs had dedicated QI staff, 74.0% had staff members who had received formal QI training, 49.8% had recently been involved in at least 1 performance-related activity, and 42.9% had developed an agency-wide strategic plan. More than one half (54.3%) of the LHDs agreed that they would seek national accreditation, whereas only one third (33.0%) agreed that their LHD would seek national
TABLE 2—Results From Spearman Correlation Analysis Between Quality Improvement Maturity and Accreditation Attributes of Local Health Departments: Nebraska, 2011 Agency Would Seek Accreditation QI Maturity Organizational culture Commitment Collaboration Capacity and competency
Agency Would Seek Accreditation Within First 2 Y
r
P
r
.16
.54
.49
.049
–.02 .03
.944 .914
National Standards Are a Good Idea
Accreditation Would Strengthen Agency
P
r
P
r
–.02
.938
.15
.543
.17
.22
.417
.46
.056
.35
.02 .30
.946 .306
–.01 .10
.957 .72
.08 .33
P
Confident in Agency’s Capacity to Obtain Accreditation
Agency Has Begun Preparing for Accreditation
r
P
r
P
.5
.22
.402
.37
.139
.151
.30
.246
.57
.015
.764 .216
.18 .33
.482 .231
.18 –.30
.478 .258
Skills
.34
.211
.32
.285
.03
.911
.09
.74
.11
.694
–.35
.165
Methods
.15
.561
–.04
.889
.08
.763
.47
.049
.47
.055
–.07
.794
Investment
.24
.381
.46
.102
.05
.866
.06
.828
.24
.398
–.36
.171
QI practice Ever implemented QI
.33
.209
.47
.081
.37
.145
.35
.168
.47
.067
.04
.879
No. of projects, past 12 mo
.30
.471
.49
.264
.29
.481
.61
.112
.29
.522
.10
.822
Length of time engaged in QI QI strategy use
.58 .37
.014 .367
.36 .76
.174 .003
.37 .51
.135 .053
.44 .37
.065 .178
.35 .57
.168 .033
.44 .30
.066 .274 .975
Alignment and spread
.22
.437
.39
.167
.27
.308
.53
.037
.44
.098
–.01
Integration
.25
.355
.37
.169
.29
.268
.58
.014
.50
.049
.08
.753
Authority
.09
.748
.16
.575
.13
.619
.03
.917
.05
.854
–.32
.217
Value Implementation
.36
.154
.23
.401
.60
.008
.31
.211
.32
.216
.27
.279
–.27
.29
.00
‡ .999
–.38
.124
–.22
.373
–.20
.441
–.21
.396
Note. QI = quality improvement. P values at less than .05 were considered significant. Source. 2011 Nebraska Local Health Department Quality Improvement Survey.
S298 | Research and Practice | Peer Reviewed | Chen et al.
American Journal of Public Health | Supplement 2, 2015, Vol 105, No. S2
RESEARCH AND PRACTICE
TABLE 3—Characteristics of Local Health Departments That Have Implemented Quality Improvement Activities: United States, 2010 No.
% (95% CI)a or Meana 6SE (Median)
449
100.0
< 25 000 25 000–49 999
122 88
39.3 (34.3, 44.3) 20.9 (17.0, 24.8)
50 000–99 999
66
15.0 (11.7, 18.4)
100 000–249 999
72
13.2 (10.2, 16.2)
250 000–499 999
42
5.7 (4.0, 7.4)
‡ 500 000
59
5.9 (4.3, 7.4)
LHD Characteristics Total LHDs Population size
Jurisdiction type Single county or city
390
89.5 (86.8, 92.2)
59
10.5 (7.8, 13.2)
Multicounty or multicity (i.e., regional) Local board of health No
111
21.8 (17.9, 25.7)
‡ 1 local boards of health
336
78.2 (74.3, 82.1)
Budget cut Yes, budget is less than previous y
191
47.7 (42.6, 52.9)
No, budget is same or greater than previous y
211
52.3 (47.1, 57.4)
375
57.5 63.4 (40.7)
Expenditures per capita, $ Staffing cuts in previous y, % Employee layoff Employee loss through attrition
409
1.9 60.3 (0)
408
3.2 60.3 (0)
Employees with reduced hours
401
2.8 60.7 (0)
Employees on mandatory furlough
408
9.2 61.3 (0)
QI characteristics No. of formal QI activities in past y None ‡ 1 projects No. of QI frameworks used in past y
106
26.9 (22.4, 31.4)
339
73.1 (68.6, 77.6)
None
252
61.3 (56.5, 66.1)
‡ 1 QI frameworks
179
38.7 (33.9, 43.5)
No. of QI techniques used in past y None
82
21.7 (17.5, 25.9)
357
78.3 (74.1, 82.5)
291 150
68.4 (63.9, 72.9) 31.6 (27.1, 36.1)
No, LHD does not have staff formally trained in QI
102
26.0 (21.6, 30.5)
Yes, LHD has staff formally trained in QI
340
74.0 (69.5, 78.4)
‡ 1 QI techniques Dedicated QI staff No, LHD does not have dedicated QI staff Yes, LHD has dedicated QI staff Staff formally trained in QI
No. of performance-related activities None
196
50.2 (45.2, 55.3)
‡ 1 performance-related activities
228
49.8 (44.7, 54.8)
Agency-wide strategic plan No, LHD has not developed plan
237
57.1 (52.3, 61.9)
Yes, LHD has developed plan
210
42.9 (38.1, 47.7) Continued
Supplement 2, 2015, Vol 105, No. S2 | American Journal of Public Health
accreditation within the first 2 years of the program. Table 4 shows the results of the regression analyses. LHDs that had implemented at least 1 formal QI project in the past year were 0.50 times more likely (95% CI = 1.03, 2.19) to agree that their LHD would seek accreditation than LHDs that had not implemented any formal QI project in the past year. LHDs that had recently been involved in at least 1 performance-related activity were 1.79 times more likely (95% CI = 2.12, 3.68) to agree that their LHD would seek accreditation than LHDs that had not been involved in any performance-related activities. LHDs with at least 1 local board of health were also 0.58 times more likely (95% CI = 1.12, 2.23) to agree that their LHD would seek accreditation than LHDs without a local board of health. LHDs with higher per capita expenditures (odds ratio [OR] = 0.997; 95% CI = 0.994, 0.999), a higher percentage of employees lost through attrition in the previous year (OR = 0.964; 95% CI = 0.934, 0.995), and a higher percentage of employees with reduced hours in the previous year (OR = 0.98; 95% CI = 0.96, 0.99) were less likely to agree that their LHD would seek accreditation. LHDs that had used at least 1 QI framework (OR = 1.94; 95% CI = 1.43, 2.63) and at least 1 QI technique (OR = 3.41; 95% CI = 1.96, 5.95) in the past year were 0.94 and 2.41 times more likely, respectively, to agree that their LHD would seek accreditation within 2 years of the program. LHDs serving a population of 25 000 to 49 999 (OR = 0.46; 95% CI = 0.31, 0.70) as well as those serving a population of 250 000 to 499 999 (OR = 0.26; 95% CI = 0.13, 0.53) were less likely to agree that their LHD would seek accreditation within 2 years than LHDs serving a population of less than 25 000. LHDs with higher per capita expenditures (OR = 0.997; 95% CI = 0.994, 0.999), a higher percentage of employees laid off in the previous year (OR = 0.96; 95% CI = 0.93, 0.99), a higher percentage of employees lost through attrition in the previous year (OR = 0.94; 95% CI = 0.90, 0.98), and a higher percentage of employees with reduced hours in the previous year (OR = 0.972; 95% CI = 0.947, 0.998) were also less likely to agree that their LHD would seek accreditation within 2 years.
Chen et al. | Peer Reviewed | Research and Practice | S299
RESEARCH AND PRACTICE
TABLE 3—Continued Seek accreditation under voluntary national program LHD would not seek accreditation
173
45.7 (40.5, 50.9)
LHD would seek accreditation
227
54.3 (49.1, 59.5)
LHD would not seek accreditation £ first 2 y
259
67.0 (62.1, 71.9)
LHD would seek accreditation £ first 2 y
135
33.0 (28.1, 37.9)
Seek accreditation under voluntary national program £ first 2 y of program
Note. CI = confidence interval; LHD = local health department; QI = quality improvement. Source. National Association of County and City Health Official’s 2010 National Profile of Local Health Departments.14 a Weighted values.
DISCUSSION This study has implications for how policymakers and public health practitioners may better integrate QI and accreditation strategies in the LHD setting. Our results from the Nebraska analysis suggest that QI maturity is generally positively associated with an LHD’s attitude toward, beliefs about, and readiness for seeking national accreditation. In particular, an LHD in which QI methods are better applied and QI practice is better supported and diffused through policies and procedures tends to believe that accreditation would strengthen the agency. Not surprisingly, LHDs that value QI more highly tend to believe that national standards are a good idea. These results suggest that the nature of the pursuit of QI and accreditation share significant common ground so that the benefits of accreditation are better recognized by LHDs that value and support QI more. Our results also indicate that LHDs with leaders who have a higher commitment to QI and in which QI engagement has been in place longer tended to express a greater desire to seek accreditation. Nevertheless, only the LHDs that had actually used more formal QI strategies expressed a greater desire to seek accreditation within 2 years. These results highlight the important role of actual QI implementation and suggest that a greater desire to seek accreditation may result from a greater number of QI experiences and greater QI expertise. In addition, our results indicate that LHDs that have used more formal QI strategies and have better supported QI through policies and procedures have higher confidence in their ability to obtain accreditation. These results further suggest that the actual experience of implementing QI
systematically not only increases an LHD’s desire to seek accreditation but also enhances its perceived readiness to obtain accreditation. The results from the Nebraska analysis are based on bivariate analyses without controlling for potential confounding factors; therefore, the regression analysis using the US sample was conducted to remedy that limitation. Our results from the US analysis confirmed the positive relationship between QI strategy use and accreditation attitude. In particular, the LHDs that had implemented formal QI projects in the past year, had prior experience in other performance-related activities, or both were more likely to express a desire to seek accreditation. However, only LHDs that had actually adopted a formal QI framework or formal QI techniques in the past year were more likely to express a desire to seek accreditation within the next 2 years. These results further highlight the importance of an LHD’s experiences and expertise accumulated through actual QI implementation in its decision to seek accreditation. In addition, prior experience using a more systematic QI approach (i.e., specific QI frameworks or techniques rather than general QI project experience or performance-related activities) significantly enhanced an LHD’s attitude toward seeking accreditation in the short term. Our results on other control variables indicate that LHDs with a local board of health were more likely to express a desire to seek accreditation generally than their counterparts without such governance. This finding is consistent with the conventional wisdom that LHDs with a local board of health may be under greater pressure to pursue accreditation. Nevertheless, our study suggested that the presence of a local board of health does not
S300 | Research and Practice | Peer Reviewed | Chen et al.
increase the likelihood of an LHD’s expressing a desire to seek accreditation in the short term. This may suggest that the impetus for an LHD’s seeking accreditation in the short term is driven more by internal forces (e.g., prior QI experiences and expertise) than by external forces. In addition, LHDs that had experienced more staffing cuts in the previous year were less likely to express a desire to seek accreditation (both in general and within 2 years) than their counterparts with fewer staffing cuts. Given that many LHDs perceive the PHAB accreditation process to be time and resource consuming, the LHDs that had experienced more recent staffing cuts would face more challenges in moving the accreditation process forward while meeting their daily operational needs. Our results also indicate that LHDs with higher per capita expenditures were less likely to express a desire to seek accreditation. Although this result was inconsistent with our expectation, its effect size was very marginal. LHDs serving a population of 25 000 to 49 999 or 250 000 to 499 999 persons were less likely to express a desire to seek accreditation within 2 years than LHDs serving a population of less than 25 000 persons. Because our results did not demonstrate a clear relationship pattern between population size and accreditation attitude, future research is needed to explore this area further. Our study suggests that an LHD’s QI maturity and strategies and its attitude toward seeking national accreditation are positively related. LHDs may benefit from implementing QI activities (especially adopting more formal and systematic QI strategies) in their seeking national accreditation. Given the similar nature of QI and accreditation activities, LHDs may benefit from the scope economies between these 2 types of tasks. Early QI adopters may also benefit from using their greater momentum and culture of seeking excellence, better established and documented policies and procedures, more supportive and collaborative working environment, more efficient operation, and greater QI experience and expertise during their process of seeking national accreditation. Although considerable staff time and cost associated with seeking accreditation can be a challenge for many LHDs, seeking accreditation can be an incentive for LHDs to establish a formal continuous QI plan to improve their
American Journal of Public Health | Supplement 2, 2015, Vol 105, No. S2
RESEARCH AND PRACTICE
TABLE 4—Logistic Regression Results on the Likelihood of Local Health Departments Expressing a Desire to Seek National Accreditation: United States, 2010 LHD Would Seek Accreditation, OR (95% CI)
Characteristic
LHD Would Seek Accreditation £ 2 Years, OR (95% CI)
Quality improvement characteristics No. of formal QI activities in past y None (Ref)
1.00
‡ 1 projects
1.50* (1.03, 2.19)
1.00 0.67 (0.43, 1.02)
programs and services. Overall, LHDs would be inclined to seek accreditation for a variety of reasons, including improving performance, establishing policies and procedures, and enhancing accountability to the community. Policymakers should develop policies to incentivize late QI-adopting LHDs to implement formal QI techniques and strategies to facilitate their accreditation seeking and improve their practice. j
No. of QI frameworks used in past y None (Ref)
1.00
‡ 1 QI frameworks
1.26 (0.94, 1.69)
1.00
About the Authors
1.94** (1.43, 2.63)
Li-Wu Chen, Anh Nguyen, Janelle J. Jacobson, Niodita Gupta, and Sarbinaz Bekmuratova are with the Department of Health Services Research and Administration, College of Public Health, University of Nebraska Medical Center, Omaha. David Palm is with the Office of Community and Rural Health, Division of Public Health, Nebraska Department of Health and Human Services, Lincoln. Correspondence should be sent to Li-Wu Chen, MHSA, PhD, University of Nebraska Medical Center, College of Public Health, 984350 Nebraska Medical Center, Omaha, NE 68198-4350 (e-mail: [email protected]). Reprints can be ordered at http://www.ajph.org by clicking the “Reprints” link. This article was accepted August 19, 2014.
No. of QI techniques used in past y None (Ref)
1.00
‡ 1 QI techniques
0.84 (0.55, 1.28)
1.00 3.41** (1.96, 5.95)
Dedicated QI staff No, LHD does not have dedicated QI staff (Ref)
1.00
1.00
Yes, LHD has dedicated QI staff
1.33 (1.00, 1.76)
1.27 (0.94, 1.71)
No, LHD does not have staff formally trained in QI (Ref)
1.00
1.00
Yes, LHD has staff formally trained in QI
1.10 (0.79, 1.52)
1.17 (0.79, 1.72)
1.00
1.00
Staff formally trained in QI
Number of performance related activities None (Ref) ‡ 1 performance-related activities
2.79** (2.12, 3.68)
1.29 (0.96, 1.74)
Contributors
Agencywide strategic plan No, LHD has not developed plan (Ref) Yes, LHD has developed plan
1.00
1.00
1.12 (0.85, 1.47)
1.05 (0.78, 1.41)
L.-W. Chen was principal investigator for the study; conceptualized, designed, and implemented the study; and developed the article. A. Nguyen conducted data analysis for the study and contributed to the development of the method section. J. J. Jacobson, N. Gupta, S. Bekmuratova, and D. Palm all contributed to survey instrument development, study implementation, and article development.
LHD characteristics Population size < 25 000 (Ref)
1.00
25 000–49 999
0.81 (0.56, 1.16)
1.00 0.46** (0.31, 0.70)
50 000–99 999
1.12 (0.74, 1.68)
0.96 (0.63, 1.47)
100 000–249 999
1.18 (0.79, 1.76)
0.74 (0.49, 1.10)
250 000–499 999
0.96 (0.51, 1.79)
0.26** (0.13, 0.53)
‡ 500 000
1.10 (0.63, 1.93)
1.45 (0.84, 2.49)
Single county or city (Ref)
1.00
1.00
Multicounty or multicity (i.e., regional)
1.29 (0.83, 2.01)
1.42 (0.91, 2.22)
1.00
1.00
Acknowledgments Funding for this study was provided by Robert Wood Johnson Foundation. We thank Sue Nardie for her help with editing this article. We also thank the Public Health Practice-Based Research Network National Coordinating Center for their technical assistance support in the implementation of the study. We also thank the local health departments in Nebraska that participated in this study. We also thank the National Association of County and City Health Officials for providing the 2010 National Profile of Local Health Departments data.
Jurisdiction type
Local board of health No (Ref) ‡ 1 local boards of health
1.58* (1.12, 2.23)
1.07 (0.74, 1.55)
Budget cut No, budget is same or greater than previous y (Ref)
1.00
1.00
Yes, budget is less than previous y
0.96 (0.74, 1.25)
0.85 (0.64, 1.14)
0.997* (0.994, 0.999)
0.997* (0.994, 0.999)
Expenditures per capita, $
Human Participant Protection Approval was sought and received from the University of Nebraska Medical Center institutional review board.
Staffing cuts in previous y, % 0.99 (0.96, 1.01)
0.96* (0.93, 0.99)
References
Employee loss through attrition
0.964* (0.934, 0.995)
0.94* (0.90, 0.98)
Employees with reduced hours
0.98** (0.96, 0.99)
1. Lenihan P, Welter C, Chang C, Gorenflo G. The operational definition of a functional local public health agency: the next strategic step in the quest for identity and relevance. J Public Health Manag Pract. 2007;13 (4):357---363.
Employee layoff
Employees on mandatory furlough
0.998 (0.990, 1.000)
0.972* (0.947, 0.998) 0.999 (0.990, 1.000)
Note. CI = confidence interval; LHD = local health department; OR = odds ratio; QI = quality improvement. Source. National Association of County and City Health Officials’ 2010 National Profile of Local Health Departments.14 *P < .05; **P < .001.
Supplement 2, 2015, Vol 105, No. S2 | American Journal of Public Health
2. Corso LC, Landrum LB, Lenaway D, Brooks R, Halverson PK. Building a bridge to accreditation—the role of the National Public Health Performance Standards
Chen et al. | Peer Reviewed | Research and Practice | S301
RESEARCH AND PRACTICE
Program. J Public Health Manag Pract. 2007;13(4): 374---377. 3. Beitsch LM, Thielen L, Mays G, et al. The multistate learning collaborative, states as laboratories: informing the national public health accreditation dialogue. J Public Health Manag Pract. 2006;12(3):217---231. 4. Public Health Accreditation Board. Welcome to the Public Health Accreditation Board. Available at: http:// www.phaboard.org/index.php/about. Accessed July 21, 2011. 5. Menkens AJ, Stone D, Wood BW, Reed JF. Local public health department accreditation: thoughts from North Carolina. J Public Health Manag Pract. 2009; 15(5):443---445. 6. Beitsch LM, Landrum LB, Chang C, Wojciehowski K. Public health laws and implications for a national accreditation program: parallel roadways without intersection? J Public Health Manag Pract. 2007;13(4):383---387. 7. Landrum LB, Bassler E, Polyak G, Edgar M, Giangreco C, Dopkeen J. Local public health certification and accreditation in Illinois: blending the old and the new. J Public Health Manag Pract. 2007;13(4):415---421. 8. Nolan P, Bialek R, Kushion ML, Lenaway D, Hamm MS. Financing and creating incentives for a voluntary national accreditation system for public health. J Public Health Manag Pract. 2007;13(4):378---382. 9. Riley W, Lownik E. Process analysis in local health departments: using quality improvement methods and techniques to identify failure modes. Paper presented at: 10th Annual AcademyHealth Research Meeting; June 15, 2011; Seattle, WA. 10. Public Health Accreditation Board. Standards and Measures Version 1.5. Available at: http://www. phaboard.org/wp-content/uploads/SM-Version-1.5Board-adopted-FINAL-01-24-2014.docx.pdf. Accessed June 9, 2014. 11. Baker SL, Beitsch L, Landrum LB, Head R. The role of performance management and quality improvement in a national voluntary public health accreditation system. J Public Health Manag Pract. 2007;13(4): 427---429. 12. Shah GH, Beatty K, Leep C. Do PHAB accreditation prerequisites predict local health departments’ intentions to seek voluntary national accreditation? Front Public Health Serv Syst Res. 2013;2(3):Article 4. 13. Joly BM, Booth M, Mittal P, Shaler G. Measuring quality improvement in public health: the development and psychometric testing of a QI Maturity Tool. Eval Health Prof. 2012;35(2):119---147. 14. National Association of County and City Health Officials. 2010 National Profile of Local Health Departments. Washington, DC: National Association of County and City Health Officials; 2010.
S302 | Research and Practice | Peer Reviewed | Chen et al.
American Journal of Public Health | Supplement 2, 2015, Vol 105, No. S2
Relationship Between Quality Improvement Implementation and Accreditation Seeking in Local Health Departments Li-Wu Chen, PhD, MHSA, Anh Nguyen, MSPH, Janelle J. Jacobson, MPH, Niodita Gupta, MD, MPH, Sarbinaz Bekmuratova, MS, and David Palm, PhD
Since early in this century, national, state, and local organizations have sought to develop a process for accreditation of local health departments (LHDs). Several attempts have been made to identify public health accreditation criteria at the state and local levels. In 2002, the National Association of County and City Health Officials developed a set of 45 public health standards matched to the 10 essential public health services and associated metrics, which became known as an operational definition of a functional LHD.1 The operational definition served as a base for the development of other accreditation tools. In addition, the National Public Health Performance Standards Program was initiated in 1998 as a collaborative effort of 7 national public health organizations that were coordinated and funded by the Centers for Disease Control and Prevention.2 The National Public Health Performance Standards Program represented an attempt to define public health practice standards for state and local public health systems and governing bodies.2 In 2002, it released 3 assessment instruments framed around the 10 essential public health services.2 These tools provided the basis for a national voluntary accreditation program. Building on these previous efforts, in 2004 the Exploring Accreditation Project was developed as a joint initiative of the National Association of County and City Health Officials, the Association of State and Territorial Health Officials, the National Association of Local Boards of Health, and the American Public Health Association to determine the implications and feasibility of a national public health accreditation system.3 In accordance with the recommendations of the Exploring Accreditation Project Steering Committee, the Public Health Accreditation Board (PHAB) was formed to implement and oversee national
Objectives. We examined the relationship between quality improvement (QI) maturity and accreditation attributes of US local health departments (LHDs), specifically those in Nebraska. Methods. Using 2011 Nebraska LHD QI survey data, we conducted Spearman correlation analyses between QI maturity domains and accreditation attributes. Using the 2010 National Association of County and City Health Officials’ National Profile of LHDs, we conducted logistic regression analyses to examine the relationships between specific QI strategies and attitude toward seeking accreditation. Results. Leaders’ commitment to and length of time engaged in QI were positively associated with LHDs’ general attitude toward seeking accreditation. Use of QI strategies and integration of QI policies and practices were positively associated with LHDs’ confidence in their capacity to obtain accreditation. LHDs that had used at least 1 QI framework and at least 1 QI technique in the past year were more likely to agree that they would seek accreditation within 2 years of the national accreditation program. Conclusions. Experience with and expertise in QI implementation play an important role in LHDs’ decision to seek accreditation, and their accreditationseeking efforts may benefit from prior implementation of systematic QI strategies. (Am J Public Health. 2015;105:S295–S302. doi:10.2105/AJPH.2014. 302278)
public health department accreditation.4 The key focus of the PHAB accreditation program is to strengthen the public health infrastructure.5 Although the process of accreditation can be arduous, it has several benefits to the LHD, including identifying performance improvement areas and improving management, leadership, and relationships within the community. 6 Paralleling the PHAB’s efforts, the Multistate Learning Collaborative on Performance and Capacity Assessment or Accreditation of Public Health Departments worked with states with existing assessment or accreditation as natural laboratories to enhance existing initiatives and disseminate the findings of these mature programs.3 The collaborative’s efforts have yielded valuable information. For instance, some LHDs observed benefits of the accreditation process in the standardization of measures for accountability and the development
Supplement 2, 2015, Vol 105, No. S2 | American Journal of Public Health
of morale among LHD staff members.5 However, other LHDs experienced problems during the accreditation process, including the redundant documentation of standards and difficulty understanding how to meet the documentation standards.3 In addition, some LHDs found that the short period of onsite review and the lack of funding made accreditation difficult.6 Moreover, researchers have noted that marketing accreditation to LHDs without offering incentives could be challenging, and such processes may cause anxiety and reluctance to participate. These challenges are also likely to be exacerbated by the different types and sizes of LHDs.7 Much previous research on accreditation has focused on the role of financing and incentives in voluntary adoption of accreditation standards. Accreditation can be an expensive and time-consuming process for LHDs, but these efforts are rewarded during the accreditation
Chen et al. | Peer Reviewed | Research and Practice | S295
RESEARCH AND PRACTICE
process when the LHD’s quality efforts are documented.8 One of the key features that has been highlighted throughout the literature is the role of quality improvement (QI) in the accreditation process. QI, regardless of where a department stands on accreditation, is an essential activity for all health departments.9 QI has been incorporated as a central concept throughout PHAB’s Standards and Measures Version 1.5,10 with a strong focus on the QI processes used by the LHD. Domain 9 of the PHAB standards and measures focuses on the evaluation of all programs, interventions, and the key public health process, as well as the implementation of a formalized QI process.10 LHDs that have already implemented a continuous QI plan or performance management system can document this existing system and submit it to fulfill this requirement.10 Although QI implementation is essential in LHDs, a number of barriers to its implementation exist, including program requirements and lack of funding, incentives for implementation, leadership, and additional technical assistance resources.11 In addition, the language used to describe QI processes is not uniform across LHDs, making it difficult to study QI activities. More important, although QI activities are essential for accreditation, little has been written about the relationship between QI and accreditation and how LHDs can better integrate QI and accreditation strategies. In 2013, Shah et al.12 examined the relationship between PHAB accreditation prerequisites and LHDs’ intention to seek PHAB accreditation. Although their study provides an understanding of how conducting a community health assessment, community health improvement planning, and strategic planning are associated with LHDs’ intent to seek accreditation, the specific characteristics of QI implementation and the effect of implementation on LHDs’ intention to seek accreditation have not been examined. We examined the relationship between QI maturity and strategies and accreditation attributes in the United States, specifically Nebraska.
METHODS We used different methods and data sources for the analysis of US and Nebraska LHDs, respectively. They are described as follows.
Data Sources For the Nebraska sample, we used 2011 Nebraska LHD QI survey data. This survey was electronically administered by our research team to all 21 LHD directors in Nebraska from May 2011 to August 2011. A total of 19 LHD directors (90.5% of the sample) responded to the online survey. We designed the survey instrument using the QI taxonomy developed by the University of Minnesota, input from the Nebraska Public Health Practice---Based Research Network Steering Committee, and questions adapted from the Multistate Learning Collaborative on Performance and Capacity Assessment or Accreditation of Public Health Departments 2011 Annual Survey, designed by the University of Southern Maine.9,13 The questions covered QI maturity domains and accreditation attributes (attitudes, beliefs, and readiness).13 Respondents were asked to rate the level to which they agreed or disagreed with statements on a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree). Measuring QI maturity within the public health department setting is a fairly new concept and is based on 4 domains, including organizational culture, capacity and competency, practice, and alignment and spread.13 Organizational culture refers to the QI values and norms that influence the internal and external interactions at the agency. The 2 organizational culture dimensions include commitment (a desire to adopt new ideas and improve services) and collaboration (a mutually supportive work environment in which QI can thrive). Capacity and competency refer to the skills, functions, and approaches used for QI. The 3 capacity and competency dimensions include skills (training in basic QI methods), methods (application of QI tools, approaches, and data), and investment (designated and established QI staff and priorities). Alignment and spread refer to the extent to which QI is supported and diffused within the organization. The 4 alignment and spread dimensions include integration (agency policies and practices that support QI), authority (staff ability to make and implement QI decisions), value (the perception that QI is an important investment), and implementation (the ease of implementing QI).13 For the US sample, we obtained data from National Association of County and City Health
S296 | Research and Practice | Peer Reviewed | Chen et al.
Officials’ 2010 National Profile of LHDs.14 The 2010 National Profile of LHDs included data from a set of core questions that was sent to all LHDs in the United States and from supplemental questions that were grouped into 2 modules and randomly sent to select LHDs. We used data on governance, funding, population size, and staffing from the core questions and data on QI strategies and accreditation attitudes from module 1. We used an indicator from module 1 to select only LHDs that reported having implemented formal or informal QI activities. We included 449 of 531 LHD respondents from module 1 in the analysis. Estimation weights were provided to account for sampling and to obtain national estimates for all LHDs in the United States.
Data Analysis Using 2011 Nebraska LHD QI survey data, we created measures for QI maturity domains and their respective dimensions by summing the scores for the items that fell within that domain or dimension. We used reverse coding as appropriate so that a higher score was better, and we treated “don’t know” responses as missing. We measured QI practice by whether an LHD had ever implemented a QI project, how many QI projects had been implemented in the past 12 months, how long an LHD had engaged in QI efforts, and the extent to which an LHD had used QI models, techniques, and metrics. We used Spearman correlation analyses, a nonparametric correlation technique often used when 1 or both variables are ordinal or not normally distributed or when the sample size is small, to analyze the relationship between QI maturity (organizational culture, capacity and competency, practice, and alignment and spread) and accreditation attitudes, beliefs, and readiness for Nebraska LHDs. Using the National Association of County and City Health Officials’ 2010 National Profile of LHDs, we created the following QI strategy-related measures: whether an LHD had implemented at least 1 formal QI project in the past year, whether an LHD had used at least 1 QI framework (e.g., Baldrige, Lean, Six Sigma) in the past year, whether an LHD had used at least 1 QI technique (e.g., process map, root-cause analysis) in the past year, whether an LHD had dedicated QI staff, and whether LHD staff had received any formal training in QI
American Journal of Public Health | Supplement 2, 2015, Vol 105, No. S2
RESEARCH AND PRACTICE
methods. We created 2 accreditation attitude measures to indicate whether an LHD would seek national accreditation and whether an LHD would seek national accreditation within the first 2 years of the program. We conducted multivariable logistic regression analyses to examine the association between QI strategies and accreditation attitudes among LHDs that had implemented QI activities in the United States. Specifically, we examined how implementing particular QI strategies affected an LHD’s likelihood of wanting to seek accreditation in general and wanting to seek accreditation within 2 years, controlling for an LHD’s previous involvement in performance-related activities (e.g., MAPP (Mobilizing for Action through Planning and Partnerships), Turning Point), the presence of an agency-wide strategic plan, the size of the total population served by an LHD, jurisdiction type, governance (i.e., the presence of local board of health), budget cuts, per capita expenditures, and staffing cuts. We analyzed all data using SAS version 9.3 (SAS Institute Inc., Cary, NC).
RESULTS Table 1 shows the levels of QI maturity and accreditation attributes in Nebraska’s LHDs in 2011. Median scores at the upper end or above the midpoint value of the possible score range are considered favorable. The median score for organizational culture was 32 (of 35), suggesting that Nebraska’s LHDs were highly mature in fostering an environment with QI embedded in organizational values and norms. The median score for capacity and competency was 37 (of 50), suggesting that Nebraska’s LHDs struggled with applying appropriate QI strategies and lacked the capacity to implement QI. The median score for alignment and spread was 36 (of 55), suggesting that Nebraska’s LHDs had many opportunities to improve policies and practices that support QI and perceptions about implementing QI. The majority (83.3%) of responding LHDs indicated that their agency had engaged in implementing a QI process. A median of 3 formal projects had been implemented by Nebraska LHDs within the past 12 months. About half (47.4%) of responding LHDs indicated that their QI efforts had been consistent for more than 5 years. More than half (53.3%) indicated that their LHD had
TABLE 1—Quality Improvement Maturity and Accreditation Attitudes, Beliefs, and Readiness in Local Health Departments (n = 19): Nebraska, 2011 QI Maturity Domain and Dimensions
No. of Items
Score Range
%
Mean
Median
7
7–35
...
31.4
32.0
Commitment
3
3–15
...
13.5
14.0
Collaboration
4
4–20
...
18.0
18.0
10
10–50
...
36.5
37.0
Skills Methods
2 6
2–10 6–30
... ...
7.7 22.6
8.0 22.0
Investment
2
2–10
...
6.1
6.0
Ever implemented QI: yes
...
...
83.3
...
...
Ever implemented QI: no
...
...
16.7
...
...
No. of projects, past 12 mo
...
...
...
3.8
3.0
No systematic QI efforts in place
...
...
15.8
...
...
Engaged in QI < 1 y Engaged in QI 1–2 y
... ...
... ...
5.3 21.1
... ...
... ...
Engaged in QI 3–4 y
...
...
10.5
...
...
Engaged in QI ‡ 5 y
...
...
47.4
...
...
No use of QI strategies
...
...
13.3
...
...
Use of QI model, technique, or metric
...
...
13.3
...
...
Use of combination of QI model and technique,
...
...
20.0
...
...
... 11
... 11–55
53.3 ...
... 37.3
... 36.0
Integration
8
8–40
...
27.2
26.5
Authority
1
1–5
...
4.0
4.0
Value
1
1–5
...
4.3
4.0
1
1–5
...
1.8
2.0
1–5 1–5
... ...
3.9 3.0
4.0 3.0
Organizational culture
Capacity and competency
Practice
QI model and metric, or QI technique and metric Use of QI model, technique, and metric Alignment and spread
Implementation
Accreditation measuresa Attitude toward accreditation Agency will seek accreditation Agency will seek accreditation within first 2 y
1 1
Belief in accreditation Believe that national standards are a good idea
1
1–5
...
4.1
4.0
Believe accreditation will strengthen agency
1
1–5
...
3.9
4.0
Confident in agency’s capacity to obtain accreditation
1
1–5
...
2.9
3.0
Agency has begun preparing for accreditation
1
1–5
...
3.8
4.0
Perceived readiness for accreditation
Note. QI = quality improvement. Source. 2011 Nebraska Local Health Department Quality Improvement Survey. a The scale for each item was based on a 5-point Likert scale (1 = strongly disagree to 5 = strongly agree). We calculated QI maturity domain and dimension scores by summing the scores for the items that fell within that domain or dimension.
used all 3 types of QI strategies (models, techniques, and metrics). Items related to accreditation (attitudes, beliefs, and readiness) were rated using a 5-point Likert scale (5 = strongly agree, 4 = agree, 3 = neither agree nor disagree, 2 = disagree, 1 = strongly disagree). Therefore, on the basis
Supplement 2, 2015, Vol 105, No. S2 | American Journal of Public Health
of this scale and using the median scores of all respondents, the results suggest that although LHD directors in Nebraska generally agreed that their agency would seek national accreditation (median = 4), they were unsure whether accreditation would be sought within the first 2 years of the program (median = 3). LHD
Chen et al. | Peer Reviewed | Research and Practice | S297
RESEARCH AND PRACTICE
(r = .60; P = .008). Integration of QI policies and practices (r = .58; P = .014) and the alignment and spread of QI (r = .53; P = .037) within an LHD were positively associated with the belief that accreditation would strengthen the agency. The ability to apply QI methods was also positively associated with the belief that accreditation would strengthen the agency; however, the association was marginally significant (r = .47; P = .049). LHD leaders’ commitment to QI was also positively associated with the agency’s having begun to prepare for accreditation (r = .57; P = .015). The use of QI strategies (r = .57; P = .033) and the integration of QI policies and practices (r = .50; P = .049) within the agency were positively associated with an LHD’s confidence in its capacity to obtain accreditation. Table 3 shows the characteristics of US LHDs that have implemented QI activities. In 2010, an estimated 84.4% of all US LHDs indicated that their LHD had implemented some formal or informal QI activities. More than one third (39.3%) of these LHDs served
directors in Nebraska generally believed that national standards were a good idea and would strengthen their agencies (median = 4). Although LHDs in Nebraska had generally begun preparation for accreditation (median = 4), respondents were unsure whether their agency had sufficient capacity to obtain accreditation (median = 3). Table 2 shows the results of the correlation analysis between QI maturity and accreditation attributes for Nebraska LHDs. The length of time engaged in QI (r = .58; P = .014) was positively associated with an LHD’s general attitude toward seeking accreditation. A commitment to QI among leaders was also positively associated with an LHD’s general attitude toward seeking accreditation; however, the association was marginally significant (r = .49; P = .049). The use of QI strategies was positively associated with an LHD’s attitude toward seeking accreditation within the first 2 years of the program (r = .76; P = .003). Valuing QI was positively associated with the belief that national standards were a good idea
a population of less than 25 000. The majority were also LHDs that served a single county or city (89.5%) and had at least 1 local board of health (78.2%). Almost one half (47.7%) of these LHDs experienced budget cuts, and the estimated average percentages of employees who were laid off, who were lost through attrition, whose work hours were reduced, and who were placed on mandatory furlough in the previous year were, respectively, 1.9%, 3.2%, 2.8%, and 9.2%. In 2010, 73.1% of LHDs nationwide had implemented at least 1 formal QI project, 38.7% had used at least 1 QI framework, and 78.3% had used at least 1 QI technique in the past year. Also, 31.6% of the LHDs had dedicated QI staff, 74.0% had staff members who had received formal QI training, 49.8% had recently been involved in at least 1 performance-related activity, and 42.9% had developed an agency-wide strategic plan. More than one half (54.3%) of the LHDs agreed that they would seek national accreditation, whereas only one third (33.0%) agreed that their LHD would seek national
TABLE 2—Results From Spearman Correlation Analysis Between Quality Improvement Maturity and Accreditation Attributes of Local Health Departments: Nebraska, 2011 Agency Would Seek Accreditation QI Maturity Organizational culture Commitment Collaboration Capacity and competency
Agency Would Seek Accreditation Within First 2 Y
r
P
r
.16
.54
.49
.049
–.02 .03
.944 .914
National Standards Are a Good Idea
Accreditation Would Strengthen Agency
P
r
P
r
–.02
.938
.15
.543
.17
.22
.417
.46
.056
.35
.02 .30
.946 .306
–.01 .10
.957 .72
.08 .33
P
Confident in Agency’s Capacity to Obtain Accreditation
Agency Has Begun Preparing for Accreditation
r
P
r
P
.5
.22
.402
.37
.139
.151
.30
.246
.57
.015
.764 .216
.18 .33
.482 .231
.18 –.30
.478 .258
Skills
.34
.211
.32
.285
.03
.911
.09
.74
.11
.694
–.35
.165
Methods
.15
.561
–.04
.889
.08
.763
.47
.049
.47
.055
–.07
.794
Investment
.24
.381
.46
.102
.05
.866
.06
.828
.24
.398
–.36
.171
QI practice Ever implemented QI
.33
.209
.47
.081
.37
.145
.35
.168
.47
.067
.04
.879
No. of projects, past 12 mo
.30
.471
.49
.264
.29
.481
.61
.112
.29
.522
.10
.822
Length of time engaged in QI QI strategy use
.58 .37
.014 .367
.36 .76
.174 .003
.37 .51
.135 .053
.44 .37
.065 .178
.35 .57
.168 .033
.44 .30
.066 .274 .975
Alignment and spread
.22
.437
.39
.167
.27
.308
.53
.037
.44
.098
–.01
Integration
.25
.355
.37
.169
.29
.268
.58
.014
.50
.049
.08
.753
Authority
.09
.748
.16
.575
.13
.619
.03
.917
.05
.854
–.32
.217
Value Implementation
.36
.154
.23
.401
.60
.008
.31
.211
.32
.216
.27
.279
–.27
.29
.00
‡ .999
–.38
.124
–.22
.373
–.20
.441
–.21
.396
Note. QI = quality improvement. P values at less than .05 were considered significant. Source. 2011 Nebraska Local Health Department Quality Improvement Survey.
S298 | Research and Practice | Peer Reviewed | Chen et al.
American Journal of Public Health | Supplement 2, 2015, Vol 105, No. S2
RESEARCH AND PRACTICE
TABLE 3—Characteristics of Local Health Departments That Have Implemented Quality Improvement Activities: United States, 2010 No.
% (95% CI)a or Meana 6SE (Median)
449
100.0
< 25 000 25 000–49 999
122 88
39.3 (34.3, 44.3) 20.9 (17.0, 24.8)
50 000–99 999
66
15.0 (11.7, 18.4)
100 000–249 999
72
13.2 (10.2, 16.2)
250 000–499 999
42
5.7 (4.0, 7.4)
‡ 500 000
59
5.9 (4.3, 7.4)
LHD Characteristics Total LHDs Population size
Jurisdiction type Single county or city
390
89.5 (86.8, 92.2)
59
10.5 (7.8, 13.2)
Multicounty or multicity (i.e., regional) Local board of health No
111
21.8 (17.9, 25.7)
‡ 1 local boards of health
336
78.2 (74.3, 82.1)
Budget cut Yes, budget is less than previous y
191
47.7 (42.6, 52.9)
No, budget is same or greater than previous y
211
52.3 (47.1, 57.4)
375
57.5 63.4 (40.7)
Expenditures per capita, $ Staffing cuts in previous y, % Employee layoff Employee loss through attrition
409
1.9 60.3 (0)
408
3.2 60.3 (0)
Employees with reduced hours
401
2.8 60.7 (0)
Employees on mandatory furlough
408
9.2 61.3 (0)
QI characteristics No. of formal QI activities in past y None ‡ 1 projects No. of QI frameworks used in past y
106
26.9 (22.4, 31.4)
339
73.1 (68.6, 77.6)
None
252
61.3 (56.5, 66.1)
‡ 1 QI frameworks
179
38.7 (33.9, 43.5)
No. of QI techniques used in past y None
82
21.7 (17.5, 25.9)
357
78.3 (74.1, 82.5)
291 150
68.4 (63.9, 72.9) 31.6 (27.1, 36.1)
No, LHD does not have staff formally trained in QI
102
26.0 (21.6, 30.5)
Yes, LHD has staff formally trained in QI
340
74.0 (69.5, 78.4)
‡ 1 QI techniques Dedicated QI staff No, LHD does not have dedicated QI staff Yes, LHD has dedicated QI staff Staff formally trained in QI
No. of performance-related activities None
196
50.2 (45.2, 55.3)
‡ 1 performance-related activities
228
49.8 (44.7, 54.8)
Agency-wide strategic plan No, LHD has not developed plan
237
57.1 (52.3, 61.9)
Yes, LHD has developed plan
210
42.9 (38.1, 47.7) Continued
Supplement 2, 2015, Vol 105, No. S2 | American Journal of Public Health
accreditation within the first 2 years of the program. Table 4 shows the results of the regression analyses. LHDs that had implemented at least 1 formal QI project in the past year were 0.50 times more likely (95% CI = 1.03, 2.19) to agree that their LHD would seek accreditation than LHDs that had not implemented any formal QI project in the past year. LHDs that had recently been involved in at least 1 performance-related activity were 1.79 times more likely (95% CI = 2.12, 3.68) to agree that their LHD would seek accreditation than LHDs that had not been involved in any performance-related activities. LHDs with at least 1 local board of health were also 0.58 times more likely (95% CI = 1.12, 2.23) to agree that their LHD would seek accreditation than LHDs without a local board of health. LHDs with higher per capita expenditures (odds ratio [OR] = 0.997; 95% CI = 0.994, 0.999), a higher percentage of employees lost through attrition in the previous year (OR = 0.964; 95% CI = 0.934, 0.995), and a higher percentage of employees with reduced hours in the previous year (OR = 0.98; 95% CI = 0.96, 0.99) were less likely to agree that their LHD would seek accreditation. LHDs that had used at least 1 QI framework (OR = 1.94; 95% CI = 1.43, 2.63) and at least 1 QI technique (OR = 3.41; 95% CI = 1.96, 5.95) in the past year were 0.94 and 2.41 times more likely, respectively, to agree that their LHD would seek accreditation within 2 years of the program. LHDs serving a population of 25 000 to 49 999 (OR = 0.46; 95% CI = 0.31, 0.70) as well as those serving a population of 250 000 to 499 999 (OR = 0.26; 95% CI = 0.13, 0.53) were less likely to agree that their LHD would seek accreditation within 2 years than LHDs serving a population of less than 25 000. LHDs with higher per capita expenditures (OR = 0.997; 95% CI = 0.994, 0.999), a higher percentage of employees laid off in the previous year (OR = 0.96; 95% CI = 0.93, 0.99), a higher percentage of employees lost through attrition in the previous year (OR = 0.94; 95% CI = 0.90, 0.98), and a higher percentage of employees with reduced hours in the previous year (OR = 0.972; 95% CI = 0.947, 0.998) were also less likely to agree that their LHD would seek accreditation within 2 years.
Chen et al. | Peer Reviewed | Research and Practice | S299
RESEARCH AND PRACTICE
TABLE 3—Continued Seek accreditation under voluntary national program LHD would not seek accreditation
173
45.7 (40.5, 50.9)
LHD would seek accreditation
227
54.3 (49.1, 59.5)
LHD would not seek accreditation £ first 2 y
259
67.0 (62.1, 71.9)
LHD would seek accreditation £ first 2 y
135
33.0 (28.1, 37.9)
Seek accreditation under voluntary national program £ first 2 y of program
Note. CI = confidence interval; LHD = local health department; QI = quality improvement. Source. National Association of County and City Health Official’s 2010 National Profile of Local Health Departments.14 a Weighted values.
DISCUSSION This study has implications for how policymakers and public health practitioners may better integrate QI and accreditation strategies in the LHD setting. Our results from the Nebraska analysis suggest that QI maturity is generally positively associated with an LHD’s attitude toward, beliefs about, and readiness for seeking national accreditation. In particular, an LHD in which QI methods are better applied and QI practice is better supported and diffused through policies and procedures tends to believe that accreditation would strengthen the agency. Not surprisingly, LHDs that value QI more highly tend to believe that national standards are a good idea. These results suggest that the nature of the pursuit of QI and accreditation share significant common ground so that the benefits of accreditation are better recognized by LHDs that value and support QI more. Our results also indicate that LHDs with leaders who have a higher commitment to QI and in which QI engagement has been in place longer tended to express a greater desire to seek accreditation. Nevertheless, only the LHDs that had actually used more formal QI strategies expressed a greater desire to seek accreditation within 2 years. These results highlight the important role of actual QI implementation and suggest that a greater desire to seek accreditation may result from a greater number of QI experiences and greater QI expertise. In addition, our results indicate that LHDs that have used more formal QI strategies and have better supported QI through policies and procedures have higher confidence in their ability to obtain accreditation. These results further suggest that the actual experience of implementing QI
systematically not only increases an LHD’s desire to seek accreditation but also enhances its perceived readiness to obtain accreditation. The results from the Nebraska analysis are based on bivariate analyses without controlling for potential confounding factors; therefore, the regression analysis using the US sample was conducted to remedy that limitation. Our results from the US analysis confirmed the positive relationship between QI strategy use and accreditation attitude. In particular, the LHDs that had implemented formal QI projects in the past year, had prior experience in other performance-related activities, or both were more likely to express a desire to seek accreditation. However, only LHDs that had actually adopted a formal QI framework or formal QI techniques in the past year were more likely to express a desire to seek accreditation within the next 2 years. These results further highlight the importance of an LHD’s experiences and expertise accumulated through actual QI implementation in its decision to seek accreditation. In addition, prior experience using a more systematic QI approach (i.e., specific QI frameworks or techniques rather than general QI project experience or performance-related activities) significantly enhanced an LHD’s attitude toward seeking accreditation in the short term. Our results on other control variables indicate that LHDs with a local board of health were more likely to express a desire to seek accreditation generally than their counterparts without such governance. This finding is consistent with the conventional wisdom that LHDs with a local board of health may be under greater pressure to pursue accreditation. Nevertheless, our study suggested that the presence of a local board of health does not
S300 | Research and Practice | Peer Reviewed | Chen et al.
increase the likelihood of an LHD’s expressing a desire to seek accreditation in the short term. This may suggest that the impetus for an LHD’s seeking accreditation in the short term is driven more by internal forces (e.g., prior QI experiences and expertise) than by external forces. In addition, LHDs that had experienced more staffing cuts in the previous year were less likely to express a desire to seek accreditation (both in general and within 2 years) than their counterparts with fewer staffing cuts. Given that many LHDs perceive the PHAB accreditation process to be time and resource consuming, the LHDs that had experienced more recent staffing cuts would face more challenges in moving the accreditation process forward while meeting their daily operational needs. Our results also indicate that LHDs with higher per capita expenditures were less likely to express a desire to seek accreditation. Although this result was inconsistent with our expectation, its effect size was very marginal. LHDs serving a population of 25 000 to 49 999 or 250 000 to 499 999 persons were less likely to express a desire to seek accreditation within 2 years than LHDs serving a population of less than 25 000 persons. Because our results did not demonstrate a clear relationship pattern between population size and accreditation attitude, future research is needed to explore this area further. Our study suggests that an LHD’s QI maturity and strategies and its attitude toward seeking national accreditation are positively related. LHDs may benefit from implementing QI activities (especially adopting more formal and systematic QI strategies) in their seeking national accreditation. Given the similar nature of QI and accreditation activities, LHDs may benefit from the scope economies between these 2 types of tasks. Early QI adopters may also benefit from using their greater momentum and culture of seeking excellence, better established and documented policies and procedures, more supportive and collaborative working environment, more efficient operation, and greater QI experience and expertise during their process of seeking national accreditation. Although considerable staff time and cost associated with seeking accreditation can be a challenge for many LHDs, seeking accreditation can be an incentive for LHDs to establish a formal continuous QI plan to improve their
American Journal of Public Health | Supplement 2, 2015, Vol 105, No. S2
RESEARCH AND PRACTICE
TABLE 4—Logistic Regression Results on the Likelihood of Local Health Departments Expressing a Desire to Seek National Accreditation: United States, 2010 LHD Would Seek Accreditation, OR (95% CI)
Characteristic
LHD Would Seek Accreditation £ 2 Years, OR (95% CI)
Quality improvement characteristics No. of formal QI activities in past y None (Ref)
1.00
‡ 1 projects
1.50* (1.03, 2.19)
1.00 0.67 (0.43, 1.02)
programs and services. Overall, LHDs would be inclined to seek accreditation for a variety of reasons, including improving performance, establishing policies and procedures, and enhancing accountability to the community. Policymakers should develop policies to incentivize late QI-adopting LHDs to implement formal QI techniques and strategies to facilitate their accreditation seeking and improve their practice. j
No. of QI frameworks used in past y None (Ref)
1.00
‡ 1 QI frameworks
1.26 (0.94, 1.69)
1.00
About the Authors
1.94** (1.43, 2.63)
Li-Wu Chen, Anh Nguyen, Janelle J. Jacobson, Niodita Gupta, and Sarbinaz Bekmuratova are with the Department of Health Services Research and Administration, College of Public Health, University of Nebraska Medical Center, Omaha. David Palm is with the Office of Community and Rural Health, Division of Public Health, Nebraska Department of Health and Human Services, Lincoln. Correspondence should be sent to Li-Wu Chen, MHSA, PhD, University of Nebraska Medical Center, College of Public Health, 984350 Nebraska Medical Center, Omaha, NE 68198-4350 (e-mail: [email protected]). Reprints can be ordered at http://www.ajph.org by clicking the “Reprints” link. This article was accepted August 19, 2014.
No. of QI techniques used in past y None (Ref)
1.00
‡ 1 QI techniques
0.84 (0.55, 1.28)
1.00 3.41** (1.96, 5.95)
Dedicated QI staff No, LHD does not have dedicated QI staff (Ref)
1.00
1.00
Yes, LHD has dedicated QI staff
1.33 (1.00, 1.76)
1.27 (0.94, 1.71)
No, LHD does not have staff formally trained in QI (Ref)
1.00
1.00
Yes, LHD has staff formally trained in QI
1.10 (0.79, 1.52)
1.17 (0.79, 1.72)
1.00
1.00
Staff formally trained in QI
Number of performance related activities None (Ref) ‡ 1 performance-related activities
2.79** (2.12, 3.68)
1.29 (0.96, 1.74)
Contributors
Agencywide strategic plan No, LHD has not developed plan (Ref) Yes, LHD has developed plan
1.00
1.00
1.12 (0.85, 1.47)
1.05 (0.78, 1.41)
L.-W. Chen was principal investigator for the study; conceptualized, designed, and implemented the study; and developed the article. A. Nguyen conducted data analysis for the study and contributed to the development of the method section. J. J. Jacobson, N. Gupta, S. Bekmuratova, and D. Palm all contributed to survey instrument development, study implementation, and article development.
LHD characteristics Population size < 25 000 (Ref)
1.00
25 000–49 999
0.81 (0.56, 1.16)
1.00 0.46** (0.31, 0.70)
50 000–99 999
1.12 (0.74, 1.68)
0.96 (0.63, 1.47)
100 000–249 999
1.18 (0.79, 1.76)
0.74 (0.49, 1.10)
250 000–499 999
0.96 (0.51, 1.79)
0.26** (0.13, 0.53)
‡ 500 000
1.10 (0.63, 1.93)
1.45 (0.84, 2.49)
Single county or city (Ref)
1.00
1.00
Multicounty or multicity (i.e., regional)
1.29 (0.83, 2.01)
1.42 (0.91, 2.22)
1.00
1.00
Acknowledgments Funding for this study was provided by Robert Wood Johnson Foundation. We thank Sue Nardie for her help with editing this article. We also thank the Public Health Practice-Based Research Network National Coordinating Center for their technical assistance support in the implementation of the study. We also thank the local health departments in Nebraska that participated in this study. We also thank the National Association of County and City Health Officials for providing the 2010 National Profile of Local Health Departments data.
Jurisdiction type
Local board of health No (Ref) ‡ 1 local boards of health
1.58* (1.12, 2.23)
1.07 (0.74, 1.55)
Budget cut No, budget is same or greater than previous y (Ref)
1.00
1.00
Yes, budget is less than previous y
0.96 (0.74, 1.25)
0.85 (0.64, 1.14)
0.997* (0.994, 0.999)
0.997* (0.994, 0.999)
Expenditures per capita, $
Human Participant Protection Approval was sought and received from the University of Nebraska Medical Center institutional review board.
Staffing cuts in previous y, % 0.99 (0.96, 1.01)
0.96* (0.93, 0.99)
References
Employee loss through attrition
0.964* (0.934, 0.995)
0.94* (0.90, 0.98)
Employees with reduced hours
0.98** (0.96, 0.99)
1. Lenihan P, Welter C, Chang C, Gorenflo G. The operational definition of a functional local public health agency: the next strategic step in the quest for identity and relevance. J Public Health Manag Pract. 2007;13 (4):357---363.
Employee layoff
Employees on mandatory furlough
0.998 (0.990, 1.000)
0.972* (0.947, 0.998) 0.999 (0.990, 1.000)
Note. CI = confidence interval; LHD = local health department; OR = odds ratio; QI = quality improvement. Source. National Association of County and City Health Officials’ 2010 National Profile of Local Health Departments.14 *P < .05; **P < .001.
Supplement 2, 2015, Vol 105, No. S2 | American Journal of Public Health
2. Corso LC, Landrum LB, Lenaway D, Brooks R, Halverson PK. Building a bridge to accreditation—the role of the National Public Health Performance Standards
Chen et al. | Peer Reviewed | Research and Practice | S301
RESEARCH AND PRACTICE
Program. J Public Health Manag Pract. 2007;13(4): 374---377. 3. Beitsch LM, Thielen L, Mays G, et al. The multistate learning collaborative, states as laboratories: informing the national public health accreditation dialogue. J Public Health Manag Pract. 2006;12(3):217---231. 4. Public Health Accreditation Board. Welcome to the Public Health Accreditation Board. Available at: http:// www.phaboard.org/index.php/about. Accessed July 21, 2011. 5. Menkens AJ, Stone D, Wood BW, Reed JF. Local public health department accreditation: thoughts from North Carolina. J Public Health Manag Pract. 2009; 15(5):443---445. 6. Beitsch LM, Landrum LB, Chang C, Wojciehowski K. Public health laws and implications for a national accreditation program: parallel roadways without intersection? J Public Health Manag Pract. 2007;13(4):383---387. 7. Landrum LB, Bassler E, Polyak G, Edgar M, Giangreco C, Dopkeen J. Local public health certification and accreditation in Illinois: blending the old and the new. J Public Health Manag Pract. 2007;13(4):415---421. 8. Nolan P, Bialek R, Kushion ML, Lenaway D, Hamm MS. Financing and creating incentives for a voluntary national accreditation system for public health. J Public Health Manag Pract. 2007;13(4):378---382. 9. Riley W, Lownik E. Process analysis in local health departments: using quality improvement methods and techniques to identify failure modes. Paper presented at: 10th Annual AcademyHealth Research Meeting; June 15, 2011; Seattle, WA. 10. Public Health Accreditation Board. Standards and Measures Version 1.5. Available at: http://www. phaboard.org/wp-content/uploads/SM-Version-1.5Board-adopted-FINAL-01-24-2014.docx.pdf. Accessed June 9, 2014. 11. Baker SL, Beitsch L, Landrum LB, Head R. The role of performance management and quality improvement in a national voluntary public health accreditation system. J Public Health Manag Pract. 2007;13(4): 427---429. 12. Shah GH, Beatty K, Leep C. Do PHAB accreditation prerequisites predict local health departments’ intentions to seek voluntary national accreditation? Front Public Health Serv Syst Res. 2013;2(3):Article 4. 13. Joly BM, Booth M, Mittal P, Shaler G. Measuring quality improvement in public health: the development and psychometric testing of a QI Maturity Tool. Eval Health Prof. 2012;35(2):119---147. 14. National Association of County and City Health Officials. 2010 National Profile of Local Health Departments. Washington, DC: National Association of County and City Health Officials; 2010.
S302 | Research and Practice | Peer Reviewed | Chen et al.
American Journal of Public Health | Supplement 2, 2015, Vol 105, No. S2
