J Clin Epidemiol Vol.44, No. 7, Printedin Great Britain

pp. 685499,

0895-4356/91 163.00 + 0.00 PergamonPressplc

1991

THE EVALUATION OF THE HENRY J. KAISER FAMILY FOUNDATION’S COMMUNITY HEALTH PROMOTION GRANT PROGRAM: DESIGN EDWARD H. WAGNER,‘,* THOMAS D. KOEPSELL,*~~ CAROLYN ANDERMAN,’ BRUCE M. PsATY,“~,~ ALLEN CHEADLE,*SUSAN G. CURRY,‘~* MICHAEL VON KORFF, I**THOMAS M. WICKIZER,* WILLIAM L. BEERY,’ PAULA K. DIEHR,~~*JENIFER L. EHRETH,* BARBARA H. K_EHRER,’

DAVIDC. PEARSON’ and EDWARDB. PERRIN* ‘Center for Health Studies, Group Health Cooperative of Puget Sound, Seattle, WA 98101, 2Department of Health Services, University of Washington, ‘Department of Epidemiology, University of Washington, 4Department of Medicine, Harborview Medical Center-University of Washington, Seattle, WA 98195, ‘Center for Health Promotion, Group Health Cooperative of Puget Sound, 6Department of Biostatistics, University of Washington, Seattle, WA 98195 and ‘Henry J. Kaiser Family Foundation, Menlo Park, California, U.S.A. (Received in revised form 26 September 1990)

Abstract-The Kaiser Family Foundation’s Community Health Promotion Grant Program (CHPGP) provides funding and technical assistance in support of communitybased efforts to prevent major health problems. The first phase of the program was implemented in 11 communities in the western United States. This paper describes the evaluation design of the CHPGP in the West, the methods of data collection, and the baseline comparability of intervention and control communities. Major features of the evaluation design include: (1) the randomization of qualified communities making application into funded and unfunded comparison groups; (2) a second set of matched control communities for some intervention sites; (3) data gathering through repeated surveys of community residents (probability samples of adults and adolescents) and institutions (health-related organizations and randomly sampled grocery stores and restaurants); and (4) the use of secondary data to monitor health events. Selected baseline data show that intervention and control communities differ in racial/ethnic composition, but relevant health behaviors and ratings of community activation for health promotion appear comparable. Community

health promotion

Evaluation

THE COMMUNITYHEALTH PROMOTION GRANT PROGRAM

methods

Standard Three-Community and Five-City Studies [l-3] and the North Karelia Project [4-6]. There is also evidence of reduced cardiovascular morbidity and mortality rates in North Karelia, but whether this is attributable to the program remains uncertain [7]. The high costs of cardiovascular prevention interventions directed at high risk individuals and the promising results of community-wide efforts have convinced many experts [8,9] that the latter may be

Community health promotion programs attempt to reduce disease risk in the population through mass media messages, environmental modifications and other interventions aimed at the public at large, rather than through services delivered selectively to high risk individuals. This approach may have achieved beneficial effects on cardiovascular risk factors in the 685

686

EDWARDH. WAGNERet al.

“potentially far more effective and ultimately the only acceptable answer” [lo]. On this basis, the Henry J. Kaiser Family Foundation embarked on a major initiative to foster community health promotion activities directed at five leading health problems: cardiovascular disease, cancer, substance abuse, adolescent pregnancy, and injuries [l 13. An important element of this effort is the Community Health Promotion Grant Program (CHPGP). The CHPGP supports, through grants and technical assistance, community-based programs designed to promote healthier habits of living. The first phase of the CHPGP began in the western U.S. in April 1987 with grants of approximately $150,00O/year made to each of 11 communities, and technical assistance provided by the Stanford University Health Promotion Resource Center. The Foundation desired a comprehensive, independent and rigorous evaluation of the CHPGP in the West and selected a research team based at the University of Washington and Group Health Cooperative of Puget Sound to conduct the evaluation. The CHPGP program and evaluation differ in several important respects from the cardiovascular disease community prevention programs mentioned above, and those currently underway in Minnesota [12] and Pawtucket, RI [13]. The CHPGP is primarily a grant program, rather than a research project, and the evaluation had to be adapted accordingly. Communities applied for funds and were chosen on the merits of their applications and not on their suitability for research purposes. The Foundation expected local programs to organize the community and develop interventions tailored to local needs, interests and resources, rather than to be iterations of a single, centrally developed programmatic approach. The evalution seeks: (1) to assess whether each CHPGP program is successful in improving a variety of health behaviors and, ultimately, health outcomes in its community; (2) to contribute new information about the effectiveness of community-based, as opposed to academically-based, health promotion efforts; (3) to provide information about community organization and activation, resource needs, and intervention effects that could be useful to other communities considering health promotion efforts; and (4) to develop and test methods useful in the evaluation of community-based health promotion programs. In this paper, we describe the evaluation design, data collection

approaches, and baseline data in intervention and control communities. INTERVENTION MODEL

The conceptual framework for the program derives from a position paper prepared for the Foundation by Wallack et al. [14], the Stanford experience with community-based interventions [15], and the work of Green and colleagues [16, 171. The intervention strategy, based in social learning theory [18], gives emphasis to modifying community norms, and inducing changes in the physical, regulatory, and socioeconomic environments to make them more supportive of healthful behaviors and behavior change [19]. We incorporated these ideas in the intervention model illustrated in Fig. 1. The model postulates that successful CHPGP projects must first activate their communities in order to develop interventions capable of producing noticeable changes in community norms and environments. Community activation in this context includes the development of a broadly-based consensus among leading community organizations to address a health problem, coordinated planning of interventions, shared allocation of intervention resources, and broad citizen involvement. One rationale for this emphasis on community activation is that individual behavior change is more likely to be influenced by multiple messages and environmental changes in many different contexts than by any single intervention approach. Thus, the “activated community” may reach individual citizens through: (1) changed norms-the shared expectations of the community will shift toward approval of healthful behaviors and disapproval of unhealthful ones; (2) changed environments-the community’s environments will be altered to encourage healthful behaviors and discourage unhealthful ones; or (3) more behavioral models-the availability of individuals who have adopted healthful norms and behaviors and can serve as models for others will increase. Our approach to evaluation has been to plan measurements at all steps in the intervention model, and to observe whether predicted relationships among steps can be confirmed. EVALUATION DESIGN

The variability among local programs in the health problems addressed, the target popu-

The Henry J. Kaiser Family Foundation

Community organizations

t el

Programs

Community activation

687

1

Exposure

Program quality

Behavior-change

process

Individual outcome

Fig. 1. Conceptual framework for evaluating the Community Health Promotion Grants Program.

lations served, and the specific interventions developed represented a considerable departure from previous community preventive efforts. The uniqueness of each community program necessitated an evaluation design that would provide an estimate of the success or failure of each CHPGP grantee by comparing changes in activation, perceptions and health behaviors over time in the intervention community with concurrent changes in control communities not receiving CHPGP support. Selection of intervention and applicant control communities To meet its programmatic objectives, the Foundation identified communities eligible to receive a grant award through a competitive application process. A general mailing of information about the CHPGP was first sent to communities throughout the West. Nearly 700 letters of intent were received. From these, the Foundation staff invited the 60 most promising communities to submit full applications. After review of the written applications, 26 communities were site visited. These site visits narrowed the list of potentially fundable applicants to 18, and each of the 18 was ultimately assigned a quality ranking by Foundation staff and consultants. At each stage in the application review process, reviewers used written criteria for selection that assessed: the definition and

appropriateness of the health problem and target population (priority to disadvantaged and/or minority populations), the appropriateness and scientific soundness of intervention strategies, program emphasis on environmental change, the quality of the proposed community organization, the level of current community support, the potential for project continuation beyond the grant, and the qualifications of project staff. Four of the 18 finalists were designated by the Foundation for funding because of special merit, community need, and a special interest in statewide programs. Sufficient funds remained available to make seven additional awards to communities chosen from among the other 14 finalist applicants. Although program quality rankings were assigned, the Foundation’s reviewers felt that all 14 were meritorious and that they could not discern major differences in the potential for success among them. Therefore, the reviewers authorized the evaluation team to randomly divide the 14 remaining communities into a funded intervention group and an unfunded comparison group. While the advantages associated with randomizing communities are well known [20,21], it has rarely been used because of concerns about political acceptability, and the risk of obtaining poorly balanced groups by chance when randomizing small numbers of study units

EDWARD H. WAGNER et al.

688

[12,22]. In this instance, to achieve better balance the 14 finalist communities were first organized into three strata according to population size and the level of political organization: urban (population at least 350,000), suburban/rural, and Native American regions or reservations. (Two of the four communities funded prior to randomization fell into a fourth stratum: entire states.) The Foundation required, in addition, that the final group of funded communities also satisfy certain distributional constraints designed to assure the desired mix of health problems, minority target populations, and geographic locations. The Finite Selection Model, originally developed for the Rand Health Insurance Experiment by Morris and colleagues [23], provided a method for randomly dividing funded communities within strata, while satisfying the various distributional constraints and promoting similar distributions of program quality rankings between funded and unfunded groups. Application of the model involved stepwise formation of the funded and unfunded groups by using a random number to determine whether the funded group or the unfunded group would “choose” first from the remaining sites. Each addition to a group was then chosen to minimize differences in mean program quality

Qualified

applicant

ratings without violating a distributional constraint. As noted earlier, four finalists-two statewide programs, one urban community, and one Native American reservation-had been previously selected for funding by the Foundation. The 14 remaining finalists were randomly divided into intervention and control groups containing three and four urban communities, three and two suburban/rural communities, and one Native American community apiece, respectively (Fig. 2). The process of evaluating proposals from applicant communities resulted, of course, in a highly selected group of finalists. The selection process favored applications targeting minority populations and demonstrating that some level of community organization had already taken place to address the health problem of interest. Such selection clearly limits the generalizability of the CHPGP and hence its evaluation results. However, it enhances internal validity by comparing intervention and control communities selected through the same process and assigned at random. Non -applicant controls Restriction of the evaluation design exclusively to finalist applicant communities would have had two major drawbacks, both of which

communities

I

I

n=4

4 Urban

1 Urban

2 Suburban/ Rural

3 Suburban/

4 Suburban/ Rural

1 Native American Reservation

1 Native American Reservation

“D” Communities’

n-7

“A” Communities’

“B” Communities’

‘Letter designations for communities used in tables 3-4

Fig. 2. Overall evaluation design.

“C” Communities’

The Henry J. Kaiser Family Foundation

could contribute to missing a true program effect. First, a finding of no difference between highly selected funded and unfunded applicant communities could indicate either a failure of the CHPGP approach, or the ability of unfunded communities to mount effective programs without a CHPGP grant. Many applicant communities had a variety of program elements already in place, but not community-wide efforts. Second, if the heterogeneity among the controls, even within strata, carried over to health behaviors and other outcomes, this variability would reduce the power of the evaluation to detect program effects. We therefore added a second control group selected from among communities that did not make application to the CHPGP. This group consisted of communities matched to randomized intervention sites with regard to geographic location, size, ethnic mix, and socioeconomic factors. Budgetary constraints allowed us to match non-applicant controls to only two intervention communities. Each of these two intervention communities was matched to two control communities, thus creating two matched sets of three communities each. Since we were concerned about missing a program effect, this strategy was implemented in the suburban/rural stratum on the assumption that a grant award of modest size might have greater and more readily measurable impact in a small community than in a larger metropolitan area. The final evaluation design, depicted in Fig. 2, includes 22 communities: 11 intervention and

689

11 control communities. Comparison data for the four non-randomized intervention communities will come from the aforementioned control communities and from survey data available from other sources such as the Centers for Disease Control (CDC) Behavioral Risk Factor Surveillance System [24], or surveys of adults and adolescents on Native American reservations sponsored by the Indian Health Service. DATA COLLECTION

The intervention model shown in Fig. 1 directly influenced our choice of data collection approaches and measures. Table 1 relates the constructs included in the model to the data collection activities. Information about community activation, health promotion activities and other community characteristics comes primarily from a survey of peer-nominated leaders of important community organizations (key informants). The details of CHPGP program content and the acquisition and use of resources are obtained from program status reports completed by the program leadership. Surveys of randomly sampled adults and adolescents furnish data concerning program exposure, correlates of behavior change (perceived norms and environment, behavioral models), health behaviors, and selected health outcomes such as pregnancy, accidents and injuries. Data collected from representative samples of grocery stores and restaurants provide additional information

Table 1. Intervention model components and data collection activities Data collection activity

Intervention model component

Community activation survey

Community activation Community organizations Programs

Status report

Programs Program quality Program exposure

Adult survey

Program exposure Norms Perceived environment Behavioral models Behavior outcomes Health outcomes

Adolescent survey

Program exposure Norms Perceived environment Behavioral models Behavior outcomes Health outcomes

Grocery store and restaurant (environmental indicator) surveys

Environment

EDWARDH. WAGNERet al.

690

about community environments related to diet and the use of tobacco and alcohol. In addition, we are collecting secondary data about births, deaths, abortions, hospitalizations, and the census in order to assess the potential impact of the CHPGP on health outcomes such as: total mortality; mortality from cardiovascular disease, cancer, injuries, and substance abuse; hospitalizations for the same causes; and teenage pregnancies and abortions. Cohort vs cross-sectional samples

Intervention effects will be assessed by comparing changes in key constructs (e.g. healthful behaviors) over time in intervention and control communities. Surveys can be conducted over time either by interviewing fresh cross-sectional samples or by resurveying a cohort or panel identified at baseline. The advantages and disadvantages of each approach have been discussed in the context of this evaluation more thoroughly by Koepsell [25]. In brief, using fresh cross-sectional samples on each survey occasion avoids the biases associated with attrition from a cohort, or learning effects due to exposure to prior surveys, and allows direct assessment of the effects of in- and out-migration on community composition and the prevalence of health behaviors. Cohorts provide opportunities to relate behavior changes directly

to baseline characteristics or program exposure at the level of the individual, and they confer greater statistical power because repeated measures on the same individuals eliminate the between-individual component of variance. The community activation (key informant) survey involves only cohorts, while both cohorts and repeated cross-sectional samples of adult residents are being surveyed in communities focusing on adult health problems. In order to preserve the confidentiality of adolescents surveyed in their classrooms, we rely on crosssectional adolescent samples. Table 2 summarizes the data collection methods, nature of the respondent samples, and data collection schedule for each survey. Baseline surveys were largely completed in 1988. We had hoped to gather information on the knowledge, attitudes, and behaviors of both adults and adolescents in all communities. Budgetary limitations required that we tailor the survey populations and measurement approaches to the health problems addressed by the intervention communities. Adolescent data were collected only in intervention communities targeting adolescent health problems, and adult surveying in those same communities was limited to repeated cross-sectional samples. We collected both cohort and cross-sectional adult data in communities targeting adult health problems. In the urban community targeting

Table 2. Data collection methods

Survey

Sample

Survey schedule

Cophort or cross-sectional

Data collection method

Community activation

Reputational sample of key organizations and informants

1988,199O 1992

Cohort

Telephone interview

Status report

Program directors in funded communities

Every 6 months since 1987

Cohort

Written protocol

Adult

Random digit dialing &/or sample of commercial list

1988,1990, 1992

Both

Telephone interview, In person at one site

All 9th &/or 12th graders in randomly sampled schools

1988, 1990 1992

Cross-sectional

Self-administered in class

Grocery store

Randomly sampled from Yellow Pages

1988, 1990, 1992

Both

Observation in store

Restaurant

Randomly sampled from Yellow Pages

1988, 1990, 1992

Both

Telephone interviews with manager

The Henry J. Kaiser Family Foundation

injury prevention in the elderly, the adult survey sample included only individuals 65 and above. Because the randomized applicant control communities serve as comparators for all intervention communities in their stratum, full data were collected in each of these. Data collected in the non-applicant matched comparison communities mirrored that collected in the intervention sites. The definition of community All communities were asked, during the application process, to delimit their target population geographically, and, when relevant, demographically. For the intervention and applicant control communities, we used the boundaries

691

included in their application. For the nonapplicant matched comparisons, we chose the entire county as the community for study since their respective intervention communities were entire counties. When an intervention community further targeted its interventions to a subgroup of the population such as the elderly, sampling was adjusted accordingly in both the intervention and relevant comparison sites. All surveying, whether of individuals or institutions, adhered to this definition of community in the specification of the sampling frame. Tables 3 and 4 summarize the survey activities and sample sizes for each survey in the 22 communities involved in the evaluation, respectively. For example, Table 3 shows that commu-

Table 3. Data collection activities in evaluation communities*

Community

Health target

Community activation survey

Adult survey

Adolescent survey?

Grocery store survey

Restaurant survey

X X X

X X X X

Randomized sites Urban intervention

Al A2 A3

CVD/cancer Teen pregnancy Injuries (elderly)

X X X

X

X

D

X

Urban control X

Bl B2 B3 B4 Suburban/rural A4

A5 A6 Suburban/rural

X X X X

D X D

X X X X

X

X X

X

X X

X X

X

X

X

X

X X

X X

X X

X X

X X X intervention

CVD/cancer Teen pregnancy/ substance abuse Substance abuse

X

X

X X

control

B5 B6

X

Native American intervention

Al

Substance abuse

Native American control B7

X

X

X

X

Matched controls Surburban /rural

X X

C4a C4b CSa CSb

X

X

X

X

X

X X

X X

X Other funded sites

States

Dl D2

Teen pregnancy Multiple

X X

X

Substance abuse

X

X

Native American reservation D4 Substance abuse

X

Urban D3

F

X

*Community identifying letters and numbers include group (A = randomized intervention, B = randomized control, C = matched control, D = nonrandomized intervention), specific community identifier, and for C communities, the A to which it is matched. TX means survey conducted; D means school district access denied; F means face-to-face interviews.

EDWARD

692

H. WAGNERet al.

Table 4. Data collection activities in evaluation communities: sample sizes at baseline

Community

Community activation survey

Health target

Adult survey

Adolescent survey*

Grocery store survey

Restaurant survey

Randomized sites Urban intervention

Al A2 A3

CVD/cancer Teen pregnancy Injuries (elderly)

22 29 38

954 503 889

D

2 3 15

25 9 29

23 38 51 22

935 866 948 947

3083/5 D 2049/4 D

12 8 15 14

25

26 24

896 486

887/3

17 5

25 25

41

501

52213

26 27

672 621

92615 49314

14 10

27 38

15 15

25 25

145

315

Urban control

Bl B2 B3 B4 Suburban/rural A4

A5 A6 Suburban/rural

36 26

intervention

CVD/cancer Teen pregnancy/ substance abuse Substance abuse control

BS B6 Native American intervention

Al

Substance abuse

34

193/o

32

282120

Native American control

Bl

Matched controls Suburban/rural

C4a C4b C5a C5b

27 31 22 19

669 623 70218 84113

Other fuoded sites States

Dl D2 Urban D3

Teen pregnancy Multiple

59 68

2606/20

Substance abuse

39

17612

Native American reservation D4 Substance abuse

Total sample sizes *Number of students/number

22

436

720

10,946

122419 14,614/96

of schools; D means school district access denied.

nity A4 is a suburban/rural randomized intervention community targeting cardiovascular disease and cancer, in which community activation, adult, and grocery/restaurant data are being collected. In the analysis it will be compared with randomized controls B5 and B6, as well as matched controls C4a and C4b. Table 4 gives the number of respondents to each of the baseline surveys. Community activation survey A telephone survey of key informants nominated by their peers in the community supplied baseline information about community activation. Reputational sampling was used to select health leaders representing important community organizations. In each community,

whether intervention or comparison, we first contacted the local health department and identified the person most knowledgeable about health promotion activities in the community. The health department contact was then interviewed in person and asked to name persons, affiliated with important community organizations, with the greatest knowledge or influence concerning the five health targets of the CHPGP. Approximately 12 such individuals were selected, interviewed by telephone and the nomination process repeated. Final samples of 15-47 leaders were selected so as to ensure coverage of each health area, and major organizational sectors in the community: the health department, local government, schools, media, and voluntary agencies. At baseline, 765 key

The Henry J. Kaiser Family Foundation

informants were identified and 720 (94%) were successfully interviewed. The baseline interview elicited ratings of the strength of community activation for health promotion/disease prevention directed at adults in general, adolescents in general, teenage pregnancy, alcohol/drug abuse, cardiovascular disease/cancer, and tobacco use. The ratings were obtained using lo-item scales assessing the strength (from 1 [very weak] to 5 [very strong]) of community leadership, public awareness, political support, and various aspects of programs in the community. The mean rating of the items answered provides a “community activation” scale score (Table 4). Open-ended items gathered descriptive information about each health problem including: current health promotion activities, campaigns, or laws; and community factors that impede health promotion efforts. The survey instrument also included a set of questions assessing inter-organizational relationships targeting specific health problems in a way that enables us to evaluate these relationships using network analytic techniques [26,27]. Adult suruey

The baseline adult survey was conducted by telephone interview with samples selected by random-digit dialing except in two intervention communities: (1) an urban community (and its relevant comparison communities) where the intervention was restricted to a specific subpopulation, the elderly, and commercial telemarketing lists were used to identify the samples, and (2) a Native American reservation where telephone coverage was so low that faceto-face interviewing was required. Tapes provided by Donnelley Market Information Services produced a set of telephone exchanges that most closely corresponded to each community’s geographic definition. With the exception of one program serving an inner city population, where we screened respondents for residence within the targeted community, all residents 18 years and older at the selected exchanges were eligible for the adult survey regardless of address. The Waksberg method of random-digit dialing was employed to identify a probability sample of households [28]. From among eligible household members enumerated during a screening interview, one person 18 years and older was selected at random for the extended telephone interview.

693

Because only about one-quarter of households include at least one person 60 years of age or older, random-digit dialing is an inefficient method of sampling the elderly. The Polk Company produces commercially available mailing lists that can be requested by age of household head. We tested the use of Polk lists as a method of sampling the elderly in Seattle [29] and found that they provided reasonable coverage of the older population except among low-income households. We used Polk lists to identify the total sample in the urban community targeting injury prevention in seniors, and to supplement the elderly interviewed by random-digit dialing in the four control communities in the urban stratum. Households selected for us by Polk included those with at least one person who was retired or 65 years of age or older. Sampled housholds were then stratified by the median income (low, medium, high) of their census tract and sampled equally from each stratum. The initial response rates were low for people identified from the Polk lists and called directly, and thereafter telephone contact was preceded by a brief letter describing the study. The baseline adult telephone survey was conducted under contract by the Westat Corporation. Eighty-five interviewers administered the survey in 15 communities during 4 months in Fall 1988. Quality-control procedures included monitoring 10% on interviews, a specially trained refusal-conversion staff who recontacted particularly difficult respondents, and computer-assisted interviewing techniques to reduce errors in dialing, sampling, and following skip patterns. A total of 14,254 households were screened and extended interviews were completed with 10,510 adults. The response rates for the baseline screening interview, extended interview, and overall response (the product of the two) were 73, 74 and 54%, respectively, with little difference between random digit dialing and Polk samples. The baseline adult survey interview covered 10 areas: (1) home safety; (2) general health practices; (3) physical activity; (4) diet; (5) tobacco, alcohol and other drug use; (6) physical and mental health status; (7) behavioral process (perceived norms and environmental conditions which may help or hinder health behavior change); (8) injuries; (9) communitybased program exposure; and (10) demographic characteristics. Insofar as possible, we used items from ongoing risk factor and health surveys which might provide comparable data. The

694

EDWARDH. WAGNERet al.

CDC Behavioral Risk Factor Surveillance (BRFS) questionnaire served as a primary source for survey items [24,30]. The survey also included items from the instruments used in the Minnesota Heart Health Program [31], the Stanford Five-Cities Project [32], the Health Interview Surveys from the National Center for Health Statistics [33], the Medical Outcomes Study [34], and the Rand Mental Health Inventory [35]. The dietary questions were developed to assess, under the constraints of a telephone interview, the usual intake of total fat, saturated fat, and dietary fiber. The development and validation of these questions are described elsewhere [36]. Adolescent survey Adolescent data were obtained by means of a school-based survey utilizing a self-administered questionnaire. This approach to surveying adolescents was selected because the school setting provides efficient access to a large, representative sample of adolescents, and, given the highly sensitive nature of some of the information about sexuality and substance use, respondent privacy and anonymity can be better assured. The goal was to interview all 9th and 12th graders in six schools in each community. Schools targeted for intervention were automatically chosen in those CHPGP communities with school-based programs. Otherwise schools were randomly sampled with probability proportional to the size of their student enrollment in the relevant grades. The sampling frame for each community consisted of all public and private schools with 9th and/or 12th grades where at least 50% of enrolled students resided in the defined geographic area. Recruitment of schools proved to be a formidable undertaking, particularly in urban areas, because school administrators were reluctant to give up instructional time for such activities, and some found the proposed survey content objectionable or politically threatening. In the majority of communities and all communities in the suburban/rural stratum, negotiations were conducted with individual school principals. In these communities, two-thirds of the sampled schools agreed to participate in the survey. In five urban communities, a formal review by the school district was required prior to contacting individual school officials. In three of the five, we have been unable to obtain approval for the survey at the district level despite multiple appeals. Adolescent survey

data thus will be of primary importance in evaluating programs targeting adolescents in the suburban/rural and Native American strata, while secondary data about adolescent pregnancies and abortions are critical for assessing the impact of the urban program targeting teen pregnancy. The baseline adolescent survey included items in nine areas: (1) substance use, (2) sexual activity and contraception, (3) vehicular safety, (4) injuries, (5) physical activity, (6) diet, (7) physical and mental health status, (8) exposure to health promotion programs, and (9) demographic characteristics. The survey was selfadministered during a single class period (approximately 50 minutes). Insofar as possible, it included well-tested measures that will permit comparisons with national survey data. Primary sources included Monitoring the Future [37] for substance use items, and the National Survey of Family Growth [38] for items concerning sexual activity and contraceptive practices. Within each participating school, we attempted to survey all 9th and 12th grade students. Outside of California, parents were sent a letter informing them of the nature and purpose of the survey, and were given the opportunity to decline their child’s participation by signing and returning a form. Within California, a state regulation requiring that active written parental consent be obtained prior to asking students sex-related questions necessitated a different approach. California parents were given three options: (1) return a signed form giving permission for their child to be given the complete survey including sex-related questions, (2) not return the form which would indicate their preference for their child to be given a “restricted” version of the survey (no sexrelated questions), or (3) return a signed form indicating that they did not want their child to participate in the survey at all. As a result of this “triple option” consent process, we obtained active parental consent for the complete questionnaire from one-third of California students, and passive consent for the limited questionnaire from all but 1% of the remainder. Every effort was made during data collection to identify and gain responses from all consenting adolescents present on the day of the survey. We were unable to do follow-up surveying for those who were absent. Survey administration took place during regularly scheduled classes and was supervised by trained survey administrators. Teachers were present for classroom

695

The Henry J. Kaiser Family Foundation

management only. Overall, a total of 14,614 students from 96 schools completed the baseline survey, representing a student response rate of 79% of those enrolled. Environmental indicator surveys

Surveys of randomly selected grocery stores (conducted by direct observation) and restaurants (conducted by telephone interview of managers) afford more direct measures of the community environment relevant to diet and the use of cigarettes and alcohol. Other indicators of the community’s environment are derived from secondary data about laws, regulations and law enforcement. Grocery stores and restaurants were randomly sampled from the yellow pages of local telephone directories. This sampling frame was compared to others such as industry organization lists and health department licensing files and found to be more current and nearly as complete. The grocery survey approach involved direct observation by a trained data collector of four general product areas within the store: fresh produce, meat, milk and bread [39]. For each major product area, two types of information were recorded: the presence of healthpromotion items and the physical dimensions of product displays. Health-promotion items included health-education activities on or near the displays, which provide nutritional information and/or increased awareness of healthy food choices. The second component of the instrument consisted of measurements of the physical dimensions of key product displays to determine the proportion of healthful products such as those low in fat relative to the overall size of the display. A pre-test of the measurement protocol has been reported [39]. The grocery store survey was conducted in 13 communities. Each zip code in a community was assigned to one of three income strata (high, middle and low) based on income and population figures obtained from the 1980 Census. Five stores were sampled at random from each of the high, middle and low strata, resulting in a target sample size of 15 for each community. If one stratum had fewer than five stores in the frame, additional stores from the other two strata were drawn (if available) until the quota of 15 was reached for each community. One trained rater traveled to all of the sites over a 3 month period. Consent was obtained either from regional offices of chain stores, or from the manager of independent stores. A total of 145 CEW--F

stores were surveyed at baseline with completion rates of 87% for chain stores and 92% for independent stores. The restaurant survey had three objectives: (1) to identify the prevalence and characteristics of non-smoking areas in restaurants; (2) to ascertain how often restaurants point out healthful menu options; and (3) to examine alcohol serving practices at restaurants with bars. The sampling frame consisted of all restaurants with indoor seating within the community. As in the grocery store survey, the actual sampling frame was obtained from listings in the Yellow Pages. The target sample size was 25 completed interviews in each community. The questions on non-smoking seating were adapted from an instrument developed at the Fred Hutchinson Cancer Research Center, Seattle. Specific items queried the size of nonsmoking areas, the proportion of customers who choose non-smoking seating, and the reaction of staff and customers to those who smoke in designated non-smoking areas. Additional items explored the availability of low fat, low salt and low cholesterol entrees, the frequency of problems with intoxicated customers, and whether bartenders are required to have training in recognizing and handling intoxicated customers. Interviews were conducted by telephone with the managers of sampled restaurants of whom 315 (90°/ of those sampled) completed the survey. DATA

ANALYSIS

The general approach to analysis of data collected for the evaluated is based on the mixed-model analysis of variance (ANOVA) for nested experimental designs and is described more fully elsewhere [40]. It can be considered a generalization of the time-series regression model described by Salonen et al. [22]. The ANOVA approach is based on a well-developed body of statistical theory and provides a unifying framework to deal with several analysis issues that arise as a consequence of the complex evaluation design just described. These include: (1) allocation of entire communities, rather than individuals, to alternative treatment groups; (2) use of both cohort and repeated cross-sectional samples in survey data gathering; (3) the expectation that program effects may result in different time paths in the response variables between intervention and control communities; and (4) the desire to compare individual intervention

EDWARD H. WAGNER et al.

696

programs with a set of two or more control communities. The ANOVA model could also be used to estimate sample size and statistical power under alternative design scenarios. Ultimately, the numbers of individual-level observations per community and per survey occasion will almost surely be unequal, so that the balanced-data requirements of ANOVA will not be strictly met. This will likely necessitate use of more specialized analytic methds and software for unbalanced designs; nonetheless, the ANOVA framework provided reasonable guidance for design purposes.

BASELINE COMPARISON OF INTERVENTION AND CONTROL COMMUNITIES

Table 5 shows selected baseline characteristics among the intervention and control communi-

ties involved in the randomized component of the evaluation design. The intervention and control communities vary widely by racial/ ethnic composition and family income, but appear to be fairly similar with regard to the health behaviors and community activation ratings examined. The prevalences of high-risk behaviors are quite consistent with recent statewide and national estimates. Reported community activation was higher for adult than for adolescent health problems, and highest for cardiovascular disease and cancer prevention and lowest for adolescent pregnancy prevention. Differences in socio-demographic mix were smaller when the intervention communities were compared to the matched controls. Nevertheless, socio-demographic differences will still need to be accounted for in all analyses of program effects by comparing behavior changes

Table 5. Selected baseline characteristics in studv communities Urban stratum communities Intervention Characteristic Adults (random digit dial): % Male % White % African-American % Hispanic % Asian % Below poverty % Current smokers % Cholesterol checked in last year % favoring dispensing contraceptives to teens

Al

A2

44

44 30 21 31 11 39 26 39 66

46 6 3-l 9 32 21 43 63

Older adults (Polk sample): Average age % Male % White % African-American % Hispanic % Asian % Falling in last year Community activation survey*: CVD/cancer Adolescent pregnancy

Random control A3

12 40 69 12 5 13 12

Bl

B2

B3

B4

43 85 1 11 3 11 21 39 62

42 26 2 I 61 33 29 43 65

44

75 9 9 6 :I:

44 43 34 34 8 22 30

45 65

42 II

11 41 95 0 3 1 8

11 52 12 0 5 19 6

12 35 81 6 3 3 I

13 36 51 41 3 5 14

3.5 2.8

:::

3.3 3.1

Suburban/rural

3.3 3.0

2.9 3.0

communities Matched control

Random control

Intervention Characteristic

A4

A5

A6

B5

B6

C4a

C4b

Adults (random digit dial): % Male % White % African-American % Hispanic % Asian % Below poverty % Current smokers % Cholesterol checked in last year % Favoring dispensing contraceptives to teens

42 13 10 9 I 19 28 38 70

43 92 0 5 1 28 30 42 64

41 91 0 2 0 18 11 30 55

45 16 1 16 6 34 18 41 13

41 64 4 21 10 21 22 40 69

43 90 1 7 2 19 22 40 73

45 81 1 14 3 29 21 35 68

C5a

C5b

-continued

The Henry J. Kaiser Family Foundation

697

Table S-continued Adolescents: % 12th grade % Male % White % African-American % Hispanic % Current smokers % Recent binge drinking % Using marijuana past year % Sexually active % Rarely/never using birth control Community activation survey*: Adolescent pregnancy Alcohol/drugs

44 53 81

41 51 84

0

0

12 12 29 30 52 35

13 8 18 17 37 51

2.3 3.1

2.6 3.4

2.6 3.3

42 51 63 4 18 11 26 35 56 50

29 32 28 10 38 17 20 29 55 53

2.5 3.4

2.1 2.8

51 81

2.2 3.4

2.6 3.2

50 84

1

0

10 10 36 19 52 49

13 11 34 30 52 38

2.5 3.4

3.0 3.2

Native American communities Intervention

Random control

Characteristic

Al

B7

Adolescents (Grades 7-12): % 12th Grade % Male % Native American % Current smokers % Drunk in last year % Using marijuana in last year % Sexually active % Rarely/never using birth control

10 50 80 21 21 28 35 46

17 54 73 23 38 20 38 45

Community activation survey*: Alcohol/drugs

3.0

3.0

*Values are the mean ratings (ranging from 1 lverv weak1 to 5 [very strong]) for the IO-item scales assessing community activation related to the healih t&get named: _

among similar subgroups in intervention control communities.

and

CONCLUSION

The Henry J. Kaiser Family Foundation’s Community Health Promotion Grant Program and its evaluation design build upon two decades of experience with population-targeted efforts to prevent chronic disease. The program differs from most prior efforts by virtue of the diversity of health targets (as opposed to a focus on cancer or cardiovascular disease prevention, for example), the substantial local control over the nature of the specific interventions employed (as opposed to a program designed by a central group such as an academic institution or funding agency), and the separation of program and evaluation components (as opposed to most academically-based programs). These characteristics added to the well-described difficulties of evaluating community-based health promotion programs that are attempting to affect the public at large [41]. The random assignment of applicant communities decreases the likelihood of bias in comparisons between the intervention and control communities, but the community selection pro-

cess will limit the generalizability of the findings to communities that can make credible application for grant funds. A second set of matched non-applicant controls should enhance our ability to identify and interpret program effects. Similarly, linking data collection approaches directly to a model outlining the intervention components and their hypothesized effects provides opportunities to confirm program effects on the community as a whole, as well as on individual residents, and to isolate major elements of the intervention that appear to be associated with program success. In addition to providing evidence of program success or failure, we hope that this evaluation of communitybased efforts to reduce risk factors may contribute to our currently limited understanding of the role of the community in the initiation and maintenance of health relevant behaviors. Acknowledgement-This

study was supported by a grant from the Henry J. Kaiser Family Foundation.

REFERENCES 1. Farquhar JW, Maccoby N, Wood PD et al. Community education for cardiovascular health. Lancet 1977; i: 1192-l 195. 2. Maccoby N, Farquhar JW, Wood PD et al. Reducing the risk of cardiovascular disease: effects of a

698

3.

4.

5.

6.

7.

8.

9.

10. 11.

12.

13.

14.

15.

16.

17. 18. 19. 20. 21.

EDWARDH. WAGNERet al, community-based campaign on knowledge and behavior. J Commun Health 1977; 3: 100-114. Farquhar JW, Fortmann SP, Flora JA et al. Effects of communitywide education on cardiovascular disease risk factors: the Stanford five-city project. JAMA 1990; 264: 359-365. Salonen JT, Puska P, Mustaniemi H. Changes in morbidity and mortality during comprehensive community programme to control cardiovascular diseases during 1972-7 in North Karelia. Br Med J 1979; 2: 1178-l 183. McAlister A, Pusck P, Solonen JT et al. Theory and action for health promotion: illustrations from the North Karelia Project. Am J Public Health 1982; 72(l): 43-50. Puska P, Salonen JT, Nissinen A er al. Change in risk factors for coronary heart disease during 10 years of a community intervention programme (North Karelia project). Br Med J 1983; 287: 1840-1844. Puska P, Nissinen A, Tuomilehto J et al. The community-based strategy to prevent coronary heart disease: conclusions from the ten years of the North Karelia Project. Prev Med 1986; 15: 176-191. Kottke TE. Puska P, Salonen JT, Tuomilehto J, Nissinen A. Projected effects of high-risk versus population-based prevention strategies in coronary heart disease. Am J Epidemiol 1985; 121(5): 697-704. Expert Committee on Prevention of Coronary Heart Disease Prevention of Coronary Heart Disease. Technical Report Series 678. Geneva: WHO; 1982. Rose G. Strategy of prevention: lessons from cardiovascular disease. Br Med J 1981; 282: 1847-1951. Tarlov AR, Kehrer BH, Hall DP, Samuels SE, Brown GS, Felix MRJ, Ross JA. Foundation work: the health promotion program of the Henry J. Kaiser Family Foundation. Am J Health Promotion 1987; 2: 74-80. Blackburn H, Leupker R, Kline FG et al. The Minnesota Heart Health Program: a research and demonstration project in cardiovascular disease prevention. In: Matarazzo JG, Weiss SM, Herd JA et al., Eds. Behavioral Health: a Handbook of Health Enhancement and Disease Prevention. New York: Wiley; 1984: 1171-l 178. Lasater T, Abrams D, Artz L et al. Lay volunteer delivery of a community-based cardiovascular risk change program: the Pawtucket experiment. In: Matarazzo JD, Weiss SM, Herd JA et al., IS. Rehavioral Health: a Handbook of Health Enhancement and Disease Prevention. New York: Wiley; 1984: 1166-1170. Wallack L, Rundall TG, Syme SL, Lashof JC. An EnvironmentalApproachto Promoting HeaIth. A report prepared for the Kaiser Family Foundation. Farquhar JW. The community based model of lifestyle intervention trials. Am J Epidemiol 1978; 108: 103-111. Green LW, Raeburn JM. Health promotion. What is it? What will it become? Health Promotion 1988; 3: 151-159. Green LW, McAlister AL. Macro-intervention to support health behavior: some theoretical perspectives and uractical reflections. Health Rduc 0 1984: 11: 322-339. Bandura A. Social Learning Theory. Morristown, NJ; General Learning Press; 1971. Syme SL. Strategies for health promotion. Prev Med 1976; 15: 492-507. Rose G, Heller RF, Pedoe HT, Christie DGS. Heart disease prevention project: a randomized controlled trial in industry. Br Med J 1980; 1: 747-751. Harris JE. Macroexperiments versus Microexperiments for health policy. In: Hausman JA, Wise DA, Eds. Social Experimentation. Chicago: The University of Chicago Press; 1985.

22.

23.

24.

25.

26.

27.

28. 29.

30.

31.

32.

33.

34.

35.

36.

37.

38.

Salonen JT, Kottke TE, Jacobs DR, Hannan PJ. Analysis of community-based cardiovascular disease prevention studies-valuation issues in the North Karelia Project and the Minnesota Heart Health Program. Int J Epidemiol 1986; 15: 176182. Morris C. A finite selection model for experimental design of the Health Insurance Study. J Econ 1979; 11: 43-61. Marks JS, Hogelin GC, Gentry EM, Jones JT, Gaines KL. Forman MR. Trowbridze FL. The behavioral risk factor surveys: I State-spe&c prevalence estimates of behavioral risk factors. Am J Prev Med 1985; 1: l-8. Koepsell T. Methodological issues in evaluation of community-based health promotion and disease prevention programs. Presented at the 7th Annual Meeting of tbe American College of Epidemiology, Ann Arbor, MI; 67 October, 1988. Morrissey JP, Tausig MT, Lindsey ML. Interorganizational networks in mental health systems: assessing community support programs for the chronically mentally ill. In: Scott WR, Black BL, Eds The Organization of Mental Health Services. Beverly Hills, CA: Focus Editions; 1986. Morrissey JP, Tausig MT, Lindsey ML. Network analysis methods for mental health service system research: a comparison of two community support systems. National Institute of Mental Health Series BN No. 6, DHHS Publ. No. (ADM) 85-1383. Washington. DC: U.S. Goverment Printing Office: 1985. Waksberg J. Sampling methods fo; random digit dialing. J Am Stat Assoc 1978; 73: 40-46. Psaty BM, Cheadle A, Curry S et al. Sampling elderly in the community: a pilot study of the coverage provided by commercial telemarketing lists. Am J Rpidemiol 1991; In press. Gentrv EM. Kalsbeck WD. Houelin GC et al. The behavioral risk factor surveys: II Design, methods, and estimates from combined state data. Am J hev Med 1985; 1: 9-14. Jacobs DR, Luepker RV, Mittelmark MB er al. Community-wide prevention strategies: evaluation design of the Minnesota Heart Health Program. J Chron Dis 1986; 39: 775-788. Farquhar JW, Fortmann SP, Maccoby N er al. The Stanford Five-city Project: design and methods. Ant J Epidemiol 1985; 122: 323-334. National Center for Health Statistics. Health promotion and disease prevention, United States, 1985. Vital and Health Statistics Series 10, No. 163, PHS Publ. No. (PHS) 88-1591. Washington, DC: U.S. Government Printing Office; 1985. Stewart AL, Hays ED, Ware JE Jr. The MOS shortform general health survey: reliability and validity in a patient population. Med Care 1988; 26: 724-735. Ware JE, Johnston SA, Davies-Avery A, Brook RH Conceptualization and measurementof health for adults in the health insurance studv: Vol. III, Mental Health. R-1987/3-HEW, The RandCorporation, Santa Monica, CA: The Rand Corporation; 1979. Kristal AR. DeLett Al. Henrv HJ. Fowler AS. Rapid assessment of dietary intake of fat; fiber and saturated fat: validity of an instrument suitable for community intervention research and nutrional surveillance. Am J Health Promotion 1990; 4: 288-295. Johnston LD, G’Malley PM, Bachman JG. National trends in drua use and related factors among American high school students and young adults, 1975-1986. DHHS Publ. No. (ADM) 87-1535. Washington, DC: U.S. Government Printing Office; 1987. National Center for Health Statistics, Bachrach CA, Horn MC. Mosher WD, Shimizu I. National survey of family growth, cycle III, sample design, weighting, and

The Henry J. Kaiser Family Foundation variance estimation. Vital and Health Statistics Series 2, No. 98. DHHS Publ. No. (PHS) 85-1372. Washington, DC: U.S. Government Printing Office; 1985. 39. Cheadle A, Psaty BM, Wagner EH, Diehr PH, Koepsell TD, Curry SG, VonKorff MV. Evaluating community-based nutrition programs: assessing the reliability of a survey of grocery store product displays. Am J Public Health 1990; 80: 709-711.

40.

41.

699

Koepsell TD, Martin DC, Diehr PH, Psaty BM, Wagner EH, Perin EB, Cheadle A. Data analysis and sample size issues in evaluations of community-based health promotion and disease prevention programs: a mixed model analysis of variance approach. J Clin Epidemiol 1991; 44: 701-703. Braverman MT, Ed. Evaluating Health Promotion

Programs. San Francisco: Jossey-Bass; 1989.

The evaluation of the Henry J. Kaiser Family Foundation's Community Health Promotion Grant Program: design.

The Kaiser Family Foundation's Community Health Promotion Grant Program (CHPGP) provides funding and technical assistance in support of community-base...
1MB Sizes 0 Downloads 0 Views