Development and Reliability Testing of a Fast-Food Restaurant Observation Form.

For individual use only. Duplication or distribution prohibited by law.

Measurement Issues; Nutrition

Development and Reliability Testing of a Fast-Food Restaurant Observation Form Leah Rimkus, MPH, RD; Punam Ohri-Vachaspati, PhD, RD; Lisa M. Powell, PhD; Shannon N. Zenk, PhD; Christopher M. Quinn, MS; Dianne C. Barker, MHS; Oksana Pugach, PhD; Elissa A. Resnick, MPH; Frank J. Chaloupka, PhD Abstract Purpose. To develop a reliable observational data collection instrument to measure characteristics of the fast-food restaurant environment likely to influence consumer behaviors, including product availability, pricing, and promotion. Design. The study used observational data collection. Setting. Restaurants were in the Chicago Metropolitan Statistical Area. Subjects. A total of 131 chain fast-food restaurant outlets were included. Measures. Interrater reliability was measured for product availability, pricing, and promotion measures on a fast-food restaurant observational data collection instrument. Analysis. Analysis was done with Cohen’s j coefficient and proportion of overall agreement for categorical variables and intraclass correlation coefficient (ICC) for continuous variables. Results. Interrater reliability, as measured by average j coefficient, was .79 for menu characteristics, .84 for kids’ menu characteristics, .92 for food availability and sizes, .85 for beverage availability and sizes, .78 for measures on the availability of nutrition information,.75 for characteristics of exterior advertisements, and .62 and .90 for exterior and interior characteristics measures, respectively. For continuous measures, average ICC was .88 for food pricing measures, .83 for beverage prices, and .65 for counts of exterior advertisements. Conclusion. Over 85% of measures demonstrated substantial or almost perfect agreement. Although some measures required revision or protocol clarification, results from this study suggest that the instrument may be used to reliably measure the fast-food restaurant environment. (Am J Health Promot 2015;30[1]:9–18.) Key Words: Fast-Food Restaurants, Reliability, Environment, Marketing, Prevention Research. Manuscript format: research; Research purpose: instrument development; Study design: observational research; Outcome measure: other environmental; Setting: local community; Health focus: nutrition, weight control; Strategy: policy, built environment; Target population age: youth, adults, seniors; Target population circumstances: Chicago Metropolitan Statistical Area, all education levels, all income levels, all races/ethnicities

PURPOSE Over the past few decades, Americans have obtained an increasing proportion of food and beverages from away-from-home sources,1–3 with a significant portion coming from limitedservice or fast-food outlets.4 Between 1977–1978 and 2003–2006, the contribution of fast food to total caloric intake among children and adolescents aged 2 to 18 years increased from 2% to 13%.2,5 More recent evidence shows that from 2003–2004 to 2007– 2008, this percentage dropped from 12% to 10% among children aged 2 to 11 years, but remained constant at 17% among adolescents aged 12 to 19 years.6 Consumption of fast food has been associated with higher total energy intake and other markers of poor diet, including higher intakes of fat, saturated fat, sodium, added sugars, and carbonated soft drinks, and with lower intakes of fruits, vegetables, milk, and fiber.3,7–10 Given these findings, some public health professionals have postulated that exposure to fast-food outlets is an environmental contribu-

Leah Rimkus, MPH, RD; Lisa M. Powell, PhD; Christopher M. Quinn, MS; Oksana Pugach, PhD; Elissa A. Resnick, MPH; and Frank J. Chaloupka, PhD, are with the Institute for Health Research and Policy, University of Illinois at Chicago, Illinois. Punam Ohri-Vachaspati, PhD, RD, is with the School of Nutrition and Health Promotion, Arizona State University, Tempe, Arizona. Shannon N. Zenk, PhD, is with the College of Nursing, University of Illinois at Chicago, Illinois. Dianne C. Barker, MHS, is with Barker Bi-Coastal Health Consultants Inc, Calabasas, California. Send reprint requests to Leah Rimkus, MPH, RD, Institute for Health Research and Policy, University of Illinois at Chicago, 1747 W. Roosevelt Road, Room 558 (M/C 275), Chicago, IL 60608; e-mail: [email protected]. This manuscript was submitted July 31, 2013; revisions were requested October 28, 2013 and January 16, 2014; the manuscript was accepted for publication February 2, 2014. Copyright Ó 2015 by American Journal of Health Promotion, Inc. 0890-1171/15/$5.00 þ 0 DOI: 10.4278/ajhp.130731-QUAN-389

American Journal of Health Promotion

September/October 2015, Vol. 30, No. 1

9

For individual use only. Duplication or distribution prohibited by law. tor to poor dietary behaviors and obesity.11 However, research examining the relationship between fast-food restaurant availability and weight status has shown mixed results. Some studies found a positive association,12–16 while others reported no relationship.11,17–20 One possible reason for these inconsistencies is the lack of measurement of characteristics within restaurants.21,22 Without measured characteristics, the assumption is that all fast-food restaurants are similar in terms of availability, price, and promotion of healthy and unhealthy food and beverage options. Reliable measures of the fast-food restaurant environment are needed to assess the relationship between fastfood restaurant characteristics and weight status. Such measures are also critical for evaluating the implementation and impact of recent policy and programmatic efforts to facilitate healthy choices within fast-food restaurants. In spite of increased interest in food environment measurement, there is a lack of validated and reliable measures.23,24Among eight instruments used for measuring the restaurant environment21,25–31 identified through a recent literature review32 and the National Cancer Institute’s Measuring the Food Environment Web site,33 only three21,25,28 reported any reliability testing. Only one of these21 examined pricing/affordability. Two21,25 included measures of the availability of nutrition information and the presence of health promotion signage. The purpose of this article is to describe the development and interrater reliability of a brief observational data collection instrument focused on the fast-food restaurant environment and designed to measure restaurant characteristics, including food and beverage availability, price, and promotion—policy-relevant factors likely to influence individuals’ food and beverage purchasing and consumption behaviors.

METHODS Measures and Protocol As part of a larger, national research study of community food and physical activity environments conducted by the Bridging the Gap program, a four-page

10


Fast-Food Observation Form (BTGFFOF) was developed, building largely on two existing instruments21,34 and with input from an expert panel (a list of panelists is provided in the Acknowledgments). When determining with our expert panel what measures to include on the form, priority was given to items that were clear, provided useful detail to help characterize the fast-food restaurant environment, would not take a great deal of time to code in the field, and would not likely require asking an employee. New measures on food and beverage prices, interior and exterior marketing, and the provision of nutrition information were added to capture prices on a wider range of items than available in existing secondary data sources35 and to collect data that are especially relevant to current policy issues, including food marketing to children36 and provision of calorie and other nutrition information in restaurants.37,38 Early iterations of the instrument were tested in four locations (Boise, Idaho; Milwaukee, Wisconsin; Chicago, Illinois; and Los Angeles, California) to examine variations in the fast-food environment in different communities. The form was revised based on findings from these pilots, and a comprehensive training protocol was developed for use in reliability testing. Major topics and specific measures on the form are summarized in Table 1. Characteristics of the general menu and kids’ menu were recorded, such as the presence of healthier options (i.e., fruit, nonfried vegetable, salad, and/or yogurt) on the dollar menu and on the kids’ menu. Presence of nutritionrelated designations on the menu or menu board, such as low-fat or lowcalorie, were recorded in order to capture whether restaurants identified and advertised such food and beverage options for food patrons. Recognizing that availability of food items was greatly dependent on the type of restaurant observed, eight food items were included on the form. Three matched the fast-food prices that are monitored by the Council for Community and Economic Research (C2ER)35 and that have been used extensively in previous studies that examine impacts of food prices on diet

and obesity outcomes.39 Five other commonly available food items were added to broaden the list to encompass other menu types. Seven commonly available sweetened and unsweetened beverage items were included. Prices were recorded for all food and beverage items coded as available. For beverages, data collectors coded the price of the smallest size available, either selecting ‘‘small’’ or writing in the name of the smallest size (e.g., 12 ounce) that was available in the restaurant. For fountain drinks and french fries, the prices of both the smallest and largest sizes available were captured. Data collectors received a ‘‘menu guide’’ that provided, for the major chains, the name of the specific food item to price when multiple options on the menu board matched the general specifications on the form (e.g., sandwich with grilled/roasted chicken). These food items were identified using company Web sites and finding the closest match, based on serving size and calorie content, to a standard menu item from the top-grossing restaurant chain within a given menu category (e.g., Premium Grilled Chicken Classic Sandwich from McDonald’s). To assess the availability of nutrition information at indoor and outdoor (e.g., drive-through) points of purchase (i.e., locations where food orders are placed), the presence of six possible sources of nutrition information was recorded. For each source coded as present, the provision of calorie information, any reference to total suggested daily calories, and any statement that additional nutrition information was available on request were recorded. In addition, one employee at each location was asked for nutrition information and his/her response(s) was recorded. Data collectors counted and coded the number of food, beverage, and combined (food and beverage) advertisements sized at least 8.5 3 11 inches on the restaurant’s exterior and property (e.g., signs posted in the parking lot or on a fence). From those totals, data collectors coded the number of ads with a price promotion and other characteristics such as the presence of any dollar menu promotion, health



Table 1 Summary of Topics and Measures on the Bridging the Gap Fast-Food Observation Form Topical Category Menu characteristics

Kids’ menu characteristics

Food and beverage availability and sizes

Food and beverage prices

Availability of nutrition information

Exterior advertisements

Exterior characteristics

Interior characteristics

Measures Included Dollar menu and healthy option(s) on dollar menu Combination meals Salad options and low-fat/fat-free dressing option Fruit sides (unsweetened) Vegetable sides (nonfried, without added fat) Designations on menu or menu board for ‘‘low-fat,’’ ‘‘low-calorie,’’ or items identified by restaurants as ‘‘healthy’’ Kids’ menu/kids’ meal Healthy beverage option and healthy side option on kids’ menu Toy giveaways Food items from Cost of Living Index (quarter-pound hamburger with cheese, fried chicken leg/thigh, cheese pizza) Additional food items (french fries, sandwich with grilled/roasted chicken, entreé salad with grilled/roasted chicken, ground beef taco, turkey sub) Healthy beverage options (100% juice, low-fat milk, bottled water) Other beverage options (fountain drinks, iced/flavored coffee drink, hot/flavored espresso drink) Prices for all food items coded as available Prices of smallest and largest size french fries Prices for all beverages (smallest size) coded as available Prices of smallest and largest size fountain drink Presence of menu board, food display tags, posted material (e.g., poster, banner, decal), printed material (e.g., brochure, flyer, postcard), printed menu, and drive-through menu board at interior/exterior point of purchase Presence of calorie information on each source Presence of statement on total suggested daily calories on each source Signage that additional nutrition information is available on request Employee response to request for nutrition information Number of food, beverage, and combined (food/beverage) advertisements on building exterior Number of food, beverage, and combined (food/beverage) advertisements on restaurant property (i.e., not on the building) Number of ads from each category with a price promotion Presence of ads with a dollar menu promotion, health claim, quality claim Presence of any and of specific types of child-directed marketing (e.g., presence of poster with cartoon characters, kids’ meal toy, or poster with child-oriented TV or movie star) Restaurant type (i.e., menu type) Accessibility (whether the restaurant shares space with other retail, presence of parking on-site, bicycle parking, sidewalk, lighting) Number of exterior walls Markers of crime/disorder (presence of graffiti, bars on windows) Number of cash registers Amenities (interior seating, public restroom, free water for customers, and indoor play area for kids) Placement/promotional characteristics (cookies, cakes, pies, or brownies for sale at the counter, self-service fountain drink machine) Markers of crime/disorder (bulletproof glass at cash register)

claim (e.g., ‘‘light,’’ ‘‘low-fat’’), or food quality claim (e.g., ‘‘fresh,’’ ‘‘natural’’). Presence of any child-directed marketing was also recorded and (where present) the types of child-directed marketing were identified (e.g., presence of cartoon character or a kids’ meal toy in an advertisement). Finally, interior and exterior characteristics were coded to assess restaurant features (e.g., presence of a public


restroom, availability of free water), interior marketing (e.g., cookies, cakes, pies, or brownies for sale at the counter), restaurant accessibility (e.g., shared space with other retail, presence of any sidewalk on street or bicycle parking), and markers of disorder (e.g., presence of graffiti on the restaurant building and/or exterior signs, presence of bars covering the restaurant windows).

Sample A reliability study was conducted in the Chicago Metropolitan Statistical Area (MSA). Internet research was conducted using company Web sites to identify outlets for the 36 top-grossing fast-food chains40 located in Chicago and nearby suburban communities. The 1958 fast-food outlets identified in the study area were mapped to census tracts, which were categorized into


11

For individual use only. Duplication or distribution prohibited by law. income tertiles (based on median household income) and two racial groups using U.S. 2000 Census data.41 Tracts with 70% or more white residents were considered predominantly white, while the remaining were considered non-white. To minimize field staff travel time, clusters of retail activity were identified throughout the MSA, and a stratified convenience sample of tracts was drawn based on an algorithm to ensure adequate variation in income, race, and urbanicity. The final sample of 135 restaurants (representing 24 different chains) contained an equal percentage of fast-food outlets in low-, middle- and high-income tracts, with one-half of outlets each in predominantly white and non-white tracts and one-half each within and outside of Chicago’s city limits.

variable, and distinguishes between j values for dichotomous versus categorical variables with three or more response categories. A j value of .81 to 1.00 was considered to be almost perfect agreement, .61 to .80 substantial, .41 to .60 moderate, .21 to .40 fair, 0 to .20 slight, and ,0 poor agreement.42 ICC values were interpreted using the same ranges. For those variables with no variation across the sample (all ‘‘yes’’ or all ‘‘no’’ responses), a valid j value could not be computed. Therefore, only overall agreement is presented for these measures. Any measure coded by one or both observers as ‘‘don’t know’’ or ‘‘not applicable’’ was considered missing and was excluded from the reliability analysis.

Data Collection Data collection occurred during a 2week period in December 2009. Following a 2½-day training, two field observers visited each sampled restaurant and independently coded the full observation form following a standardized protocol. Data collectors were instructed to record what was visible during their visit. However, in instances where key information (e.g., price of a food item) was not posted on the menu board or visible elsewhere, they were instructed to ask (jointly) for the necessary information at the end of the observation to minimize any potential burden on employees. Restaurants in the sample were not contacted for permission prior to the field team visits; however, data collectors were given ID badges from the University of Illinois at Chicago and a letter that explained the purpose of the study. The university’s institutional review board determined that the data collection did not constitute human subjects research and was thus exempt from review.

RESULTS

Analysis All data were analyzed using SAS version 9.2 (released 2008, SAS Institute, Cary, North Carolina). Overall percent agreement and simple j values were computed for categorical variables, and intraclass correlation coefficients (ICC) were computed for continuous variables. Table 2 indicates which test statistic was used for each

12


Data are presented from 131 completed observations (97% of the original sample). Four observations were not completed because the business was not found or was permanently closed (n ¼ 3) or because the data collectors felt unsafe conducting the observation (n ¼ 1). A variety of restaurant types were observed, including burger and fries outlets (24%), sandwich shops (20%), and fried chicken/fried fish outlets (18%) (Table 3). The average time for completing the observation form was 25 minutes (standard deviation [SD] ¼ 7.9 minutes). Reliability results for all measures on the BTG-FFOF (100 categorical and 31 continuous variables) are presented in Table 2. Sixteen of the categorical measures did not have valid j values because there was no variation across sampled restaurants (e.g., menu board at point of purchase, designation on menu for low-saturated fat items). These variables are excluded from the data summaries and averages provided hereafter. Among the measures related to menu characteristics, 45% had substantial agreement (j .61, but ,.81), and 45% had almost perfect agreement (j .81). Lower agreement (j ¼ .53) was found for presence of a healthy option on the dollar menu. In the kids’ menu characteristics section, none of the measures had substantial

agreement, while 83% had almost perfect agreement. Lower agreement (j ¼ .50) was seen on the measure assessing whether the healthy side option on the kids’ menu was the default option. For food availability and sizes, 18% of items had substantial and 82% had almost perfect agreement. Among the food price measures, 33% had substantial and 67% had almost perfect agreement. For beverage availability and sizes, 29% of items had substantial and 64% had almost perfect agreement. Lower agreement (j ¼ .36) was found on the size of the smallest 100% juice. Similar to the food pricing, 38% of the beverage prices had substantial agreement; 63% had almost perfect agreement. Within the nutrition information availability section, nine measures did not have a valid j value owing to lack of variation, i.e., all ‘‘yes’’ or all ‘‘no’’ responses. Of the remainder, 43% had substantial agreement, and 36% had almost perfect agreement. Lower agreement (,.61) was found for measures relating to posted material at the point of purchase. Two variables in the exterior advertisements section did not have a valid j value; of the remainder, seven variables (33%) had substantial agreement and eight measures (38%) had almost perfect agreement. The measures demonstrating lower agreement included number of beverage ads on the building exterior with a price promotion, number of food/beverage ads on the building exterior, number of food/ beverage ads on the property, number of food/beverage ads on the property with a price promotion, presence of any ad with a quality claim, and presence of ‘‘other’’ child-directed marketing. Among the exterior characteristics measures, three (23%) had a j/ICC between .61 and .80 and six (46%) had a j/ICC .81. Sidewalk lighting, parking lot lighting, and bars on window all had slight agreement (j ¼ 0 to .20) and presence of bike parking had moderate agreement. Finally, two measures (25%) under interior restaurant characteristics had substantial agreement, and the remaining 75% of measures had almost perfect agreement.



Table 2 Interrater Reliability for All Measures on the Bridging the Gap Fast-Food Observation Form Measure Menu characteristics (14 categorical measures) Dollar menu available Dollar menu has healthy option Combo meals available Salad options available Low-fat or fat-free salad dressing available Menu has vegetable side (nonfried, without added fat) Menu has fruit side (unsweetened) Designation for low-calorie or light items Designation for low-fat items Designation for low-saturated fat items Designation for low-sodium items Designation for low-carbohydrate items Designation for heart healthy items Designation for other ‘‘healthy’’ items Kids’ menu characteristics (6 categorical measures) Kids’ menu available Kids’ meal has healthy beverage option Healthy beverage option is the default Kids’ meal has healthy side option Healthy side option is the default Toys or giveaways with kids’ meal Food availability and sizes (11 categorical measures) French fries available (smallest) Size of smallest french fries French fries available (largest) Size of largest french fries Hamburger (quarter pound with cheese) available Sandwich with grilled/roasted chicken available Entreé salad with grilled/roasted chicken available Fried chicken (1 leg and 1 thigh) available Cheese pizza (thin crust 10–12 inches) available Ground beef taco available Turkey and cheese sub sandwich (6 inches) available Food prices (9 continuous measures) Price of smallest french fries Price of largest french fries Price of hamburger (quarter pound with cheese) Price of sandwich with grilled/roasted chicken Price of entree salad with grilled/roasted chicken Price of fried chicken (1 leg and 1 thigh) Price of cheese pizza (thin crust, 10–12 inches) Price of ground beef taco Price of turkey and cheese sub sandwich (6 inches) Beverage availability and sizes (16 categorical measures) Fountain drink available (smallest) Size of smallest fountain drink Fountain drink available (largest) Size of largest fountain drink Bottled soda available Size of smallest bottled soda 100% juice available Size of smallest 100% juice Low-fat milk available Size of smallest low-fat milk Bottled water available Size of smallest bottled water Flavored iced coffee drink available Size of smallest flavored iced coffee drink


No.*

Test Statistic

129 50 129 129 62 129 128 116 116 116 116 116 116 117

j j j j j j j j j j j j j j

2 2 2 2 2 2 2 2 2 2 2 2 2 2

3 3 3 3 3 3 3 3 3 3 3 3 3 3

2 2 2 2 2 2 2 2 2 2 2 2 2 2

129 90 57 89 61 88

j j j j j j

2 2 2 2 2 2

3 3 3 3 3 3

2 2 2 2 2 2

130 37 130 33 129 129 129 133 129 132 132

j j j j j j j j j j j

many many many many many many many many many many many

88 82 58 98 88 39 25 25 52 129 100 128 95 129 25 129 59 128 32 127 62 128 24

ICC ICC ICC ICC ICC ICC ICC ICC ICC j j j j j j j j j j j j j j

2 2 2 2 2 2 2 2 2 2 2 2 2 2

3 3 3 3 3 3 3 3 3 3 3 3 3 3

2 2 2 2 2 2 2 2 2 2 2 2 2 2

j/ICC

Overall Agreement

0.79 0.77 0.53 0.73 0.94 0.90 0.70 0.97 0.79 0.81 † † 0.92 † 0.64 0.84 0.98 0.85 1.00 0.83 0.50 0.87 0.92 0.95 0.76 0.92 0.71 0.98 0.97 0.99 0.97 0.96 0.93 0.96 0.88 0.87 0.69 1.00 1.00 1.00 0.72 0.62 0.99 1.00 0.85 0.97 0.78 0.98 0.73 0.78 1.00 0.97 0.36 0.94 † 0.87 † 1.00 0.82

0.93 0.88 0.78 0.93 0.97 0.97 0.85 0.98 0.91 0.91 1.00 1.00 0.99 1.00 0.82 0.94 0.99 0.93 1.00 0.93 0.85 0.94 0.96 0.97 0.89 0.95 0.91 0.98 0.97 0.98 0.98 0.99 0.97 0.97 — — — — — — — — — — 0.95 0.99 0.89 0.99 0.86 0.91 1.00 0.98 0.86 0.98 1.00 0.94 1.00 1.00 0.92


13


Table 2, Continued Measure

No.*

Test Statistic

j/ICC

Overall Agreement

Hot, flavored espresso drink available Size of smallest flavored hot espresso drink Beverage prices (8 continuous measures) Price of smallest fountain drink Price of largest fountain drink Price of smallest bottled soda Price of smallest 100% juice Price of smallest low-fat milk Price of smallest bottled water Price of smallest flavored iced coffee drink Price of smallest flavored hot espresso drink Availability of nutrition information (23 categorical measures) Menu board at the point of purchase Menu board has calorie information for menu items Menu board has statement on daily calories Menu board has nutrition information on request statement Food display tags at point of purchase Food display tags have calorie information for menu items Food display tags have statement on daily calories Posted material at point of purchase Posted material has calorie information for menu items Posted material has statement on daily calories Posted material has nutrition information on request statement Printed material at point of purchase Printed material has calorie information for menu items Printed material has statement on daily calories Printed material has nutrition information on request statement Printed menu at point of purchase Printed menu has calorie information for menu items Printed menu has statement on daily calories Printed menu has nutrition information on request statement Restaurant has a drive-through Drive-through menu board has calorie information for menu items Drive-through menu board has statement on daily calories Drive-through menu board has nutrition information on request statement Exterior advertisements (11 categorical, 12 continuous measures)

128 21

j232 j232

178 164 51 96 62 127 44 38

ICC ICC ICC ICC ICC ICC ICC ICC

130 128 125 124 130 37 34 130 91 91 74 125 33 33 22 118 2 2 1 126 59 59 57

j232 j232 j232 j232 j232 j232 j232 j232 j many j232 j232 j232 j232 j232 j232 j232 j many j232 j232 j232 j232 j232 j232

1.00 0.90 — — — — — — — — — 0.97 1.00 0.92 1.00 0.98 0.96 1.00 1.00 0.86 0.84 0.98 0.97 0.90 0.94 0.97 1.00 0.98 1.00 1.00 1.00 0.98 1.00 1.00 0.96 0.93

251 249 215 108 246 245 57 33 249 245 92 33 124 120 120 111 14 14 14 15 14 14 15

ICC ICC ICC ICC ICC ICC ICC ICC ICC ICC ICC ICC 23 23 23 23 23 23 23 23 23 23 23

1.00 0.74 0.83 0.78 0.95 1.00 0.94 0.88 0.82 0.68 0.61 0.78 † 0.77 † 0.79 0.91 † † 0.59 0.60 0.74 0.49 0.78 0.89 0.93 † 0.74 † † † 0.97 1.00 † 0.73 0.75 (j) 0.65 (ICC) 0.75 0.85 0.78 0.79 0.68 0.79 0.42 0.91 0.48 0.51 0.61 0.26 0.82 0.85 0.35 0.62 † 1.00 † 0.86 0.81 0.86 0.55

Number of food ads on the building exterior Number of food ads on the property Number of food ads on building with price promotion Number of food ads on property with price promotion Number of beverage ads on the building exterior Number of beverage ads on the property Number of beverage ads on building with price promotion Number of beverage ads on property with price promotion Number of food and beverage ads on the building exterior Number of food and beverage ads on the property Number of food and beverage ads on building with price promotion Number of food and beverage ads on property with price promotion Any ads with a dollar menu promotion Any ads with a health claim Any ads with a quality claim Any child-directed marketing on exterior or property Presence of exterior play area Presence of indoor play area visible from outside Presence of a 3-D cartoon character Presence of a poster with a cartoon character Presence of a poster with child-oriented TV or movie star Presence of a poster showing kids Other child-directed marketing

14


j j j j j j j j j j j

2 2 2 2 2 2 2 2 2 2 2

— — — — — — — — — — — — 0.94 0.97 0.81 0.87 1.00 1.00 1.00 0.93 0.93 0.93 0.80



Table 2, Continued Measure Exterior characteristics (12 categorical, 1 continuous measures) Restaurant type In shared space with grocery or general merchandise store In shared space with gas or convenience store In shared space with another restaurant Located within a food court or mall Sidewalk present on street Sidewalk lighting present Parking present on-site Parking lot lighting present Bicycle parking present Bars on restaurant windows Graffiti/tagging on building or exterior signage Number of exterior walls facing street or parking Interior characteristics (7 categorical, 1 continuous measures) Restaurant has seating Restaurant has public restroom Restaurant has interior play area for kids Restaurant has self-service fountain drinks Restaurant has free water for customers Restaurant has cookies, cakes, pies, or brownies for sale at counter Restaurant has glass or plastic safety divider at counter Number of cash registers

No.*

Test Statistic

131 129 129 129 131 123 103 126 106 114 123 126 250

j many j232 j232 j232 j232 j232 j232 j232 j232 j232 j232 j many ICC

130 128 130 130 129 128 130 259

j j j j j j j

23 23 23 23 23 23 23 ICC

2 2 2 2 2 2 2

j/ICC

Overall Agreement

0.62‡ 0.99 0.80 0.92 0.83 1.00 0.62 0.03 0.96 0.15 0.57 0 0.62 0.89 0.90‡ 0.95 0.89 0.79 0.98 0.86 0.86 1.00 0.61

0.95 0.99 0.99 0.99 0.98 1.00 0.92 0.73 0.99 0.92 0.90 0.99 0.97 — 0.97 0.99 0.97 0.98 0.99 0.93 0.93 1.00 —

ICC indicates intraclass correlation coefficient. Note: For continuous variables, overall agreement was not calculated and thus is designated by a dash (—). Bold values indicate the average j/ICC or overall agreement for a given section. * Represents the number of restaurants in the case of j values and number of individual observations in the case of ICC values. † No valid j value due to insufficient variation (all ‘‘yes’’ or all ‘‘no’’ responses). ‡ Continuous measure (ICC value) was omitted from average j displayed for this section.

DISCUSSION Findings from this study indicate that measurement of the fast-food restaurant environment using the BTG-FFOF was feasible and reliable. The vast majority of measures had substantial or almost perfect agreement between two independent observers. Overall agreement was greater than 80% for all but two categorical variables. Twenty-two (26%) of the categorical variables had substantial agreement, and 50 (60%) had almost perfect agreement. Among the continuous variables, 13 (42%) had substantial agreement and 14 (45%) had almost perfect agreement. Measures related to menu characteristics, kids’ menu characteristics, food/beverage availability and sizes, food/beverage prices, availability of nutrition information, and interior restaurant characteristics demonstrated high agreement. Lower j/ICC values were more common among


measures related to exterior advertisements and characteristics. Earlier restaurant studies that included a reliability-testing component reported high interrater reliability on measures of fruit and vegetable availability.25,28 This is consistent with the present study, which demonstrated substantial or almost perfect agreement for measures in the menu characteristics and food/beverage item availability and sizes sections. While these two earlier studies included a thorough review of the restaurant menu, neither included observation of the broader restaurant environment, such as the availability of nutrition information or the presence of signage and other forms of marketing. The Nutrition Environment Measures Study in Restaurants (NEMS-R) instrument contains a thorough review of the restaurant menu as well as observation of other characteristics of the restaurant environment, including pricing, signage, as well as facilitators of and barriers to healthy eating.

Results from interrater reliability testing of the NEMS-R instrument in both fast-food and full-service restaurants showed high agreement (over 75% agreement and j greater than .80) for most measures, but lower j values (,.60) for the availability of a healthy option among the main-dish salads and

Table 3 Restaurants Observed, by Type Restaurant Type Burger and fries Sandwich or sub shop Fried chicken/fried fish Mexican Pizzeria Coffee shop Chinese/Pan-Asian Other* Total

Frequency Percent 31 26 24 13 13 10 9 5 131

23.7 19.9 18.3 9.9 9.9 7.6 6.9 3.8 100.0

* Includes outlets with multiple brands or menu types in one shared location.


15

For individual use only. Duplication or distribution prohibited by law. on the kids’ menu, nutrition information availability on the menu, and many of the signage items.21 Even though the NEMS-R measures did not completely match those on the BTGFFOF, the findings from the two instruments are comparable with regard to the lower agreement found for healthy options on the dollar menu and availability of nutritional information via posted material (i.e., signage). Unlike the NEMS-R instrument, the BTG-FFOF does not comprehensively examine restaurants’ menus, consider caloric and other nutrient content, or code the number or proportion of ‘‘healthy’’ entrees and other options. Instead, it was designed to incorporate quick markers of ‘‘healthy’’ options (e.g., fruit and vegetable sides, entreé salads, low-fat/nonfat salad dressing, low-fat/nonfat milk, 100% juice, bottled water), to reduce the training burden and data collectors’ time in the field, and to measure additional policyrelevant items. Some of the lower j values found in the present study could be attributable to low occurrence rates and/or lack of variation in the study sample. For example, given the low prevalence of printed menus (expected primarily in pizzerias), any measures related to the characteristics of printed menus had a sample size of fewer than five restaurants; therefore, additional testing would help to more precisely determine the interrater reliability of these measures. Additional measures that showed little variation (e.g., parking lot lighting, bars on windows) had very low j values in spite of very high overall agreement. This suggests a possible prevalence effect because a highly skewed prevalence within the sample can result in a low j value, even with high overall agreement.43 In the case of the parking lot lighting and bars on windows, only one restaurant from the total sample was coded with a negative response and two were coded with a positive response, respectively, so the marginal totals were very unbalanced. More generally, despite efforts to sample restaurants from a fairly diverse set of neighborhoods, little variation was seen in the present study for a number of exterior restaurant characteristics. Future research using a more diverse restaurant sample, including non-

16


chain/independent fast-food restaurants, would be helpful to test agreement for these measures. For measures that had lower agreement in the present study, discussion with data collectors after fieldwork helped elucidate possible reasons for inconsistencies, and measures and training protocols were modified accordingly. For example, data collectors commented that the term ‘‘healthy’’ applied to options on the dollar menu and kids’ menu was subjective and questioned what items would qualify, even though these details were provided in the training manual. On the final instrument, this was changed to specify ‘‘a fruit, vegetable, or yogurt side’’ rather than ‘‘a healthy side.’’ Additional specifications and practice exercises were added to trainings to clarify the definition of posted material and the point-of-purchase area in restaurants with varying sizes and layouts. The exterior advertisements section of the form was modified substantially to remove counts of food and beverage (combined) ads and instead count the number of all advertisements, and from that total, all ads that had a food and that had a beverage; furthermore, price promotions were assessed across all ads. Similarly, the child-directed marketing measures were modified to eliminate any skip pattern and instead to code the presence (or absence) of all types of child-directed marketing in all restaurants. Given low reliability and data collectors’ feedback regarding lack of clarity, the quality claim measure was removed from the exterior advertisements section. Additional detail was provided in subsequent trainings to describe the area considered the restaurants’ ‘‘property’’ and to define price promotions in exterior ads. Finally, additional instructions were added for coding the presence of sidewalk lighting, including guidance regarding allowable sources of lighting and minimum coverage. Strengths and Limitations The BTG-FFOF, developed by an interdisciplinary group of researchers with input from an expert panel, is well-suited to capture policy-relevant characteristics of fast-food restaurants and adds measures and details not previously captured in restaurant envi-

ronment measurement tools, particularly the manner in which nutrition information is (or is not) made available and the absolute prices for 17 food and beverage items. With input from an expert panel during the developmental stages, the face validity of the instrument and its measures was affirmed. Researchers and practitioners interested in measuring policyrelevant characteristics of the fast-food restaurant environment that can influence consumers’ purchases, such as pricing, marketing, price promotions, and nutrition signage, can consider using the BTG-FFOF. Different from existing instruments, this tool captures absolute prices for food and beverage items, rather than relative comparisons of price within the restaurant, allowing researchers to assess differences across menu items, restaurants, and communities. It also captures detailed information on the manner in which nutrition information is (or is not) provided on site, which will help in the evaluation of the menu-labeling implementation and in identifying possible areas for improvement. While the BTG-FFOF includes some markers of healthier options on the menu, authors omitted any comprehensive menu analysis so as to reduce time required in the field. Markers of healthy options, such as low-fat milk and fruit and vegetable sides, were drawn largely from the 2010 Dietary Guidelines for Americans.44 It is important to note that the BTG-FFOF only captured the presence of items identified by the restaurant as ‘‘low-fat,’’ ‘‘low-calorie,’’ or ‘‘healthy,’’ as a more comprehensive analysis or evaluation of the actual nutritional content of the fast-food restaurant’s menu offerings was beyond the scope and objectives of the study. Future research should go beyond these quick markers of healthy options to more comprehensively consider restaurants’ offerings of whole, nutrient-dense foods versus processed foods high in refined sugars and starches. More detailed analyses of the nutritional quality of restaurants’ menu items could be undertaken using existing tools, such as the NEMS-R, and/or ingredient and nutrition information provided on companies’ Web sites. Furthermore, as the nutrition science evidence base continues to


For individual use only. Duplication or distribution prohibited by law. develop and new consensus and guidelines emerge, with regard to the role of total fat and saturated fat in chronic disease etiology,45–47 for example, instruments developed to capture the healthfulness of food environments will need to be updated. The primary limitation of the present study is that it was conducted in national chain fast-food restaurants. The sampling did not include local or regional chains or independent fastfood outlets. While nearly all measures on the BTG-FFOF are likely to be applicable in independent fast-food establishments, more information is needed on the ability to capture data reliably using the BTG-FFOF in these settings. Additionally, regional and/or urbanization-based differences in restaurant layout, menu offerings, signage, and/or pricing mechanisms might influence the ability of trained staff to capture these data reliably. Further reliability testing in other regions or in rural communities would be helpful. It is worth noting that agreement may be overestimated for select measures in the present study, given the data collection protocol. Because data collectors were instructed to ask employees jointly about measures that could not be observed, some measures, particularly food and beverage prices, may have higher agreement than would be found if data collectors asked these questions separately. Information was not collected on how often or for how many items data collectors asked employees for information; however, preliminary results from the research team’s 2010 fieldwork in 256 fast-food outlets, representing 65 major chains located across 20 U.S. communities, indicated that data collectors had to ask for the prices of 12.5% of all available food and beverage items. Implications for Research and Practice The BTG-FFOF adds to the field of food environment research by providing a tool that includes tested markers of the availability of some healthier options, numerous food and beverage prices (beyond the three items tracked by C2ER), marketing and promotional techniques (including price promotions and child-directed marketing), and detailed measures on the manner


in which nutritional information is provided on-site. Using this tool, researchers can track changes in the fastfood restaurant environment over time and assess differences across types of fast-food restaurants and communities in the availability of healthier options and nutrition information, prices of food/beverage items, and promotion of food/beverage items inside and outside of the restaurant.

SO WHAT? Implications for Health Promotion Practitioners and Researchers What is already known on this topic? A limited number of tools developed to measure characteristics of the restaurant environment have been tested for reliability. To our knowledge, none of these tools focus specifically on the fast-food environment and assess all of these characteristics: menu options, pricing, marketing, and availability of nutrition information. What does the article add? This article adds a new tool for research on the restaurant environment that includes reliable measures of healthy option availability, food and beverage prices, marketing and promotional techniques, and the manner in which nutritional information is provided on-site. What are the implications for health promotion practice or research? Future research can use the BTGFFOF to gather data on availability of food and beverage products, price and promotion of items, and the availability of nutrition information. These data will be useful for tracking trends, measuring disparities in environments, and analyzing the impact of key policy and programmatic changes.

Acknowledgments Support for this study was provided by the Robert Wood Johnson Foundation to the Bridging the Gap Program at the University of Illinois at Chicago. The authors would like to gratefully acknowledge the input provided by our expert panel when creating the original fast-food restaurant observation form: Tracy Fox, MPH, RD, consultant; Joel Gittelsohn, PhD, Johns Hopkins University; Karen Glanz, PhD, University of Pennsylvania; Valerie Quinn, MEd, California Department of Health Services; Donald (Diego) Rose, PhD, Tulane University; Joseph Sharkey, PhD, Texas A&M University; and Mary Story, PhD, Healthy Eating Research program at the University of Minnesota.

References

1. Nielsen SJ, Siega-Riz AM, Popkin B. Trends in food locations and sources among adolescents and young adults. Prev Med. 2002;35:107–113. 2. Guthrie JF, Lin BH, Frazaõ E. Role of food prepared away from home in the American diet, 1977-78 versus 1994-96: changes and consequences. J Nutr Educ Behav. 2002;34:140–150. 3. Paeratakul S, Ferdinand DP, Champagne CM, Ryan DH, Bray GA. Fast food consumption among US adults and children: dietary and nutrient intake profile. J Am Diet Assoc. 2003;103:1332– 1338. 4. US Department of Agriculture. Table 15— Sales of meals and snacks away from home by type of outlet. Available at: http://www. ers.usda.gov/Briefing/ CPIFoodAndExpenditures/Data/ Expenditures_tables/table15.htm. Accessed March 14, 2013. 5. Poti JM, Popkin BM. Trends in energy intake among US children by eating location and food source, 1977–2006. J Am Diet Assoc. 2011;111:1156–1164. 6. Powell LM, Nguyen BT, Han E. Energy intake from restaurants: demographics and socioeconomics, 2003–2008. Am J Prev Med. 2012;43:498–504. 7. French SA, Story M, Neumark-Sztainer D, et al. Fast food restaurant use among adolescents: associations with nutrient intake, food choices and behavioral and psychosocial variables. Int J Obes Relat Metab Disord. 2001;25:1823–1833. 8. Bowman S, Gortmaker S, Ebbeling C, et al. Effects of fast food consumption on energy intake and diet quality among children in a national household survey. Pediatrics. 2004;113:112–118. 9. Bowman S, Vinyard B. Fast food consumption of US adults: impacts on energy and nutrient intakes and overweight status. J Am College Nutr. 2004; 23:163–168. 10. Powell LM, Nguyen BT. Fast-food and fullservice restaurant consumption among children and adolescents. JAMA Pediatr. 2013;167(10):14–20. 11. Jeffery RW, Baxter J, McGuire M, Linde J. Are fast food restaurants an environmental risk factor for obesity? Int J Behav Nutr Phys Act. 2006;3:2. 12. Powell LM, Chaloupka FJ, Bao Y. The availability of fast-food and full-service restaurants in the United States: Associations with neighborhood characteristics. Am J Prev Med. 2007; 33(suppl 4):240–245. 13. Larson NI, Story MT, Nelson MC. Neighborhood environments: disparities in access to healthy foods in the US. Am J Prev Med. 2009;36:74–81. 14. Maddock J. The relationship between obesity and the prevalence of fast food restaurants: state-level analysis. Am J Health Promot. 2004;9:137–143. 15. Chou S-Y, Rashad I, Grossman M. Fast-Food Restaurant Advertising on Television and Its Influence on Childhood Obesity. Cambridge,


17


16. 17.

18.

19.

20.

21.

22.

23.

24.

25.

26.

18

Mass: National Bureau of Economic Research; 2006. Mehta N, Chang V. Weight status and restaurant availability: a multilevel analysis. Am J Prev Med. 2008;34:127–133. Bodor JN, Rice JC, Farley TA, et al. The association between obesity and urban food environments. J Urban Health. 2010; 87:771–781. Sturm R, Datar A. Body mass index in elementary school children, metropolitan area food prices and food outlet density. Public Health. 2005;119:1059–1068. Simmons D, McKenzie A, Eaton S, et al. Choice and availability of takeaway and restaurant food is not related to the prevalence of adult obesity in rural communities in Australia. Int J Obes (Lond). 2005;29:703–710. Powell LM, Auld MC, Chaloupka FJ, et al. Access to fast food and food prices: relationship with fruit and vegetable consumption and overweight among adolescents. Adv Health Econ Health Serv Res. 2007;17:23–48. Saelens BE, Glanz K, Sallis JF, Frank LD. Nutrition Environment Measures Study in Restaurants (NEMS-R): development and evaluation. Am J Prev Med. 2007;32:273– 281. Cerin E, Frank LD, Sallis JF, et al. From neighborhood design and food options to residents’ weight status. Appetite. 2011;56: 693–703. Glanz K, Sallis JF, Saelens BE, Frank LD. Healthy nutrition environments: concepts and measures. Am J Health Promot. 2005;19: 330–333, ii. McKinnon RA, Reedy J, Morrissette MA, et al. Measures of the food environment: a compilation of the literature, 1990–2007. Am J Prev Med. 2009;36(suppl 4):124–133. Cassady D, Housemann R, Dagher C. Measuring cues for healthy choices on restaurant menus: development and testing of a measurement instrument. Am J Health Promot. 2004;18:444–449. Story M, Sherwood NE, Himes JH, et al. An after-school obesity prevention program for African-American girls: The Minnesota GEMS pilot study. Ethn Dis. 2003;13(suppl 1):54–64.


27. Gittelsohn J, Franceschini MC, Rasooly IR, et al. Understanding the food environment in a low income urban setting: implications for food store interventions. J Hunger Environ Nutr. 2007; 2:33–50. 28. Edmonds J, Baranowski T, Baranowski J, et al. Ecological and socioeconomic correlates of fruit, juice, and vegetable consumption among African-American boys. Prev Med. 2001;32:476–481. 29. Minaker LM, Raine KD, Cash SB. Measuring the food service environment: development and implementation of assessment tools. Can J Public Health. 2009; 100:421–425. 30. Wang J, Williams M, Rush E, et al. Mapping the availability and accessibility of healthy food in rural and urban New Zealand—Te Wai o Rona: Diabetes Prevention Strategy. Public Health Nutr. 2010;13:1049–1055. 31. Baker EA, Schootman M, Barnidge E, Kelly C. The role of race and poverty in access to foods that enable individuals to adhere to dietary guidelines. Prevent Chronic Dis. 2006;3:A76. 32. Ohri-Vachaspati PC, Leviton LC. Measuring food environments: guide to available instruments. Am J Health Promot. 2010;24:410–426. 33. National Cancer Institute, Cancer Control and Population Sciences, Risk Factor Monitoring and Methods Branch. Measures of the food environment. Available at: https://riskfactor.cancer. gov/mfe/instruments/. Accessed March 14, 2013. 34. California Department of Public Health. CX3 Tier 2—Fast food marketing environment. Available at: http://www. cdph.ca.gov/programs/cpns/Pages/ CX3_T2_FF_MktEnvironSurvey.aspx. Accessed March 14, 2013. 35. Council for Community and Economic Research. Cost of Living Index Manual. Arlington, Va: Council for Community and Economic Research; 2008. (Available at: http://www.coli.org/surveyforms/ colimanual.pdf.) 36. Institute of Medicine of the National Academies. Food Marketing to Children and

37.

38.

39.

40.

41.

42.

43.

44.

45.

46.

47.

Youth: Threat or Opportunity? Washington, DC: National Academies Press; 2005. Wootan MG, Osborn M. Availability of nutrition information from chain restaurants in the United States. Am J Prev Med. 2006;30:266–268. Center for Science in the Public Interest. Menu labeling. Available at: http://www. cspinet.org/menulabeling. Accessed March 14, 2013. Powell LM, Chriqui JF, Khan T, et al. Assessing the potential effectiveness of food and beverage taxes and subsidies for improving public health: a systematic review of prices, demand and body weight outcomes. Obes Rev. 2013;14:110–128. Restaurants & Institutions Magazine. 2009 Top 400 restaurant chains. Available at: http://www.rimag.com/article/372414-R_ I_2009_Top_400_Restaurants_Chains. php. Accessed January 6, 2010. US Census Bureau. Summary file I. Technical documentation. Available at: www.census.gov/prod/cen2000/doc/sf1. pdf. Accessed May 17, 2013. Landis JR, Koch G. The measurement of observer agreement for categorical data. Biometrics. 1977;33:159–174. Cicchetti DV, Feinstein AR. High agreement but low kappa: II. Resolving the paradoxes. J Clin Epidemiol. 1990;43: 551–558. US Department of Agriculture, US Department of Health and Human Services. Dietary Guidelines for Americans, 2010. 7th ed. Washington, DC: US Government Printing Office; 2010. Siri-Tarino PW, Sun Q, Hu FB, Krauss RM. Meta-analysis of prospective cohort studies evaluating the association of saturated fat with cardiovascular disease. Am J Clin Nutr. 2010;91:535–546. Mozaffarian D. The great fat debate: taking the focus off of saturated fat. J Am Dietetic Assoc. 2011;111:665–666. Dalen JE, Devries S. Diets to prevent coronary heart disease 1957–2013: what have we learned? Am J Med [Epub ahead of print 2013 Dec 30]. doi: 10.1016/j. amjmed.2013.12.014.


www.HealthPromotionJournal.com EDITOR IN CHIEF Michael P. O’Donnell, PhD, MBA, MPH ASSOCIATE EDITORS IN CHIEF Jennifer E. Taylor, PhD Jennie Jacobs Kronenfeld, PhD Kwame Owusu-Edusei Jr., PhD* Kerry J. Redican, MPH, PhD, CHES

The Wisdom of Practice and the Rigor of Research

“The American Journal of Health Promotion provides a forum for that rare commodity — practical and intellectual exchange between researchers and practitioners.” Kenneth E. Warner, PhD Dean and Avedis Donabedian Distinguished University Professor of Public Health School of Public Health, University of Michigan

“The contents of the American Journal of Health Promotion are timely, relevant, and most important, written and reviewed by the most respected researchers in our field.” David R. Anderson, PhD, LP Senior Vice President & Chief Health Officer, StayWell Health Management

Be the first to know.

Available exclusively to ONLINE SUBSCRIBERS The American Journal of Health Promotion is now publishing all articles online, ahead of print. Articles are available as a PDF document for download as soon as they have completed the review process. This means you can access the very latest papers in the field of health promotion – in some cases up to a year before they appear in print. Subscribe Online at www.HealthPromotionJournal.com CUSTOMER SERVICE (US only) or 785-865-9402

Subscribe Today. 6 Issues/Year ISSN 0890-1171 (PRINT) ISSN 2168-6602 (ONLINE)

*Kwame Owusu-Edusei, Jr. is serving in his personal capacity. The views expressed are his own and do not necessarily represent the views of the Centers for Disease Control and Prevention or the United States Government.

AMJ_1417_AD_subscription_2015_V2NoDisc.indd 1

ANNUAL SUBSCRIPTION RATES (Effective 1-1-2015 through 12-31-2015) SUBSCRIPTION USA CANADA/ OTHER AGENCY MEXICO COUNTRIES DISCOUNT

Individual Print & Online*

$145

$154

$163

N/A

Institutional Print Only**

$191

$200

$209

10%

Tier 1: Institutional Print & Online Institutional Online Only

$373 $373

$382 $373

$391 $373

10% 10%


$477 $477

$486 $477

$495 $477

10% 10%


$581 $581

$590 $581

$599 $581

10% 10%

$904

$913

10%

University w/Archive Posting Privileges*** $895

*Individual Subscriptions must be set up in the name of a single individual and mailed to a residential address. ** Print subscriptions are one print copy per issue. For multi-site institutions wishing to have a copy sent to each location, additional subscriptions are required. Tier 1 — Most Employers and Corporations except Health Organizations, Libraries and Schools Tier 2 — Health Organizations including Hospitals, Clinics, Health Promotion Providers, Insurance Companies and Voluntary Health Agencies Tier 3 — Libraries, Colleges and Universities ***University w.Archive Posting Privileges — Allows an unlimited number of faculty, students and staff to post an unlimited number of typeset accepted manuscripts on the school’s internal archive website. Includes print and online.

12/22/14 3:44 PM

Development of a Saudi Food Frequency Questionnaire and testing its reliability and validity.

Endoscopy nurse-administered propofol sedation performance. Development of an assessment tool and a reliability testing model.

Development and psychometric testing of ventilator bundle questionnaire and observation schedule.

Student Practice Evaluation Form-Revised Edition online comment bank: development and reliability analysis.

A protocol for testing expert-system reliability.

Validity and reliability testing of the Scoliometer.

Development and reliability testing of the quality clinical placement evaluation tool.

Development of the Hirai Cancer Fatigue Scale: Testing its reliability and validity.

The Healthcare Complaints Analysis Tool: development and reliability testing of a method for service monitoring and organisational learning.

Further Development of the Assessment of Military Multitasking Performance: Iterative Reliability Testing.

Direct Observation Assessment of Milestones: Problems with Reliability.

Development and Reliability Testing of a Health Action Process Approach Inventory for Physical Activity Participation among Individuals with Schizophrenia.

[Chronic form of bird breeder's disease: about one observation].

Validity and reliability of the critical care pain observation tool: a replication study.

Reliability of indicators of nursing care quality: testing interexaminer agreement and reliability.

Development and Alternate Form Reliability of the Complex Task Performance Assessment (CTPA) for People With Mild Stroke.

The Aggression Observation Short Form identified episodes not reported on the Staff Observation Aggression Scale--Revised.

Reliability Concerns in Measuring Respondent Skin Tone by Interviewer Observation.

Reliability and validity testing of an archery chronometer.

Restaurant-associated outbreak of giardiasis.

Reliability of isometric subtalar pronator and supinator strength testing.

Validity and reliability of critical power field testing.

Development and testing of mobile technology for community park improvements: validity and reliability of the eCPAT application with youth.

ACSC Indicator: testing reliability for hypertension.