ALCOHOLISM: CLINICAL AND EXPERIMENTAL RESEARCH
Vol. 39, No. 4 April 2015
The Effects of Survey Modality on Adolescents’ Responses to Alcohol Use Items Melvin D. Livingston, Kelli A. Komro, and Alexander C. Wagenaar
Background: We examined differences in response to self-reported alcohol use items by survey mode, whether self-report differences were the result of modality effects or self-selection, and whether these differences varied across the treatment and control arms of a preventative intervention trial. Methods: Data from an existing alcohol prevention trial were used to estimate the effect of survey modality on adolescent’s self-reported alcohol use at ages 17 to 18. Estimates were derived from regression models controlling for self-reported alcohol use during 8th grade, measured using a single survey modality, as well as time invariant selection factors. Results: No statistically significant survey modality effects were found. No differential effects of survey modality were observed by assigned intervention group. Conclusions: We provide initial evidence that adolescent alcohol prevention trials may use multiple survey modalities when necessary to increase response rates without harming interpretation of intervention effects. Key Words: Mode, Survey, Alcohol, Intervention, Adolescent.
M
IXED-MODE DATA COLLECTION designs are increasingly common in survey research, surveillance systems, and epidemiological and intervention research (Fleming et al., 2010; Goldsmith et al., 2008; Komro et al., 2008; Pabst et al., 2010; Wells et al., 2008; White et al., 2008). The most commonly cited reasons for the use of mixedmode surveys are increased response rates and decreased study costs (de Leeuw, 2005). However, the increase in response rate gained from using mixed modes is not without potential complications. Participants’ response may vary by mode of the survey itself, introducing unfortunate confounding (Aquilino, 1994; Gfroerer et al., 1997; Wang et al., 2005). Studies to date examining modality effects of web versus paper surveys specific to alcohol questionnaires have found mixed results. Three published studies found little evidence for modality effects for web-based survey of self-reported alcohol use compared to mailed paper surveys among university students (Kypri et al., 2004; McCabe et al., 2002, 2006). Similarly, there were no significant differences in alcohol responses seen in a laboratory-based experiment of university students comparing web-based and paper and pencil surveys (Miller et al., 2002). Two studies found that the interaction of survey mode with questions of high sensitivity is likely to lead to significant survey mode effects (Aquilino,
From the Department of Health Outcomes & Policy and Institute for Child Health Policy (MDL, KAK, ACW), College of Medicine, University of Florida, Gainesville, Florida. Received for publication April 4, 2014; accepted December 19, 2014. Reprint requests: Melvin D. Livingston, PhD, P.O. Box 100177, Gainesville, FL 32610; Tel.: 352-265-2525; Fax: 352-265-8047; E-mail: [email protected] Copyright © 2015 by the Research Society on Alcoholism. DOI: 10.1111/acer.12659 710
1994; Kays et al., 2012). Different sensitivity levels may explain inconsistency in the literature on studies reporting modality effects. Studies finding little evidence of modality effects for self-reported alcohol use primarily use adult or college-aged samples, while studies finding significant modality effects occur in samples drawn from adolescent populations and diverse administration contexts (Aquilino, 1994; Gfroerer et al., 1997; Kann et al., 2002; Kays et al., 2012; McCabe et al., 2005; Rootman and Smart, 1985; Wang et al., 2005). This is consistent with the literature that has found significant differences in perceived anonymity and confidentiality between web-based and paper surveys regardless of the context the surveys were given in (Bates and Cox, 2008; Dilillo et al., 2006). As adolescents face greater social pressure not to drink compared to adults, the difference in perceived anonymity of different survey modes may be of greater importance in how truthfully adolescents respond. Studies comparing in-school paper surveys versus homeadministered paper surveys have consistently found higher reported rates of alcohol and drug use in-school-administered surveys (Gfroerer et al., 1997; Kann et al., 2002; Rootman and Smart, 1985). Kann and colleagues (2002) found significant differences in reported prevalence rates of problem behaviors between the school-based Youth Risk Behavior Survey (YRBS) and the home-based National Health Interview Survey (NHIS), including measures of alcohol use. Kann and colleagues (2002) found significant differences in the reported prevalence of the following alcohol measures: lifetime alcohol use (YRBS = 79.8%, NHIS = 68.5%, t = 6.56, p < 0.001), episodic heavy drinking (YRBS = 28.2%, NHIS = 22.5%, t = 3.21, p < 0.001), and drank alcohol before age 13 (YRBS = 33.8%, NHIS = 19.7%, t = 11.56, p < 0.001); significant differences were not found for self-reported current alcohol use (Kann Alcohol Clin Exp Res, Vol 39, No 4, 2015: pp 710–715
SURVEY MODE ON ADOLESCENT ALCOHOL ITEMS
et al., 2002). The largest differences between the school- versus home-based surveys occurred with questions regarding illegal or stigmatized behavior, leading the authors to conclude the higher rates reported via an anonymous paper and pencil survey in school to be less affected by social desirability or underreporting bias than an anonymous home-based paper and pencil survey. The resulting differences between modes are attributed to the perceived lower levels of confidentiality in household-based surveys. While studies have been carried out on the effects of survey modality on adolescent responses to alcohol use items, most of this research has focused on comparisons from national population-based surveys to assess error in population-level prevalence estimates (Gfroerer et al., 1997; Kann et al., 2002; Kays et al., 2012; McCabe et al., 2002, 2006; Rootman and Smart, 1985). National survey comparisons are useful for assessing general population-wide effects of survey modality and validating population prevalences, they do not necessarily demonstrate the presence or absence of modality effects in more specialized settings, or the potential threat to the interpretation of intervention effects in controlled trials. Exposure to an intervention could itself be selection factor when participants are allowed to choose their own survey modality. As a result, there remains a need to explore effects of survey modality in settings common to adolescent alcohol prevention trials. Previous studies have also largely examined college-(McCabe et al., 2002, 2006) or elementary-(McCabe et al., 2005; Wang et al., 2005) aged students. As a result, little is known about the influence of survey modality in trials of adolescent alcohol prevention. Therefore, we examined effects of survey modality on responses to alcohol use questions in an adolescent population drawn from a recent alcohol prevention trial. Specifically, we evaluate differences in responses to self-reported alcohol use items by survey mode, whether differences are the result of modality effects or self-selection into survey mode, and whether modality effects differed by treatment versus control condition in the underlying trial.
MATERIALS AND METHODS Study Design Data were collected as part of the Project Northland Chicago (PNC) trial (Komro et al., 2004b, 2008). PNC was a longitudinal group-randomized trial of Chicago public schools to test an alcohol and drug use preventative intervention within a multi-ethnic urban setting. Public schools in Chicago were eligible for recruitment if they included 5th through 8th grade, had 30 or more students per grade, and had low rates of annual mobility (
Vol. 39, No. 4 April 2015
The Effects of Survey Modality on Adolescents’ Responses to Alcohol Use Items Melvin D. Livingston, Kelli A. Komro, and Alexander C. Wagenaar
Background: We examined differences in response to self-reported alcohol use items by survey mode, whether self-report differences were the result of modality effects or self-selection, and whether these differences varied across the treatment and control arms of a preventative intervention trial. Methods: Data from an existing alcohol prevention trial were used to estimate the effect of survey modality on adolescent’s self-reported alcohol use at ages 17 to 18. Estimates were derived from regression models controlling for self-reported alcohol use during 8th grade, measured using a single survey modality, as well as time invariant selection factors. Results: No statistically significant survey modality effects were found. No differential effects of survey modality were observed by assigned intervention group. Conclusions: We provide initial evidence that adolescent alcohol prevention trials may use multiple survey modalities when necessary to increase response rates without harming interpretation of intervention effects. Key Words: Mode, Survey, Alcohol, Intervention, Adolescent.
M
IXED-MODE DATA COLLECTION designs are increasingly common in survey research, surveillance systems, and epidemiological and intervention research (Fleming et al., 2010; Goldsmith et al., 2008; Komro et al., 2008; Pabst et al., 2010; Wells et al., 2008; White et al., 2008). The most commonly cited reasons for the use of mixedmode surveys are increased response rates and decreased study costs (de Leeuw, 2005). However, the increase in response rate gained from using mixed modes is not without potential complications. Participants’ response may vary by mode of the survey itself, introducing unfortunate confounding (Aquilino, 1994; Gfroerer et al., 1997; Wang et al., 2005). Studies to date examining modality effects of web versus paper surveys specific to alcohol questionnaires have found mixed results. Three published studies found little evidence for modality effects for web-based survey of self-reported alcohol use compared to mailed paper surveys among university students (Kypri et al., 2004; McCabe et al., 2002, 2006). Similarly, there were no significant differences in alcohol responses seen in a laboratory-based experiment of university students comparing web-based and paper and pencil surveys (Miller et al., 2002). Two studies found that the interaction of survey mode with questions of high sensitivity is likely to lead to significant survey mode effects (Aquilino,
From the Department of Health Outcomes & Policy and Institute for Child Health Policy (MDL, KAK, ACW), College of Medicine, University of Florida, Gainesville, Florida. Received for publication April 4, 2014; accepted December 19, 2014. Reprint requests: Melvin D. Livingston, PhD, P.O. Box 100177, Gainesville, FL 32610; Tel.: 352-265-2525; Fax: 352-265-8047; E-mail: [email protected] Copyright © 2015 by the Research Society on Alcoholism. DOI: 10.1111/acer.12659 710
1994; Kays et al., 2012). Different sensitivity levels may explain inconsistency in the literature on studies reporting modality effects. Studies finding little evidence of modality effects for self-reported alcohol use primarily use adult or college-aged samples, while studies finding significant modality effects occur in samples drawn from adolescent populations and diverse administration contexts (Aquilino, 1994; Gfroerer et al., 1997; Kann et al., 2002; Kays et al., 2012; McCabe et al., 2005; Rootman and Smart, 1985; Wang et al., 2005). This is consistent with the literature that has found significant differences in perceived anonymity and confidentiality between web-based and paper surveys regardless of the context the surveys were given in (Bates and Cox, 2008; Dilillo et al., 2006). As adolescents face greater social pressure not to drink compared to adults, the difference in perceived anonymity of different survey modes may be of greater importance in how truthfully adolescents respond. Studies comparing in-school paper surveys versus homeadministered paper surveys have consistently found higher reported rates of alcohol and drug use in-school-administered surveys (Gfroerer et al., 1997; Kann et al., 2002; Rootman and Smart, 1985). Kann and colleagues (2002) found significant differences in reported prevalence rates of problem behaviors between the school-based Youth Risk Behavior Survey (YRBS) and the home-based National Health Interview Survey (NHIS), including measures of alcohol use. Kann and colleagues (2002) found significant differences in the reported prevalence of the following alcohol measures: lifetime alcohol use (YRBS = 79.8%, NHIS = 68.5%, t = 6.56, p < 0.001), episodic heavy drinking (YRBS = 28.2%, NHIS = 22.5%, t = 3.21, p < 0.001), and drank alcohol before age 13 (YRBS = 33.8%, NHIS = 19.7%, t = 11.56, p < 0.001); significant differences were not found for self-reported current alcohol use (Kann Alcohol Clin Exp Res, Vol 39, No 4, 2015: pp 710–715
SURVEY MODE ON ADOLESCENT ALCOHOL ITEMS
et al., 2002). The largest differences between the school- versus home-based surveys occurred with questions regarding illegal or stigmatized behavior, leading the authors to conclude the higher rates reported via an anonymous paper and pencil survey in school to be less affected by social desirability or underreporting bias than an anonymous home-based paper and pencil survey. The resulting differences between modes are attributed to the perceived lower levels of confidentiality in household-based surveys. While studies have been carried out on the effects of survey modality on adolescent responses to alcohol use items, most of this research has focused on comparisons from national population-based surveys to assess error in population-level prevalence estimates (Gfroerer et al., 1997; Kann et al., 2002; Kays et al., 2012; McCabe et al., 2002, 2006; Rootman and Smart, 1985). National survey comparisons are useful for assessing general population-wide effects of survey modality and validating population prevalences, they do not necessarily demonstrate the presence or absence of modality effects in more specialized settings, or the potential threat to the interpretation of intervention effects in controlled trials. Exposure to an intervention could itself be selection factor when participants are allowed to choose their own survey modality. As a result, there remains a need to explore effects of survey modality in settings common to adolescent alcohol prevention trials. Previous studies have also largely examined college-(McCabe et al., 2002, 2006) or elementary-(McCabe et al., 2005; Wang et al., 2005) aged students. As a result, little is known about the influence of survey modality in trials of adolescent alcohol prevention. Therefore, we examined effects of survey modality on responses to alcohol use questions in an adolescent population drawn from a recent alcohol prevention trial. Specifically, we evaluate differences in responses to self-reported alcohol use items by survey mode, whether differences are the result of modality effects or self-selection into survey mode, and whether modality effects differed by treatment versus control condition in the underlying trial.
MATERIALS AND METHODS Study Design Data were collected as part of the Project Northland Chicago (PNC) trial (Komro et al., 2004b, 2008). PNC was a longitudinal group-randomized trial of Chicago public schools to test an alcohol and drug use preventative intervention within a multi-ethnic urban setting. Public schools in Chicago were eligible for recruitment if they included 5th through 8th grade, had 30 or more students per grade, and had low rates of annual mobility (
