JOURNAL OF SEX RESEARCH, 52(2), 220–230, 2015 Copyright # The Society for the Scientific Study of Sexuality ISSN: 0022-4499 print=1559-8519 online DOI: 10.1080/00224499.2013.867924
Feasibility, Acceptability, and Initial Efficacy of an Online Sexual Health Promotion Program for LGBT Youth: The Queer Sex Ed Intervention Brian Mustanski, George J. Greene, and Daniel Ryan Feinberg School of Medicine, Northwestern University
Sarah W. Whitton University of Cincinnati Lesbian, gay, bisexual, and transgender (LGBT) youth experience multiple sexual health inequities driven, in part, by deficits in parental and peer support, school-based sex education programs, and community services. Research suggests that the Internet may be an important resource in the development of sexual health among LGBT youth. We examined the feasibility of recruiting youth in same-sex relationships into an online sexual health intervention, evaluated intervention acceptability, and obtained initial estimates of intervention efficacy. LGBT youth (16 to 20 years old) completed Queer Sex Ed (QSE), an online, multimedia sexual health intervention consisting of five modules. The final sample (N ¼ 202) completed the pretest, intervention, and posttest assessments. The primary study outcomes were sexual orientation identity and self-acceptance (e.g., coming-out self-efficacy), sexual health knowledge (e.g., sexual functioning), relationship variables (e.g., communication skills), and safer sex (e.g., sexual assertiveness). Analyses indicated that 15 of the 17 outcomes were found to be significant (p < .05). Effect sizes ranged from small for sexual orientation (e.g., internalized homophobia) and relationship variables (e.g., communication skills) to moderate for safer sex (e.g., contraceptive knowledge) outcomes. This study demonstrated the feasibility, acceptability, and initial efficacy of QSE, an innovative online comprehensive sexual health program for LGBT youth.
Lesbian, gay, bisexual, and transgender (LGBT) youth experience inequities in multiple areas of sexual health. Men who have sex with men (MSM) represent the majority of human immunodeficiency virus (HIV) diagnoses among youth in the United States and show the largest increase in new infections (Mustanski, Newcomb, Du Bois, Garcia, & Grov, 2011). Among female-born youth, there is some evidence of increased odds of teen pregnancy among lesbian and bisexual girls (Blake et al., 2001; Saewyc, Bearinger, Blum, & Resnick, 1999), although little research has been conducted in this area. The World Health Organization (WHO, 2006) emphasizes that sexual health is more than the absence of disease or unplanned pregnancy; it also involves a positive and respectful approach to sexuality and sexual relationships, as well as the possibility of having pleasurable and safe sexual experiences. In this definition, accepting one’s sexual orientation and gender identity Correspondence should be addressed to Brian Mustanski, Department of Medical Social Sciences, Northwestern University, 625 North Michigan Avenue, Suite 2700, Chicago, IL 60657. E-mail: [email protected]
is a core aspect of sexual health (Mustanski, Lyons, & Garcia, 2011). LGBT youth can be challenged in this respect as members of a minority group that experience sexual prejudice (Herek & McLemore, 2013); the internalization of this prejudice has been shown to have adverse mental health consequences (Hatzenbuehler, 2009; Newcomb & Mustanski, 2010). Little is known about issues related to sexual functioning (e.g., pain, dysfunction) among young people of any sexual orientation (Bancroft, 2003; for an exception, see Landry & Bergeron, 2009), and among adults, the few studies of sexual orientation differences have been inconsistent (e.g., Burri et al., 2012; Laumann, Paik, & Rosen, 1999; Sandfort & de Keizer, 2001). Finally, healthy romantic relationships are another aspect of sexual health that has been understudied among LGBT youth. Studies have documented most LGBT youth desire to be in a romantic relationship (D’Augelli, Rendina, Grossman, & Sinclair, 2007; DeHaan, Kuper, Magee, Bigelow, & Mustanski, 2013), yet they face difficulties finding partners given the relatively small number of ‘‘out’’ youth in most contexts frequented by teenagers (Mustanski, Birkett, Greene, Hatzenbuehler, & Newcomb, 2014). As with
THE QUEER SEX ED INTERVENTION
other youth, intimate partner violence is a problem among LGBT youth (Edwards & Sylaska, 2013; Halpern, Young, Waller, Martin, & Kupper, 2004). Across all of these aspects of sexual health, research on transgender youth is extraordinarily limited. The social determinants of sexual health issues among LGBT youth are manifested formally and informally at multiple ecodevelopmental levels (Mustanski, Birkett, et al., 2014; Mustanski, Newcomb, Du Bois, et al., 2011). LGBT youth frequently lack support in settings that promote the development of sexual health (Kubicek, Beyer, Weiss, Iverson, & Kipke, 2010; Mustanski, Lyons, & Garcia, 2011). Most school-based sex education programs do not address the needs of LGBT youth (Kubicek et al., 2010; Santelli et al., 2006). Parents, who play a critical role in the development of sexual health among heterosexual youth (Miller, Benson, & Galbraith, 2001), may become silent if they are unaware of or unwilling to acknowledge their child’s sexual orientation (Doty, Willoughby, Lindahl, & Malik, 2010; Friedman & Morgan, 2009; Kubicek et al., 2010). The predominance of heterosexuality means most peers are not eligible romantic partners for gay and lesbian youth, which limits their ability to explore romantic relationships and develop this competency during adolescence. LGBT youth also have few accessible role models of healthy same-sex relationships and may have less parental support for and monitoring of their romantic and sexual activities. Finally, there are relatively few community resources to meet the sexual and relationship health needs of LGBT youth. In the Centers for Disease Control and Prevention (CDC) compendium of evidence-based HIV prevention programs, there are seven best evidence interventions for youth, none of which targets adolescent MSM (CDC, 2012). Relationship education programs, with established efficacy in improving couple outcomes (Hawkins, Blanchard, Baldwin, & Fawcett, 2008), have successfully been adapted for adolescents (Adler-Baeder, Kerpelman, Schramm, Higginbotham, & Paulk, 2007) and for adult male same-sex couples (Buzzella, Whitton, & Tompson, 2012), but no programs have been developed for LGBT youth. With the shortage of positive influences in these contexts, research suggests that the Internet may be a critical resource. Research has shown that LGBT youth use the Internet to overcome deficits in offline access to sexual health information (DeHaan et al., 2013). A large survey of young MSM found the majority used the Internet to find HIV information (Mustanski, Lyons, & Garcia, 2011), and another survey found that LGBT youth were substantially more likely than their heterosexual peers to search online for sexual health information (Mitchell, Ybarra, Korchmaros, & Kosciw, 2014). Furthermore, the LGBT youth in this study were more likely to report they looked for information online because they did not have anyone to ask offline. For LGBT youth to benefit
from information technology they need to be able to access quality content that ideally has proven positive effects on sexual health. In this regard, current options are very limited. Reviews and meta-analyses of technology-based interventions show significant effects for adult MSM and heterosexual adolescents, but no studies of interventions for LGBT or MSM youth (Guse et al., 2012; Noar, Pierce, & Black, 2010). Internetdelivered relationship education programs show positive effects with adult heterosexual couples (Loew et al., 2012) but are not available for youth or LGBT individuals. Since these reviews, one study has reported significant effects of an online HIV prevention program for young MSM (Mustanski, Garofalo, Monahan, Gratzer, & Andrews, 2013), but to our knowledge there is no evaluation of online comprehensive sexual health education for LGBT youth. Here we report an initial evaluation of the Queer Sex Ed (QSE) online intervention designed to promote comprehensive sexual health of LGBT youth. Prior publications have described the mixed methods research that was used to inform the development of QSE (DeHaan et al., 2013; Magee, Bigelow, Dehaan, & Mustanski, 2012). The aims of the current study were to (1) determine the feasibility of recruiting and enrolling LGBT youth in same-sex relationships into an online sexual health intervention; (2) use mixed methods to evaluate the acceptability of and engagement with the intervention; (3) and obtain initial estimates of intervention efficacy using a one-group pretest-posttest design. Method Procedures All participants were required to complete a pretest survey, an online sexual health curriculum, and a posttest survey. Once participants completed the pretest survey they had instant access to the intervention. Posttest surveys were completed at least two weeks after intervention completion. Eligible criteria included (1) identified as LGBT or queer or reported same-sex attraction or behaviors; (2) 16 to 20 years of age; (3) lived in the United States; and (4) engaged in a romantic relationship of any duration with someone of the same biological sex, defined as ‘‘a serious partner that you’ve had vaginal, anal, or oral sex with and who you’ve had an ongoing relationship with, like a boyfriend or girlfriend, or someone you dated for a while and feel very close to.’’ The relationship requirement was included to give participants an immediate opportunity to practice the relationship and sexual health education skills they learned in the intervention. It was also adopted because research has shown the majority of unprotected sex and HIV transmission among young MSM occur in the context of a serious or main relationship (Mustanski, Newcomb, & Clerkin, 221
MUSTANSKI, GREENE, RYAN, AND WHITTON
2011; Newcomb, Ryan, Garofalo, & Mustanski, 2014; Sullivan, Salazar, Buchbinder, & Sanchez, 2009), and therefore we had a specific goal of intervening in the context of such relationships to prevent risk of sexually transmissed infection (STI) transmission. To avoid fraud and reduce the likelihood of participants completing the study multiple times, verification procedures were enacted: participants’ Internet protocol (IP) addresses were checked to verify U.S. location; participants’ names, e-mail addresses, mailing addresses, and IP addresses were checked for duplication; and public Facebook information was reviewed. To receive compensation, participants provided their name and mailing address for electronic gift cards and were informed that due to credit card fraud detection policies the card would be unusable unless the shipping address used for purchases matched the address they provided. Any suspicious cases were sent an e-mail requesting they provide further information to verify their identity and=or eligibility. Recruitment and Retention Recruitment was conducted though targeted Facebook advertisements, messages posted to TrevorSpace (a social networking site for LGBTQ youth), and organizations sharing our messages through social media. Enrollment occurred between November 2012 and April 2013.
Figure 1.
222
Figure 1 contains a flowchart illustrating recruitment and retention statistics. The final analytic sample was 202 participants that completed the pretest, intervention, and posttest assessments. Participants were compensated $35 after they had completed the posttest survey. This study was approved by the institutional review board. Intervention Description The development of the QSE intervention was guided by the information-motivation-behavioral (IMB) skills model (Fisher & Fisher, 2002). According to the IMB model, enactment of health behaviors is the result of one’s information about the health issue, one’s motivation to engage in health behaviors, and one’s behavioral skills and abilities in engaging in those health behaviors. Motivation is a particularly important element of the model for adolescent enactment of sexual health behaviors such as condom use (Mustanski, Donenberg, & Emerson, 2006). Motivation to initiate and maintain health behaviors consists of perceived vulnerability to adverse health outcomes, as well as attitudes, intentions, and perceived social norms regarding health behaviors. The QSE intervention consisted of an introduction and five intervention modules that followed a common sequencing. A female-bodied avatar named ‘‘Ed’’ served
QSE participant recruitment process Note. Characteristics are not mutually exclusive.
THE QUEER SEX ED INTERVENTION
as the intervention moderator. A video of Ed introduced the QSE program and provided a brief overview at the beginning of each module. Each module ended with a quiz about the presented materials, which was used as a didactic method to reinforce learning. If a question was answered incorrectly, the correct answer was explained. These quizzes were not part of the outcome assessment in this study and there was no item overlap. Module content was delivered in diverse media formats to appeal to different learning styles and to keep the intervention engaging. Table 1 provides an overview of the content in each module. The software that hosted the QSE intervention was configured so that participants selected user names and passwords so they could log out and log in later to continue where they left off, if necessary. Measures Demographics. The demographic questionnaire assessed participant age, race=ethnicity, birth sex, gender identity, self-reported sexual orientation, highest earned education level, and zip code. Zip code was used to classify participants as urban or rural residence by linking to a metropolitan statistical area code provided by SAS (Hadden & Zdeb, 2006). Sexual risk behavior. The HIV-Risk Assessment for Sexual Partnerships (H-RASP; Mustanski, Starks, & Newcomb, 2014) was designed to assess sexual behavior and associated situational=contextual variables at the level of the sexual partnership. The following sexual risk behavior variables were calculated: (1) number of sexual partners, (2) number of unprotected vaginal or anal sex acts with all partners, and (3) number of unprotected vaginal or anal sex acts with casual partners. The H-RASP was administered only during the pretest survey to describe the sample’s level of sexual risk engagement. Pretest-posttest measures. The measures used in evaluating the QSE intervention are shown in Table 2. Participants were offered the option to respond ‘‘I do not want to answer’’ for each item, so there was a small degree of missingness at the item level. For ordinal scale measures, the final score was calculated by taking the mean of all items as long as missing data for those items were
Feasibility, Acceptability, and Initial Efficacy of an Online Sexual Health Promotion Program for LGBT Youth: The Queer Sex Ed Intervention Brian Mustanski, George J. Greene, and Daniel Ryan Feinberg School of Medicine, Northwestern University
Sarah W. Whitton University of Cincinnati Lesbian, gay, bisexual, and transgender (LGBT) youth experience multiple sexual health inequities driven, in part, by deficits in parental and peer support, school-based sex education programs, and community services. Research suggests that the Internet may be an important resource in the development of sexual health among LGBT youth. We examined the feasibility of recruiting youth in same-sex relationships into an online sexual health intervention, evaluated intervention acceptability, and obtained initial estimates of intervention efficacy. LGBT youth (16 to 20 years old) completed Queer Sex Ed (QSE), an online, multimedia sexual health intervention consisting of five modules. The final sample (N ¼ 202) completed the pretest, intervention, and posttest assessments. The primary study outcomes were sexual orientation identity and self-acceptance (e.g., coming-out self-efficacy), sexual health knowledge (e.g., sexual functioning), relationship variables (e.g., communication skills), and safer sex (e.g., sexual assertiveness). Analyses indicated that 15 of the 17 outcomes were found to be significant (p < .05). Effect sizes ranged from small for sexual orientation (e.g., internalized homophobia) and relationship variables (e.g., communication skills) to moderate for safer sex (e.g., contraceptive knowledge) outcomes. This study demonstrated the feasibility, acceptability, and initial efficacy of QSE, an innovative online comprehensive sexual health program for LGBT youth.
Lesbian, gay, bisexual, and transgender (LGBT) youth experience inequities in multiple areas of sexual health. Men who have sex with men (MSM) represent the majority of human immunodeficiency virus (HIV) diagnoses among youth in the United States and show the largest increase in new infections (Mustanski, Newcomb, Du Bois, Garcia, & Grov, 2011). Among female-born youth, there is some evidence of increased odds of teen pregnancy among lesbian and bisexual girls (Blake et al., 2001; Saewyc, Bearinger, Blum, & Resnick, 1999), although little research has been conducted in this area. The World Health Organization (WHO, 2006) emphasizes that sexual health is more than the absence of disease or unplanned pregnancy; it also involves a positive and respectful approach to sexuality and sexual relationships, as well as the possibility of having pleasurable and safe sexual experiences. In this definition, accepting one’s sexual orientation and gender identity Correspondence should be addressed to Brian Mustanski, Department of Medical Social Sciences, Northwestern University, 625 North Michigan Avenue, Suite 2700, Chicago, IL 60657. E-mail: [email protected]
is a core aspect of sexual health (Mustanski, Lyons, & Garcia, 2011). LGBT youth can be challenged in this respect as members of a minority group that experience sexual prejudice (Herek & McLemore, 2013); the internalization of this prejudice has been shown to have adverse mental health consequences (Hatzenbuehler, 2009; Newcomb & Mustanski, 2010). Little is known about issues related to sexual functioning (e.g., pain, dysfunction) among young people of any sexual orientation (Bancroft, 2003; for an exception, see Landry & Bergeron, 2009), and among adults, the few studies of sexual orientation differences have been inconsistent (e.g., Burri et al., 2012; Laumann, Paik, & Rosen, 1999; Sandfort & de Keizer, 2001). Finally, healthy romantic relationships are another aspect of sexual health that has been understudied among LGBT youth. Studies have documented most LGBT youth desire to be in a romantic relationship (D’Augelli, Rendina, Grossman, & Sinclair, 2007; DeHaan, Kuper, Magee, Bigelow, & Mustanski, 2013), yet they face difficulties finding partners given the relatively small number of ‘‘out’’ youth in most contexts frequented by teenagers (Mustanski, Birkett, Greene, Hatzenbuehler, & Newcomb, 2014). As with
THE QUEER SEX ED INTERVENTION
other youth, intimate partner violence is a problem among LGBT youth (Edwards & Sylaska, 2013; Halpern, Young, Waller, Martin, & Kupper, 2004). Across all of these aspects of sexual health, research on transgender youth is extraordinarily limited. The social determinants of sexual health issues among LGBT youth are manifested formally and informally at multiple ecodevelopmental levels (Mustanski, Birkett, et al., 2014; Mustanski, Newcomb, Du Bois, et al., 2011). LGBT youth frequently lack support in settings that promote the development of sexual health (Kubicek, Beyer, Weiss, Iverson, & Kipke, 2010; Mustanski, Lyons, & Garcia, 2011). Most school-based sex education programs do not address the needs of LGBT youth (Kubicek et al., 2010; Santelli et al., 2006). Parents, who play a critical role in the development of sexual health among heterosexual youth (Miller, Benson, & Galbraith, 2001), may become silent if they are unaware of or unwilling to acknowledge their child’s sexual orientation (Doty, Willoughby, Lindahl, & Malik, 2010; Friedman & Morgan, 2009; Kubicek et al., 2010). The predominance of heterosexuality means most peers are not eligible romantic partners for gay and lesbian youth, which limits their ability to explore romantic relationships and develop this competency during adolescence. LGBT youth also have few accessible role models of healthy same-sex relationships and may have less parental support for and monitoring of their romantic and sexual activities. Finally, there are relatively few community resources to meet the sexual and relationship health needs of LGBT youth. In the Centers for Disease Control and Prevention (CDC) compendium of evidence-based HIV prevention programs, there are seven best evidence interventions for youth, none of which targets adolescent MSM (CDC, 2012). Relationship education programs, with established efficacy in improving couple outcomes (Hawkins, Blanchard, Baldwin, & Fawcett, 2008), have successfully been adapted for adolescents (Adler-Baeder, Kerpelman, Schramm, Higginbotham, & Paulk, 2007) and for adult male same-sex couples (Buzzella, Whitton, & Tompson, 2012), but no programs have been developed for LGBT youth. With the shortage of positive influences in these contexts, research suggests that the Internet may be a critical resource. Research has shown that LGBT youth use the Internet to overcome deficits in offline access to sexual health information (DeHaan et al., 2013). A large survey of young MSM found the majority used the Internet to find HIV information (Mustanski, Lyons, & Garcia, 2011), and another survey found that LGBT youth were substantially more likely than their heterosexual peers to search online for sexual health information (Mitchell, Ybarra, Korchmaros, & Kosciw, 2014). Furthermore, the LGBT youth in this study were more likely to report they looked for information online because they did not have anyone to ask offline. For LGBT youth to benefit
from information technology they need to be able to access quality content that ideally has proven positive effects on sexual health. In this regard, current options are very limited. Reviews and meta-analyses of technology-based interventions show significant effects for adult MSM and heterosexual adolescents, but no studies of interventions for LGBT or MSM youth (Guse et al., 2012; Noar, Pierce, & Black, 2010). Internetdelivered relationship education programs show positive effects with adult heterosexual couples (Loew et al., 2012) but are not available for youth or LGBT individuals. Since these reviews, one study has reported significant effects of an online HIV prevention program for young MSM (Mustanski, Garofalo, Monahan, Gratzer, & Andrews, 2013), but to our knowledge there is no evaluation of online comprehensive sexual health education for LGBT youth. Here we report an initial evaluation of the Queer Sex Ed (QSE) online intervention designed to promote comprehensive sexual health of LGBT youth. Prior publications have described the mixed methods research that was used to inform the development of QSE (DeHaan et al., 2013; Magee, Bigelow, Dehaan, & Mustanski, 2012). The aims of the current study were to (1) determine the feasibility of recruiting and enrolling LGBT youth in same-sex relationships into an online sexual health intervention; (2) use mixed methods to evaluate the acceptability of and engagement with the intervention; (3) and obtain initial estimates of intervention efficacy using a one-group pretest-posttest design. Method Procedures All participants were required to complete a pretest survey, an online sexual health curriculum, and a posttest survey. Once participants completed the pretest survey they had instant access to the intervention. Posttest surveys were completed at least two weeks after intervention completion. Eligible criteria included (1) identified as LGBT or queer or reported same-sex attraction or behaviors; (2) 16 to 20 years of age; (3) lived in the United States; and (4) engaged in a romantic relationship of any duration with someone of the same biological sex, defined as ‘‘a serious partner that you’ve had vaginal, anal, or oral sex with and who you’ve had an ongoing relationship with, like a boyfriend or girlfriend, or someone you dated for a while and feel very close to.’’ The relationship requirement was included to give participants an immediate opportunity to practice the relationship and sexual health education skills they learned in the intervention. It was also adopted because research has shown the majority of unprotected sex and HIV transmission among young MSM occur in the context of a serious or main relationship (Mustanski, Newcomb, & Clerkin, 221
MUSTANSKI, GREENE, RYAN, AND WHITTON
2011; Newcomb, Ryan, Garofalo, & Mustanski, 2014; Sullivan, Salazar, Buchbinder, & Sanchez, 2009), and therefore we had a specific goal of intervening in the context of such relationships to prevent risk of sexually transmissed infection (STI) transmission. To avoid fraud and reduce the likelihood of participants completing the study multiple times, verification procedures were enacted: participants’ Internet protocol (IP) addresses were checked to verify U.S. location; participants’ names, e-mail addresses, mailing addresses, and IP addresses were checked for duplication; and public Facebook information was reviewed. To receive compensation, participants provided their name and mailing address for electronic gift cards and were informed that due to credit card fraud detection policies the card would be unusable unless the shipping address used for purchases matched the address they provided. Any suspicious cases were sent an e-mail requesting they provide further information to verify their identity and=or eligibility. Recruitment and Retention Recruitment was conducted though targeted Facebook advertisements, messages posted to TrevorSpace (a social networking site for LGBTQ youth), and organizations sharing our messages through social media. Enrollment occurred between November 2012 and April 2013.
Figure 1.
222
Figure 1 contains a flowchart illustrating recruitment and retention statistics. The final analytic sample was 202 participants that completed the pretest, intervention, and posttest assessments. Participants were compensated $35 after they had completed the posttest survey. This study was approved by the institutional review board. Intervention Description The development of the QSE intervention was guided by the information-motivation-behavioral (IMB) skills model (Fisher & Fisher, 2002). According to the IMB model, enactment of health behaviors is the result of one’s information about the health issue, one’s motivation to engage in health behaviors, and one’s behavioral skills and abilities in engaging in those health behaviors. Motivation is a particularly important element of the model for adolescent enactment of sexual health behaviors such as condom use (Mustanski, Donenberg, & Emerson, 2006). Motivation to initiate and maintain health behaviors consists of perceived vulnerability to adverse health outcomes, as well as attitudes, intentions, and perceived social norms regarding health behaviors. The QSE intervention consisted of an introduction and five intervention modules that followed a common sequencing. A female-bodied avatar named ‘‘Ed’’ served
QSE participant recruitment process Note. Characteristics are not mutually exclusive.
THE QUEER SEX ED INTERVENTION
as the intervention moderator. A video of Ed introduced the QSE program and provided a brief overview at the beginning of each module. Each module ended with a quiz about the presented materials, which was used as a didactic method to reinforce learning. If a question was answered incorrectly, the correct answer was explained. These quizzes were not part of the outcome assessment in this study and there was no item overlap. Module content was delivered in diverse media formats to appeal to different learning styles and to keep the intervention engaging. Table 1 provides an overview of the content in each module. The software that hosted the QSE intervention was configured so that participants selected user names and passwords so they could log out and log in later to continue where they left off, if necessary. Measures Demographics. The demographic questionnaire assessed participant age, race=ethnicity, birth sex, gender identity, self-reported sexual orientation, highest earned education level, and zip code. Zip code was used to classify participants as urban or rural residence by linking to a metropolitan statistical area code provided by SAS (Hadden & Zdeb, 2006). Sexual risk behavior. The HIV-Risk Assessment for Sexual Partnerships (H-RASP; Mustanski, Starks, & Newcomb, 2014) was designed to assess sexual behavior and associated situational=contextual variables at the level of the sexual partnership. The following sexual risk behavior variables were calculated: (1) number of sexual partners, (2) number of unprotected vaginal or anal sex acts with all partners, and (3) number of unprotected vaginal or anal sex acts with casual partners. The H-RASP was administered only during the pretest survey to describe the sample’s level of sexual risk engagement. Pretest-posttest measures. The measures used in evaluating the QSE intervention are shown in Table 2. Participants were offered the option to respond ‘‘I do not want to answer’’ for each item, so there was a small degree of missingness at the item level. For ordinal scale measures, the final score was calculated by taking the mean of all items as long as missing data for those items were
