568030
research-article2015
VAWXXX10.1177/1077801214568030Violence Against WomenMacy et al.
Practice Note
Providers’ Perspectives About Helpful Information for Evaluating Domestic Violence and Sexual Assault Services: A Practice Note
Violence Against Women 2015, Vol. 21(3) 416–429 © The Author(s) 2015 Reprints and permissions: sagepub.com/journalsPermissions.nav DOI: 10.1177/1077801214568030 vaw.sagepub.com
Rebecca J. Macy1, Ijeoma Nwabuzor Ogbonnaya2, and Sandra L. Martin1
Abstract This practice note presents findings from a statewide survey of domestic violence and sexual assault agency directors (n = 80; 77% response rate), regarding their opinions about the outcome information that should be collected from victims during evaluations of five commonly provided services: legal advocacy, medical advocacy, group services, individual counseling, and shelter. The findings showed that four information types were repeatedly ranked among the most important to collect to understand whether services helped victims including victims’ satisfaction with services, victims’ progress toward meeting their goals, changes in the extent of violence and/or trauma that victims experienced, and changes in victims’ knowledge. Keywords domestic violence, evaluation, partner violence, rape, sexual assault
Domestic violence and sexual assault agencies are located in many communities throughout the United States. Typically, domestic violence and sexual assault agencies are community-based, nonprofit organizations that offer intimate and interpersonal violence victims a variety of services. These include the following: (a) legal advocacy
1University 2University
of North Carolina at Chapel Hill, USA of Southern California, Los Angeles, USA
Corresponding Author: Rebecca J. Macy, School of Social Work, University of North Carolina at Chapel Hill, Tate-Turner-Kuralt Building, 325 Pittsboro Street CB #3550, Chapel Hill, NC 27599, USA. Email: [email protected]
Downloaded from vaw.sagepub.com at The University of Auckland Library on March 24, 2015
417
Macy et al.
services to offer victims information about and support with accessing legal remedies (e.g., protection orders); (b) medical advocacy services to offer victims information and support with accessing relevant health care services (e.g., emergency contraception, rape kits, treatments for sexually transmitted infections); (c) group services to provide victims with emotional support, information, safety planning strategies, as well as connections to others who are coping with violent victimization; (d) individual counseling services to provide victims with one-on-one emotional support, information, and safety planning; and (e) shelter to provide victims and their children with a safe living environment, as well as emotional support, information, and referrals to other community services (Macy, Giattina, Sangster, Crosby, & Montijo, 2009; Riger et al., 2002). Although there are numerous domestic violence and sexual assault agencies within our country that provide important services to many violence victims, little is known about the outcomes of these services. Domestic violence and sexual assault agencies rarely conduct rigorous program evaluations (Riger et al., 2002; Wasco et al., 2004) because conducting evaluation research within domestic violence and sexual assault agencies poses many challenges. One key challenge that has been discussed extensively among both service providers and researchers concerns the type of outcome information that should be gathered from violence victims at the completion of services to assess the helpfulness of the many types of interventions provided by these agencies (Bennett, Riger, Schewe, Howard, & Wasco, 2004; Edleson & Frick, 1997; Riger et al., 2002; Sullivan, 2011; Sullivan & Alexy, 2001; Wasco et al., 2004). Despite the challenges of conducting evaluation research within domestic violence and sexual assault agencies, there are growing calls from granting agencies and policymakers for agencies to document their service outcomes (Riger et al., 2002; Sullivan 2011). Likewise, many domestic violence and sexual assault service providers would like to know how helpful their services are and to what extent their services make a difference in victims’ lives (Edleson & Frick, 1997; Sullivan & Alexy, 2001). In response to this situation, we conducted a study of domestic violence and sexual assault agency directors’ opinions regarding the types of information that should be collected from victims during outcome evaluation research. We focused on the opinions of service providers because previous research emphasized the importance of including providers’ perspectives in developing outcome evaluation procedures and tools (Edleson & Frick, 1997; Riger et al., 2002). To the best of our knowledge, however, no systematic research has been conducted to determine what types of information domestic violence and sexual assault service providers consider helpful for outcome evaluations. Furthermore, little attention has been paid to the potential harm to victims that might be posed from collecting particular types of information during outcome evaluations. Thus, evidence is needed about what types of information service providers consider most helpful to collect from victims in outcome evaluations, as well as what types of information might be potentially harmful to collect from victims. In this study, we targeted agency directors because these service providers represent a rich resource for outcome evaluation strategies that might work well in their
Downloaded from vaw.sagepub.com at The University of Auckland Library on March 24, 2015
418
Violence Against Women 21(3)
communities given their leadership positions. Accordingly, we undertook a descriptive study in which we surveyed domestic violence and sexual assault agency directors to ask them about the types of outcome information they felt would be helpful or harmful in evaluating the effectiveness of five types of commonly provided domestic violence and sexual assault services, including the following: legal advocacy, medical advocacy, support groups, individual counseling, and shelter.
Method Study Sample This study was part of a larger project that aimed to develop standardized assessment instruments and standardized outcome evaluation instruments for North Carolina domestic violence and sexual assault services (Macy, Martin, Nwabuzor, & Rizo, in press). Accordingly, potential study participants included all directors of North Carolina domestic violence and sexual assault agencies. We created a list of these directors by using information on websites of the following organizations: the North Carolina Council for Women and Domestic Violence Commission (NCCWDVC), the North Carolina Coalition Against Domestic Violence (NCCADV), and the North Carolina Coalition Against Sexual Assault (NCCASA). We confirmed the accuracy of this information by contacting each agency. The final list of 104 directors served as the study sample.
Survey Instrument The survey instrument was designed to solicit directors’ opinions about the types of outcome information they thought would be helpful or harmful in evaluating the helpfulness of five types of domestic violence and sexual assault services. The instrument was based on an extensive review of the literature concerned with domestic violence and sexual assault services, as well as our team’s prior research on these services (Macy et al., 2009; Macy, Rizo, Johns, & Ermentrout, 2013). In developing the instruments, we were also guided by prior work on the evaluation of domestic violence and sexual assault services indicating that service outcomes should be conceptualized as observable changes in victims’ behaviors, beliefs, emotional status, knowledge, life circumstances, and skills (Riger et al., 2002; Sullivan & Alexy, 2001). The draft instrument was pilot tested by staff members at NCCADV, NCCASA, and individuals with experience working in domestic violence and/or sexual assault agencies. The pilot respondents were not members of the survey sample. Feedback from the pilot was used to finalize the survey instrument. When finalized, the instrument asked about the helpfulness of assessing particular types of outcomes of five types of services commonly offered by many domestic violence and sexual assault agencies (Macy et al., 2009; Riger et al., 2002). These services included legal advocacy, medical advocacy, group services, individual counseling, and shelter that were detailed in this practice note’s introduction.
Downloaded from vaw.sagepub.com at The University of Auckland Library on March 24, 2015
419
Macy et al.
Respondents were asked to answer questions only about the types of services provided by their agency (e.g., if the agency did not provide shelter, the respondent was not to answer questions concerning shelter). For each of the five types of services, the survey instrument asked the following: “To understand whether (or to what extent) services helped victims, how helpful would it be to know the following types of information about victims after they have received services?” This question was followed by a listing of various types of outcome information (e.g., progress toward meeting goals, changes in the extent of violence and trauma experienced, changes in victims’ knowledge about emotional consequences of violence, changes in physical health). Respondents rated the helpfulness of each type of outcome information using a 7-point scale: 1 (harmful), 2 (not helpful), 3 (a little helpful), 4 (somewhat helpful), 5 (very helpful), 6 (extremely helpful), 7 (absolutely necessary). The survey also collected information concerning the characteristics of the respondent and the agency in which she or he worked, including whether the agency provided domestic violence services exclusively, sexual assault services exclusively, or both domestic violence and sexual assault services (a combined agency). Information was also collected on the respondents’ sex, race, number of years in their current position, and number of years in any position providing services related to domestic violence and/or sexual assault, and education level. A copy of the survey instrument is available from the first author upon request.
Survey Administration First, research protocols and data collection instruments were approved by the Institutional Review Board at our university. The survey was then administered over a 20-week period. Respondents could complete the survey either electronically or by using a mailed paper copy. Potential respondents received information about the webbased survey in a personalized email, as well as a paper version of the survey delivered via U.S. mail with a personalized cover letter. Typically, the director was the best person to respond to the survey. However, some agencies had an associate director or other staff member who led services. Therefore, the survey invitation asked the directors to either complete the survey themselves or designate the staff member whom they felt was most knowledgeable about their agencies’ services to complete the survey. Potential respondents were contacted several times, with all contacts offering recipients ways of opting out. Of the 104 potential respondents, 80 responded (participation rate of 77%), with 40 completing the web survey and 40 completing the mailed survey.
Analyses Descriptive statistics, including percentages, means, and standard deviations, were used to describe the characteristics of the study respondents. Means and standard deviations were also used to summarize respondents’ ratings concerning the helpfulness of
Downloaded from vaw.sagepub.com at The University of Auckland Library on March 24, 2015
420
Violence Against Women 21(3)
Table 1. Providers’ Perspectives (n = 76) Concerning the Relative Usefulness of Various Types of Outcome Information That Could Be Used to Evaluate Legal Advocacy in Helping Victims. Outcome information type Legal outcomes of victim’s case Victim’s satisfaction with legal services Progress toward meeting legal advocacy goals Changes in the extent of violence experienced Changes in knowledge concerning laws Mental health changes Changes in knowledge about emotional consequences of violence Coping strategy changes Social support changes Substance use/abuse changes Housing changes Legal/criminal status changes Parenting changes Change in ability to carry out everyday tasks Physical health changes School/employment changes
M (SD)
Rank
6.34 (0.73) 6.22 (0.70) 5.89 (0.79) 5.76 (0.97) 5.57 (0.90) 5.52 (0.99) 5.49 (0.92) 5.49 (0.97) 5.26 (0.97) 5.22 (1.28) 5.21 (1.13) 5.14 (1.54) 5.07 (1.33) 5.03 (1.22) 5.01 (1.07) 4.97 (1.08)
1 2 3 4 5 6 7 7 9 10 11 12 13 14 15 16
Note. Mean helpfulness scores of 1 = “harmful,” 2 = “not helpful,” 3 = “a little helpful,” 4 = “somewhat helpful,” 5 = “very helpful,” 6 = “extremely helpful,” and 7 = “absolutely necessary.” Ranks were assigned based on the mean helpfulness scores, with the top ranked item being the item with the highest mean score indicating the most helpfulness. If two (or more) items had the same mean score, they were ranked as tied.
the various types of information that could be used to assess the outcomes of the five types of services—that is, legal advocacy, medical advocacy, support group, individual counseling, and shelter—with higher mean scores indicating more helpful information. (The first columns of Tables 1-5 present the complete list of outcome information types in the survey by each service type.) In addition, ranks were assigned based on the respondents’ mean ratings, with the rank of “1” indicating that the information type was viewed as being most helpful by the respondents. If two or more items had the same mean rating, the items were ranked as being tied and were assigned the same ranking. In addition, we examined the number and percentage of the respondents who viewed the collection of the various types of outcome information as being potentially harmful to victims.
Results Description of the Survey Respondents Most respondents (n = 54; 69.20%) worked in combined agencies that provided both domestic violence and sexual assault services. Some (n = 14; 17.50%) worked in
Downloaded from vaw.sagepub.com at The University of Auckland Library on March 24, 2015
421
Macy et al.
Table 2. Providers’ Perspectives (n = 51) Concerning the Relative Usefulness of Various Types of Outcome Information That Could Be Used to Evaluate Medical Advocacy in Helping Victims. Outcome information type Satisfaction with medical advocacy services Knowledge concerning post-assault medical options Progress toward meeting medical goals Mental health changes Changes in the extent of violence experienced Physical health changes Changes in knowledge about emotional consequences of violence Coping strategies changes Social support changes Substance use/abuse changes Changes in ability to carry out everyday tasks Housing changes Parenting changes School/employment changes Legal/criminal status changes
M (SD)
Rank
6.27 (0.72) 6.04 (0.75) 6.02 (0.71) 5.86 (0.87) 5.76 (0.97) 5.61 (0.98) 5.59 (1.00) 5.55 (1.06) 5.37 (1.06) 5.36 (1.31) 5.08 (1.18) 4.96 (1.13) 4.78 (1.36) 4.74 (1.32) 4.63 (1.64)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Note. Mean helpfulness scores of 1 = “harmful,” 2 = “not helpful,” 3 = “a little helpful,” 4 = “somewhat helpful,” 5 = “very helpful,” 6 = “extremely helpful,” and 7 = “absolutely necessary.” Ranks were assigned based on the mean helpfulness scores, with the top ranked item being the item with the highest mean score indicating the most helpfulness. If two (or more) items had the same mean score, they were ranked as tied.
agencies that provided domestic violence services exclusively, and some (n = 10; 12.50%) worked in agencies that provided sexual assault services exclusively. Nearly half the respondents (n = 36, 45%) described their current role at their agencies as “executive director” or “program director”; 20% (n = 16) described their role as “associate director,” “coordinator,” “manager,” “shelter director,” or “supervisor”; and the others (n = 10; 12.5%) described their role as “advocate,” “case manager,” or “counselor.” More than half of the respondents (n = 50; 63.3%) reported being employed in their current position for 5 years or less. However, most respondents had longer histories of providing domestic violence or sexual assault services than was reflected by years in their current position. Among those who had previous experience working with intimate partner violence (IPV) victims, more than half (n = 46; 58.2%) had provided domestic violence services for 6 or more years. Among those who had previous experience working with sexual assault victims, nearly half (n = 37; 48.1%) had provided sexual assault services for 6 or more years. Nearly 95% (n = 75) of the respondents continued to provide direct services as some portion of their current work responsibilities, with nearly half (n = 36; 45.6%) of these respondents reporting that at least half of their work time was spent providing direct services to survivors.
Downloaded from vaw.sagepub.com at The University of Auckland Library on March 24, 2015
422
Violence Against Women 21(3)
Table 3. Providers’ Perspectives (n = 76) Concerning the Relative Usefulness of Various Types of Outcome Information That Could Be Used to Evaluate Group Services in Helping Victims. Outcome information type Satisfaction with group services Progress toward meeting their group goals Changes in knowledge concerning safety planning/strategies to increase safety Changes in the extent of violence experienced Changes in victim’s knowledge about emotional consequences of violence Coping strategies changes Mental health changes Social support changes Substance use/abuse changes Change in ability to carry out everyday tasks Parenting changes Housing changes Physical health changes School/employment changes Legal/criminal status changes
M (SD)
Rank
6.39 (0.63) 6.07 (0.65) 5.88 (0.89)
1 2 3
5.87 (0.78) 5.86 (0.67)
4 5
5.83 (0.72) 5.68 (0.93) 5.53 (0.80) 5.37 (1.12) 5.11 (1.12) 5.11 (1.09) 4.91 (1.06) 4.91 (1.16) 4.84 (1.16) 4.68 (1.46)
6 7 8 9 10 10 12 12 14 15
Note. Mean helpfulness scores of 1 = “harmful,” 2 = “not helpful,” 3 = “a little helpful,” 4 = “somewhat helpful,” 5 = “very helpful,” 6 = “extremely helpful,” and 7 = “absolutely necessary.” Ranks were assigned based on the mean helpfulness scores, with the top ranked item being the item with the highest mean score indicating the most helpfulness. If two (or more) items had the same mean score, they were ranked as tied.
Providers’ Perspectives on Outcomes to Evaluate Legal Advocacy Services Table 1 presents findings on the domestic violence and sexual assault providers’ perspectives concerning the relative usefulness of various types of outcome information that could be used to evaluate the helpfulness of legal advocacy. To help readers with interpreting the findings, we remind them of the survey’s response options: 1 = “harmful,” 2 = “not helpful,” 3 = “a little helpful,” 4 = “somewhat helpful,” 5 = “very helpful,” 6 = “extremely helpful,” and 7 = “absolutely necessary.” The study respondents were of the opinion that the top five most helpful outcomes that should be used to evaluate the effectiveness of legal advocacy were on average (a) legal outcomes of victims’ cases, rated as extremely helpful (M = 6.34; SD = 0.73); (b) victims’ satisfaction with legal advocacy services, rated as extremely helpful (M = 6.22; SD = 0.70); (c) victims’ progress toward meeting their goals for legal advocacy services, rated as very helpful (M = 5.89; SD = 0.79); (d) changes in the extent of violence and trauma experienced by victims, rated as very helpful (M = 5.76; SD = 0.97); and (e) changes in victims’ knowledge concerning laws and legal options, rated as very helpful (M = 5.57; SD = 0.90).
Downloaded from vaw.sagepub.com at The University of Auckland Library on March 24, 2015
423
Macy et al. Table 4. Providers’ Perspectives (n = 72) Concerning the Relative Usefulness of Various Types of Outcome Information That Could Be Used to Evaluate Individual Counseling in Helping Victims. Outcome information type Satisfaction with individual counseling Progress toward meeting individual counseling goals Changes in knowledge concerning safety planning/safety strategies Coping strategies changes Changes in the extent of violence experienced Changes in knowledge about emotional consequences of violence Mental health changes Social support changes Substance use/abuse changes Change in ability to carry out everyday tasks Parenting changes Physical health changes Housing changes School/employment changes Legal/criminal status changes
M (SD)
Rank
6.24 (0.62) 6.22 (0.76) 6.10 (0.83) 6.01 (0.75) 6.00 (0.82) 5.97 (0.77) 5.86 (1.04) 5.68 (0.87) 5.63 (1.18) 5.45 (1.17) 5.31 (1.18) 5.19 (1.23) 5.17 (1.03) 5.01 (1.03) 4.89 (1.45)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Note. Mean helpfulness scores of 1 = “harmful,” 2 = “not helpful,” 3 = “a little helpful,” 4 = “somewhat helpful,” 5 = “very helpful,” 6 = “extremely helpful,” and 7 = “absolutely necessary.” Ranks were assigned based on the mean helpfulness scores, with the top ranked item being the item with the highest mean score indicating the most helpfulness. If two (or more) items had the same mean score, they were ranked as tied.
Providers’ Perspectives on Outcomes to Evaluate Medical Advocacy Services Table 2 presents findings on the domestic violence and sexual assault providers’ perspectives concerning the relative usefulness of various types of outcome information that could be used to evaluate the helpfulness of medical advocacy. The study respondents were of the opinion that the top five most helpful outcomes that should be used to evaluate the effectiveness of medical advocacy were on average (a) victims’ satisfaction with medical advocacy services, rated as extremely helpful (M = 6.27; SD = 0.72); (b) changes in victims’ knowledge of post-assault medical options, rated as extremely helpful (M = 6.04; SD = 0.75); (c) victims’ progress toward meeting their goals for medical advocacy services, rated as extremely helpful (M = 6.02; SD = 0.71); (d) changes in victims’ mental health, rated as very helpful (M = 5.86; SD = 0.87); and (e) changes in the extent of violence and trauma experienced by victims, rated as very helpful (M = 5.76; SD = 0.97).
Providers’ Perspectives on Outcomes to Evaluate Group Services Table 3 presents findings on the domestic violence and sexual assault providers’ perspectives concerning the relative usefulness of various types of outcome information that could be used to evaluate the helpfulness of group services. The study respondents were
Downloaded from vaw.sagepub.com at The University of Auckland Library on March 24, 2015
424
Violence Against Women 21(3)
Table 5. Providers’ Perspectives (n = 58) Concerning the Relative Usefulness of Various Types of Outcome Information That Could Be Used to Evaluate Shelter Services in Helping Victims. Outcome information type Satisfaction with the shelter services Changes in knowledge concerning safety planning/safety strategies Progress toward meeting shelter goals Changes in the extent of violence experienced Changes in knowledge about emotional consequences of violence Legal outcomes Coping strategies changes Substance use/abuse changes Social support changes Mental health changes Housing changes Parenting changes Change in ability to carry out everyday tasks Physical health changes School/employment changes Legal/criminal status changes
M (SD)
Rank
6.34 (0.70) 6.26 (0.70) 6.19 (0.79) 6.07 (0.75) 5.98 (0.79) 5.95 (0.86) 5.91 (0.72) 5.75 (0.99) 5.74 (0.84) 5.74 (0.88) 5.63 (1.00) 5.58 (1.00) 5.42 (1.18) 5.37 (1.08) 5.29 (1.10) 5.02 (1.37)
1 2 3 4 5 6 7 8 9 9 11 12 13 14 15 16
Note. Mean helpfulness scores of 1 = “harmful,” 2 = “not helpful,” 3 = “a little helpful,” 4 = “somewhat helpful,” 5 = “very helpful,” 6 = “extremely helpful,” and 7 = “absolutely necessary.” Ranks were assigned based on the mean helpfulness scores, with the top ranked item being the item with the highest mean score indicating the most helpfulness. If two (or more) items had the same mean score, they were ranked as tied.
of the opinion that the top five most helpful outcomes that should be used to evaluate the effectiveness of group services were on average (a) victims’ satisfaction with group services, rated as extremely helpful (M = 6.39; SD = 0.63); (b) victims’ progress toward meeting their group goals, rated as extremely helpful (M = 6.07; SD = 0.65); (c) changes in victims’ knowledge regarding safety planning, rated as very helpful (M = 5.88; SD = 0.89); (d) changes in the extent of violence and trauma experienced by victims, rated as very helpful (M = 5.87; SD = 0.78); and (e) changes in victims’ knowledge about the emotional consequences of violence, rated as very helpful (M = 5.86; SD = 0.67).
Providers’ Perspectives on Outcomes to Evaluate Individual Counseling Services Table 4 presents findings on the domestic violence and sexual assault providers’ perspectives concerning the relative usefulness of various types of outcome information that could be used to evaluate the helpfulness of individual counseling. The respondents were of the opinion that the top five most helpful outcomes that should be used to evaluate the effectiveness of individual counseling services were on average (a) victims’ satisfaction with individual counseling services (M = 6.24; SD = 0.62), (b) victims’ progress toward
Downloaded from vaw.sagepub.com at The University of Auckland Library on March 24, 2015
425
Macy et al.
meeting their counseling goals (M = 6.22; SD = 0.76), (c) changes in victims’ knowledge regarding safety planning (M = 6.10; SD = 0.83), (d) changes in the victims’ coping strategies (M = 6.01; SD = 0.75), and (e) changes in the extent of violence and trauma experienced by victims (M = 5.97; SD = 0.77). All five information types were rated as being extremely helpful outcomes to assess the effects of counseling services.
Providers’ Perspectives on Outcomes to Evaluate Shelter Services Table 5 presents findings on the providers’ perspectives concerning the relative usefulness of various types of outcome information that could be used to evaluate the helpfulness of shelter services. None of the respondents from agencies that exclusively provided sexual assault services reported providing shelter services. Thus, the findings concerning the relative usefulness of various types of outcome information that could be used to evaluate shelter services are from respondents at agencies that either provided domestic violence services exclusively or respondents at agencies that provided domestic violence services in combination with sexual assault services. The study respondents were of the opinion that the top five most helpful outcomes that should be used to evaluate the effectiveness of shelter services were on average (a) victims’ satisfaction with shelter services, rated as extremely helpful (M = 6.34; SD = 0.70); (b) changes in victims’ knowledge regarding safety planning, rated as extremely helpful (M = 6.26; SD = 0.70); (c) victims’ progress toward meeting their shelter goals, rated as extremely helpful (M = 6.19; SD = 0.79); (d) changes in the extent of violence and trauma that victims experience, rated as extremely helpful (M = 6.07; SD = 0.75); and (e) changes in victims’ knowledge about the emotional consequences of violence, rated as very helpful (M = 5.98; SD = 0.79).
Providers’ Perspectives on Outcome Information That Could be Harmful to Victims As shown in Tables 1 to 5, the findings show that, on average, respondents did not rate any type of outcome information as being harmful to collect from victims as part of an outcome evaluation. Very few respondents rated any of the outcome information as being potentially harmful to collect from victims. Specifically, our review of the individual participant data showed that one respondent characterized outcome information about changes in victims’ legal and criminal status as being harmful to collect in the context of all five types of services investigated here. Three additional respondents replied that collecting outcome information on changes in the victims’ legal and criminal status could be potentially harmful to victims receiving legal advocacy services.
Discussion To the best of our knowledge, this statewide survey is among the first efforts to systematically investigate the types of information that domestic violence and sexual assault service providers consider helpful in evaluating the outcomes of their services. The study’s
Downloaded from vaw.sagepub.com at The University of Auckland Library on March 24, 2015
426
Violence Against Women 21(3)
findings offer important insights regarding the helpfulness of outcome information across different domestic violence and sexual assault types of services (i.e., legal advocacy, medical advocacy, support group services, individual counseling, and shelter). Overall, four outcome types were repeatedly ranked among the top five most important types of information to collect to understand whether services helped victims, regardless of the type of service being evaluated. These outcome information types were as follows: (a) victims’ satisfaction with services, (b) victims’ progress toward meeting their individual goals for that service, (c) changes in the extent of violence and/or trauma that the victims experienced, and (d) changes in the victims’ knowledge relevant to the specific type of service (e.g., knowledge of legal options in relation to legal advocacy, knowledge of the emotional consequences of violence in relation to shelter services). These findings are consistent with the literature concerned with recommended practices for domestic violence and sexual assault services (Macy et al., 2009; Sullivan, 2011). Providers’ perspectives on the helpfulness of collecting information concerning victims’ service satisfaction may reflect the importance that providers place on the empowerment philosophy in helping victims. Services are considered to be empowering when providers (a) collaboratively and actively involve the survivor in service planning, (b) emphasize survivors’ strengths and resources, and (c) provide information and teach skills to enhance survivors’ self-efficacy (Busch & Valentine, 2000). Victims’ satisfaction may be one way for providers to assess to what extent their services are empowering for victims. Nonetheless, prior research on the evaluation of domestic violence and sexual assault services distinguishes between victims’ satisfaction with services and service outcomes (Riger et al., 2002; Sullivan, 2011). Victims may be satisfied with services, but not helped by services in observable and measurable ways. However, this prior work also suggests that victims are unlikely to be helped with services in observable and measurable ways if they are not also satisfied with services. Thus, the current finding about the value of gathering information of victims’ satisfaction with services appears consistent with prior work positing that services should at least be satisfactory to be beneficial. Similarly, providers’ views that outcome information about victims’ progress toward meeting their individualized goals would be very helpful to collect from victims may reflect the importance that service providers place on helping victims in individualized and personalized ways (Macy et al., 2009). Consistent with this interpretation, Sullivan (2011) recommends that evaluation outcomes should account for victims’ unique and varied needs. Providers’ views on the helpfulness of collecting information about changes in the extent of trauma and violence that victims experience likely reflect the importance that service providers place on ensuring that victims are safe. Becoming safe is one of the first steps toward helping victims recover from the traumatic consequences of violent victimization (Macy et al., 2009; Sullivan, 2011). Thus, the relative importance that these respondents placed on collecting information on changes in victims’ trauma and violence may reflect the import of safety and violence cessation for victims’ recovery and well-being.
Downloaded from vaw.sagepub.com at The University of Auckland Library on March 24, 2015
427
Macy et al.
Providers’ views on the helpfulness of collecting information about changes in victims’ knowledge may reflect the importance that service providers place on the provision of information to help victims (Macy et al., 2009). Domestic violence and sexual assault service providers typically aim to help victims secure safety and trauma recovery by providing information so that victims can make cognizant decisions about the best steps for their own lives. Another important finding was that so few providers in this study were of the opinion that the collection of the various outcome information types investigated here could pose potential harm to victims. Specifically, only four of the survey respondents deemed any of the information listed in the survey as being potentially harmful to victims. In addition to the findings presented here, our team also conducted inferential analyses on the data we collected (i.e., MANOVA and post-hoc ANOVA to investigate mean differences) to assess whether there were significant average differences among providers who worked in (a) combined agencies that provided both domestic violence and sexual assault services, (b) agencies that provided domestic violence services exclusively, and (c) agencies that provided sexual assault services exclusively. These analyses showed there was considerable similarity regarding how outcome information types were ranked for understanding whether services helped victims despite differences in the types of agencies at which the respondents worked. In light of the very few significant differences determined by these analyses, we have not presented them here. We conservatively interpreted the few significant differences to be likely spurious given the number of analyses we conducted and the size of our sample. Accordingly, these nonsignificant findings suggest that researchers and service providers may find it helpful to collect similar outcome information, whether the organization provides domestic violence services, sexual assault services, or both types of services. In considering all the findings, we acknowledge our study’s limitations. The comprehensiveness of the sampling frame and high response rate (77%) provide assurance that the findings are representative of the state. However, the sample included only respondents from North Carolina, thereby limiting the study’s generalizability. Furthermore, our study had a relatively small sample overall (n = 80), with a small number of agencies providing sexual assault services in a dedicated fashion (n = 10). Consequently, our nonsignificant findings (i.e., the MANOVA and ANOVA findings described above) among participants from different agency types may be due to sample size and a lack of statistical power. Moreover, the findings are based exclusively on service providers’ perspectives. Thus, we encourage future research to investigate these issues using national samples and to include the perspectives of both service providers and victims. Furthermore, this study did not investigate one commonly offered domestic violence and sexual assault intervention, namely, crisis services, because this study’s focus was on the outcomes of generally longer term services, rather than brief services. Our decision to omit crisis services from our survey was also informed by prior research showing the problems with the feasibility, reliability, and validity of collecting data from victims who are in emergency crisis situations (Bennett et al., 2004; Riger et al., 2002; Sullivan,
Downloaded from vaw.sagepub.com at The University of Auckland Library on March 24, 2015
428
Violence Against Women 21(3)
2011; Wasco et al., 2004). Unfortunately, it was beyond the scope of this study to investigate other important challenges related to evaluation of domestic violence and sexual assault services, such as operationalizing these information types presented here for practical and feasible data collection, as well as recommending procedures for data collection in the busy service environment of a domestic violence shelter or a rape crisis center. Nonetheless, we hope that the findings from this study will offer providers and researchers guidance to inform such next-step efforts. Even with this study’s limitations, the findings provide important information about recommended domestic violence and sexual assault service outcome information. Both researchers and providers face considerable challenges in conducting ethical, rigorous, and meaningful evaluations of domestic violence and sexual assault services. Thus, we hope that this study’s findings will be used to inform the development of beneficial outcome instruments for domestic violence and sexual assault services. In turn, we hope that such instruments will be fruitfully used both by researchers and service providers to conduct valuable evaluations. It is only through such efforts that researchers and service providers will have substantial evidence regarding whether domestic violence and sexual assault services make a positive difference in victims’ lives. Acknowledgments We wish to acknowledge Carmen Crosby, Natalie Johns, Emily Pelino, Katherine Spinney, and Siobhan Young for their contributions to this research, as well as Susan White for her comments on earlier drafts of this manuscript.
Declaration of Conflicting Interests The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This project was supported by the U.S. Department of Justice, through the N.C. Department of Crime Control & Public Safety/Governor’s Crime Commission (Award No. 180-1-07-4VD-AW-192).
References Bennett, L., Riger, S., Schewe, P., Howard, A., & Wasco, S. (2004). Effectiveness of hotline, advocacy, counseling, and shelter services for victims of domestic violence: A statewide evaluation. Journal of Interpersonal Violence, 19, 815-829. Busch, N. B., & Valentine, D. (2000). Empowerment practice: A focus on battered women. Affilia, 15, 82-95. Edleson, J. L., & Frick, C. (1997). Evaluating domestic violence programs manual. Minneapolis, MN: Domestic Abuse Project. Macy, R. J., Giattina, M., Sangster, T. H., Crosby, C., & Montijo, N. J. (2009). Domestic violence and sexual assault services: Inside the blackbox. Aggression and Violent Behavior, 14, 359-373.
Downloaded from vaw.sagepub.com at The University of Auckland Library on March 24, 2015
429
Macy et al.
Macy, R. J., Martin, S., Nwabuzor, I., & Rizo, C. F. (in press). What do domestic violence and sexual assault service providers need to know about survivors to deliver services? Violence Against Women. Macy, R. J., Rizo, C. F., Johns, N. B., & Ermentrout, D. M. (2013). North Carolina directors’ opinions about domestic violence and sexual assault service strategies that help survivors. Journal of Interpersonal Violence, 28, 1040-1066. Riger, S., Bennett, L., Wasco, S. M., Schewe, P. A., Frohmann, L., Camacho, J. M., & Campbell, R. (2002). Evaluating services for survivors of domestic violence and sexual assault. Thousand Oaks, CA: Sage. Sullivan, C. M. (2011). Evaluating domestic violence support service programs: Waste of time, necessary evil, or opportunity for growth? Aggression and Violent Behavior, 16, 354-360. Sullivan, C. M., & Alexy, C. (2001). Evaluating the outcomes of domestic violence service programs: Some practical considerations and strategies. Harrisburg, PA: VAWnet. Available from http://www.vawnet.org Wasco, S. M., Campbell, R., Howard, A., Mason, G. E., Staggs, S. L., Schewe, P. A., & Riger, S. (2004). A statewide evaluation of services provided to rape survivors. Journal of Interpersonal Violence, 19, 252-263.
Author Biographies Rebecca J. Macy, PhD, ACSW, LCSW, is the L. Richardson Preyer distinguished chair for strengthening families and the associate dean for academic affairs at the University of North Carolina at Chapel Hill School of Social Work. She joined the faculty in 2002, with her doctoral degree in social welfare from the University of Washington and her MSW from Tulane University. Her research is concerned with partner violence, sexual violence, and human trafficking. Her research activities focus on the health consequences of victimization, repeated victimizations across the life span, and the development of community-based preventions and interventions to promote violence survivors’ resilience and well-being. Ijeoma Nwabuzor Ogbonnaya is a postdoctoral fellow in the School of Social Work at the University of Southern California. She earned a BA in psychology from Spelman College, and graduated with an MA in social work from the University of Pennsylvania. Her doctoral degree is from the School of Social Work at the University of North Carolina at Chapel Hill. Her research interests include the co-occurrence of domestic violence and child abuse/neglect, child welfare, domestic violence, and African American children and families. Sandra L. Martin, PhD, is an epidemiologist who currently serves as the associate dean for research in the School of Public Health and as a professor and associate chair for research within the Department of Maternal and Child Health at the University of North Carolina at Chapel Hill. Her research, teaching, and public health service focus on the health of women and children, with particular attention paid to the role that physical and sexual violence play in their lives. Her research has examined violence during pregnancy and the postpartum period, adolescent dating violence, and violence in military families.
Downloaded from vaw.sagepub.com at The University of Auckland Library on March 24, 2015
research-article2015
VAWXXX10.1177/1077801214568030Violence Against WomenMacy et al.
Practice Note
Providers’ Perspectives About Helpful Information for Evaluating Domestic Violence and Sexual Assault Services: A Practice Note
Violence Against Women 2015, Vol. 21(3) 416–429 © The Author(s) 2015 Reprints and permissions: sagepub.com/journalsPermissions.nav DOI: 10.1177/1077801214568030 vaw.sagepub.com
Rebecca J. Macy1, Ijeoma Nwabuzor Ogbonnaya2, and Sandra L. Martin1
Abstract This practice note presents findings from a statewide survey of domestic violence and sexual assault agency directors (n = 80; 77% response rate), regarding their opinions about the outcome information that should be collected from victims during evaluations of five commonly provided services: legal advocacy, medical advocacy, group services, individual counseling, and shelter. The findings showed that four information types were repeatedly ranked among the most important to collect to understand whether services helped victims including victims’ satisfaction with services, victims’ progress toward meeting their goals, changes in the extent of violence and/or trauma that victims experienced, and changes in victims’ knowledge. Keywords domestic violence, evaluation, partner violence, rape, sexual assault
Domestic violence and sexual assault agencies are located in many communities throughout the United States. Typically, domestic violence and sexual assault agencies are community-based, nonprofit organizations that offer intimate and interpersonal violence victims a variety of services. These include the following: (a) legal advocacy
1University 2University
of North Carolina at Chapel Hill, USA of Southern California, Los Angeles, USA
Corresponding Author: Rebecca J. Macy, School of Social Work, University of North Carolina at Chapel Hill, Tate-Turner-Kuralt Building, 325 Pittsboro Street CB #3550, Chapel Hill, NC 27599, USA. Email: [email protected]
Downloaded from vaw.sagepub.com at The University of Auckland Library on March 24, 2015
417
Macy et al.
services to offer victims information about and support with accessing legal remedies (e.g., protection orders); (b) medical advocacy services to offer victims information and support with accessing relevant health care services (e.g., emergency contraception, rape kits, treatments for sexually transmitted infections); (c) group services to provide victims with emotional support, information, safety planning strategies, as well as connections to others who are coping with violent victimization; (d) individual counseling services to provide victims with one-on-one emotional support, information, and safety planning; and (e) shelter to provide victims and their children with a safe living environment, as well as emotional support, information, and referrals to other community services (Macy, Giattina, Sangster, Crosby, & Montijo, 2009; Riger et al., 2002). Although there are numerous domestic violence and sexual assault agencies within our country that provide important services to many violence victims, little is known about the outcomes of these services. Domestic violence and sexual assault agencies rarely conduct rigorous program evaluations (Riger et al., 2002; Wasco et al., 2004) because conducting evaluation research within domestic violence and sexual assault agencies poses many challenges. One key challenge that has been discussed extensively among both service providers and researchers concerns the type of outcome information that should be gathered from violence victims at the completion of services to assess the helpfulness of the many types of interventions provided by these agencies (Bennett, Riger, Schewe, Howard, & Wasco, 2004; Edleson & Frick, 1997; Riger et al., 2002; Sullivan, 2011; Sullivan & Alexy, 2001; Wasco et al., 2004). Despite the challenges of conducting evaluation research within domestic violence and sexual assault agencies, there are growing calls from granting agencies and policymakers for agencies to document their service outcomes (Riger et al., 2002; Sullivan 2011). Likewise, many domestic violence and sexual assault service providers would like to know how helpful their services are and to what extent their services make a difference in victims’ lives (Edleson & Frick, 1997; Sullivan & Alexy, 2001). In response to this situation, we conducted a study of domestic violence and sexual assault agency directors’ opinions regarding the types of information that should be collected from victims during outcome evaluation research. We focused on the opinions of service providers because previous research emphasized the importance of including providers’ perspectives in developing outcome evaluation procedures and tools (Edleson & Frick, 1997; Riger et al., 2002). To the best of our knowledge, however, no systematic research has been conducted to determine what types of information domestic violence and sexual assault service providers consider helpful for outcome evaluations. Furthermore, little attention has been paid to the potential harm to victims that might be posed from collecting particular types of information during outcome evaluations. Thus, evidence is needed about what types of information service providers consider most helpful to collect from victims in outcome evaluations, as well as what types of information might be potentially harmful to collect from victims. In this study, we targeted agency directors because these service providers represent a rich resource for outcome evaluation strategies that might work well in their
Downloaded from vaw.sagepub.com at The University of Auckland Library on March 24, 2015
418
Violence Against Women 21(3)
communities given their leadership positions. Accordingly, we undertook a descriptive study in which we surveyed domestic violence and sexual assault agency directors to ask them about the types of outcome information they felt would be helpful or harmful in evaluating the effectiveness of five types of commonly provided domestic violence and sexual assault services, including the following: legal advocacy, medical advocacy, support groups, individual counseling, and shelter.
Method Study Sample This study was part of a larger project that aimed to develop standardized assessment instruments and standardized outcome evaluation instruments for North Carolina domestic violence and sexual assault services (Macy, Martin, Nwabuzor, & Rizo, in press). Accordingly, potential study participants included all directors of North Carolina domestic violence and sexual assault agencies. We created a list of these directors by using information on websites of the following organizations: the North Carolina Council for Women and Domestic Violence Commission (NCCWDVC), the North Carolina Coalition Against Domestic Violence (NCCADV), and the North Carolina Coalition Against Sexual Assault (NCCASA). We confirmed the accuracy of this information by contacting each agency. The final list of 104 directors served as the study sample.
Survey Instrument The survey instrument was designed to solicit directors’ opinions about the types of outcome information they thought would be helpful or harmful in evaluating the helpfulness of five types of domestic violence and sexual assault services. The instrument was based on an extensive review of the literature concerned with domestic violence and sexual assault services, as well as our team’s prior research on these services (Macy et al., 2009; Macy, Rizo, Johns, & Ermentrout, 2013). In developing the instruments, we were also guided by prior work on the evaluation of domestic violence and sexual assault services indicating that service outcomes should be conceptualized as observable changes in victims’ behaviors, beliefs, emotional status, knowledge, life circumstances, and skills (Riger et al., 2002; Sullivan & Alexy, 2001). The draft instrument was pilot tested by staff members at NCCADV, NCCASA, and individuals with experience working in domestic violence and/or sexual assault agencies. The pilot respondents were not members of the survey sample. Feedback from the pilot was used to finalize the survey instrument. When finalized, the instrument asked about the helpfulness of assessing particular types of outcomes of five types of services commonly offered by many domestic violence and sexual assault agencies (Macy et al., 2009; Riger et al., 2002). These services included legal advocacy, medical advocacy, group services, individual counseling, and shelter that were detailed in this practice note’s introduction.
Downloaded from vaw.sagepub.com at The University of Auckland Library on March 24, 2015
419
Macy et al.
Respondents were asked to answer questions only about the types of services provided by their agency (e.g., if the agency did not provide shelter, the respondent was not to answer questions concerning shelter). For each of the five types of services, the survey instrument asked the following: “To understand whether (or to what extent) services helped victims, how helpful would it be to know the following types of information about victims after they have received services?” This question was followed by a listing of various types of outcome information (e.g., progress toward meeting goals, changes in the extent of violence and trauma experienced, changes in victims’ knowledge about emotional consequences of violence, changes in physical health). Respondents rated the helpfulness of each type of outcome information using a 7-point scale: 1 (harmful), 2 (not helpful), 3 (a little helpful), 4 (somewhat helpful), 5 (very helpful), 6 (extremely helpful), 7 (absolutely necessary). The survey also collected information concerning the characteristics of the respondent and the agency in which she or he worked, including whether the agency provided domestic violence services exclusively, sexual assault services exclusively, or both domestic violence and sexual assault services (a combined agency). Information was also collected on the respondents’ sex, race, number of years in their current position, and number of years in any position providing services related to domestic violence and/or sexual assault, and education level. A copy of the survey instrument is available from the first author upon request.
Survey Administration First, research protocols and data collection instruments were approved by the Institutional Review Board at our university. The survey was then administered over a 20-week period. Respondents could complete the survey either electronically or by using a mailed paper copy. Potential respondents received information about the webbased survey in a personalized email, as well as a paper version of the survey delivered via U.S. mail with a personalized cover letter. Typically, the director was the best person to respond to the survey. However, some agencies had an associate director or other staff member who led services. Therefore, the survey invitation asked the directors to either complete the survey themselves or designate the staff member whom they felt was most knowledgeable about their agencies’ services to complete the survey. Potential respondents were contacted several times, with all contacts offering recipients ways of opting out. Of the 104 potential respondents, 80 responded (participation rate of 77%), with 40 completing the web survey and 40 completing the mailed survey.
Analyses Descriptive statistics, including percentages, means, and standard deviations, were used to describe the characteristics of the study respondents. Means and standard deviations were also used to summarize respondents’ ratings concerning the helpfulness of
Downloaded from vaw.sagepub.com at The University of Auckland Library on March 24, 2015
420
Violence Against Women 21(3)
Table 1. Providers’ Perspectives (n = 76) Concerning the Relative Usefulness of Various Types of Outcome Information That Could Be Used to Evaluate Legal Advocacy in Helping Victims. Outcome information type Legal outcomes of victim’s case Victim’s satisfaction with legal services Progress toward meeting legal advocacy goals Changes in the extent of violence experienced Changes in knowledge concerning laws Mental health changes Changes in knowledge about emotional consequences of violence Coping strategy changes Social support changes Substance use/abuse changes Housing changes Legal/criminal status changes Parenting changes Change in ability to carry out everyday tasks Physical health changes School/employment changes
M (SD)
Rank
6.34 (0.73) 6.22 (0.70) 5.89 (0.79) 5.76 (0.97) 5.57 (0.90) 5.52 (0.99) 5.49 (0.92) 5.49 (0.97) 5.26 (0.97) 5.22 (1.28) 5.21 (1.13) 5.14 (1.54) 5.07 (1.33) 5.03 (1.22) 5.01 (1.07) 4.97 (1.08)
1 2 3 4 5 6 7 7 9 10 11 12 13 14 15 16
Note. Mean helpfulness scores of 1 = “harmful,” 2 = “not helpful,” 3 = “a little helpful,” 4 = “somewhat helpful,” 5 = “very helpful,” 6 = “extremely helpful,” and 7 = “absolutely necessary.” Ranks were assigned based on the mean helpfulness scores, with the top ranked item being the item with the highest mean score indicating the most helpfulness. If two (or more) items had the same mean score, they were ranked as tied.
the various types of information that could be used to assess the outcomes of the five types of services—that is, legal advocacy, medical advocacy, support group, individual counseling, and shelter—with higher mean scores indicating more helpful information. (The first columns of Tables 1-5 present the complete list of outcome information types in the survey by each service type.) In addition, ranks were assigned based on the respondents’ mean ratings, with the rank of “1” indicating that the information type was viewed as being most helpful by the respondents. If two or more items had the same mean rating, the items were ranked as being tied and were assigned the same ranking. In addition, we examined the number and percentage of the respondents who viewed the collection of the various types of outcome information as being potentially harmful to victims.
Results Description of the Survey Respondents Most respondents (n = 54; 69.20%) worked in combined agencies that provided both domestic violence and sexual assault services. Some (n = 14; 17.50%) worked in
Downloaded from vaw.sagepub.com at The University of Auckland Library on March 24, 2015
421
Macy et al.
Table 2. Providers’ Perspectives (n = 51) Concerning the Relative Usefulness of Various Types of Outcome Information That Could Be Used to Evaluate Medical Advocacy in Helping Victims. Outcome information type Satisfaction with medical advocacy services Knowledge concerning post-assault medical options Progress toward meeting medical goals Mental health changes Changes in the extent of violence experienced Physical health changes Changes in knowledge about emotional consequences of violence Coping strategies changes Social support changes Substance use/abuse changes Changes in ability to carry out everyday tasks Housing changes Parenting changes School/employment changes Legal/criminal status changes
M (SD)
Rank
6.27 (0.72) 6.04 (0.75) 6.02 (0.71) 5.86 (0.87) 5.76 (0.97) 5.61 (0.98) 5.59 (1.00) 5.55 (1.06) 5.37 (1.06) 5.36 (1.31) 5.08 (1.18) 4.96 (1.13) 4.78 (1.36) 4.74 (1.32) 4.63 (1.64)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Note. Mean helpfulness scores of 1 = “harmful,” 2 = “not helpful,” 3 = “a little helpful,” 4 = “somewhat helpful,” 5 = “very helpful,” 6 = “extremely helpful,” and 7 = “absolutely necessary.” Ranks were assigned based on the mean helpfulness scores, with the top ranked item being the item with the highest mean score indicating the most helpfulness. If two (or more) items had the same mean score, they were ranked as tied.
agencies that provided domestic violence services exclusively, and some (n = 10; 12.50%) worked in agencies that provided sexual assault services exclusively. Nearly half the respondents (n = 36, 45%) described their current role at their agencies as “executive director” or “program director”; 20% (n = 16) described their role as “associate director,” “coordinator,” “manager,” “shelter director,” or “supervisor”; and the others (n = 10; 12.5%) described their role as “advocate,” “case manager,” or “counselor.” More than half of the respondents (n = 50; 63.3%) reported being employed in their current position for 5 years or less. However, most respondents had longer histories of providing domestic violence or sexual assault services than was reflected by years in their current position. Among those who had previous experience working with intimate partner violence (IPV) victims, more than half (n = 46; 58.2%) had provided domestic violence services for 6 or more years. Among those who had previous experience working with sexual assault victims, nearly half (n = 37; 48.1%) had provided sexual assault services for 6 or more years. Nearly 95% (n = 75) of the respondents continued to provide direct services as some portion of their current work responsibilities, with nearly half (n = 36; 45.6%) of these respondents reporting that at least half of their work time was spent providing direct services to survivors.
Downloaded from vaw.sagepub.com at The University of Auckland Library on March 24, 2015
422
Violence Against Women 21(3)
Table 3. Providers’ Perspectives (n = 76) Concerning the Relative Usefulness of Various Types of Outcome Information That Could Be Used to Evaluate Group Services in Helping Victims. Outcome information type Satisfaction with group services Progress toward meeting their group goals Changes in knowledge concerning safety planning/strategies to increase safety Changes in the extent of violence experienced Changes in victim’s knowledge about emotional consequences of violence Coping strategies changes Mental health changes Social support changes Substance use/abuse changes Change in ability to carry out everyday tasks Parenting changes Housing changes Physical health changes School/employment changes Legal/criminal status changes
M (SD)
Rank
6.39 (0.63) 6.07 (0.65) 5.88 (0.89)
1 2 3
5.87 (0.78) 5.86 (0.67)
4 5
5.83 (0.72) 5.68 (0.93) 5.53 (0.80) 5.37 (1.12) 5.11 (1.12) 5.11 (1.09) 4.91 (1.06) 4.91 (1.16) 4.84 (1.16) 4.68 (1.46)
6 7 8 9 10 10 12 12 14 15
Note. Mean helpfulness scores of 1 = “harmful,” 2 = “not helpful,” 3 = “a little helpful,” 4 = “somewhat helpful,” 5 = “very helpful,” 6 = “extremely helpful,” and 7 = “absolutely necessary.” Ranks were assigned based on the mean helpfulness scores, with the top ranked item being the item with the highest mean score indicating the most helpfulness. If two (or more) items had the same mean score, they were ranked as tied.
Providers’ Perspectives on Outcomes to Evaluate Legal Advocacy Services Table 1 presents findings on the domestic violence and sexual assault providers’ perspectives concerning the relative usefulness of various types of outcome information that could be used to evaluate the helpfulness of legal advocacy. To help readers with interpreting the findings, we remind them of the survey’s response options: 1 = “harmful,” 2 = “not helpful,” 3 = “a little helpful,” 4 = “somewhat helpful,” 5 = “very helpful,” 6 = “extremely helpful,” and 7 = “absolutely necessary.” The study respondents were of the opinion that the top five most helpful outcomes that should be used to evaluate the effectiveness of legal advocacy were on average (a) legal outcomes of victims’ cases, rated as extremely helpful (M = 6.34; SD = 0.73); (b) victims’ satisfaction with legal advocacy services, rated as extremely helpful (M = 6.22; SD = 0.70); (c) victims’ progress toward meeting their goals for legal advocacy services, rated as very helpful (M = 5.89; SD = 0.79); (d) changes in the extent of violence and trauma experienced by victims, rated as very helpful (M = 5.76; SD = 0.97); and (e) changes in victims’ knowledge concerning laws and legal options, rated as very helpful (M = 5.57; SD = 0.90).
Downloaded from vaw.sagepub.com at The University of Auckland Library on March 24, 2015
423
Macy et al. Table 4. Providers’ Perspectives (n = 72) Concerning the Relative Usefulness of Various Types of Outcome Information That Could Be Used to Evaluate Individual Counseling in Helping Victims. Outcome information type Satisfaction with individual counseling Progress toward meeting individual counseling goals Changes in knowledge concerning safety planning/safety strategies Coping strategies changes Changes in the extent of violence experienced Changes in knowledge about emotional consequences of violence Mental health changes Social support changes Substance use/abuse changes Change in ability to carry out everyday tasks Parenting changes Physical health changes Housing changes School/employment changes Legal/criminal status changes
M (SD)
Rank
6.24 (0.62) 6.22 (0.76) 6.10 (0.83) 6.01 (0.75) 6.00 (0.82) 5.97 (0.77) 5.86 (1.04) 5.68 (0.87) 5.63 (1.18) 5.45 (1.17) 5.31 (1.18) 5.19 (1.23) 5.17 (1.03) 5.01 (1.03) 4.89 (1.45)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Note. Mean helpfulness scores of 1 = “harmful,” 2 = “not helpful,” 3 = “a little helpful,” 4 = “somewhat helpful,” 5 = “very helpful,” 6 = “extremely helpful,” and 7 = “absolutely necessary.” Ranks were assigned based on the mean helpfulness scores, with the top ranked item being the item with the highest mean score indicating the most helpfulness. If two (or more) items had the same mean score, they were ranked as tied.
Providers’ Perspectives on Outcomes to Evaluate Medical Advocacy Services Table 2 presents findings on the domestic violence and sexual assault providers’ perspectives concerning the relative usefulness of various types of outcome information that could be used to evaluate the helpfulness of medical advocacy. The study respondents were of the opinion that the top five most helpful outcomes that should be used to evaluate the effectiveness of medical advocacy were on average (a) victims’ satisfaction with medical advocacy services, rated as extremely helpful (M = 6.27; SD = 0.72); (b) changes in victims’ knowledge of post-assault medical options, rated as extremely helpful (M = 6.04; SD = 0.75); (c) victims’ progress toward meeting their goals for medical advocacy services, rated as extremely helpful (M = 6.02; SD = 0.71); (d) changes in victims’ mental health, rated as very helpful (M = 5.86; SD = 0.87); and (e) changes in the extent of violence and trauma experienced by victims, rated as very helpful (M = 5.76; SD = 0.97).
Providers’ Perspectives on Outcomes to Evaluate Group Services Table 3 presents findings on the domestic violence and sexual assault providers’ perspectives concerning the relative usefulness of various types of outcome information that could be used to evaluate the helpfulness of group services. The study respondents were
Downloaded from vaw.sagepub.com at The University of Auckland Library on March 24, 2015
424
Violence Against Women 21(3)
Table 5. Providers’ Perspectives (n = 58) Concerning the Relative Usefulness of Various Types of Outcome Information That Could Be Used to Evaluate Shelter Services in Helping Victims. Outcome information type Satisfaction with the shelter services Changes in knowledge concerning safety planning/safety strategies Progress toward meeting shelter goals Changes in the extent of violence experienced Changes in knowledge about emotional consequences of violence Legal outcomes Coping strategies changes Substance use/abuse changes Social support changes Mental health changes Housing changes Parenting changes Change in ability to carry out everyday tasks Physical health changes School/employment changes Legal/criminal status changes
M (SD)
Rank
6.34 (0.70) 6.26 (0.70) 6.19 (0.79) 6.07 (0.75) 5.98 (0.79) 5.95 (0.86) 5.91 (0.72) 5.75 (0.99) 5.74 (0.84) 5.74 (0.88) 5.63 (1.00) 5.58 (1.00) 5.42 (1.18) 5.37 (1.08) 5.29 (1.10) 5.02 (1.37)
1 2 3 4 5 6 7 8 9 9 11 12 13 14 15 16
Note. Mean helpfulness scores of 1 = “harmful,” 2 = “not helpful,” 3 = “a little helpful,” 4 = “somewhat helpful,” 5 = “very helpful,” 6 = “extremely helpful,” and 7 = “absolutely necessary.” Ranks were assigned based on the mean helpfulness scores, with the top ranked item being the item with the highest mean score indicating the most helpfulness. If two (or more) items had the same mean score, they were ranked as tied.
of the opinion that the top five most helpful outcomes that should be used to evaluate the effectiveness of group services were on average (a) victims’ satisfaction with group services, rated as extremely helpful (M = 6.39; SD = 0.63); (b) victims’ progress toward meeting their group goals, rated as extremely helpful (M = 6.07; SD = 0.65); (c) changes in victims’ knowledge regarding safety planning, rated as very helpful (M = 5.88; SD = 0.89); (d) changes in the extent of violence and trauma experienced by victims, rated as very helpful (M = 5.87; SD = 0.78); and (e) changes in victims’ knowledge about the emotional consequences of violence, rated as very helpful (M = 5.86; SD = 0.67).
Providers’ Perspectives on Outcomes to Evaluate Individual Counseling Services Table 4 presents findings on the domestic violence and sexual assault providers’ perspectives concerning the relative usefulness of various types of outcome information that could be used to evaluate the helpfulness of individual counseling. The respondents were of the opinion that the top five most helpful outcomes that should be used to evaluate the effectiveness of individual counseling services were on average (a) victims’ satisfaction with individual counseling services (M = 6.24; SD = 0.62), (b) victims’ progress toward
Downloaded from vaw.sagepub.com at The University of Auckland Library on March 24, 2015
425
Macy et al.
meeting their counseling goals (M = 6.22; SD = 0.76), (c) changes in victims’ knowledge regarding safety planning (M = 6.10; SD = 0.83), (d) changes in the victims’ coping strategies (M = 6.01; SD = 0.75), and (e) changes in the extent of violence and trauma experienced by victims (M = 5.97; SD = 0.77). All five information types were rated as being extremely helpful outcomes to assess the effects of counseling services.
Providers’ Perspectives on Outcomes to Evaluate Shelter Services Table 5 presents findings on the providers’ perspectives concerning the relative usefulness of various types of outcome information that could be used to evaluate the helpfulness of shelter services. None of the respondents from agencies that exclusively provided sexual assault services reported providing shelter services. Thus, the findings concerning the relative usefulness of various types of outcome information that could be used to evaluate shelter services are from respondents at agencies that either provided domestic violence services exclusively or respondents at agencies that provided domestic violence services in combination with sexual assault services. The study respondents were of the opinion that the top five most helpful outcomes that should be used to evaluate the effectiveness of shelter services were on average (a) victims’ satisfaction with shelter services, rated as extremely helpful (M = 6.34; SD = 0.70); (b) changes in victims’ knowledge regarding safety planning, rated as extremely helpful (M = 6.26; SD = 0.70); (c) victims’ progress toward meeting their shelter goals, rated as extremely helpful (M = 6.19; SD = 0.79); (d) changes in the extent of violence and trauma that victims experience, rated as extremely helpful (M = 6.07; SD = 0.75); and (e) changes in victims’ knowledge about the emotional consequences of violence, rated as very helpful (M = 5.98; SD = 0.79).
Providers’ Perspectives on Outcome Information That Could be Harmful to Victims As shown in Tables 1 to 5, the findings show that, on average, respondents did not rate any type of outcome information as being harmful to collect from victims as part of an outcome evaluation. Very few respondents rated any of the outcome information as being potentially harmful to collect from victims. Specifically, our review of the individual participant data showed that one respondent characterized outcome information about changes in victims’ legal and criminal status as being harmful to collect in the context of all five types of services investigated here. Three additional respondents replied that collecting outcome information on changes in the victims’ legal and criminal status could be potentially harmful to victims receiving legal advocacy services.
Discussion To the best of our knowledge, this statewide survey is among the first efforts to systematically investigate the types of information that domestic violence and sexual assault service providers consider helpful in evaluating the outcomes of their services. The study’s
Downloaded from vaw.sagepub.com at The University of Auckland Library on March 24, 2015
426
Violence Against Women 21(3)
findings offer important insights regarding the helpfulness of outcome information across different domestic violence and sexual assault types of services (i.e., legal advocacy, medical advocacy, support group services, individual counseling, and shelter). Overall, four outcome types were repeatedly ranked among the top five most important types of information to collect to understand whether services helped victims, regardless of the type of service being evaluated. These outcome information types were as follows: (a) victims’ satisfaction with services, (b) victims’ progress toward meeting their individual goals for that service, (c) changes in the extent of violence and/or trauma that the victims experienced, and (d) changes in the victims’ knowledge relevant to the specific type of service (e.g., knowledge of legal options in relation to legal advocacy, knowledge of the emotional consequences of violence in relation to shelter services). These findings are consistent with the literature concerned with recommended practices for domestic violence and sexual assault services (Macy et al., 2009; Sullivan, 2011). Providers’ perspectives on the helpfulness of collecting information concerning victims’ service satisfaction may reflect the importance that providers place on the empowerment philosophy in helping victims. Services are considered to be empowering when providers (a) collaboratively and actively involve the survivor in service planning, (b) emphasize survivors’ strengths and resources, and (c) provide information and teach skills to enhance survivors’ self-efficacy (Busch & Valentine, 2000). Victims’ satisfaction may be one way for providers to assess to what extent their services are empowering for victims. Nonetheless, prior research on the evaluation of domestic violence and sexual assault services distinguishes between victims’ satisfaction with services and service outcomes (Riger et al., 2002; Sullivan, 2011). Victims may be satisfied with services, but not helped by services in observable and measurable ways. However, this prior work also suggests that victims are unlikely to be helped with services in observable and measurable ways if they are not also satisfied with services. Thus, the current finding about the value of gathering information of victims’ satisfaction with services appears consistent with prior work positing that services should at least be satisfactory to be beneficial. Similarly, providers’ views that outcome information about victims’ progress toward meeting their individualized goals would be very helpful to collect from victims may reflect the importance that service providers place on helping victims in individualized and personalized ways (Macy et al., 2009). Consistent with this interpretation, Sullivan (2011) recommends that evaluation outcomes should account for victims’ unique and varied needs. Providers’ views on the helpfulness of collecting information about changes in the extent of trauma and violence that victims experience likely reflect the importance that service providers place on ensuring that victims are safe. Becoming safe is one of the first steps toward helping victims recover from the traumatic consequences of violent victimization (Macy et al., 2009; Sullivan, 2011). Thus, the relative importance that these respondents placed on collecting information on changes in victims’ trauma and violence may reflect the import of safety and violence cessation for victims’ recovery and well-being.
Downloaded from vaw.sagepub.com at The University of Auckland Library on March 24, 2015
427
Macy et al.
Providers’ views on the helpfulness of collecting information about changes in victims’ knowledge may reflect the importance that service providers place on the provision of information to help victims (Macy et al., 2009). Domestic violence and sexual assault service providers typically aim to help victims secure safety and trauma recovery by providing information so that victims can make cognizant decisions about the best steps for their own lives. Another important finding was that so few providers in this study were of the opinion that the collection of the various outcome information types investigated here could pose potential harm to victims. Specifically, only four of the survey respondents deemed any of the information listed in the survey as being potentially harmful to victims. In addition to the findings presented here, our team also conducted inferential analyses on the data we collected (i.e., MANOVA and post-hoc ANOVA to investigate mean differences) to assess whether there were significant average differences among providers who worked in (a) combined agencies that provided both domestic violence and sexual assault services, (b) agencies that provided domestic violence services exclusively, and (c) agencies that provided sexual assault services exclusively. These analyses showed there was considerable similarity regarding how outcome information types were ranked for understanding whether services helped victims despite differences in the types of agencies at which the respondents worked. In light of the very few significant differences determined by these analyses, we have not presented them here. We conservatively interpreted the few significant differences to be likely spurious given the number of analyses we conducted and the size of our sample. Accordingly, these nonsignificant findings suggest that researchers and service providers may find it helpful to collect similar outcome information, whether the organization provides domestic violence services, sexual assault services, or both types of services. In considering all the findings, we acknowledge our study’s limitations. The comprehensiveness of the sampling frame and high response rate (77%) provide assurance that the findings are representative of the state. However, the sample included only respondents from North Carolina, thereby limiting the study’s generalizability. Furthermore, our study had a relatively small sample overall (n = 80), with a small number of agencies providing sexual assault services in a dedicated fashion (n = 10). Consequently, our nonsignificant findings (i.e., the MANOVA and ANOVA findings described above) among participants from different agency types may be due to sample size and a lack of statistical power. Moreover, the findings are based exclusively on service providers’ perspectives. Thus, we encourage future research to investigate these issues using national samples and to include the perspectives of both service providers and victims. Furthermore, this study did not investigate one commonly offered domestic violence and sexual assault intervention, namely, crisis services, because this study’s focus was on the outcomes of generally longer term services, rather than brief services. Our decision to omit crisis services from our survey was also informed by prior research showing the problems with the feasibility, reliability, and validity of collecting data from victims who are in emergency crisis situations (Bennett et al., 2004; Riger et al., 2002; Sullivan,
Downloaded from vaw.sagepub.com at The University of Auckland Library on March 24, 2015
428
Violence Against Women 21(3)
2011; Wasco et al., 2004). Unfortunately, it was beyond the scope of this study to investigate other important challenges related to evaluation of domestic violence and sexual assault services, such as operationalizing these information types presented here for practical and feasible data collection, as well as recommending procedures for data collection in the busy service environment of a domestic violence shelter or a rape crisis center. Nonetheless, we hope that the findings from this study will offer providers and researchers guidance to inform such next-step efforts. Even with this study’s limitations, the findings provide important information about recommended domestic violence and sexual assault service outcome information. Both researchers and providers face considerable challenges in conducting ethical, rigorous, and meaningful evaluations of domestic violence and sexual assault services. Thus, we hope that this study’s findings will be used to inform the development of beneficial outcome instruments for domestic violence and sexual assault services. In turn, we hope that such instruments will be fruitfully used both by researchers and service providers to conduct valuable evaluations. It is only through such efforts that researchers and service providers will have substantial evidence regarding whether domestic violence and sexual assault services make a positive difference in victims’ lives. Acknowledgments We wish to acknowledge Carmen Crosby, Natalie Johns, Emily Pelino, Katherine Spinney, and Siobhan Young for their contributions to this research, as well as Susan White for her comments on earlier drafts of this manuscript.
Declaration of Conflicting Interests The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This project was supported by the U.S. Department of Justice, through the N.C. Department of Crime Control & Public Safety/Governor’s Crime Commission (Award No. 180-1-07-4VD-AW-192).
References Bennett, L., Riger, S., Schewe, P., Howard, A., & Wasco, S. (2004). Effectiveness of hotline, advocacy, counseling, and shelter services for victims of domestic violence: A statewide evaluation. Journal of Interpersonal Violence, 19, 815-829. Busch, N. B., & Valentine, D. (2000). Empowerment practice: A focus on battered women. Affilia, 15, 82-95. Edleson, J. L., & Frick, C. (1997). Evaluating domestic violence programs manual. Minneapolis, MN: Domestic Abuse Project. Macy, R. J., Giattina, M., Sangster, T. H., Crosby, C., & Montijo, N. J. (2009). Domestic violence and sexual assault services: Inside the blackbox. Aggression and Violent Behavior, 14, 359-373.
Downloaded from vaw.sagepub.com at The University of Auckland Library on March 24, 2015
429
Macy et al.
Macy, R. J., Martin, S., Nwabuzor, I., & Rizo, C. F. (in press). What do domestic violence and sexual assault service providers need to know about survivors to deliver services? Violence Against Women. Macy, R. J., Rizo, C. F., Johns, N. B., & Ermentrout, D. M. (2013). North Carolina directors’ opinions about domestic violence and sexual assault service strategies that help survivors. Journal of Interpersonal Violence, 28, 1040-1066. Riger, S., Bennett, L., Wasco, S. M., Schewe, P. A., Frohmann, L., Camacho, J. M., & Campbell, R. (2002). Evaluating services for survivors of domestic violence and sexual assault. Thousand Oaks, CA: Sage. Sullivan, C. M. (2011). Evaluating domestic violence support service programs: Waste of time, necessary evil, or opportunity for growth? Aggression and Violent Behavior, 16, 354-360. Sullivan, C. M., & Alexy, C. (2001). Evaluating the outcomes of domestic violence service programs: Some practical considerations and strategies. Harrisburg, PA: VAWnet. Available from http://www.vawnet.org Wasco, S. M., Campbell, R., Howard, A., Mason, G. E., Staggs, S. L., Schewe, P. A., & Riger, S. (2004). A statewide evaluation of services provided to rape survivors. Journal of Interpersonal Violence, 19, 252-263.
Author Biographies Rebecca J. Macy, PhD, ACSW, LCSW, is the L. Richardson Preyer distinguished chair for strengthening families and the associate dean for academic affairs at the University of North Carolina at Chapel Hill School of Social Work. She joined the faculty in 2002, with her doctoral degree in social welfare from the University of Washington and her MSW from Tulane University. Her research is concerned with partner violence, sexual violence, and human trafficking. Her research activities focus on the health consequences of victimization, repeated victimizations across the life span, and the development of community-based preventions and interventions to promote violence survivors’ resilience and well-being. Ijeoma Nwabuzor Ogbonnaya is a postdoctoral fellow in the School of Social Work at the University of Southern California. She earned a BA in psychology from Spelman College, and graduated with an MA in social work from the University of Pennsylvania. Her doctoral degree is from the School of Social Work at the University of North Carolina at Chapel Hill. Her research interests include the co-occurrence of domestic violence and child abuse/neglect, child welfare, domestic violence, and African American children and families. Sandra L. Martin, PhD, is an epidemiologist who currently serves as the associate dean for research in the School of Public Health and as a professor and associate chair for research within the Department of Maternal and Child Health at the University of North Carolina at Chapel Hill. Her research, teaching, and public health service focus on the health of women and children, with particular attention paid to the role that physical and sexual violence play in their lives. Her research has examined violence during pregnancy and the postpartum period, adolescent dating violence, and violence in military families.
Downloaded from vaw.sagepub.com at The University of Auckland Library on March 24, 2015
