Evaluation and Program Planning 45 (2014) 1–8

Contents lists available at ScienceDirect

Evaluation and Program Planning journal homepage: www.elsevier.com/locate/evalprogplan

Using implementation evaluation to uncover a child neglect prevention program§ Annie Be´rube´ a,*, Vicky Lafantaisie b, Diane Dubeau a, Sylvain Coutu a, Jose´e Caron c, Annie Devault a a

Universite´ du Que´bec en Outaouais, Canada Universite´ du Que´bec a` Trois-Rivie`res, Canada c Centre de services sociaux et de la sante´ de Gatineau, Canada b

A R T I C L E I N F O

A B S T R A C T

Article history: Received 26 October 2012 Received in revised form 8 January 2014 Accepted 1 March 2014 Available online 11 March 2014

Child neglect is an ecosystemic problem with a great variety of risk factors to consider and, therefore, it requires a multimodal and individualized intervention. Although such an intervention is better for the families, it represents a great challenge for the evaluation process. Objectives: The purpose of this study is to document, using Dane and Schneider’s model (1998), the differences between the services received by parents participating in a parental group designed to prevent the presence or the recurrence of child neglect. Methods: Quantitative program implementation data was collected from 50 families who took part in a four-module program over a two-year period. Results: The results demonstrate uniformity with regard to the program’s central elements despite the differences in the services each family received. Adherence to the program was mainly respected despite slight variations in the number of sessions offered and in the group sizes. On the other hand, dosage varied greatly, with families attending from one to four offered modules. For each module, attendance varied from participation in one group session to participation in all ten group sessions. Moreover, for families who participated in at least two modules, attendance significantly increased between the first and second module. The families’ level of participation also differed, with families being rated from low to highly engaged at the end of each group session. Conclusions: Interventions must be adjusted to the specific needs of the clientele and to the characteristics of the environment in which they will be implemented. These variations could have important impacts on the effects of the intervention on the families. Therefore, evaluating these programs requires the consideration of these variations and of their repercussions on the program’s effects. ß 2014 Elsevier Ltd. All rights reserved.

Keywords: Program evaluation Child neglect Implementation

1. Introduction The evaluation of programs advocating multimodal intervention is a daunting task due to the variety of services offered. A single program may offer the possibility of participating in a great range of services such as individualized and group activities or home visits that tackle various themes. This is the case with a vast

§ This article received financial support from ministe`re de la Sante´ et des Services Sociaux, as well as from the Groupe de recherche et d’action sur la victimisation des enfants (GRAVE). * Corresponding author at: De´partement de psychologie et de psychoe´ducation, Universite´ du Que´bec en Outaouais, C.P. 1250, succ. Hull, Gatineau, Que´bec, Canada J8X 3X7. Tel.: +1 819 595 3900x2547. E-mail address: [email protected] (A. Be´rube´).

http://dx.doi.org/10.1016/j.evalprogplan.2014.03.001 0149-7189/ß 2014 Elsevier Ltd. All rights reserved.

majority of programs aiming to support families where child neglect is an issue. Two meta-analyses on interventions targeting child neglect concluded that family interventions often lead to minor changes that can be observed through small, or very small, effect sizes (MacMillan et al., 2009; Pinquart & Teubert, 2010). Indeed, the MacMillan et al. (2009) study found that there were no programs capable of preventing the reoccurrence of neglectful behavior. Furthermore, the results shown in the study performed by Barlow, Johnston, Kendrick, Polnay, and Stewart-Brown (2006) demonstrated the difficulty of modifying the trajectory of families when it came to preventing the reoccurrence of child neglect. This article aims to highlight the possible complexity and variability of an intervention program instated to meet the various

2

A. Be´rube´ et al. / Evaluation and Program Planning 45 (2014) 1–8

needs of families living in a context of neglect. What each family was exposed to should be measured before to conduct an impact evaluation. Doing otherwise could lead to great difficulties in detecting statistically significant changes when measuring effects. This article follows the special issue recently published by Administration Policy in Mental Health (2011, 38), in which many authors studied the methodology adopted in research surrounding the evaluation of services provided to children under child welfare services (Horwitz & Landsverk, 2011; Landsverk, Brown, Reutz, Palinkas, & Horwitz, 2011; Schoenwald et al., 2011). 1.1. Child neglect Child neglect is a very serious problem that modern society has not yet succeeded in stemming. In fact, current data shows that child neglect is still the most prevailing form of child maltreatment (Association des Centres jeunesse du Que´bec, 2010). According to the most recent U.S. National Incidence Study of Child Abuse and Neglect (NIS-4), when applying the Harm Standard, neglect accounts for 61% of all forms of maltreatment, affecting roughly 770 000 children in the U.S. Whereas child abuse has been declining since it reached its peak in 1993s NIS-3, child neglect has stayed at the same level. Moreover, when the more inclusive Endangerment standard is applied, taking into account injured children as well as those considered to be in danger, the data show an important increase in emotional neglect. Indeed, the number of children in that situation has more than doubled (Sedlak et al., 2010). The consequences of child neglect on children can be severe, ranging from physical and mental health problems to an increased risk of developing patterns of behavior leading to aggression and criminality (Gilbert et al., 2009). For instance, a history of either childhood emotional or physical neglect increases the odd ratio of illicit drug abuse in early adolescence by about 150% (OR of 2.4 and 2.5 for emotional and physical abuse respectively), and the lifetime odd ratio by 30–80% (OR 1.3–1.8) (Dube et al., 2006; see also Dunn et al., 2002; Topitzes, Mersky, & Reynolds, 2010). Moreover, children living in a neglectful environment appear to show difficulties mostly regarding their social and emotional development. Findings show that, after controlling for the effect of maternal depression and socio-demographic risk factors, psychologically neglectful environments are linked to internalizing and externalizing problem behaviors in children as young as three years old (Dubowitz, Pitts, & Black, 2004; Dunn et al., 2002). Furthermore, children having experienced neglect during early childhood are four times more likely to show delinquency in adulthood than children living in similar socio-demographic conditions but not exposed to this form of maltreatment (Kazemian, Spatz Widom, & Farrington, 2011). Growing up in a neglectful environment also impacts children’s emotional development. Children raised in such a context exhibit a weaker understanding of others’ negative emotions as well as a decreased capacity for emotional self-regulation (Shipman, Edwards, Brown, Swisher, & Jennings, 2005). Similarly, neglect strongly influences the intellectual capacities of the child, as demonstrated by results nearly a standard deviation below average in reading, mathematics, and intelligence (Lounds, Borkowski, & Withman, 2006). As a matter of fact, neglect’s repercussions on the cognitive, social and emotional development are more severe than those caused by physical abuse (Hildyard & Wolfe, 2002). Over the years, researchers have attempted to understand the phenomenon by which neglect is formed and perpetuated within families. There seems to be a consensus regarding the necessity of adopting an ecosystemic and developmental approach focused on the environmental components that influence the answer to the

developmental needs of the child rather than simply focusing on inadequate parental behavior (Brown, Cohen, Johnson, & Salzinger, 1998; Dubowitz & Bennet, 2007; Tanner & Turney, 2003). According to this point of view, child neglect is defined more as the absence of behaviors that are beneficial to the child than by toxic parental behavior (E´thier, Bourassa, Klapper, & Dionne, 2006). Supporting this affirmation are the findings that neglect is linked to many social factors such as single parenting, unemployment, and the lack of attendance to childcare much more so than any other forms of abuse (Mersky, Berger, Reynolds, & Gromoske, 2009). According to Trocme´ et al. (2005), neglect occurs as a result of a double disruption: (a) a disruption of the parent–child relationship characterized by the parent’s significant difficulty in demonstrating an emotional availability for the child, thus jeopardizing his/ her physical integrity and development on many levels, whether it be physical, cognitive, emotional, or social; and (b) a disruption of relationships between the family and the community, characterized by a functional and social isolation of members of the family, which leads to the absence of adequate alternatives for answering the child’s needs. 1.2. Evaluating interventions targeting child neglect The scope of the problem of neglect and its consequences on children motivates the creation of interventions targeting affected families and, the effectiveness of these interventions is at the center of a growing body of research. Many studies concentrate on maltreatment in general, thus including abusive behavior and neglect. The meta-analyses of Pinquart and Teubert (2010) and of Mikton and Butchart (2009) compile the evaluations of more than a hundred studies on the matter. Studies evaluating programs particularly aimed at neglect are rarer despite many authors insisting on the necessity of answering this problem specifically (MacMillan et al., 2009; Mersky et al., 2009). Amid the studies examining neglect, the one of DePanfillis and Dubowitz (2005) evaluated the effectiveness of home visits paired with collective activities with families identified as being at risk of neglect. Results indicated an attenuation of many risk factors associated with neglect such as parental stress, parental depression, and daily stress, as well as an increase in protective factors (i.e., parental competencies, social support, and family functioning). Overall, this resulted in an improvement of the physical and psychological care and security of the child and in a decrease of external and internal behavioral disorders in children. A similar study was completed by Barlow et al. (2007). An intensive home visit program was compared to a control program where home visits were less frequent (an average of 41.2 compared to 9.2 home visits over 18 months). The results showed no difference between the two groups for quality of the home environment and with regard to protective issues concerning the children. However, mothers’ level of sensibility significantly increased over the intervention period for the more intensive intervention. The study performed by E´thier, Couture, Lacharite´, and Gagnier (2000), which focused on families facing many risk factors but never having benefitted from services of child welfare services, drew the same conclusion. The study compared the effects of two services offered to neglectful families, one psychosocial intervention embedded in regular services to at-risk families and one multimodal program offering individualized and group interventions to parents and their children. Both interventions showed positive results by improving parent–child relationships and decreasing parental stress, depression and potential for abuse and neglect. However, only the multimodal program was associated with an improvement in social support and marital relationships.

A. Be´rube´ et al. / Evaluation and Program Planning 45 (2014) 1–8

On the other hand, other reviews of child neglect interventions highlight the fact that few studies have demonstrated solid results and that most are based on a small sample size (Allin, Wathen, & MacMillan, 2005; Barlow et al., 2006; MacMillan et al., 2009). The results of these studies tend to demonstrate the difficulty of influencing the trajectory of families already living in a context of neglect. Given the importance of the problem and the severity of its consequences on those that experience neglect, it is important to reach a consensus on the practices that lead to a noticeable improvement of the child’s situation. Two issues require further research. First, little attention has been given to the fidelity of the interventions received by participants and to how it influences the programs’ effects (Horwitz & Landsverk, 2011; Mildon & Shlonsky, 2011; Pinquart & Teubert, 2010). Many authors underline the importance of taking into account the implementation fidelity when it comes to analyzing services offered to children living in a context of great vulnerability (Landsverk et al., 2011; Pinquart & Teubert, 2010; Schoenwald et al., 2011). In fact, individualized programs show a wide variety in the kind of intervention each family actually receives. For example, in their meta-analysis, Pinquart and Teubert (2010) reviewed 142 papers documenting the effects of different prevention interventions. The average length of an intervention was of 15 months, ranging from one day to 60 months, with participants attending an average of 29 meetings, with a range of one to 421 meetings. This example illustrates how widely different the interventions received by the parents of the study sample could be. Despite these differences, the data are all agglomerated and presented together in the results section. The authors recommend verifying elements such as the duration of the intervention and the qualifications of the worker as well as the type of intervention used in order to better interpret the changes observed in the families. Mildon and Shlonsky (2011) share the same line of thinking when they underline the little attention that is given to the implementation of services offered to clients of child protective services. Implementation data allows a better understanding of specific program components and a better interpretation of the effects of the intervention in a context where uniformity of services is neither wanted nor desirable. Indeed, interventions targeting child neglect require a swift and situation-specific intervention. When such an approach is adopted, matters of dosing, quality, and participation in the program are not pre-established but rather measured in order to understand their contribution to the repercussions of the program on its participants. The World Health Organization recently released a report that highlighted the necessity of documenting the effectiveness of interventions targeting child neglect (Butchart, Harvey, Mian, & Fu¨rniss, 2006). What researchers have yet to determine is which approach is best suited to evaluating programs whose goal is to prevent or to treat the problem of neglect while numerous risk factors push parents to focus on their own needs rather than on those of their children (Brown et al., 1998; Dubowitz & Bennet, 2007). Given the adverse consequences of neglect and the contradictory conclusions to which researchers have arrived so far, we believe the evaluation of interventions targeting child neglect needs to be rethought. An evaluation combining a study of the program’s implementation and of its effects could provide answers to still unanswered questions as well as provide a deeper understanding of the characteristics of the core components of those interventions. 1.3. Implementation fidelity evaluation A growing number of studies demonstrate that it is easier to determine changes linked to a program by adopting an approach

3

that analyses its effects and takes into account the contextual elements to which participants are exposed (Domitrovich & Greenberg, 2000; Durlak & DuPre, 2008; Fagan, Hanson, Hawkins, & Arthur, 2008; Lee et al., 2008). Durlak and DuPre (2008) conclude in their meta-analysis that effect sizes are two to three times larger when analyses account for program fidelity. Gradually, researchers have reached a consensus on a definition of the elements that characterize what many refer to as an ‘‘implementation fidelity evaluation’’. Dane and Schneider (1998) have elaborated a conceptual model that aims to document the factors of a program that are likely to affect its outcome. This model has been used in other studies, such as those by Durlak and DuPre (2008), Domitrovich and Greenberg (2000), Fagan et al. (2008), Lee et al. (2008), and Paquette, Tourigny, and Joly (2009), among others. The model is composed of the following five aspects: (1) adherence or consistency between practices suggested by the program and practices actually implemented; (2) quality of the intervention (e.g., the degree of conversion of the program by the worker); (3) dosage or exposure of the participants to the program; (4) participation, such as the response of participants to the program; (5) differentiation or the difference between what is suggested by the program and what has already been implemented. The advantage of this model is that it allows for a categorization of the majority of the information gathered in the context of the implementation evaluation and that it helps establish links between the way the intervention has been implemented and the observed changes. The purpose of this study is to document, using Dane and Schneider’s model (1998), the differences between the services received by parents participating in a parental group designed to prevent the presence or the recurrence of child neglect. The research question is therefore to identify what were the program’s variations over two years regarding adherence, dosage, quality, parent participation, and differentiation. More specifically, we will study the program adherence in terms of frequency and duration of the meetings, number of parents in each group and themes broached during the meetings. For dosage, we will look at the number of sessions attended by participants and whether the attendance is related to the parents’ characteristics, the length of participation in the program or program’s topic. Regarding quality, we will examine if the quality of the group atmosphere, as well as participants’ appreciation of the program were similar for each group and whether parents consider that the program responded to their needs. Concerning the degree of participation, the idea is to identify if it varies from one participant to another and if this variability is related to parents’ characteristics, length of participation in the program or its topic. Finally, in terms of differentiation, we will examine if participants were similarly involved in other social services. This is an exploratory study conducted in order to verify the pertinence of documenting these dimensions when considering a broader evaluation of the program’s effects on participants. 2. Method 2.1. Participants Data collection was completed between the fall of 2008 and the spring of 2010. In total, 49 parents (45 mothers and 4 fathers) agreed to take part in at least one parent group. Four inclusion criteria and one exclusion criterion were observed to recruit families willing to participate in the program. First, participants must be parental figures to children aged 0–12 years old. The second criterion stipulates that families must be admitted on the basis of neglectful behavior that compromises the development or security of their children. Both of these criteria were respected by

4

A. Be´rube´ et al. / Evaluation and Program Planning 45 (2014) 1–8

all of the families enrolled. According to the third inclusion criterion, the situations of neglect must be enduring within the family despite previous interventions. It appears that all families, except for one, were currently receiving or had recently received services from the Child protection services. The final eligibility criterion concerns a factor of exclusion allowing stopping the group component of the program for parents whose issues were problematic in parent groups. During the two years the study was conducted, no parent was excluded from a group. The socio-demographic portrait of the families reveals that 38.7% of the families were headed by a single parent. About threequarters of the parents had custody of their child (69.4%). The data on the socio-demographic portrait completed by the intervenors revealed that all participants but two were considered as having financial difficulties. Drug use was prevalent in more than half of the participants (51.0%), whereas alcohol problems were present for 18.4% of parents. Symptoms of depression were also present in more than one third of the sample (36.1%). Moreover, 26.5% of participants showed signs of slow learning or intellectual disability. Finally, 85.7% of parents were considered as having a weak social network.

program philosophy, components and content are described in the program manual by Lacharite´ et al. (2005). The current study looks only at parent group activities. These activities encourage parents to look at their family’s situation through four different modules: the parent–child relationship, the parent as a parent, the parent–environment relationship, and the relationship with themselves. According to the manual, the program is based on a series of three to five modules that each includes 8–20 weekly meetings, lasting 2 or 2½ h, and gathering 8–12 participants. Themes and activities are scheduled for four modules parents (Lacharite´ et al., 2005). However, the program advocates flexibility, thus allowing parents’ individual and collective experiences to be expressed. The parent groups are co-facilitated by two psychosocial intervenors, one from a first line service (social services system) and the other from a second line service (Child protection services). Each facilitator has experience with neglecting families and received an initial program training provided by the program designer, which is followed by three to four annual training and supervision meetings. The groups take place in a community organization, putting parents into contact with three family services in their community.

2.2. The program

2.3. Measures

The Programme d’aide personnelle, familiale et communautaire – deuxie`me ge´ne´ration (PAPFC2) (Personal, family, and community assistance program, second generation) was launched by Carl Lacharite´ and his team in 2005. The first generation of this program was evaluated by E´thier et al. (2000). The results obtained at the time were modest yet positive: following their participation in the PAPFC, parents felt more competent in their roles; their support system was less restricted; and their potential for abuse and high stress levels decreased slightly. Many changes were effectuated following this evaluation, which led to the creation of the PAPFC2, now implemented in many regions of Quebec. The PAPFC2 differs from other programs devised to treat neglect. Rather than being centered on the education of the parents, it is ecosystemic-based (E´thier et al., 2000). The ecosystemic approach is often mentioned to explain the occurrence of situations of neglect, suggesting that individually observed parental characteristics are not the cause of neglect but rather that the interaction and the accumulation of various personal, interpersonal, and social factors (e.g., drug addiction, spousal abuse and poverty) are at the root of neglect (Lacharite´, E´thier, & Nolin, 2006; Tanner & Turney, 2003). Considering the fact that deficiencies are possible at different levels within these families, the PAPFC2 suggests a multi-modal approach and targets many different dimensions (e.g., the child, the parents, the parent– child relationship, the family, and the community). The program is aimed at children between the ages of 0 and 12 years and at their parents. These families are at a high risk of facing or are already facing personal, relationship, and social difficulties stemming directly from situations of neglect. In these cases, the development or safety of the participating children is either considered a concern or is compromised, and previous access to services for the children or their families did not contribute to improving their situation. The PAPFC2 consists of a global approach through which various partners (i.e., first line services, child care services, community organizations, etc.) are involved with the child and his/her family. Five main activities compose this program: (1) analysis of the child’s needs and planning of services and interventions; (2) professional, customized support; (3) direct actions aimed at the child; (4) parental group; and (5) extra-curricular support. The

Family information sheet. The program’s information sheet is filled out by the family’s main intervenor in order to refer the family to the program. The intervenor has to identify which of the children’s needs are not being met using a list (e.g., sufficient nutrition, home security, attention, affection, stimulation. . .). Intervenors also have to indicate which family risk factors characterize the child’s environment. They can chose as many factors as applicable among a list of the parents’ personal difficulties (e.g., depression, alcohol abuse, drug abuse, slow learning, intellectual disabilities, others) or social difficulties (e.g., unemployment, financial difficulties, weak social support, others). The children’s needs and the list of family risk factors were elaborated using Ward and Rose’s (2002) model of needs assessment in children. The information sheet also contains the program inclusion checklist and a history of previous services provided to the family. This last piece of information gives an overview of the challenges related to differentiating the current program from the previous services received. Log books. Log books were filled out by the group facilitators at the end of each meeting. They contained data about themes discussed during the workshop. The information was collapsed to form program adherence indicators. For each of the four proposed modules, 5–9 themes are suggested. The facilitators have to indicate which themes were tackled during the meeting and which related activities were done. For each parent group, a score was established by calculating the proportion of themes pertaining to each module. The log books also indicated the frequency and the duration of the family groups, which were then compared to the manual’s recommended frequency and duration (a measure of adherence). The presence or absence of each participant was also noted in the logbook, providing an individual measure of dosage. Finally, the log books were used to measure parents’ participation, a dimension included in Dane and Schneider’s model (1998). Facilitators indicated in the log books the names of participating parents and their individual level of participation, evaluated according to a three point scale (0 representing a lack of participation in the group, 1 representing minimal participation, and 2 being considered an excellent participation in the group). Scores were combined for each participant in order to calculate an individual participation score for each module attended.

A. Be´rube´ et al. / Evaluation and Program Planning 45 (2014) 1–8

Parents’ Workshop Appreciation Scale. Using a brief questionnaire at the end of each meeting, facilitators asked parents to rate their appreciation of the meeting. On a scale of 1–10, illustrated at both ends by a happy and an unhappy face, they rated: their appreciation of the relationship they had with the facilitators during the activity; the way the facilitators lead the meeting; the content of the meeting; and their global appreciation. This scale measures the quality of the activity from the parent’s perspective. Parents’ Global Appreciation Scale. This questionnaire was filled out by the parents at the end of their group module. In this questionnaire, parents rated their degree of satisfaction regarding the parent group. On a scale of 1–10, they rated their level of satisfaction regarding the workshops attended and whether or not their expectations concerning the parent groups were met for themselves and for their children. This scale provides a more global appreciation of the program’s quality. 2.4. Data analysis In total, eight groups of parents received the intervention during this data collection period. We conducted a series of descriptive analyses in order to obtain a description of the program’s level of adherence, by analysing the themes broached during each session as well as the frequency and duration of the meetings. We also performed descriptive analyses on the parents’ appreciation scales to account for the quality of the activities included in each session. Finally, we used inferential statistics (correlations and mean comparison) to verify if the dosage and participation level were related to the parents’ personal difficulties, to time exposure to the program or to the module’s themes.

5

one module. In the two years during which data was collected, parents participated in an average of two out of four workshop modules offered (M = 1.98, SD = 0.99). The average number of meetings parents attended during a workshop module was 5.39 (SD = 2.470), with individual scores ranging from attendance to a single meeting to attendance to all eight to ten meetings proposed within the module. Correlation analyses revealed that dosage is neither related to participants’ personal difficulties nor to depression, alcohol or drug abuse, slow learning or intellectual disabilities. Variance analyses were undertaken to verify if attendance to workshops increased with parents’ participation in the modules. It appears that the attendance of parents participating for the first time in the program was not as good as that of those participating in a second module. The analyses were performed on 30 participants that attended at least two modules. During their first participation, parents attended an average of 3.77 (SD = 2.28) meetings, while those that signed up for a second module attended an average of 6.33 (SD = 2.60) meetings (T(29) = 4.53; p < .00). This significant variation in dosage of services offered to the participants reiterates the importance of the integration of data in the implementation fidelity evaluation and of their consideration when analyzing the effects of the program. Other analyses were performed to verify whether or not the topics broached during a module had an effect on the number of workshops parents attended. The one-way analyses of variance indicated a significant difference in the participation of parents depending on the module they were exposed to (F(3,94) = 3.29; p = .024). Post hoc analyses using the Tukey HSD test indicated that the only difference occurred between participation in workshop 1, where the delicate subject of parent–child relationships was broached (M = 4.28, SD = 2.29), and in workshop 4, which concerned parents’ interactions with their environment and the services available to them (M = 6.30, SD = 2.60).

3. Results 3.3. Quality 3.1. Program adherence For this study, few components were observed for adherence: the frequency and the duration of the meetings, the number of parents per groups and the themes broached during the meetings. The program advocates the importance of offering a flexible intervention that allows for adaptations in response to the parents’ characteristics. The Outaouais region offers parents 4 types of modules. In the past years, parents have participated in 8–10 meetings per module. That fluctuation remains between the numbers of meetings recommended. The program duration also conforms to the program prescription as every meeting lasted two hours or two hours and a half. The number of participants recruited for these various models fluctuated between 8 and 16. Five of the eight groups included more than the recommended maximum of 12 participants. However, when looking at the number of participants actually present during the meetings, only one group had more than 12 parents that showed up to more than one meeting. Each module proposed between four and eight themes. The proportion of themes tackled during a module was stable between groups. However, for the two modules where the number of themes was the highest, not all the themes were tackled. In one module, eight or nine themes were presented, whereas for the other module, the groups discussed only five of the eight suggested themes. 3.2. Dosage Data related to dosage was collected from a group comprised of 49 parents (45 mothers and 4 fathers) who participated in at least

Facilitators’ evaluation of the global atmosphere during the meetings varied from one group to another. For three groups, the facilitators rated the atmosphere as excellent, with a mean of 2 out of 2. The mean atmosphere for the five other groups varied from 1.67 to 1.80. Although based on a small sample of groups (n = 8), facilitators’ perception of the atmosphere was not related to the parents’ appreciation of the activities. The relationship with the facilitators was evaluated by participants as being greatly appreciated, with scores ranging from 9.10 to 9.75 on a 10-point scale. The participants also scored their appreciation of the facilitators’ attitude as really positive, ranging from 9.22 to 9.77. Still under the umbrella of quality, the appropriateness of the services offered in relation to the needs of the family must be considered. Thus, a questionnaire aiming to address this issue was distributed to the parents at the end of the workshops and the results indicate that all of the parents except for one considered that the workshops met their needs as parents as well as the needs of their children. 3.4. Participation Results indicate that the parents’ average level of participation hovered at 1.62 (.38). For 16% of the sample, the average global participation level was below 1 whereas for 20% of the parents, the score was at the complete opposite of the spectrum, with an average score of 2 out of 2. No significant difference was noticeable when comparing parents’ level of participation during the first and second workshop. Similarly, it appears that the parents’ level of participation was not influenced by the topic of the module they

6

A. Be´rube´ et al. / Evaluation and Program Planning 45 (2014) 1–8

were participating in. Correlation analyses were performed in order to examine the relationship between parents’ participation and certain personal characteristics. The degree of parental participation during the group meetings was not related to depression, alcohol or drug abuse, slow learning or intellectual disabilities. 3.5. Differentiation We looked at services received by the families prior to the intervention. More than two thirds of the sample had received services from a first line services (68%). A majority of participants (53.1%) had received services from both the social services system and the Child protection services before taking part in the program. Only two participants had not used one service or the other. Finally, facilitators reported the presence of a community organization in nearly a quarter of the families (26%). However, current data does not allow for an in-depth study on the difference between what these programs offer or have offered to participants and the specific components of the program. 4. Discussion Neglect is an important societal problem, whether in terms of prevalence or in terms of the consequences for the children. Many different initiatives attempt to answer this worrisome problem. However, it seems to be particularly difficult to influence parental response once neglect is a reality, as is demonstrated by the failure of existing programs in producing significant or meaningful changes in these families (Allin et al., 2005; Barlow et al., 2006; MacMillan et al., 2009). This situation highlights the importance of reconsidering the way interventions are evaluated, thus leading to a better understanding of which conditions are required in order to make a difference in the lives of children and of their parents. The evaluation must consider exactly what is offered to participants, which is referred to by many authors as program implementation fidelity (Dane & Schneider, 1998; Domitrovich & Greenberg, 2000; Durlak & DuPre; Fagan et al., 2008; Paquette et al., 2009). This type of analysis answers many questions such as: the impact of dosage or level of families’ participation on the program’s effectiveness; the minimal quality required of an intervention and what it is composed of; and how interventions offered through so-called innovative programs differ from rather traditional services. The results of the current study demonstrate a uniformity of the program’s central elements despite variances occurring with regard to the services each participant benefitted from. We based our analysis on Dane and Schneider’s model (1998) in order to examine how much variation in the program could be disregarded by a traditional effect evaluation that would compare participants’ results before and after the intervention. We first looked at program’s adherence, which requires a comparison between the various elements of the program as they were implemented in order to verify if they conform to the program’s theoretical description. Despite all of the changes that can occur when the program is disseminated by a variety of workers in a diverse range of settings, adherence is rarely documented (Horwitz & Landsverk, 2011; Mildon & Shlonsky, 2011; Pinquart & Teubert, 2010) as are the impacts of variations in the program implementation on the intervention’s effects. In the case of the program presented herein, many factors reduced the level of variation, such as planning that clearly stipulated which elements were central to the program. This program structure promotes flexibility, thus allowing parents’ individual and collective experiences to be expressed (Lacharite´ et al., 2005). Results demonstrate a fluctuation of certain components of the program,

such as the number of workshops and the topics addressed. Far from being problematic, this flexibility translates into a fair respect of the program’s philosophy. However, when considering an evaluation of the program’s effects in order to verify the relation between those factors and the program efficacy, it should be noted that families were not exposed to the same number of themes and that some groups were larger than others. Dosage is another important component of program fidelity. The effect of dosage variation on the effectiveness of a program is the subject of a growing number of studies. Interestingly, Pinquart and Teubert (2010) concluded from their meta-analysis that early parenting education programs lasting 3–6 months have larger positive effects on parenting skills than shorter interventions or longer ones. However, DePanfillis and Dubowitz (2005) found no significant difference between families having participated in a program for 3 or 9 months, noting that even an intervention spanning over a short period of time can bring about significant changes in families at risk of neglect. Data from the current sample demonstrates important variations in the number of workshops parents were exposed to. On average, the parents were present at half of the meetings. Their presence was more related to their experience with the program and to the topic of the module than to their personal characteristics. These results can be regarded as encouraging since the facilitators can exert a greater influence on a program topic and on the construction of a relationship with participants than on families’ characteristics. Nevertheless, the significant variation in dosage of services offered to participants reiterates the importance of the integration of dosage in the implementation fidelity evaluation and of its consideration when analyzing the effects of the program. The quality of the intervention concerns the attitude of the staff in charge of the implementation, their ability to use the methods recommended by the program or to deliver the prescribed content (Paquette et al., 2009). The effects of the quality of the intervention on the changes in the families are supported by studies on prevention programs among others. An experimental study conducted by Olds et al. (2002) demonstrated that the program is beneficial to parents and to their child when the follow-up is monitored by a qualified nurse. Subsequently, it is noted that, in these families, pregnancies are more spaced, women return to work more quickly and social, cognitive, and language development in the children is improved. These effects are not significant in families where the follow-up is monitored by paraprofessionals. The evaluation of the PAPFC2 is simplified by the stability of the facilitators, which ensures a reduction in fluctuations in attitudes and in program facilitation knowledge. Furthermore, the quality of this operation is confirmed based on parents’ opinions collected via workshop evaluations. Parents highly appreciated the relationship they established with the intervenors and the attitude of those professionals. The group atmosphere was also reported as being highly positive by the facilitators. A more neutral evaluation of quality provided, for example, by a structured observation of the activity could be integrated in further studies as a third measure of the program quality and later be used as a control measure in an impact evaluation of the program. The association between the parents’ participation and the effect of an intervention on neglectful parenting has rarely been taken into consideration despite the seemingly important consequences it could have. Studies on child problem behavior, for example, have demonstrated that parents’ participation is related to important variations in major parenting outcomes such as the use of physical punishment, parental warmth and parental implication in school (Nix, Berman, & McMahon, 2009). This aspect must be considered in subsequent studies aiming to document the effects of the PAPFC2, given the important differences noted in the parents’ level of commitment to the

A. Be´rube´ et al. / Evaluation and Program Planning 45 (2014) 1–8

workshops. Our results show that parental participation varies greatly from one parent to another and that it is relatively stable within a participant. Parents who truly engaged in the program tended to be evaluated as highly participative during each meeting. Other participants, however, stayed disengaged during the whole program. These families did not engage more over time or depending on the module’s topic. Surprisingly, parents’ dosage and participation level are not related to their personal difficulties. Many parents displayed characteristics that could result in difficulty learning throughout the program, like depression, substance dependence or slow learning. These factors did not affect parental implication during the program. Finally, differentiation describes how the program differs compared to the ones parents were previously exposed to. This subject has not been given much attention and it brings about serious questions when offering a program to families known for years by youth care services. Data for the current study does not allow for an in-depth study on the difference between what these programs offer or have offered to participants and the specific components of the PAPFC2. A review of literature pertaining to the particularities of each service is essential. Indeed, only a comparative study would allow for a better understanding of how the program differs from other services and whether or not its specific elements result in significant changes in the families’ functioning. 4.1. Limitations In the present study, we looked at only one aspect of the program, being the parent’s group. It would be interesting to have an implementation study that would integrate the other components of the program (individualized services to parents and children, as well as collective activities). Such an evaluation would represent a greater challenge, because of the amount of data to collect. However, it is necessary to eventually better understand the effect of a multimodal program. Furthermore, an implementation evaluation would have to incorporate self-reported measures, as well as more objective measures like observations or standardized measures. The use of standardized measures is a great challenge since each program has its own components. However, there is a need to develop more uniform way of measuring adherence or quality in order to better generalize the results for implementation studies. Finally, the data did not allow for a true evaluation of differentiation. Measuring the novelty of a program could be a key element in changing families used to social services for many years. 4.2. Implications Incidentally, insuring the program’s fidelity represents a great challenge, in particular when it comes to transferring efficacy studies into a real-world context. This is the reason why many well-documented studies are reevaluated in the context of an effectiveness study once introduced into the field (Klimes-Dougan et al., 2009; Letarte, Normandeau, & Allard, 2010). Intervention fidelity, however, heeds a fair warning: interventions must adjust to the specific needs of its clientele and to the characteristics of the environment in which it is implemented. It is the evaluation that has to document these variations and to look at their repercussions on the program’s effects. The PAPFC2 is a program that could be qualified as ‘‘a`-la-carte’’ since, although many activities are prescheduled, services are later personalized to suit the families’ characteristics. This situation entails particular challenges when comes the time to evaluate the effects of the program. A compilation of the changes observed in the families is insufficiently informative as it then becomes impossible to separate which

7

conditions bring about the most significant changes. Needless to say, such evaluation must be considered for interventions related not only to neglect, but to various issues, particularly those that require a personalized and multimodal approach.

References Allin, H., Wathen, C. N., & MacMillan, H. (2005). Treatment of child neglect: A systematic review. Canadian Journal of Psychiatry, 50, 497–504. Association des Centres jeunesses du Que´bec. (2010). La ne´gligence: Faites-lui face. Montre´al: Association des Centres jeunesses du Que´bec. Barlow, J., Davis, H., McIntosh, E., Jarrett, P., Mockford, C., & Stewart-Brown, S. (2007). Role of home visiting in improving parenting and health in families at risk of abuse and neglect: Results of a multicentre randomised controlled trial and economic evaluation. Archive of Disease in Childhood, 92, 229–233. Barlow, J., Johnston, I., Kendrick, D., Polnay, L., & Stewart-Brown, S. (2006). Individual and group-based parenting programmes for the treatment of physical child abuse and neglect. Cochrane Database of Systematic Reviews, 3, CD005463. Brown, J., Cohen, P., Johnson, J. G., & Salzinger, S. (1998). A longitudinal analysis of risk factors for child maltreatment: Findings of a 17-year prospective study of officially recorded and self-reported child abuse and neglect. Child Abuse and Neglect, 22, 1065–1083. Butchart, A., Harvey, A. P., Mian, M., & Fu¨rniss, T. (2006). Preventing child maltreatment. A guide to taking action and generating evidence. Geneva: World Health Organization. Dane, A. V., & Schneider, B. H. (1998). Program integrity in primary and early secondary prevention: Are implementation effects out of control? Clinical Psychology Review, 18, 23–45. DePanfillis, D., & Dubowitz, H. (2005). Family connections: A program for preventing child neglect. Child Maltreatment, 10, 108–123. Domitrovich, C. E., & Greenberg, M. T. (2000). The study of implementation: Current findings from effective programs that prevent mental disorders in school-aged children. Journal of Educational and Psychological Consultation, 11, 193–221. Dube, S. R., Miller, J. W., Brown, D. W., Giles, W. H., Felitti, V. J., Dong, M., et al. (2006). Adverse childhood experiences and the association with ever using alcohol and initiating alcohol use during adolescence. Journal of Adolescent Health, 38, 444.e1– 444.e10. Dubowitz, H., & Bennet, S. (2007). Physical abuse and neglect of children. Lancet, 369, 1891–1899. Dubowitz, H., Pitts, S. C., & Black, M. M. (2004). Measurement of three major subtypes of child neglect. Child Maltreatment, 9, 344–356. Dunn, M. G., Tarter, R. E., Mezzich, A. C., Vanyukov, M., Kirisci, L., & Kirillova, G. (2002). Origins and consequences of child neglect in substance abuse families. Clinical Psychology Review, 22, 1063–1090. Durlak, J. A., & DuPre, E. P. (2008). Implementation matters: A review on the influence of implementation on program outcomes and the factors affecting implementation. American Journal of Community Psychology, 41, 327–350. ´Ethier, L. S., Bourassa, L., Klapper, U., & Dionne, M. (2006). L’e´volution des familles ne´gligentes: Chronicite´ et typologie. E´tat de suivi 1992 a` 2005. Rapport de recherche du Fond Que´be´cois de la Recherche de la Socie´te´ et de la Culture, Gouvernement du Que´bec. E´thier, L. S., Couture, G., Lacharite´, C., & Gagnier, J.-P. (2000). Impact of a multidimensional intervention program applied to families at risk for child neglect. Child Abuse Review, 9, 9–36. Fagan, A. A., Hanson, K., Hawkins, D. J., & Arthur, M. W. (2008). Bridging science to practice: Achieving prevention program implementation fidelity in the Community Youth Development Program. American Journal of Community Psychology, 41, 249–253. Gilbert, R., Widom, C. S., Browne, K., Fergusson, D., Webb, E., & Janson, S. (2009). Burden and consequences of child maltreatment in high-income countries. Lancet, 373(9657), 68–81. Hildyard, K. L., & Wolfe, D. A. (2002). Child neglect: Developmental issues and outcomes. Child Abuse and Neglect, 26, 679–695. Horwitz, S. M., & Landsverk, J. (2011). Methodological issues in child welfare and children’s mental health implementation research. Administration and Policy in Mental Health and Mental Health Services Research, 38, 1–3. Kazemian, L., Spatz Widom, C., & Farrington, P. (2011). A prospective examination of the relationship between childhood neglect and juvenile delinquency in the Cambridge study in delinquent development. International Journal of Child, Youth and Family Studies, 1 & 2, 65–82. Klimes-Dougan, B., August, G. J., Lee, C.-Y.S., Realmuto, G. M., Bloomquist, M. L., Horowitz, J. L., et al. (2009). Practitioner and site characteristics that relate to fidelity of implementation: The Early Risers prevention program in a going-toscale intervention trial. Professional Psychology: Research and Practice, 40, 467–475. Lacharite´, C., E´thier, L. S., & Nolin, P. (2006). Vers une the´orie e´cosyste´mique de la ne´gligence envers les enfants. Bulletin de psychologie, 59, 381–394. Lacharite´, C., Fafard, G., Bourassa, L., et al. (2005). Le programme d’aide personnelle, familiale et communautaire: nouvelle ge´ne´ration. Guide de programme. Trois-Rivie`res: Groupe de recherche et d’intervention sur la ne´gligence. Landsverk, J., Brown, H. C., Reutz, J. R., Palinkas, L., & Horwitz, S. M. (2011). Design elements in implementation research: A structured review of child welfare and child mental health studies. Administration and Policy in Mental Health and Mental Health Services Research, 38, 54–63.

8

A. Be´rube´ et al. / Evaluation and Program Planning 45 (2014) 1–8

Lee, C.-Y.S., August, G. J., Realmuto, G. M., Horowitz, J. L., Bloomquist, M. L., & KlimesDougan, B. (2008). Fidelity at a distance: Assessing implementation fidelity of the Early Risers Prevention Program in a going-to-scale intervention trial. Prevention Science, 9, 215–229. Letarte, M.-J., Normandeau, S., & Allard, J. (2010). Effectiveness of a parent training program ‘‘Incredible Years’’ in a child protection service. Child Abuse and Neglect, 34, 253–261. Lounds, J. J., Borkowsky, J. G., & Withman, T. L. (2006). The potential for child neglect: The case of adolescent mothers and their children. Child Maltreatment, 11, 281– 294. MacMillan, H. L., Wathen, N. C., Barlow, J., Fergusson, D. M., Leventhal, J. M., & Taussig, H. N. (2009). Interventions to prevent child maltreatment and associated impairment. Lancet, 373, 250–266. Mersky, J. P., Berger, L. M., Reynolds, A. J., & Gromoske, A. N. (2009). Risk factors for child and adolescent maltreatment: A longitudinal investigation of a cohort of inner-city youth. Child Maltreatment, 14, 73–88. Mikton, C., & Butchart, A. (2009). Child maltreatment prevention: A systematic review of reviews. Bulletin of the World Health Organization, 87, 353–369. Mildon, R., & Shlonsky, A. (2011). Bridge over troubled water: Using implementation science to facilitate effective services in child welfare. Child Abuse and Neglect, 35, 753–756. Nix, R. L., Bierman, K. L., & McMahon, R. J. (2009). How attendance and quality of participation affect treatment response to parent management training. Journal of Consulting and Clinical Psychology, 77, 429–438. Olds, D. L., Robinson, J., O’Brien, R., Luckey, D. W., Pettitt, L. M., & Henderson, et al. (2002). Home visiting by paraprofessionals and by nurses: a randomized, controlled trial. Pediatrics, 110(3), 486–496. Paquette, G., Tourigny, M., & Joly, J. (2009). Programme d’intervention de groupe pour des adolescentes agresse´es sexuellement: E´tude des effets spe´cifiques et exploration du lien avec l’implantation. In M. Tardif (Ed.), L’agression sexuelle: Transformations et paradoxes, Cifas 2009. Textes choisis (pp. 25–36). Montre´al: Cifas-Institut Philippe-Pinel de Montre´al.http://www.cifas.ca/. Pinquart, M., & Teubert, D. (2010). Effects of parenting education with expectant and new parents: A meta-analysis. Journal of Family Psychology, 24, 316–327. Schoenwald, S. K., Garland, A. F., Chapman, J. E., Frazier, S. L., Sheidow, A. L., & SouthamGerow, M. A. (2011). Toward the effective and efficient measurement of implementation fidelity. Administration and Policy in Mental Health and Mental Health Services Research, 38, 32–43. Sedlak, A. J., Mettenburg, J., Basena, M., Petta, I., McPherson, K., Greene, A., et al. (2010). Fourth National Incidence Study of Child Abuse and Neglect (NIS-4): Report to Congress, Executive Summary. Washington, DC: US Department of Health and Human Services Administration for Children and Families. Shipman, K., Edwards, A., Brown, A., Swisher, L., & Jennings, E. (2005). Managing emotion in a maltreating context: A pilot study examining child neglect. Child Abuse and Neglect, 29, 1015–1029. Tanner, K., & Turney, D. (2003). What do we know about child neglect? A critical review of the literature and its application to social work practice. Child and Family Social Work, 8, 25–34.

Topitzes, J. D., Mersky, J. P., & Reynolds, A. J. (2010). Child maltreatment and adult cigarette smoking: A long-term developmental model. Journal of Pediatric Psychology, 35, 484–498. Trocme´, N., MacLaurin, B., Fallon, B., Daciuk, J., Felstiner, C., Black, T., et al. (2005). E´tude canadienne sur l’incidence des signalements de cas de violence et de ne´gligence envers les enfants – 2003, Donne´es principales. Ottawa, ON: Ministre des Travaux publics et des Services gouvernementaux du Canada. Ward, H., & Rose, W. (Eds.). (2002). Approaches to Needs Assessment in Childrens’ Services. London: Jessica Kingsley Publishers. Annie Be´rube´ is a Professor in the Department of Psychoeducation and Psychology at Universite´ du Que´bec en Outaouais. Her research interests focus on the evaluation of implementation and effect of program targeting children and their family. She concentrates particularly on parental involvement in their child development. She holds a Ph.D. in Community Psychology from the Universite´ du Que´bec a` Montre´al (UQA`M). Vicky Lafantaisie is a Ph.D. student in Psychoeducation at the Universite´ du Que´bec a` Trois-Rivie`res and a research coordinator for the project E´valuation de programmes en ne´gligence. Her master degree looks at social isolation of neglecting families. Her actual work concerns the sociocultural dimensions of parental neglect, as well as to ways to encourage empowerment in those families. Diane Dubeau is a Professor in the Department of Psychoeducation and Psychology at Universite´ du Que´bec en Outaouais (Campus de Saint-Je´roˆme). Her projects are articulated around two main topics, program evaluation and paternity. She holds a master in Child development and a Ph.D. in Psychology (UQA`M). She has published in 2009 a book on paternity in the XXth century: La paternite´ au XXIe`me sie`cle published at Presses de l’Universite´ Laval. This volume is a synthesis of her work over the 15 last years on an intervention aimed at supporting paternal involvement. Sylvain Coutu is a Professor in the Department of Psychoeducation and Psychology at Universite´ du Que´bec en Outaouais (Campus de Saint-Je´roˆme) since 1988. He holds a master in Psychoeducation (Universite´ de Montre´al) and a Ph.D in Psychology (UQA`M). He is an associate researcher for l’E´quipe de recherche sur la Qualite´ e´ducative des services de garde et petite enfance. His work concentrates on the socioemotional development of preschool children. He also has an interest for programs targeting children from vulnerable families. Jose´e Caron is working with neglected population for more than 10 years. She developped a detailed understanding of that population. She is currently the coordinator of a program helping neglecting families in the Outaouais region. Annie Devault is a Professor at Social work Department at the Universite´ du Que´bec en Outaouais. Since the last ten years, her work explores paternal involvement in vulnerable contexts. She also has an interest for the intervention with neglect families Dr. Devault holds a Ph.D. in community psychology from UQA`M.

Using implementation evaluation to uncover a child neglect prevention program.

Child neglect is an ecosystemic problem with a great variety of risk factors to consider and, therefore, it requires a multimodal and individualized i...
375KB Sizes 2 Downloads 3 Views