Child Abuse & Neglect 44 (2015) 194–206

Contents lists available at ScienceDirect

Child Abuse & Neglect

Research article

Developing and testing a framework for evaluating the quality of comprehensive family assessment in child welfare夽 Cheryl Smithgall a,∗ , Elizabeth Jarpe-Ratner a , Natalya Gnedko-Berry a,1 , Sally Mason b a b

Chapin Hall at the University of Chicago, 1313 E. 60th Street, Chicago, IL 60637, United States Institute for Juvenile Research/University of Illinois at Chicago (M/C 747), 1747W. Roosevelt, Rm. 155, Chicago, IL 60608, United States

a r t i c l e

i n f o

Article history: Received 6 September 2014 Received in revised form 27 November 2014 Accepted 1 December 2014 Available online 4 January 2015 Keywords: Quality Assessment Child welfare

a b s t r a c t Over the last decade, Comprehensive Family Assessment (CFA) has become a best practice in child welfare. Comprehensive Family Assessments go beyond risk assessment to develop a full picture of the child’s and family’s situation. When appropriately synthesized, assessment information can lead to a clear articulation of the patterns of child or family functioning which are related to child abuse and maltreatment or which can be strengthened to facilitate change. This study defines and provides concrete examples of dimensions of quality in child welfare assessment reports that are consistent with the CFA guidelines and best practices embraced by child welfare agencies, courts, and other key stakeholders. Leveraging a random assignment design, the study compares the quality of reports written by a caseworker alone versus those written by a caseworker paired with a licensed Integrated Assessment (IA) screener. Findings are discussed in the context of the dual professional model and factors contributing to the timely completion of high quality assessment reports. © 2014 Elsevier Ltd. All rights reserved.

Introduction Over the last decade, Comprehensive Family Assessment (CFA) has become a best practice in child welfare. Comprehensive Family Assessments go beyond risk assessment to develop a full picture of the child’s and family’s situation which led to the current circumstances and which can drive service provision (Schene, 2005). The goal of conducting an assessment is to provide a picture which is broad and in-depth, exploring multiple factors which may contribute to risk and support well-being. Breadth is characterized by the examination of the child’s developmental and behavioral needs as well as the functioning and needs of people and systems involved in the child’s life—parents, siblings, extended family, community, and school. Depth is understood as examining the history or duration and context of problems and the nature of relationships which prevent or support well-being. The people raising and interacting with the child are critical to supporting safety and well-being. Those relationships—especially between parent and child but also between a parent and other adults—are integral to understanding the child’s and family’s needs (Budd, 2005; Johnson et al., 2006).

夽 This work conducted as part of a cooperative agreement (Grant number 90-CA-1752) between the Illinois Department of Children and Family Services (DCFS) and the U.S. Department of Health and Human Services, Administration for Children and Families, Children’s Bureau. The authors appreciate the support—through funding and the sharing of information and resources—that both the Children’s Bureau and Illinois DCFS provided to the evaluation team. ∗ Corresponding author. 1 Present address: Loyola University Chicago, 820 N. Michigan, Ave., Chicago, IL 60611, United States. http://dx.doi.org/10.1016/j.chiabu.2014.12.001 0145-2134/© 2014 Elsevier Ltd. All rights reserved.

C. Smithgall et al. / Child Abuse & Neglect 44 (2015) 194–206

195

This type of full-picture assessment is achieved through involving multiple family members as well as conducting a thorough review of other available records, whether current or historical. Gathering information from multiple sources contributes to a more accurate and reliable picture of overall family functioning over time (Budd, 2001; Budd, 2005). Collaboration with families is also key to a strong assessment (Schene, 2005) with rapport and relationship building as components of collaboration. The assessor’s relationship with the parent, children, and other family members promotes the insight necessary to understand and interpret the family’s needs and strengths in context (Reinders, 2010). Context includes assessing the family members’ perceptions—which may or may not be the same—on current circumstances, past events, and relationships and observing interactions between family members, particularly between parent and child (Budd, 2001). Patterns of parental behavior over time are especially useful in understanding what needs to change to support the child’s safety and well-being (Budd, 2001; Schene, 2005). When appropriately synthesized, assessment information can lead to a clear articulation of the patterns of child or family functioning which are related to child abuse and maltreatment or which can be strengthened to facilitate change. Integration requires reflection and judgment based on training and clinical experience and decisions are made best in collaboration with others who know the family (Cash, 2001; Schene, 2005). Staff training and clinical supervision offer frameworks and reflective support for delving into the areas that are necessary for a comprehensive family assessment (Schene, 2005). The ensuing decisions and recommendations flow clearly from a holistic assessment through the integration or formulation of the case to the service plan. An exemplary assessment report also documents the purposes of the assessment, disclosure of its uses, and evidence supporting any clinical judgments (Budd, 2012; Budde, 2012; APA Committee on Professional Practice and Standards, 2011). High quality child welfare assessments are comprehensive in nature, yet clear with respect to the elements that must be addressed in order to accomplish case closure, reunification, or other case goals. That clarity supports sustainable long-term change and focuses the activities and services on behaviors rather than compliance with service plans. Research on the Quality of Child Welfare Assessments Although practitioners and policymakers attest to the importance of quality in assessment reporting, few studies have examined this documentation of child welfare assessments. Rather, research has focused on the assessment process (e.g., Boutanquoi, Bournel-Bosson, & Minary, 2013), the use of analytic frameworks to guide assessment (e.g., Leveille & Chamberland, 2010); comparisons of assessment tools and instruments (e.g., Johnson et al., 2006), the fit between assessment data and the services provided (e.g., Cash & Berry, 2002), and the assessment of specific areas of concern in child welfare (e.g., domestic violence or mental health) (Budd, Felix, Poindexter, Naik-Polan, & Sloss, 2002; Petrucci & Mills, 2002). Only a few published studies have examined assessment content. Using grounded theory and case studies, Holland (2001) and Thomas and Holland (2010) analyzed content from 16 reports and interviews with social workers for inclusion of material about the children. Holland (2001) notes that she approached the assessments “in the same position as other audiences, e.g., the judiciary, in that access to the children was mediated” through the reports and interviews with the social workers (pp. 324–325). They found that the children were “minor characters” in the reports and that the description of their identities focused on the negative aspects, rather than including strengths. Jent and colleagues (2009) reviewed 845 assessments of children with complex child welfare cases who were referred for a multidisciplinary assessment. The researchers found that collaterals were rarely consulted for assessments, family weaknesses were emphasized with little reporting on strengths, findings were interpreted less conservatively than recommended by the assessment guidelines, and that assessors did not discuss the limitations of the assessment. Jent and coauthors identify “the lack of research on assessment quality and content as a system-wide problem” (2009, p. 896). The Illinois Integrated Assessment Program as a Model of Comprehensive Family Assessment in Child Welfare In the late 1990s, child welfare administrators in Illinois identified the quality and expense of psychological evaluations as a critical problem. A 1997 review of psychological evaluations conducted by the Illinois Department of Children and Family Services (DCFS) revealed that despite spending $14 million annually, the evaluations obtained were not useful, “... referral questions were lacking, the evaluations focused excessively on individual pathology and overlooked family dynamics, they were sometimes unnecessarily duplicated, and they were used in a way that competed with rather than complemented caseworker judgment” (Cross, 2009–10, p. 91). In response to this problem, DCFS developed the Integrated Assessment (IA) process in 2002 and 2003. Launched in 2005, the IA process is designed to examine the medical, social, developmental, mental health, and educational domains of the child and of the adults who figure prominently in the child’s life. When the initial assessment is completed as part of the IA program, child welfare caseworkers and licensed clinicians interview the children and adults and gather and review all investigation screenings, past provider assessments, background reports, treatment and school records, and other case documentation. An extensive semistructured interview protocol guides the interview and report writing process and covers the following topics: • Reason for opening case, history of involvement with the child welfare system, and family composition • Assessment of each parent including: family of origin, history of underlying conditions, and any serious losses or traumatic events; education, cognitive functioning, and employment history; social and romantic relationships; criminal behavior

196

C. Smithgall et al. / Child Abuse & Neglect 44 (2015) 194–206

and background; substance use history; medical and developmental conditions; emotional functioning; current living arrangement or housing; resiliency factors and support system; abilities as a parent and own understanding of role as a parent • Assessment of each child including: history of loss or trauma, residential mobility history and family history of abuse; educational status and needs; medical and developmental history and needs; mental health, social, and emotional functioning; strengths and resiliency factors; parent–child interaction; child’s adjustment to placement setting • Assessment of each child’s substitute caregiver including: strengths and areas of concern, respite or other needs The information gathered during the assessment process is then integrated into a report incorporating overall family functioning, clinical impressions, recommendations for all family members, and a family prognosis. The report is to be completed within 45 days of the child being taken into custody. Once completed, the written assessment report is reviewed by both the caseworker’s supervisor and IA screener’s supervisor. The information in the IA report and the collaborative process between caseworker and IA screener are intended to lead to earlier and more appropriate interventions for the child and family (see Smithgall et al., 2009 for a more detailed description of the IA model). The IA program was initially designed for and implemented with placement cases. In 2007, DCFS received a grant from the Children’s Bureau, an office of the U.S. Department of Health and Human Services, Administration for Children & Families, to adapt the model for use with intact family services cases and examine the process and practice of the IA program as part of an evaluation with a random assignment design. The evaluation was also funded by the Children’s Bureau but conducted by an independent evaluation firm. From 2010 through 2012, intact family cases were randomly assigned to receive either “services as usual”—where a caseworker conducts and writes the integrated assessment on his or her own—or the “intervention”—where a caseworker is paired with a clinical screener to conduct and write the integrated assessment. A proximal outcome for the evaluation of the program was the quality of the assessments produced as a result of the collaborative process. Contributions of this Study This study aims to define and provide concrete examples of aspects of quality in child welfare assessment reports that are consistent with the CFA guidelines and best practices embraced by child welfare agencies, courts, and other key stakeholders. Furthermore, this study leverages random assignment design to compare the quality of reports entered into the case record when written by the worker alone versus the worker-screener team. Thus, the study explores whether and how a dual professional model can contribute to the timely completion of high quality assessment reports. Methods This study was conducted as a component of a larger five-year, mixed-methods evaluation with a random assignment design. The theory of change underlying the IA program posits that high levels of engagement in a quality assessment process will yield better quality assessment reports, which, in turn, will contribute to better service linkages and improved outcomes. Intact family services cases were randomly assigned to an “intervention” group, in which the caseworker was paired with an IA screener, or a “services as usual” group, in which the caseworker completed the assessment on his or her own. Random assignment occurred at the case level immediately following the assignment of a case to a caseworker. The 53 caseworkers participating in the initiative held cases in both the intervention and services as usual groups, with no particular sequencing of cases (i.e. caseworkers could be conducting an assessment with an IA screener prior to, during, or after completing assessments for services as usual cases). The fact that caseworkers were exposed to the intervention could be presumed to make it more difficult to find significant differences in child and family outcomes between the intervention and counterfactual (i.e. services as usual) groups. However, the research team acknowledged the importance of client and situation-specific factors in assessment, and therefore the design intentionally employed random assignment of cases rather than caseworkers. This particular paper focuses on the content of the narrative assessment reports produced for cases in both the intervention and services as usual groups, and assesses differences in assessment report quality as an intermediate outcome All data analyzed for this paper were gathered and maintained by DCFS as part of their routine processes. Institutional review board approvals for this study were obtained from both the University of Chicago and Illinois DCFS. Sample Of the 53 caseworkers participating in the multi-year demonstration project, 26 had completed at least two integrated assessments with a screener at the time the sample was drawn in June 2011, approximately one year after random assignment began. Using a theoretical sampling approach (Charmaz, 2006), the evaluation team initially randomly selected 6 of the 26 eligible caseworkers and identified a pair of initial IA reports for each caseworker. One IA report was written solely by the caseworker (and served as a control group case) and one IA report was written by an IA screener working with that same caseworker (and served as a treatment group case). After the initial review, the team chose to randomly select an additional 10 caseworkers and purposively sample corresponding report pairs with completion dates in close proximity (less than one month) and with similar family size, allegation type, and degree of previous involvement with the child welfare system. The

C. Smithgall et al. / Child Abuse & Neglect 44 (2015) 194–206

197

Fig. 1. Key dimensions and subdimensions defining quality in family assessments.

rest of the cases included in the same were therefore analyzed in pairs so as to minimize any comparison biases resulting from these characteristics. After this second phase, the team determined saturation of themes in the analysis of the content had been reached and random selection of additional caseworkers and report pairs was not needed. The evaluation team extracted the 32 assessment reports and de-identified the narrative content, replacing names of individuals with an identifier for their role such as “[father of child 1]” and replacing business names and geographic information with general descriptors such as “[grocery store]” or “[town].” Analytic Approach Prior to selecting the 32 reports that were reviewed for this analysis, 6 reports were selected for an initial review to test the analytic approach. Four research team members developed the approach, delineated below, and also completed the analysis. Members of the research team were already very familiar with the structure and content of IA reports based on previous analyses (see Smithgall, Jarpe-Ratner, & Walker, 2010; Smithgall et al., 2009). This prior experience, the IA report template, and the literature on comprehensive family assessment (Budd, 2001, 2005; Schene, 2005) informed an initial rubric incorporating ten a priori themes, such as “inclusion of contextual details” and “connections made across report.” The initial review allowed the team to test and refine this set of themes. Each researcher examined all 6 reports and wrote a narrative memo describing how or whether the reports included or addressed each of the ten themes. From ensuing team discussions emerged the idea that all of the themes describe the dimensions of either thoroughness or integration. The initial list of ten themes was refined into a list of six themes or subdimensions, with three falling under each dimension (see Fig. 1). By defining subdimensions through both the analysis of the data set itself as well as the literature, this analytic approach is aligned with grounded theory approaches (Charmaz, 2006; Strauss & Corbin, 1990) which draw on both a priori knowledge as well as emerging themes and patterns in the data. Thoroughness was conceptualized as the level of detail, organization, and clarity throughout the report. The three specific subdimensions the team examined to determine thoroughness were: • Clear Timeline: whether the report included detail regarding the timeline or sequencing of events • Well-Organized Content: whether the report content was organized in a way that was easy for the reader to digest; and • Evidentiary Support: whether the foundation or evidence was provided for any characterizations or impressions in the report. Integration was conceptualized as the degree to which information was synthesized throughout the report and the ability of the writer to demonstrate how events and conditions evolved over time. The three specific subdimensions the team examined to determine integration were: • Patterns of Behavior over Time: whether the information obtained from reports provided an understanding of the development of individuals’ patterns of behavior over time;

198

C. Smithgall et al. / Child Abuse & Neglect 44 (2015) 194–206

• Family System Perspective: whether the assessment was conducted from a family systems perspective, clearly identifying and acknowledging the perspectives of multiple individuals within the family and noting how the actions of some family members may be impacting others; and • Recommendations: whether the analysis of the information obtained in the assessment resulted in clear, explicit connections between the assessment content and the recommendations and allowed for better specification of the recommendations (e.g., not only specifying the type of service to provide but also the goals or expectations for what should be addressed through that particular service). Rating of IA Reports The sample of 16 report pairs was divided among the research team. Each pair was examined together and members of the research team wrote a narrative memo examining each report with regard to the two primary dimensions and 6 subdimensions. After examining the narrative assessments, it was determined that a 5-point scale would be necessary in order to appropriately capture the degree of variability on each dimension across the reports. Each aspect was assigned a score ranging from 1 (the lowest quality) to 5 (the highest quality). A score of 1 reflected the failure to meet the criteria for that aspect. A score of 2 indicated that the writer was able to meet the criteria in some sections of the report, but not for the majority of the report. A score of 3 indicated that the writer was able to meet the criteria for the majority of the report, but for not the entire report. A score of 4 indicated that the writer was able to meet the criteria for the entire report. Finally, a score of 5 indicated that the writer was able to meet the criteria throughout the entire report as well as being able to bring an additional layer of synthesis or detail that went above and beyond the criteria. To ensure reliability, or trustworthiness (Lincoln & Guba, 1985; Sandelewski, 1993), multiple report pairs were coded, scored, and discussed by multiple researchers through a peer debriefing process (Creswell, 1998, 2003; Lincoln & Guba, 1985; Padgett, 1998). There were only two subdimensions where researchers disagreed about scores. Subsequent discussions took place and consensus was reached. For a couple reports, there was not enough evidence or information to properly assign a score for a particular sub-dimension. In these cases, at least two members of the research team reviewed the assessment. When it was agreed that a score could not be assigned, no score was entered for that particular sub-dimension. The overall score for the primary dimension was an average of the non-missing scores. For each subdimension, reports receiving a score of 4 or 5 were designated as “high quality” while reports receiving a 1 or 2 for the given subdimension were considered “low quality.” Based on the scores and identified patterns and themes, the final section of this analysis addresses findings regarding overall differences between assessment reports completed with and without an IA screener. Results In this section, the results are organized around the framework established in the analytic approach. We focus on the two key dimensions, thoroughness and integration. Excerpts from high quality examples are shared for each set of three subdimensions. Low quality assessments often reflected the absence of information or structure, making it difficult to highlight excerpts. Therefore, descriptions of aspects contributing to the low quality ratings are provided in lieu of excerpts. Finally, we offer the quantitative comparison of quality ratings for reports done by caseworkers, either with or without a screener involved in the process. Aspects of Thoroughness in Child Welfare Assessments As explained in the Methods Section, the conceptualization of thoroughness examined three aspects: clear timeline or sequencing of events, well-organized content, and characterizations with clear evidentiary support. Clear Timeline. Reports that demonstrated “high quality” with respect to establishing a clear timeline often presented information with both a date and a corresponding age or timing in relationship to another important event. The excerpt below is a good example of a report in which information about the mother’s social and romantic involvement is represented alongside other important factors. These other factors help to understand not only patterns in her relationships, but also experiences with housing, substance use, mental health, incarceration, and domestic violence issues. It also helps understand the ages of her children at the time of the events described. Significant involvements and relationships: [Mother] has a history of short-term relationships that result in pregnancy with no ongoing support from these partners. She reported no problems of domestic violence, substance abuse, or other problems affecting her relationships. [Mother] reported that she became involved in her first significant relationship when she was 18; she began dating [father1]. On [date], she gave birth to their first child, [child, age 14]. On [date], she gave birth to her second child, [child, age 12]. [Mother] stated that [child, age 12]’s father is [father1] but she could not explain why his surname is [different from that of father1]. On [date], [mother] gave birth to [child, age 10]. On [date], [mother] gave birth to [child, age 9] her fourth child. His father is [father2]. He also fathered her sixth child, [child, age 5] born on [date].

C. Smithgall et al. / Child Abuse & Neglect 44 (2015) 194–206

199

[Mother’s] fifth child, [child, age 6] was born on [date] from a relationship with [father3]. [Mother]’s seventh child [child, age 1] was born on [date]. [Child, age 1]’s father is [father4]. He was involved with another woman who also became pregnant with his child while he was seeing [mother]. [Mother] stated that when [child, age 1] was 5-months old, she learned that she was pregnant. She began to experience severe pain and was hospitalized with an ectopic pregnancy. [Mother]did not report the name of the father. None of the other fathers are involved with the children. [Mother] stated that [father1] was visiting his children occasionally a few years ago, but is no longer involved. She reported that [father2] is serving time in prison for a murder he committed in 2010. [Mother]stated that she is currently dating [mother’s current paramour] a 60-year-old, African American male who resides in [town], Illinois. She reported that he is not the father of any of her children. [Mother’s current paramour] did visit on the day of her screening and went into a bedroom with her children during her interview. Reports that were considered “low quality” with respect to establishing a clear timeline may have provided some sense of the order or sequence of events; however, for some pieces of information a timeline could be better specified or is missing altogether. An example of a low quality report was one in which information about social and romantic relationships was provided by both the mother and the father, but the timing of many of the factors was hard to discern. These included key markers of their relationship such as getting married, having a child, and moving in with her parents. There was also an incident of domestic violence that was described only minimally. By looking at both parents’ interviews and the child’s age, one could deduce that the child was born within the first two years of marriage. However, it is unclear how long the couple was living with in-laws, when the father moved to his uncle’s place of residence, and the circumstances around and timing of the arrest for domestic battery. No other sections of the IA report for this case clarified the timing of their living circumstances, the maltreatment incident, or the violence in the relationship. These are all factors that are important considerations in developing a service plan and engaging the family in a change process. Well-Organized Content. One feature of reports that demonstrated “high quality” with respect to the way in which the information was organized was their approach to formatting. These reports often used formatting that helped the reader navigate the large volume of information and quickly figure out what domain or area of assessment it was most relevant for. Formatting within the sections of the template contributed to a sense of overall quality in organization of the information. In the excerpt below, the “Work History” section of the assessment is further subdivided to identify important aspects of employment, such as first job, continuity over time, positions held, and the current status of employment. Work History First job [mother] first job was working for [fast food restaurant] when she was 16-years-old. Pattern of employment: [mother] said that she enjoys working. She has been working consistently since she was 16 years old. Positions held: [mother] worked for [restaurantA] (1 year), [restaurantB] (1 year), and now [restaurantC]. Longest Job: [mother’s] longest job was working for [restaurant] (1 year). She said that she quit the position because she did not get along with some of the people who were getting hired there. She quit [restaurant] because she and her husband moved to Kansas. Current employment status: At the time of her interview, [mother] was employed by [restaurant]. She was hired while she was pregnant but did not start until after her daughter was born. However, per [father], [fast food restaurant] is not going to keep [mother] as an employee due to her incarceration. In some of the reports rated as “low quality,” the assessment information was simply not presented in the correct sections based on the report template. In others, detail was either minimal or the information was not further subdivided, leaving the reader to deduce the critical aspects of the information that was shared, such as patterns of stability in employment over time. Evidentiary Support. A report was rated as “high quality” if clear evidentiary support was provided for any characterizations or impressions made by the report’s writer. The following excerpt was taken from a report rated as strong in evidentiary support because descriptions are provided that help the reader understand what is meant by language used to describe the mother, such as “perseverance” or “stressed.” Similarly, details provided in the developmental screening sections help the reader understand how “normal” range is defined, including ways in which the mother reports behaviors that fit with that assessment.

200

C. Smithgall et al. / Child Abuse & Neglect 44 (2015) 194–206

Individual Strengths [Mother] demonstrated perseverance. She continued to maintain proximity while her daughter was in the hospital for two months. She visited her daily and asked various questions of the hospital staff to ensure the safety of her child, while inquiring if the treatments were the best options available. Perceptions about Being a Parent [Mother] reported feeling increasingly stressed by having her daughter in her home. At first she explained that [child, age 3 months] struggled to adjust to sleeping at night. After two weeks of being home, [child, age 3 months] is sleeping at night however struggles to fall asleep and awakens approximately three times a night. [Mother] indicated that the lack of sleep has been difficult for her. She also reported that the most rewarding aspect of being a mother is the knowledge that she will be the one that teaches her daughter right from wrong. Developmental Screening - Language: In this area, [child, age 3 months] performed in the normal range. She responded to the sound of a bell ringing by widening her eyes and looking around. She was heard vocalizing and making “ah-goo” sounds. Her mother reported that she has laughed and has been heard squealing; she will turn to the sound of a rattle. In contrast, poor quality reports were ones in which descriptors and comparisons were unclear at times. For example, in one report, a description of the parent’s demeanor at the time of the interview suggests the mother is “a great deal more relaxed and less agitated” but details of exactly what was observed to support this are not offered. Similarly, the mother is described as being “overwhelmed” with the child’s behaviors, but no further detail is provided about the mother’s appearance or behaviors that led to that description. In addition to the three aspects of thoroughness presented here, the research team found that some reports exhibited exceptional quality by clearly noting discrepancies in information across the documentation or interviews or where additional follow up was needed. If evidence of discrepant information was not noted, it was difficult to determine whether such discrepancies did not exist or were not included. Therefore, this aspect was not formally scored in the analysis. Aspects of Integration in Child Welfare Assessments As explained in the methods section, the conceptualization of integration incorporated three aspects: identifying patterns of behavior over time, applying a family systems perspective, and making explicit connections between assessment content and recommendations. Patterns of Behavior. For this subdimension, a “high quality” report meant that the information obtained and its presentation within the report provide an understanding of the development or relationship of patterns of behavior over time. In the excerpt below, the assessment draws out connections between the mother’s exposure to traumatic experiences as a child, multiple unhealthy relationships as an adult, and her current mental health struggles. The end result of the assessment is a suggestion of therapy to address what may be underlying issues that trigger the depression. Mental/Emotional Health - Caretaker Adverse experiences impacting mental health: -

parents’ alcohol consumption, relationship problems, and domestic violence in their relationship physical abuse by mother father was physically abusive toward her siblings father’s struggles with PTSD parents’ deaths ongoing family problems, their lack of acceptance of her relationships involvement in abusive relationships Her stepson’s ([son of ex-husband]) death in [year]. He was intoxicated and fell from a cell phone tower at age 22. pregnancy at 15 years old limited resources Family Dynamics [Mother] has been involved in multiple unhealthy relationships, all of which included domestic violence and alcohol use by her partners. [In the social romantic relationships section of this report, a detailed history of the mother’s relationships was included, specifying dates, length of relationships, children, and relationship issues for each partner]. [child, age 15] witnessed the violence in her parents’ relationship at an early age, but does not recall the abuse. Family Prognosis and Recommendations

C. Smithgall et al. / Child Abuse & Neglect 44 (2015) 194–206

201

[Mother] is a 42-year-old mother of three. Her family recently came to the attention of the DCFS following her recent suicide attempt and concerns for her 15-year-old daughter, who was residing with her at the time. [Mother] has struggled with depression for the past two to three years. She has been treated with medication and currently reports her mood to be stable. However, considering the seriousness of the recent situation, it is strongly recommended that she seek therapy to help address any underlying triggers to the depression and her struggles. [Mother] has been exposed to multiple adverse events in her life, but has managed to maintain a purpose in spite of those difficulties. She has been the primary caregiver for her daughter and has been able to meet her basic needs thus far. In contrast, in an assessment that was rated as “low quality,” information was provided about the circumstances for involvement and substance abuse issues in the appropriate sections; however, the fact that the family was homeless seemed to be noted more as a contextual factor (i.e., simply a description of their living arrangements) in the section on “living arrangements.” There was no exploration of how the substance use and relationship problems might relate to the homelessness. The early part of the report also suggested that the mother had a difficult childhood. However, there did not seem to be anything in the assessment exploring a possible connection between her childhood experiences, substance use, relationship problems, and homelessness.

Family perspective. A “high quality” score was given to assessments in which a family systems perspective was incorporated, clearly identifying and acknowledging the perspectives of multiple individuals within the family and noting how the actions of some family members may be impacting others. In the excerpt below, even though some of the primary stressors would appear to be the ways in which the oldest child is functioning, there was clearly a lot of attention placed on how the entire family was coping and complete information given with regard to the specific needs of each child. Another strength of this report is the fact that although few family members spoke directly about their father’s incarceration and pending immigration case, the screener was able to identify the likely impact of this stressor and integrate the recognition of this into the analysis of overall family functioning. Family Dynamics Family Dynamics and Underlying Conditions The main risk factor for this family is lack of supervision and lack of support. [Mother] is a single mother who is working hard to support her family financially. Due to lack of support, she leaves the children home alone when she works her early morning paper route. If she had another option she would likely utilize it as she has in the past prior to her husband being detained. The stress of being a single caregiver to four children, one with special needs and two who are quite young, is wearing on [Mother]. She struggles to maintain a neat home environment and to keep her children engaged in meaningful activities. At the same time, she has a strong emotional bond to them and advocates for their individual needs. Family Prognosis and Recommendations Parent Recommendations for [Mother] [Mother] is the biological mother of [Male child, age 16], [Male child, age 10], [Male child, age 4], and [Male child, age 3]. Her family came to the attention of DCFS after it was discovered she was leaving [Male child, age 10] and [Male child, age 16] home alone so she could perform her paper route. [Mother] has been involved with DCFS in the past. In 2007 she was indicated for exposing her children to domestic violence. Information about her previous investigations is not available. In regards to needs, [Mother] is currently overwhelmed with her life circumstances. She has limited resources and support. Her husband is currently in detention on an immigration violation. [Mother] appears to be depressed, and is struggling to clean and organize her home. Her children have behavioral and emotional concerns. She spends a significant amount of time redirecting her children. She struggles to involve her children in meaningful activities. She appears to be putting forth effort to help her children, but has not been as effective as she wants to be. [Mother] is minimizing the problems she is having with her mood. She would benefit from counseling to address her feelings and gain support. [Mother] also has anxiety regarding the deportation status of her husband and her son’s future. In regards to strengths, [Mother] is open to services at this time. She wants to improve life for her and her children. [Mother] is a devoted mother and works hard to support her family. In contrast, reports rated “low quality” with respect to family perspective often failed to fully assess all members of the family, address relationships between individuals within the family, or acknowledge the implications of one individual’s behavior or circumstances on another family member. For example, in one report rated as being low quality, it is mentioned that the 14-year-old child has a good relationship with her mother, that she talked openly after the disclosure of sexual abuse, and that mom is supportive. However, as with several statements made throughout the report, it is unclear from the writing whether this information was based on caseworker observation or interviews with the 14-year-old or the mother. Furthermore, the majority of the report focuses on the older child (the victim of sexual abuse) while there is nothing about

202

C. Smithgall et al. / Child Abuse & Neglect 44 (2015) 194–206

observations of or interviews with the younger child. Yet, in the recommendations for the younger child (age 4), it says “counseling should be offered if [child] begins to have difficulty accepting that his father is no longer in the home.” There was nothing in the content of the report regarding the child’s attachment to the father or level of contact before or after the precipitating incident. Recommendations. Reports rated as “high quality” in the area of recommendations were written in a way that allows analysis of the information presented in the assessment. The analysis showed the information makes clear, explicit connections between the assessment content and the recommendations. The information also allows for better specification of the recommendations. Not only were the recommendations clear and grounded, as shown in the excerpt below, but there were specific suggestions about providers and programs to be used and the order in which services should be implemented. What services are recommended to address the risk factors that directly impacted the incident of maltreatment? It is recommended that [mother] complete her substance abuse program through the [provider]. [Mother] has a long history of alcohol and marijuana abuse and a year-long addiction to heroin. She has difficulty acknowledging the impact of her heroin use and makes attempts to minimize its effects on her life and her daughter’s physical adjustment. She withheld information at her substance abuse assessment related to her heroin use. It is important that she be reassessed and that her heroin use be incorporated into the understanding of the trajectory of her substance use history. She has been recommended to participate in a [level 2 substance abuse program] at [provider]. It is important that [mother] complete this program and follow any aftercare recommendations made, including continuing to attend substance-free support group such as NA and relapse prevention groups. Secondly, [mother] comply with drug screenings upon demand. Regular and random drug screens are needed to ensure that she is remaining abstinent from illicit substances throughout her involvement with DCFS. Once she has attained three months of sobriety, it is recommended that [mother] participate in long-term, individual therapy. [Mother] exhibited avoidant and numbing behaviors. She has increased anxiety, irritability, and frequent crying spells when overwhelmed. She has a history of postpartum depression and has turned to alcohol and drugs to avoid her feelings about herself and various life problems she has had. It is important that she remain sober for at least three months prior to beginning therapy so that her symptoms may be accurately understood and so that the negative feelings she has may be available to process. [Mother] will benefit from long-term, individual therapy in a neutral, therapeutic environment with a gentle, but direct therapist who can assist her in exploring how these issues relate to her current level of functioning. It will be important for her to learn relaxation strategies to help moderate her anxiety. Consultation with a psychiatrist, with mindfulness of [mother’s] addiction history, may also be valuable at her therapist’s discretion, should her anxiety and depressive symptoms remain elevated. Finally, it is recommended that [mother] participate in a parenting education course. [Mother] has been reported to be anxious and uncertain when providing care to her daughter. She asks repeated questions about how to care for her and forgets the answers. It is hoped that parenting classes will instruct her on day-to-day care and prepare her for the special needs that [child, age 3 months] may require due to her exposure to opiates in the womb. It may also be necessary for [mother] to gain an understanding of the developmental needs of children at varying ages. If additional concerns remain after completing these services, offering the services of a parenting coach would be recommended. These recommendations were connected to information obtained in the assessment in a way that (1) provided specificity related to the type of service being recommended, (2) was mindful of other recommendations or service needs, and (3) could alert the client and provider about challenges with regard to expectations about duration or follow up to the recommended services. In cases rated as “low quality,” recommendations were either not provided at all, lacked specificity, or lacked a connection to the information gathered in the assessment. In some cases, the language used in the recommendation was broad and applied repeatedly. For example, one report stated five times, “[Mother] should ensure that her child’s medical, physical, emotional, and educational needs are met.” This statement was made once for each of the five children assessed and no differentiation by age was made, despite the fact the children ranged from 6 months to 9 years old. It would be difficult for a provider to guide a parent in complying with these recommendations without conducting another assessment to understand where improvements might be expected or which needs are not already being met. Comparison of caseworker reports done with and without a screener. After the pairs of reports were scored, boxplot diagrams were constructed to depict the variability in the quality of assessment reports. The diagrams show the variability for different aspects of the thoroughness and integration dimensions. Results were separated according to whether the assessment was conducted by the caseworker alone or with an IA screener. With the exception of the Timeline dimension among reports completed with a screener, the boxplots indicate a high degree of variability in the quality of reports across all dimensions. As shown in Fig. 2, the median rating for quality was higher for reports completed with an IA screener on all three aspects of the thoroughness dimension. The middle quartile—indicated by the box in each column—shows that for each of these aspects, there was greater consistency in achieving high quality ratings among the group of assessments conducted with an IA screener. T-test comparisons confirm that for the 16 pairs of reports that were scored, the mean differences in quality

C. Smithgall et al. / Child Abuse & Neglect 44 (2015) 194–206

203

Fig. 2. Report scores on dimensions of thoroughness.

scores were statistically significant for all three aspects of thoroughness, with averages consistently higher for assessments conducted with an IA screener vs. without: Timeline (4.63 vs. 2.63, t = 2.07, p < .001); Evidentiary Support (4.19 vs. 2.25, t = 2.05, p < .001); and Organization (4.63 vs. 2.81, t = 2.09, p < .001) Fig. 3 shows the range of scores for the three aspects of integration. Again, on all three aspects, the median rating for quality was higher for reports completed with an IA screener. For this dimension, as shown in the middle quartile boxes, there was greater consistency in achieving high quality ratings among the group of assessments conducted with an IA screener. That being said, the median and middle quartile scores for the “family perspective” aspect—while higher than for assessments done by workers alone—seem to indicate that even the IA screeners may have experienced more challenges in achieving high quality with consistency. T-test comparisons confirm that for the 16 pairs of reports that were scored, the mean differences in quality scores were statistically significant for all three aspects of integration. Averages were consistently higher for assessments conducted with an IA screener than for those conducted without one: Patterns of Behaviors (4.00 vs. 2.19, t = 2.05, p < .001); Recommendations (4.38 vs. 2.06, t = 2.04, p < .001); and Family Perspective (3.31 vs. 1.56, t = 2.06, p < .001). Although the sets of scores for Thoroughness and Integration reflect a common finding that the screener played an important role in quality achievement, the thoroughness scores tend to be higher than the integration scores. In three

Fig. 3. Report scores on dimensions of integration.

204

C. Smithgall et al. / Child Abuse & Neglect 44 (2015) 194–206

reports done by caseworkers without an IA screener, the report met the standards for high quality on a number of the aspects of thoroughness and the weaknesses observed were generally with respect to aspects of integration. Discussion Building on previous research and literature on best practices in conducting family assessments, this study developed and tested a framework for thinking about the quality of child welfare assessment report content around two primary dimensions: thoroughness and integration. Thoroughness was conceptualized as the level of detail, organization, and clarity throughout the report, and the extent to which evidence was presented for any characterizations and conclusions. Integration was conceptualized as the ability of the writer to demonstrate how events and conditions evolved over time, the extent to which the report incorporated a family systems perspective, and the degree to which information was synthesized and clearly connected to well-specified recommendations for services. Concrete examples were provided from high quality reports for three aspects of thoroughness (timeline, evidentiary support, and organization) and three aspects of integration (identifying patterns of behavior over time, constructing well-synthesized and supported recommendations, and showing evidence of a family perspective being applied to the assessment). Variability in the quality of reports was observed within both sets of reports—the set done by caseworkers without an IA screener and the set of reports done with an IA screener. However, reports completed with IA screeners demonstrated higher average quality and greater consistency in achieving high quality ratings. These results may reflect the emphasis within the IA program on dual professional collaboration or they may reflect additional skills that are prioritized in the selection and supervision of IA screeners, including experience with reflective analysis and writing. Although reports completed by IA screeners had higher scores across all aspects and dimensions, the aspect defined as “family perspective” was the weakest for both sets of reports, whether completed with or without IA screeners. The focus of Comprehensive Family Assessment is not only on the child but also on the family. Therefore, these results may indicate the need for additional training for both caseworkers and IA screeners on assessment related to the functioning and needs of key people in the child’s life. Caseworkers and IA screeners may also need additional training regarding the context of problems, specifically family relationships, which may prevent or support the child’s well-being. The differences observed in the quality of the reports may reflect the different hiring processes for caseworkers and IA screeners. Implementation research identifies staff selection or hiring processes as a critical first step in building competency, and particularly critical at the practitioner level (National Implementation Research Network, 2005). The hiring process for IA screeners is selective and focuses on experience and skills considered essential for the integrated assessment program. IA screeners must have a minimum of master’s degree in social work or a related field, clinical licensure, and three years of mental health or early childhood experience. Although Illinois DCFS encourages caseworkers to obtain advance degrees, there is no requirement for an advance degree and post-degree experience. Furthermore, IA screeners must also demonstrate writing and case formulation skills during a hiring process that is conducted jointly by DCFS and the university or hospital partner employing and supervising the IA screener. Organizational and contextual factors are also critical factors in facilitating the completion of high quality assessment reports. As Schene indicates (2005), resource allocation is critical to supporting good assessment practices. The IA program is structured such that—compared to caseworkers—IA screeners carry fewer cases at any particular point in time but they complete a greater number of assessments annually (due to the shorter period of involvement in each case). Thus, the observed differences in the quality of the assessment reports may be a function of the fact that IA screeners have lower caseloads, more time to dedicate to writing and reflection, and more experience writing clinical assessments. Writing and reflection are activities that are critical for effective child welfare practice, yet they are often given lowest priority. Several scholars have pointed to the emergence of “procedurally dominated practice” in child welfare, particularly in the wake of child fatalities or other negative outcomes highlighted in the media (Parton, 2009; Ruch, 2005). Responses to such negative events often lead to the development of detailed procedural guidelines, forms, assessment tools, and related trainings, all of which are believed to increase accountability. However, these procedures also often limit worker autonomy and may actually constrain professional judgment. Well-written assessment reports are instrumental in formulating and communicating an understanding of the child and family situation and identifying targets of change. Critics may highlight the fact that caseworkers reportedly spend nearly one-third of their available case-related time on documentation in automated information systems (American Humane Association, 2011)—time that could be used to increase direct contact with families. However, having assessment information synthesized and articulated in the case documentation is especially critical in light of annual turnover rates, which are reported to be as high as 50% (Curry, McCarragher, & Dellmann-Jenkins, 2005). In conducting family assessments, caseworkers must develop and employ a variety of clinical skills. First and foremost, they must learn to engage clients and manage difficult client relationships in order to deliver effective services. This must be done while also synthesizing information from a wide range of sources in order to inform service delivery and to gauge meaningful change in circumstances and behaviors. Child welfare workers must gather, share, and monitor information about their clients while also accounting for their own decisions as well as those of other professionals and agencies with whom they work. Opportunities for workers to learn and develop professional skills in multiple areas must be ingrained within organizational structures, cultures, and working relationships. Supervisory relationships are particularly critical as they mediate the practice and management contexts within which caseworkers develop their skills. Because the Integrated Assessment program in Illinois combines relationship-based practice, high quality supervision, and information synthesis,

C. Smithgall et al. / Child Abuse & Neglect 44 (2015) 194–206

205

this practice model has considerable value as an opportunity for on-the-job professional development. It not only leads to the production of high quality assessment reports but also, as a result of its collaborative design with IA screeners, has the potential to assist workers in improving their own clinical competencies and skill development. Study Limitations Along with its strengths, the study has several limitations. As in any qualitative study, the researchers have expectations of the data which may have influenced the results. For example, we conducted a “blind” analysis of the reports by removing any indication of who wrote the reports. However, it is possible that the evaluation team would notice differences in the language and length of assessment reports conducted by the same worker, depending upon whether the reports had been completed with an IA screener. Although report length does not equate quality, this potentially weakened the rigor of the blind review. The researchers’ use of an extensive iterative analytic process between four team members was intended to enhance rigor and strengthen the validity of the findings. Also, quantifying the results through ratings may oversimplify the nuances found across the reports. The rating, however, allowed for the examination of patterns across the two samples and yielded some useful results. Conclusion As part of a multi-year evaluation of one state’s model of comprehensive family assessment, this study identifies and provides concrete examples of aspects of quality in child welfare assessment reports. This study also demonstrates the value of a dual-professional model in producing high quality reports. Child welfare jurisdictions vary considerably in their contextual circumstances such as funding, policies, and contractual arrangements with providers. Some jurisdictions may seek to replicate the dual-professional model. Where that model does not seem feasible or desirable, jurisdictions might consider adopting the assessment tools and processes of the Illinois Integrated Assessment program or perhaps change organizational factors that are embedded in the dual-professional model and potentially contributing to higher quality reports. These organizational factors include hiring and training processes, workload and time allotments for assessment activities, and supervisory relationships. Further research should focus on demonstrating the extent to which such organizational factors can be manipulated to produce high quality assessments. Also needed is research that further develops and tests the theory of change connecting high quality assessments to the receipt of services and improvements in child and family outcomes. References American Humane Association. (2011). Child welfare policy briefing: Child welfare workforce 2(3). Washington, DC: Author. Retrieved from http://www.americanhumane.org/children/stop-child-abuse/advocacy/caseworker workload paper.pdf American Psychological Association (APA) Committee on Professional Practice and Standards. (2011). Guidelines for psychological evaluations in child protection matters. Retrieved from http://www.apa.org/practice/guidelines/child-protection.pdf Boutanquoi, M., Bournel-Bosson, M., & Minary, J.-P. (2013). Evaluating situations in child welfare: From tools to workgroups. Children and Youth Services Review, 35(7), 1152–1157. Budd, K. S. (2001). Assessing parenting competence in child protection cases: A clinical practice model. Clinical Child and Family Psychology Review, 4, 1–18. Budd, K. S. (2005). Assessing parenting capacity in child welfare. Children and Youth Services Review, 27, 429–444. Budd, K. S., Felix, E. D., Poindexter, L. M., Naik-Polan, A. T., & Sloss, C. F. (2002). Clinical assessment of children in child protection cases: An empirical analysis. Professional Psychology: Research and Practice, 33(1), 3–12. Budd, K. (September 2012). Assessing parents and supporting behavioral change. Chicago, IL: Presentation at Doris Duke Fellowships Annual Meeting. Budde, S. (September 2012). Assessing parenting capacity and supporting behavior change: Challenges and strategies. Chicago, IL: Presentation at Doris Duke Fellowships Annual Meeting. Cash, S. J. (2001). Risk assessment in child welfare: The art and science. Children and Youth Services Review, 23(11), 811–830. Cash, S. J., & Berry, M. (2002). Family characteristics and child welfare services: Does the assessment drive service provision? Families in Society, 83(5/6), 499–507. Charmaz, K. (2006). Constructing grounded theory: A practical guide through qualitative analysis. London: Pine Forge Press. Cresswell, J. W. (1998). Qualitative inquiry and research design: Choosing among the five traditions. Thousand Oaks, CA: Sage. Creswell, J. W. (2003). Research design: Qualitative, quantitative, and mixed methods approaches (2nd ed.). Thousand Oaks, CA: Sage. Cross, T. (2009-2010). Obstacles and opportunities in accessing mental health services for children in foster care: Lessons from recent history in Illinois. Illinois Child Welfare, 5(1), 86–107. Retrieved from. http://www.illinoischildwelfare.org/archives/volume5/icw5-cross.pdf Curry, D., McCarragher, T., & Dellmann-Jenkins, M. (2005). Training, transfer, and turnover: Exploring the relationship among transfer of learning factors and staff retention in child welfare. Children and Youth Services Review, 27, 931–938. Holland, S. (2001). Representing children in child protection assessments? Childhood, 8(3), 322–339. Jent, J. F., Merrick, M. T., Dandes, S. K., Lamert, W. F., Haney, M. L., & Cano, N. M. (2009). Multidisciplinary assessment of child maltreatment: A multi-site pilot descriptive analysis of the Florida Child Protection Team model. Children and Youth Services Review, 31(8), 896–902. Johnson, M. A., Stone, S., Lou, C., Vu, C., Ling, J., Mizrahi, P., & Austin, M. J. (2006). Family assessment in child welfare services: Instrument comparisons. Berkeley, CA: University of California, Center for Social Services Research. Léveillé, S., & Chamberland, C. (2010). Toward a general model for child welfare and protection services: A meta-evaluation of international experiences regarding the adoption of the Framework for the Assessment of Children in Need and Their Families (FACNF). Children and Youth Services Review, 32(7), 929–944. Lincoln, Y. S., & Guba, E. G. (1985). Naturalistic inquiry. Beverly Hills, CA: Sage. National Implementation Research Network. (2005). Additional evidence for staff selection. Retrieved from. http://nirn.fpg.unc.edu/sites/ nirn.fpg.unc.edu/files/resources/NIRN-AdditionalEvidence-StaffSelection.pdf Padgett, D. K. (1998). Qualitative methods in social work research: Challenges and rewards. Thousand Oaks, CA: Sage.

206

C. Smithgall et al. / Child Abuse & Neglect 44 (2015) 194–206

Parton, N. (2009). Challenges to practice and knowledge in child welfare social work: From the social to the informational? Children and Youth Services Review, 31, 715–721. Petrucci, C., & Mills, L. G. (2002). Domestic violence assessment: Current practices and new models for improved child welfare interventions. Brief Treatment and Crisis Intervention, 2(2), 153–172. Reinders, H. (2010). The importance of tacit knowledge in practices of care. Journal of Intellectual Disability Research, 54, 28–37. Ruch, G. (2005). Relationship-based practice and reflective practice: Holistic approaches to contemporary child care social work. Child and Family Social Work, 10, 111–123. Sandelewski, M. (1993). Rigor or rigor mortis: The problem of rigor in qualitative research revisited. Advances in Nursing Science, 16(2), 1–8. Schene, P. (2005). Comprehensive family assessment guidelines for child welfare. New York, NY: National Resource Center for Family-Center Practice and Permanency Planning. Smithgall, C., Jarpe-Ratner, E., Yang, D. H., DeCoursey, J., Brooks, L., & Goerge, R. (2009). Family assessment in child welfare: The Illinois DCFS Integrated Assessment Program in policy and practice. Chicago: Chapin Hall at the University of Chicago. Smithgall, C., Jarpe-Ratner, E., & Walker, L. (2010). Looking Back, Moving Forward: Using integrated assessments to examine the educational experiences of children entering foster care. Chicago: Chapin Hall at the University of Chicago. Strauss, A., & Corbin, J. (1990). Basics of qualitative research: Grounded theory procedures and techniques. Newbury Park, CA: Sage. Thomas, J., & Holland, S. (2010). Representing children’s identities in core assessments. British Journal of Social Work, 40, 2617–2633.