Journal of Evidence-Based Social Work, 11:484–497, 2014 Copyright © Taylor & Francis Group, LLC ISSN: 1543-3714 print/1543-3722 online DOI: 10.1080/15433714.2013.853585

An Ecological Approach to Evaluating a System of Care Program: Dollars Making Sense Denise M. Green Department of Human Services, Social Work, and Rehabilitation, Troy University, Troy, Alabama, USA

Sarah E. Twill Department of Social Work, Wright State University, Dayton, Ohio, USA

Larry Nackerud and Michael Holosko School of Social Work, University of Georgia, Athens, Georgia, USA

System of care (SOC) models in North America were developed in response to the needs of children with a severe emotional disturbance. Such children experience problems across life spheres including issues at home that put them at risk of abuse and neglect, difficulties at school including special education classification and dropping-out, and involvement with the juvenile courts. SOC evaluations and research suggests that an overreliance of evaluative research efforts on standardized scales and preconceived measurable outcomes have resulted in a loss of other important data. This study’s confirmatory and holistic approach to evaluation illuminates important information concerning commonly ignored variables when using traditional evaluation models. The evaluative research study described focuses on three often overlooked behavioral variables in one SOC initiative, KidsNet Georgia, of Rockdale County, GA. These variables are: (a) using cohort analysis over time; (b) costing out services utilized; and (c) focusing on behavioral indicators and chance over time. The evaluative strategy, data collection, data, and cost analysis are discussed along with implications for practice with severe emotional disturbance youth and their families. Keywords: System of care, youth, severe emotional disturbance, program evaluation, service utilization

INTRODUCTION Children with a severe emotional disturbance (SED) experience problems across life spheres (Stroul, Blau, & Sondheimer, 2008) and responding to these needs requires an ecological perspective of care. Youth with SED faced multiple life complications including involvement in the juvenile justice system (Drerup, Croysdale, & Hoffmann, 2008; Hussey, Drinkard, Falletta, & Flannery, 2008), contact with the child welfare system (Kessler et al., 2008; Pecora, White, Jackson, & Wiggins, 2009), high rates of out of home placements (Farmer, Mustillo, Burns, & Address correspondence to Denise M. Green, Department of Human Services, Social Work, and Rehabilitation, Troy University, 107 McCartha Hall, Troy, AL 36082, USA. E-mail: [email protected]

484

ECOLOGICAL APPROACH TO SOC EVALUATIONS

485

Holden, 2008; Fontanella, 2008), and difficulty in school (Reschly & Christenson, 2006; Zigmond, 2006). Providing care for youth with SED in single systems can be expensive, both in the short and long term, and often ignores the premise of providing care in the community. Youth are classified as SED if they have a Diagnostic and Statistical Manual of Mental Disorders (DSM) diagnosis which causes significant functional impairment across life spheres such as the school, home, and community (Stroul et al., 2008). Additionally, in order to be classified as SED, youth must be served by two or more agencies such as mental health centers, schools, juvenile courts, or child welfare systems. It is estimated that the median prevalence rate for having a SED is 12% (Costello, Egger, & Angold, 2005). The Department of Health and Human Services’ Comprehensive Community Mental Health Services for Children and Their Families Program (n.d.) projected that between 4.5 and 6.3 million American youth fit the SED criteria.

LITERATURE REVIEW Scope of Problems Necessitating a System of Care (SOC) Mental Health Needs Mental health problems impact many children as well as their families, schools, and communities. Over 46% of youth 13–18 years of age had a lifetime prevalence of a mental health disorder; over 21% had a lifetime prevalence of a severe mental health disorder (Merikangas, et al., 2010). The National Institute for Mental Health (NIMH, 2006) estimated that 4.6 million youth used mental health services in a given year and that care cost $8.9 billion. Mental health services for youth cost nearly $2,000 more than care for adults. Further, NIMH (2002) estimated that 6% of the adult population had a debilitating mental illness that resulted in $300 billion per year in direct and indirect costs. Mark and Buck (2006) found that youth with SED were most likely to be caucasian, live in families with incomes above the poverty line, and have health insurance. However, they also found that youth of color were overrepresented and that the risk for being SED was higher for low income youth. Also, 28% of youth with SED and served by multiple service systems met criteria for posttraumatic stress disorder (PTSD; Mueser & Taub, 2008). Juvenile Justice Needs As a result of their externalizing behaviors, youth with emotional disturbance often come in contact with the juvenile courts. Hussey, Drinkard, and Flannery (2007) found that 65% of juveniles in a detention facility who were served by a SOC initiative met diagnostic criteria for at least one DSM diagnosis. Of those with a mental health diagnosis, 65% also met criteria for a substance abuse disorder. Drerup, Croysdale, and Hoffmann (2008) studied over 600 youth in the juvenile justice system and estimated that 92% of males and 97% of females had at least one DSM diagnosis. Ryan and Redding (2004) reviewed articles about the prevalence of mood disorders in the juvenile offender populations published after 1980. They found that prevalence rates for mood disorders varied across studies from 17–78%. This review also suggested that mood disorders exacerbated the inappropriate behaviors of the delinquent youth. Youth involved in the juvenile justice system had higher rates of SED: 46% of youth on probation and 67% of those incarcerated met SED criteria (Lyons, Baerger, Quigley, Erlich, & Griffin, 2001). Hussey et al. (2008) postulated that substance abuse, emotional problems, behavioral complexities, internal mental distress, environmental risk, and conflict tactics resulted in higher service utilization and thus, increased cost of care for youth in the juvenile justice system.

486

D. M. GREEN ET AL.

Providing juvenile justice services is indeed expensive. In 2007, over 64,000 U.S. youth were detained, costing 5.7 billion (Sickmund, Sladky, Kang, & Puzzanchera, 2008). Juvenile detention is estimated to cost upwards of $70,000 per bed (Annie E. Casey Foundation, n.d.). The cost of incarcerating an adult in the jail or prison system is also costly; adult incarceration in turn, costs approximately $26,000 per year (Schmitt, Warner, & Gupta, 2010), about double the cost of SOC services. Child Welfare Needs About 50% of children in the child welfare system had mental health issues and behavioral problems (Barth et al., 2007; Burns et al., 2004). In a study of children under six years of age, half of the sample had behavioral problems that put them at risk of poor developmental trajectories (Stahmer et al., 2005). Later in adolescence, youth served in the child welfare system were at risk of becoming delinquent (Grogan-Kaylor, Ruffolo, Ortega, & Clarke, 2008). Older youth, males, and those who had been physically abused, were more likely to engage in delinquent acts. Instability of placements also influenced the likelihood of behavioral problems (Rubin et al., 2008). When compared to youth in foster care, youth in kinship care were less likely to have behavior problems or use mental health services. Pecora et al. (2009) reviewed the literature regarding mental health outcomes for current and former recipients of foster care. They established that the most common diagnosis of alumni of the foster care system were PTSD, major depression, and alcohol dependence. Hansen and Hansen (2006) projected that child welfare spending was just under $14,000 per child per year. Rubin et al. (2004) studied mental health costs for children in foster care, and found that youth with multiple placements or those who experienced episodic foster care increased the likelihood of mental health service usage. They also found that top 10% of mental health care users accounted for 83% (almost $2 million of the $2.4 million) of mental health costs. School Needs Youth with mental health problems may also experience problems at school. Kuo, Vander Stoep, McCauley, and Kernic (2009) reported it was cost effective to provide school-based mental health screenings for all students. Such screenings cost a maximum of $14 per student and, of those students referred for services, 72% were linked to services within six weeks. A maximum cost savings of $416.90 per successful linkage was estimated. For youth at risk of and with emotional disturbances, school-based intervention programs reduced externalizing behaviors (Reddy, Newman, De Thomas, & Chun, 2009). If problematic behaviors are not detected and addressed early, childhood disorders may lead to poor school performance which can ultimately result in lack of employment opportunities, increased health care costs, and poverty in adulthood (Clark, Koroloff, Geller, & Sondheimer, 2008; Knapp, McCrone, Fombonne, Beecham, & Wostear, 2002). In 2008–2009, 420,000 students were classified as having an emotional disability which qualified them for special education services (U.S. Department of Education, 2010). For youth with SED, the consequences of school failure were significant. Students served in a self-contained classroom for emotional disabilities performed below the 25th percentile in reading, math, and written expression proficiencies (Lane, Barton-Arwood, Nelson, & Wehby, 2008). Further, about 43% percent of youth with SED dropped out, compared to 10–23% of those in other disability categories (U.S. Department of Education, 2010). Reschly and Christenson (2006) examined factors that helped predict how and why special education students dropped out. They reported that for students with emotional-behavioral disorders, grade retention was most predictive, followed by behavioral engagement factors such as discipline reports, absences, and cutting class. Following high school, youth with SED had problems keeping

ECOLOGICAL APPROACH TO SOC EVALUATIONS

487

jobs, as Zigmond (2006) found that youth with SED, both graduates and drop-outs, were employed at rates of about 50%. Their jobs were part-time, lacked health insurance, and earned close to the minimum wage, all of which may contribute to long term costs to society. The SOC Model The SOC philosophy grew out of the necessity of treating youth with SED in the most appropriate and least restrictive environments (LRE; Stroul, 2003; Stroul et al., 2008; Stroul & Friedman, 1986). Stroul and Friedman (1986) proposed the SOC philosophy for serving youth with SED. They outlined three core values and 10 guiding principles for service delivery for youth with SED. Such youth require a multitude of services ranging from health care, mental health care, legal services, foster care services, respite care, and educational care (Stroul et al., 2008; Stroul, 2003; Stroul & Friedman, 1986). The SOC model ideally unites these various organizations, with often philosophical differences, under a common umbrella with the same mission. A major premise of the SOC model is that in a “best case scenario” youth and their families are to receive comprehensive wrap around mental health services in the least restrictive and most clinically appropriate environment. Philosophically, LREs promote dignity of the youth and their families and give clients increased self-determination. Bickman et al. (1995) found that participants in a SOC program were served in increasingly LREs than those served in traditional settings. Serving youth in the LRE was both cost effective and provides youth and families treatment options integrated into their everyday lives. Barth et al. (2007) found that some community-based services were equal to or better than some out-of-home placement. Further, community-based treatment was less expensive that residential care. Daleiden, Pang, Roberts, Slacin, & Pestle (2010) found that for over 80% of participants intensive home based services were successful in keeping youth in the LRE. Older youth and those with greater behavioral impairment at intake were more likely to need residential care. At the same time that services to youth with SED were being evaluated within the mental health arena, the public educational system was reconsidering how to meet the needs of special education students. The Individuals with Disabilities Education Act (IDEA) and IDEA 1997, mandated mental health services for youth with SED (Latham, Latham, & Mandlawitz, 2008). This helped formally bring together the education system and mental health agencies, and promoted the growth of the SOC concept as we now know it today (Lourie, 2003). SOC evaluative research has largely focused on outcome variables measured by standardized scales such as the Child Behavior Checklist (CBCL), the Child and Adolescent Functional Assessment Scale (CAFAS), and the Youth Self Report (YSR), while less attention has been paid to behavioral indicators collected in the natural environment (Bickman et al., 1995; Friesen & Winters, 2003; Manteuffel, Stephens, Brashears, Krivelyova, & Fisher, 2008; U.S. Department of Health and Human Services, 2003). The results of relying solely upon aggregated standardized scale data to determine program effectiveness connotes that the richness of other data sources are ignored or are not important. This is especially true in rural communities where lengthy and time-consuming data collection protocols are viewed suspiciously by participants who reject the offer to enroll in the longitudinal study (personal communication, KidsNet Staff, August, 2004). For those communities who have had difficulty enrolling participants in a longitudinal study, a program evaluation focusing on behavioral indicators may be an appropriate way in which to judge program effectiveness. The KidsNet program of Rockdale County, GA used a SOC model and was the focus of this evaluative research study. The conceptual foundation for this evaluation research study flows from the core values and guiding principles of the SOC model. Central to that belief system is the notion that the best way to care for youth with SED is the creation of a family and youth centered system. For this evaluation, operationalization of this belief was established by drawing pre and post comparison data on

488

D. M. GREEN ET AL.

service utilization for two non-duplicate cohorts on the following SOC services: school-based services, outpatient mental health services, psychiatric inpatient services, residential services, psychiatric outpatient programs, and law enforcement contacts. Two research questions which studied the evaluation were crafted out of the assumed contention that a SOC is a better practice model of providing care for youth with SED. The first question was: Does the SOC serve participants in the LRE? The second was: How much does it cost to serve youth in the SOC? Both of these were designed to determine the worth and merit of this type of service delivery, using other data than have been reported in evaluations of SOC programs. What follows is: First, a brief description of the KidsNet Program, second, an articulation of the evaluative research methods, third, a depiction of participant demographics, and finally, evaluative research results, via the two research questions and two cohorts, are presented and discussed. Taken together, they are presented as a rationale for the inclusion of other important data in such evaluative efforts. Included in the discussion are implications for practice and programming with SED youth and families.

The KidsNet Program KidsNet of Rockdale County is part of the PeachState Wrap-around Initiative (PSWI) located in Georgia. In October 2000, a six-year federal grant was awarded to the Rockdale County Board of Commissions with the purpose of developing a coordinated community (system) of care for youth with a SED and their families. The mission statement of KidsNet is “to create family driven, culturally sensitive opportunities through the use of blended resources and collaboration to allow children with a SED to thrive in the community” (KidsNet Pamphlet, 2000). Youth qualify for KidsNet services based on the Community Mental Health Services (CMHS) initiative definition of SED. The criteria are: Age, diagnosis, disability, multiagency need, and duration and intensity of illness. KidsNet served youth with SED in the mental health catchments areas of Rockdale, Gwinnett, and Newton Counties. Youth are referred to KidsNet from a variety of sources including: therapists at the mental health center, Department of Juvenile Justice (DJJ) workers, educators, Division of Family and Children’s Services (DFCS) case managers, private treatment providers, or parents (KidsNet Pamphlet, 2000). Youth are accepted into the SOC through treatment team staffing. Once a youth is accepted, progress through the levels of care are determined by the individualized, unified treatment plan. Step-downs and re-entry into the program were determined by individual needs. Services were provided using a SOC model. Multiple community agencies combined resources and talents to best serve youth with SED. Studying the same SOC as evaluated in this evaluation, Copp, Bordnick, Traylor, and Thyer (2007) examined the effectiveness of wrap-around services for 15 participants using two standardized scales, the CAFAS and the CLBC. At the six month follow-up, no significant differences were found in client outcomes. Most recently, Stroul et al. (2010) renewed the call to remove barriers between researchers and communities providing mental health services.

METHOD This confirmative evaluation strategy incorporated a Tylerian (objective-based) research design. The program in this evaluative research study had been in existence for five years. Importing data from a program in full operation provides the researchers with a stabilizing component for data collection–specifically, staff is trained, policies are in place, and the system has recognized challenges associated with new programs and developed appropriate solutions. Also, budgetary

ECOLOGICAL APPROACH TO SOC EVALUATIONS

489

issues such spending and matching federal funds has leveled off. In addition, referral of members into the program and service delivery is efficacious due to staff expertise, collaborations, and policy establishment. All study procedures were approved by the University of Georgia Institutional Review Board and KidsNet, Georgia. Initial consent was provided by participants and caretakers to enter in and receive treatment from KidsNet (this included data collection and research). Participants and their care takers reaffirmed their willingness to participate in this study by renewing consent forms yearly. Data were collected through a survey of participants in both the 2005 and 2006 cohorts. Participants were asked to describe their service utilization and interactions with the KidsNet staff. After the participants completed a survey, the research team asked the KidsNet staff to verify information provided by the participants. Medical records were also reviewed in order to verify data provided by the participants. Triangulation of these data provided the corroborated and accurate information (Padgett, 2008). Finally, The Carter Center’s Georgia Mental Health Gap Analysis (2006), was used to assess information specific to the county in which the project and research occurred. This allowed the research team to extend verification of data beyond selfreported information from the participants thus adding reliability and validity and more confidence to the data’s integrity. Participants Two non-duplicate cohorts, Cohort One (n1 D 78 who received services in 2005), and Cohort Two (n2 D 69 who received services in 2006) were followed for one year prior to enrolling in the program, and one year during enrollment in the program. In both, approximately half of the participants were Caucasian. Additionally, for both cohorts, the court system made the most referrals to the SOC program. In some cases, data sets were not complete, therefore the reported number of participants in some data sets may be less than the original cohort number. All individuals were residents of the catchment area and enrolled in various state programs, prior to their enrollment in the KidsNet SOC (see Table 1). Some of the more significant demographic information collected concerning these youth and their families centered on poverty, and the significant life history of the primary caretakers of these youth, as reported in Table 2. Families in KidsNet SOC existed in low income with more than half living below the 2007 federal poverty line of $20,650 (U.S. Department of Health and Human Services, 2007). Even more telling was the fact that 85% of Cohort Two families lived below the living wage of $31,200. Living wage estimates were determined to be the hourly wage necessary for a family of four to procure housing, decent food, and minimum health insurance in an area. The living

TABLE 1 Participant Demographics for the Two Cohorts Demographics Age Gender Race

Primary referral source

Cohort One (n D 78)

Cohort Two (n D 69)

M D 14 years (SD D 3.40) 67% male 35% female 47% Caucasian 42% African American 11% Hispanic

M D 13.7 years (SD D 3.63) 73% male 27% female 51% Caucasian 38% African American 6% Hispanic 4.5% Other 56% juvenile justice

41% juvenile justice system

490

D. M. GREEN ET AL.

TABLE 2 Participants’ Household Income for the Two Comparative Cohorts Income Levels

Cohort One (n D 53)

Cohort Two (n D 54)

21.8% 25.5% 20.0% 16.4% 10.9% 5.5%

37% 24% 24% 13% 1.8% 0%

Less than $10,000 $10,000–$19,999 $20,000–$34,999 $35,000–$49,999 $50,000–$74,999 $75,000 and over

wage estimates for Rockdale County ranged from approximately $10–$30 per hour, depending on family composition (Glasmeier, 2011). In addition, information gathered on primary caretakers demonstrated a significant history of critical life issues. Primary caregivers to participants in Cohorts One and Two struggled with the following challenges: alcohol and substance abuse (66.7% of caregivers in Cohort One and 50.1% of caregivers in Cohort Two), family violence (63.3% and 44.9%), psychiatric hospitalization (56.5% and 29%), mental illness (56.5% and 68.1%), and criminal conviction (53.8% and 36.2%).

RESULTS I. Research Question One: LRE? The following data compare previous service utilization and SOC utilization for the Cohorts One (n1 D 78) and Two (n2 D 69) participating in KidsNet SOC for the calendar year of 2005 and 2006, respectfully. Data were drawn from multiple sources. First, families filled out a service utilization questionnaire designed to gather self-reported information about the use of services one year prior to enrolling in KidsNet. Additional triangulation of data were completed using The Carter Center’s Georgia Mental Health Gap Analysis (2006), KidsNet staff, participating service agencies, and Kids Net youth utilizing direct interviews. These data were collected exclusively in Rockdale County, Georgia (see Table 3).

TABLE 3 Service Utilization Before and During SOC Services (n1 D 78, n2 D 69) Service Menu School-based services Outpatient services Psychiatric hospitalization Residential treatment Psychiatric programs ETOH/SA treatment

Previous*

2005**

Previous*

2006**

74.5% 68.6% 40.0% 32.1% 23.2% 8.0%

81.3% 87.5% 18.8% 8.9% 18.8% 0.0%***

62.3% 55.0% 23.1% 16.0% 13.0% 1.5%

78.2% 95.6% 11.6% 13.0% 1.5% 0.0%***

*Previous indicates service utilization one year prior to enrolling in SOC services. **2005 and 2006 indicates service utilization for the current year enrolled in SOC services. ***Alcohol and substance abuse therapy is currently rolled into standard outpatient services.

ECOLOGICAL APPROACH TO SOC EVALUATIONS

491

TABLE 4 Cohort One–Cost of Service Utilization (n1 D 78) Service Used KidsNet wrap around services Residential treatment Psychiatric inpatient Community assessments Community donations Total Cost per participant

Yearly Cost of Service, 2005 $500,000.00 $499,654.93 $54,417.00 $13,975.00 $6,449.00 $1,074,495.93 $13,775.59

Cohort One The first noteworthy finding here was the consistent increase in school-based services and outpatient services while enrolled in KidsNet. Coupled with this increase was the significant decrease in the more costly inpatient, residential, psychiatric programs, and law enforcement contact (to include juvenile youth detention) during the same period of time. While the philosophy that a SOC established in the community (that is family and youth centered) is the best way to care for youth, these comparative data on service utilization provided sufficient evidence that KidsNet SOC did maintain youth in the LRE in addition to greatly reducing the more costly services which in turn, supports the efficacy of this type of service delivery. Cohort Two Cohort Two data again demonstrated the efficacy of the KidsNet SOC. Prior service utilization was significantly increased in school-based and outpatient services while the more costly inpatient, residential, and psychiatric programs are reduced. The reason that school-based and outpatient services were not at 100% had to do with families utilizing private pay tutors or private psychiatric services. There were also a small number of families that may have refused one or more services. II. Research Question Two: Cost Effectiveness? Table 4 illustrates the monetary breakout of service costs for Cohort One (n1 D 78) in calendar year 2005. These data were drawn from multiple sources. First, families filled out a service utilization questionnaire designed to gather self-reported information about the use of services for the calendar year. Additional triangulation of data was completed using the Carter Center’s Georgia Mental Health Gap Analysis (2006), KidsNet staff, participating service agencies, and KidsNet youth. These data sets were collected exclusively in Rockdale County, Georgia. Cohort One Within the category of KidsNet wrap-around services, is the embedded cost of operating the KidsNet program, outpatient services, training and travel, and staff salaries. No additional information was provided to analyze this category. The next largest category, residential treatment, comprised the bulk of service delivery costs. While only seven youth used this service, the average cost per day was $197.19, and length of stay on average was 365 days. The reasons for residential treatment included increases in severity

492

D. M. GREEN ET AL.

TABLE 5 Cohort Two–Cost of Service Utilization (n2 D 69) Services Used

Yearly Cost of Service, 2006

Residential treatment KidsNet wrap around services Psychiatric inpatient Community assessments Community donations Total Cost per participant

$663,964.20 $500,000.00 $13,152.57 $9,100.00 $8,209.00 $1,194,425.77 $17,310.52

of behavior, family breakdown, and risk to other family members. Psychiatric inpatient services were used by 15 youth at an average of eight days at a cost of $458.14 per day. Other services not reported, but should be considered in future analyze include: Respite care, juvenile detention costs, medication costs, child care costs, travel costs, and emergency financial assistance such as paying an electric bill in order to prevent homelessness or help prevent psychiatric decomposition which may result in institutional care (see Table 4). Cohort Two As previously noted, the category of wrap-around services includes the cost of operating the KidsNet program, outpatient services, training and travel, and staff salaries. The largest category, residential treatment, comprised the bulk of the service delivery costs. In 2006, a total of nine youth used this service, the average cost per day was $202.12, and length of stay on average was 365 days. A 2% cost of living increase was added to the raw figures. Psychiatric inpatient services were used by eight youth for an average of 3.5 days at a cost of $469.50 per day. Not only was there a significant decrease seen in the number of youth utilizing psychiatric inpatient services, the length of stay was reduced by more than 50% (see Table 5). Cost of Delivering SOC Services Finally, an overall summary of cost per participant, both pre and during the SOC of care, was calculated for both cohorts. For Cohort One, the year prior to utilizing a SOC cost per participant was $30,405.85, and the SOC cost was $13,513.74. For Cohort Two, the year prior to utilizing a SOC cost per participant was $49,741.15, and the SOC cost was $17,059.66. For both cohorts, the reduction in cost of providing services was over 50% per participant. The KidsNet SOC cost of service utilization is within the range of cost cited by Manteuffel et al. (2008) research as referenced in The System of Care Handbook by Stroul, Blau, and Sondheimer (2008).

DISCUSSION The rationale for utilizing additional cohort level data in this study seems warranted and it yielded some interesting findings. This evaluative research study examined two cohorts of youth served by a SOC in Georgia. Service utilization and cost were examined for a year prior to enrollment in the SOC and during the first year of service provision. In both cohorts, the use and cost of schoolbased services and outpatient mental health services went up in the first year of participation in

ECOLOGICAL APPROACH TO SOC EVALUATIONS

493

the SOC. Simultaneously, utilization of inpatient psychiatric services, residential treatment, and psychiatric services went down for both cohorts during the first year of SOC services. These data indicated that the participants were served in the LRE appropriate for their behavioral and psychiatric needs. Costly services such as residential care which are approximately $200 per day for an average 365 day stay ($73,000 per child per stay) and inpatient psychiatric treatment which cost approximately $460 per day with stays ranging from four to eight days (1,840–$3,680 per stay) were reduced. Despite this reduction, residential and psychiatric inpatient cost still exceeded $550,000 in 2005 and $670,000 in 2006. Wrap-around services which included staff salaries and all outpatient services cost $500,000 per year. In total, the operating budgets were $1,074,495.93 in 2005 and $1,194,425.77 in 2006. It cost an average of $13,775.59 in 2005 and $17,310.22 in 2006 to treat each participant. While this may be perceived as costly, service utilization costs in the year prior to participation in the SOC were $30,505 for Cohort One and $49,741 for Cohort Two. When viewed from this lens, participation in the KidsNet SOC reduced costs for both groups by a resounding 50%. In comparison, Bickman et al. (1995) found it cost $7,777 to serve youth at the Fort Bragg demonstration site (SOC), compared to $4,904 at the comparison sites (traditional services). When adjusted to reflect inflation of 2% per year which was likely to occur between 1996 when the study was conducted and 2006, the new projected costs would be $9,669 and $5,978.38, respectively. Bickman and colleagues attributed the differences in costs between the SOC service and traditional services to the fact that youth in the demonstration site spent more time in treatment, used a greater number of services (e.g., outpatient), had higher usage of the intermediate services (e.g., day treatment, wrap-around), and that costs associated with such service were higher. In this study, it was also important to examine the reduction of previous service utilization prior to enrollment into KidsNet Georgia for the 2006 cohort, as compared to the 2005 cohort. The former data revealed a trend that may demonstrate the impact that a SOC philosophy has on the entire community. For example, once a SOC model is accepted within the community and its service providers, an attitude shift occurs that affects all children provided services within that community. Communities and agencies unite under a common collective belief that youth and their families are best treated in their communities regardless of levels of service. Another interesting finding was the high percentages of life crises experienced by the caregivers of the participants. Alcohol and substance abuse, family violence, psychiatric hospitalization, mental illness, and criminal conviction were all common experiences of the participants’ caregivers. Additionally, 50% of the families were living below the federal poverty level or the living wage for the area, which may have further complicated life circumstances of the participants. Parental well-being contributes to the well-being of children (Waldfogel, Craigie, & Brooks-Gunn, 2010). The role of life circumstances on the development of the participants’ SED or how caregiver impairment (or lack thereof) exacerbates or helps alleviate issues is complicated and warrants further examination. Limitations While the results of this evaluation research were quite promising, some caution needs to be taken interpreting the findings. This study was limited to N D 147 participants in one SOC in Georgia. Demographic factors such as race or gender, or previous experiences with service providers, poverty rates, and level of impairment of the caregivers may limit the generalizability of findings to different populations. Additionally, due to ethical concerns, there was no assignment to group, either with a different intervention or no intervention. It is therefore, difficult to ascertain the absolute degree to which participation in the SOC resulted in the behavioral changes that caused different service utilization

494

D. M. GREEN ET AL.

patterns, or if time and other external forces influenced these changes. Maturation also is an internal validity threat to these data, as it is possible that youth simply “aged-out” of inappropriate behaviors, which initially resulted in services. Additionally, the participants’ caregivers may have stabilized some of their own problematic behaviors or life challenges. Modifying, and/or eliminating these concerns may have resulted in better parenting or increased involvement or supervision of the participants. There were also some practical limitations about data collection that may have influenced the findings. Data were collected from primary and secondary records. As such, missing data were unable to be captured. Further, it was assumed that the information contained in these records was accurate. As well, not all of the records of youth who participated in the KidsNet program were available. Some relocated and their case files were transferred to another mental health center, and other files could not be located. To temper these limitations, two strategies were implemented. First, the 2005 and 2006 cohorts were selected because the KidsNet program had been established and formative start-up issues were resolved. Extracting data from a program in full operation provided the researchers with a relatively stabilizing component–specifically, staff was trained, policies were in place, and the system had recognized challenges associated with new programs and developed appropriate solutions. Referral of members into the program and service delivery was relatively efficacious due to staff expertise, collaborations, and policy establishment. Finally, triangulation of these data helped establish confidence, accuracy, and reduced errors in self-report (Padgett, 2008). Implications for Practice and Future Research A major premise of the SOC model is that youth and their families are to receive comprehensive mental health services in the least restrictive and most clinically appropriate environment. Results of this evaluation showed that youth served by a SOC program increased school-based and outpatient mental health services, while decreasing restrictive placements like inpatient psychiatric settings or residential treatment programs. This corroborates the landmark research of Bickman et al. (1995) who found that participants in a SOC program were served in increasingly LREs than those served in traditional settings. Serving youth in the LRE is both cost effective and provides youth and families treatment options integrated into their everyday lives. This promotes dignity of the youth and their families and gives clients increased self-determination. At face value, agency directors, budget managers, tax payers, and politicians may see SOC services as expensive, and they are not necessarily incorrect in that perception. However, individuals who are not trained clinicians may not understand the variegated complexities and severity of behaviors and psychiatric symptoms of youth with SED. Without intensive wrap-around services, such youth may decompensate to the point of requiring inpatient psychiatric hospitalizations or residential care, both of which are costly. Further, given the behavioral issues displayed by these youth, one must also consider the affiliated cost of juvenile detention which is estimated to cost upwards of $70,000 per bed (Annie E. Casey Foundation, n.d.). If an appropriate intervention is not provided to youth with SED, the projected lifetime costs to both youth and the overall community will far exceed the cost of services. Youth with SED have problems with gainful employment including obtaining only part-time work with wages near minimum wage and without health benefits (Zigmond, 2006). Without any economic stability, individuals may turn to a life of crime to survive. Once caught, the cost of incarcerating an adult in the jail or prison system is approximately $26,000 per year (Schmitt et al., 2010), about double the cost of SOC services. In our current uncertain economic times, findings from this evaluation should not be inappropriately used to advocate for mental health budget cuts. Rather, the cost savings related to the reduction of inpatient hospitalization and residential treatment should be reallocated to more LRE

ECOLOGICAL APPROACH TO SOC EVALUATIONS

495

services. SOC services are less costly than those in inpatient settings. However, as more youth with SED are maintained in our community rather than being served in inpatient and residential settings, it is likely that more SOC community services will be needed. Future researchers should continue to evaluate whether or not SOC services are beneficial for these youth and families served by it. Research is also needed in the area of youth with SEDs in the SOC model. Researchers should continue to evaluate the treatment effectiveness of the SOC model. Previous research has disputed the effectiveness of SOC services (Bickman et al., 1995; Bickman, Noser, & Summerfelt, 1999; Evans & Banks, 1996; Mordock, 1997; Saxe & Cross, 1997). Given the expense of providing SOC services, the body of knowledge needs to continue to grow to determine the most efficient and effective manner to serve youth in crises. At the same time, future research should be conducted to determine if dollars spent today in providing SOC services reduce future expenditures on the participants as they mature. It would be important to establish that SOC services help reduce future incarceration or improve workforce readiness. Given the severity of issues youth with SED face, it is unlikely that SOC services provided during adolescence would mitigate the need for any intervention as adults. However, it would be desirable if youth with SED could become adults who need fewer, less intensive service and were able to participate in more appropriate ways in society. And as this study has contended, additional data such as these need to be incorporated into research and evaluation efforts on SOC models. Finally, for youth and their families, the necessity to be served within their own communities is essential to the preservation of families, inclusion in choices of self-care, and is the cornerstone of self-determination. In addition to these quality of life issues, the cost of SOC service delivery in this study was demonstrated to be half or more of utilizing “traditional methods” of service delivery. A SOC reduces the higher, more restrictive services, while providing community-based services designed to keep families together. As this study found, even when community-based services increased, the overall costs remained significantly lower. Future evaluative research efforts, focused on behavioral indicators such as service utilization and cost effectiveness may lend additional empirical to the strongly embraced core values and guiding principles foundation of the SOC model. As indicated in the title, this article attempts to show how more carefully accounting for dollars in such evaluations makes sense, in today’s climate of cost effectiveness and reductions in spending for health and human services.

REFERENCES Annie E. Casey Foundation (n.d.). Detention reform: A cost-savings approach. Retrieved from http://www.aecf.org// media/Pubs/Other/D/DetentionReformACostSavingApproach/JDAI_facts_1.pdf Barth, R. P., Lloyd, E. C., Green, R. L., James, S., Leslie, L. K., & Landsverk, J. (2007). Predictors of placement moves among children with and without emotional and behavioral disorders. Journal of Emotional and Behavioral Disorders, 15, 46–55. Bickman, L., Guthrie, P. R., Foster, E. M., Lambert, E. W., Summerfelt, W. T., Breda, C. S., & Heflinger, C. A. (1995). Evaluating managed mental health services: The Fort Bragg experiment. New York: Plenum Press. Bickman, L., Noser, K., & Summerfelt, W. T. (1999). Long-term effects of a system of care on children and adolescents. Journal of Behavioral Health Services, 26, 185–202. Burns, B. J., Phillips, S. D., Wagner, H. R., Barth, R. P., Kolko, D. J., Campbell, Y., & Landsverk, J. (2004). Mental health need and access to mental health services by youth involved with child welfare: A national study. Journal of the American Academy of Child and Adolescent Psychiatry, 43, 960–970. doi:10.1097/01.chi0000127590.95585.65 Clark, H. B., Koroloff, N., Geller, J., & Sondheimer, D. L. (2008). Research on transition to adulthood: Building the evidence base to inform services and supports for youth and young adults with serious mental health disorders. Journal of Behavioral Health Services and Research, 35, 365–372. doi:10.1007/s11414-008-9140-4 Copp, H. L., Bordnick, P. S., Traylor, A. C., & Thyer, B. A. (2007). Evaluating wraparound services for seriously emotionally disturbed youth: Pilot study outcomes in Georgia. Adolescence, 42(168), 723–732.

496

D. M. GREEN ET AL.

Costello, E. J., Egger, H., & Angold, A. (2005). 10-year research update review: The epidemiology of child and adolescent psychiatric disorders: I. methods and public health burden. Journal of the American Academy of Child and Adolescent Psychiatry, 44, 972–986. doi:10.1097/01.chi.0000172552.41596.6f Daleiden, E. L., Pang, D., Roberts, D., Slavin, L. A., & Pestle, S. L. (2010). Intensive home based services within a comprehensive system of care for youth. Journal of Child and Family Studies, 19(3), 318–325. doi:10.1007/s10826009-9300-z Drerup, L. C., Croysdale, A., & Hoffmann, N. G. (2008). Patterns of behavioral health conditions among adolescents in a juvenile justice system. Professional Psychology: Research and Practice, 39, 122–128. doi:10.1037/0735-7028.39. 2.122 Evans, M. E., & Banks, S. M. (1996). The Fort Bragg managed care experiment. Journal of Child and Family Studies, 5(2), 169–172. Farmer, E. M., Mustillo, S., Burns, B. J., & Holden, E. W. (2008). Use and predictors of out-of-home placements within systems of care. Journal of Emotional and Behavioral Disorders, 16, 5–14. doi:10.1177/1063426607310845 Fontanella, C. A. (2008). The influence of clinical, treatment, and healthcare system characteristics on psychiatric readmission of adolescents. American Journal of Orthopsychiatry, 78, 187–198. doi:10.1037/a0012557 Friesen, B. J., & Winters, N. C. (2003). The role of outcomes in system of care: Quality improvement and program evaluation. In A. J. Pumariega & N. C. Winters (Eds.), The handbook of child and adolescent system of care (pp. 459– 486). San Francisco, CA: John Wiley & Sons. Glasmeier, A. (2011). Living wage calculation for Rockdale Country, Georgia. Retrieved from http://www.livingwage.geog. psu.edu/counties/13247 Grogan-Kaylor, A., Ruffolo, M. C., Ortega, R. M., & Clarke, J. (2008). Behaviors of youth involved in the child welfare system. Child Abuse and Neglect, 32, 35–49. doi:10.1016/j.chiabu.2007.09.004 Hansen, M. E., & Hansen, B. A. (2006). The economics of adoption of children from foster care. Child Welfare, 85(3), 559–583. Retrieved from http://faculty.buffalostate.edu/hennesda/matching/racial%20matching.pdf Hussey, D. L., Drinkard, A. M., Falletta, L., & Flannery, D. J. (2008). Understanding clinical complexity in delinquent youth: Comorbidities, service utilization, cost, and outcomes. Journal of Psychoactive Drugs, 40, 85–95. Hussey, D. L., Drinkard, A. M., & Flannery, D. J. (2007). Comorbid substance use and mental disorders among offending youth. Journal of Social Work Practice in the Additions, 7, 117–138. doi:10.1300/J160v07n01_07 Kessler, R. C., Pecora, P. J., Williams, J., Hiripi, E., O’Brien, K., English, D., : : : Sampson, N. A. (2008). Effects of enhanced foster care on the long-term physical and mental health of foster care alumni. Archives of General Psychiatry, 65, 625–633. Knapp, M., McCrone, P., Fombonne, E., Beecham, J., & Wostear, G. (2002). The maudsley long-term follow-up of child and adolescent depression: 3. Impact of comorbid conduct disorder on service use and costs in adults. British Journal of Psychiatry, 180, 19–23. Kuo, E., Vander Stoep, A., McCauley, E., & Kernic, M. A. (2009). Cost-effectiveness of a school-based emotional health screening program. Journal of School Health, 79, 277–285. doi:10.1111/j.1746-1561.2009.00410.x Lane, K. L., Barton-Arwood, S. M., Nelson, J. R., & Wehby, J. (2008). Academic performance of students with emotional and behavioral disorders served in a self-contained setting. Journal of Behavioral Education, 17, 43–62. doi:10.1007/s10864-007-9050-1 Latham, P. S., Latham, P. H., & Mandlawitz, M. R. (2008). Special education law. Boston, MA: Pearson. Lourie, I. S. (2003). A history of community child mental health. In A. J. Pumariega & N. C. Winter (Eds.), The handbook of child and adolescent system of care (pp. 17–34). San Francisco, CA: John Wiley & Sons. Lyons, J. S., Baerger, D. R., Quigley, P., Erlich, J., & Griffin, E. (2001). Mental health service needs of juvenile offenders: A comparison of detention, incarceration, and treatment settings. Children’s Services: Social Policy, Research, and Practice, 4, 69–85. Manteuffel, B., Stephens, R. L., Brashears, F., Krivelyova, A., & Fisher, S. K. (2008). Evaluation results and systems of care. In B. A. Stroul & G. M. Blau (Eds.), The system of care handbook (pp. 25–69). Baltimore: Paul H. Brookes Publishing Co. Mark, T. L., & Buck, J. A. (2006). Characteristics of U.S. youth with serious emotional disturbance: Data from the National Health Interview Survey. Psychiatric Services, 57, 1573–1578. doi:10.1176/appi.ps.57.11.1586 Merikangas, K. R., He, J., Burstein, M., Swanson, S. A., Avenevoli, S., Cui, L., Benjet, C., : : : Swendsen, J. (2010). Lifetime prevalence of mental disorders in U.S. adolescents: Results from the national comorbidity survey replicationadolescent supplement (NCS-A). Journal of the American Academy of Child and Adolescent Psychiatry, 49(10), 980– 989. doi:10.1016/j.jaac.2010.05.017 Mordock, J. B. (1997). The Fort Bragg continuum of care demonstration project: The population served was unique and the outcomes are questionable. Child Psychiatry and Human Development, 27, 241–254. Mueser, K. T., & Taub, J. (2008). Trauma and PTSD among adolescents with severe emotional disorders involved in multiple service systems. Psychiatric Services, 59, 627–634. doi:10.1176/appi.ps.59.6.627

ECOLOGICAL APPROACH TO SOC EVALUATIONS

497

National Institute of Mental Health. (2002). Annual total direct and indirect cost of serious mental illness. Retrieved from http://www.nimh.nih.gov/statistics/4COST_TOTAN.shtml National Institute of Mental Health. (2006). Mental healthcare cost data for all Americans. Retrieved from http://www. nimh.nih.gov/statistics/4COST_AM2006.shtml Padgett, D. (2008). Qualitative methods in social work research (2nd ed.). Los Angeles, CA: Sage. Pecora, P. J., White, C. R., Jackson, L. J., & Wiggins, T. (2009). Mental health of current and former recipients of foster care: A review of recent studies in the USA. Child and Family Social Work, 14, 132–146. doi:10.1111/j.13652206.2009.00618.x Reddy, L. A., Newman, E., De Thomas, C. A., & Chun, V. (2009). Effectiveness of school-based prevention and intervention programs for children and adolescents with emotional disturbance: A meta-analysis. Journal of School Psychology, 47, 77–99. doi:10.1016/j.jsp.2008.11.001 Reschly, A. L., & Christenson, S. L. (2006). Prediction of dropout among students with mild disabilities: A case for the inclusion of student engagement variables. Remedial and Special Education, 27, 276–292. Rubin, D. M., Downes, K. J., O’Reilly, A. L., Mekonnen, R., Luan, X., & Localio, R. (2008). Impact of kinship care on behavioral well-being for children in out-of-home care. Archives of Pediatric and Adolescent Medicine, 162, 550–556. doi:10.1001/archpedi.162.6.550 Ryan, E. P., & Redding, R. E. (2004). A review of mood disorders among juvenile offenders. Psychiatric Services, 55, 1397–1407. Saxe, L., & Cross, T. P. (1997). Interpreting the Fort Bragg children’s mental health demonstration project: The cup is half full. American Psychologist, 52, 553–556. Schmitt, J., Warner, K., & Gupta, S. (2010). The high budgetary cost of incarceration. Retrieved from http://www.cepr.net/ documents/publications/incarceration-2010-06.pdf Sickmund, M., Sladky, T. J., Kang, W., & Puzzanchera, C. (2008) Easy access to the census of juveniles in residential placement. Retrieved from http://www.ojjdp.gov/ojstatbb/ezacjrp/ Stahmer, A. C., Leslie, L. K., Hurlburt, M., Barth, R. P., Webb, M. B., Landsverk, J., & Zhang, J. (2005). Developmental and behavioral needs and service use for young children in child welfare. Pediatrics, 116, 891–900. doi:10.1542/peds.20042135 Stroul, B. A. (2003). Systems of care: A framework for children’s mental health. In A. J. Pumariega & N. C. Winters (Eds.), The handbook of child and adolescent system of care (pp. 17–34). San Francisco, CA: John Wiley & Sons. Stroul, B. A., Blau, G. M., & Sondheimer, D. L. (2008). System of care: A strategy to transform children’s mental health care. In B. A. Stroul & G. M. Blau (Eds.), The system of care handbook (pp. 3–24). Baltimore: Paul H. Brookes Publishing Co. Stroul, B., & Friedman, R. (1986). A system of care for children and youth with severe emotional disturbances. Washington, DC: Georgetown University Child Development Center, National Technical Assistance Center for Children’s Mental Health. U.S. Department of Education Institute of Educational Sciences, National Center for Educational Statistics. (2010). Digest of Education Statistics-Number of 14- through 21-year-old students served under Individuals with Disabilities Education Act, Part B, who exited school, by exit reason, age, and type of disability: United States and other jurisdictions, 2006-07 and 2007-08. Retrieved from http://nces.ed.gov/programs/digest/d10/tables/dt10_117.asp U.S. Department of Health and Human Services, Assistant Secretary for Planning and Evaluation (2007). The 2007 HHS poverty guidelines. Retrieved from http://aspe.hhs.gov/poverty/07poverty.shtml U.S. Department of Health and Human Services, Substance Abuse and Mental Health Services Administration, Center for Mental Health Services. (2003). Child, adolescent & family: Comprehensive community mental health services program for children and their families in-depth. Retrieved from http://mentalhealth.samhsa.gov/cmhs/childrenscampaign/ccmhs. asp Waldfogel, J., Craigie, T., & Brooks-Gunn, J. (2010). Fragile families and child wellbeing. Future of Children, 20, 87–112. Zigmond, N. (2006). Twenty-four months after high school: Paths taken by youth diagnosed with severe emotional and behavior disorders. Journal of Emotional and Behavioral Disorders, 14, 99–107.

Copyright of Journal of Evidence-Based Social Work is the property of Taylor & Francis Ltd and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.