Integrated Multisystem Analysis in a Mental Health and Criminal Justice Ecosystem Dr. Erin Falconer1, Dr. Tal El-Hay2, Dr. Dimitris Alevras3, Dr. John Docherty1, Dr. Chen Yanover2, Alan Kalton4, Dr. Yaara Goldschmidt2, Dr. Michal Rosen-Zvi2 1 Medical Affairs, Otsuka America Pharmaceutical, Inc., Princeton, NJ, USA 2 IBM Research - Haifa, Israel 3 IBM Global Business Services, West Chester, PA, USA 4 IBM Research – Africa, Nairobi, Kenya
Abstract Patients with a serious mental illness often receive care that is fragmented due to reduced availability of or access to resources, and inadequate, discontinuous, and uncoordinated care across health, social services, and criminal justice organizations. These gaps in care may lead to increased mental health disease burden and relapse, as well as repeated incarcerations. Further, the complex health, social service, and criminal justice ecosystem within which the patient may be embedded makes it difficult to examine the role of modifiable risk factors and delivered services on patient outcomes, particularly given that agencies often maintain isolated sets of relevant data. Here we describe an approach to creating a multisystem analysis that derives insights from an integrated data set including patient access to case management services, medical services, and interactions with the criminal justice system. We combined data from electronic systems within a US mental health ecosystem that included mental health and substance abuse services, as well as data from the criminal justice system. We applied Cox models to test the associations between delivery of services and re-incarceration. Using this approach, we found an association between arrests and crisis stabilization services in this population. We also found that delivery of case management or medical services provided after release from jail was associated with a reduced risk for re-arrest. Additionally, we used machine learning to train and validate a predictive model linking non-modifiable and modifiable risk factors and outcomes. A predictive model, constructed using elastic net regularized logistic regression, and considering age, past arrests, mental health diagnosis, as well as use of a jail diversion program, outpatient, medical and case management services predicted the probability of re-arrests with fair accuracy (AUC=.67). By modeling the complex interactions between risk factors, service delivery and outcomes, we may better enable systems of care to meet patient needs and improve outcomes. Introduction The mental healthcare system in the United States is fundamentally broken – it is fragmented, inconsistent, underfunded, and rapidly deteriorating. Dominated by a lack of both consistency and continuity of care, the system allows many clients to mentally decompensate, ultimately leading to three negative crisis outcomes – homelessness, emergency hospitalization and incarceration. This fragmentation in the treatment of the mentally ill is particularly evident when one considers that it has been estimated that between 44 and 61 percent of jail inmates in the United States have a mental health problem (James et al Mental health problems in prison). Many individuals with severe mental illness are released from prison every year in the United States and re-enter the community with a need to continue treatment for their mental health issues. Continuous mental health treatment of these individuals may help prevent relapse and recidivism. Lack of continuous care for adults with serious mental illness may not only result in more decompensation and crisis as the individual navigates the mental health, social, and criminal justice systems, but it also limits our understanding of the impact and interaction between modifiable risk factors and access to multiple services on patient outcomes such as re-arrest. Given that different agencies often maintain isolated datasets, the fragmentation of data systems and lack of access to continuous patient-level data means that it is difficult to collate data across health, social, and criminal justice agencies to evaluate the interplay between multiple services and outcomes. There is a critical need to evaluate patient-level and service-level data across multiple agencies in order to understand the mechanisms through which we may intervene to prevent or delay psychiatric crises.
526
Previous work evaluated data from US Medicaid claims files and arrest records and found a reduced risk of re-arrest with receipt of outpatient services1,2,3 and psychotropic medication possession3 in adults with mental illness. Other research using county and statewide criminal justice records and archival data from health and social services found that individual risk factors including being homeless, not having outpatient mental health treatment, and having involuntary psychiatric evaluation in the previous quarter, and being black, younger than 21 years and having a cooccurring substance abuse problem increased the odds of arrest 4. Recent studies for other medical applications have used electronic medical records data to establish predictive models for illness severity in various disease domains, including preterm infants5, congestive heart failure6, septic shock7 and HIV8. In this study, we describe an approach to modeling the interplay among services and outcomes across an ecosystem of medical and social services providers and the criminal justice system in which there is a constant flow of individuals with serious mental illness in and out of the criminal justice system. We explored associations between the occurrence of arrest and crisis services outcomes tested using hazard modeling with both fixed- and timedependent covariates. We demonstrate the utility of such a combined dataset for predictive modeling by training and testing a model for arrest prediction. Preliminaries In this study, the sets of covariates used for prediction include both basic risk factors as well as indicators of access to specific services. Cox models were applied to test the associations between access to services and outcomes. Predictive models were constructed using elastic net regularized logistic regression. Association analysis using Cox models Cox proportional hazard models were used for testing associations between risk factors and the expected time for failure events to occur9.This association is modeled using a hazard rate that represents the amount of risk as a function of time. The effect of each risk factor is assumed to be multiplicative with respect to the hazard rate. In addition to predicting the effect of services given immediately after release, we examine the effect of continuous access to services. These tests involve time dependent covariates such as access to services in every month after
Figure 1: Data preprocessing and feature extraction framework release from jail. Association tests with such covariates were performed using extended Cox models (see 11 for more details). Predictive modeling using elastic nets An elastic net is a method that allows classical regression models to deal with high dimensionality of observations.
527
This method performs data-driven variable selection and results in a sparse model that includes the most informative covariates. Learning in such models involves tuning of two parameters: (1) alpha – which controls the sparseness and stability of the model, where a higher alpha increases the tendency of the learning algorithm to filter out noninformative covariates; (2) beta – a regularization parameter that prevents over-fitting of the model to the data which is employed to obtain optimal generalization performance. These parameters are usually tuned using internal crossvalidation on a training data set. The accuracy of the model is assessed on a test set. For more details see12. Methods Framework – integrated view of patients ecosystem This study uses information extracted from electronic systems resident within a mental health ecosystem in the southern US. This included data used to support the claims process for medical and social services delivered to mental health and substance abuse patients, and data collected by the criminal justice system. All individual patient data used for the analysis were collected after obtaining appropriate consents and agreements, and personal information was removed to protect patient privacy. Patient data collected for this analysis span 21 months and describe the engagements that patients have with service providers in the ecosystem. An essential part of analyzing such longitudinal data is defining an index date, where data collected before that date serve as input to decision making and data from after that point in time define the outcome values and characterizes the treatment. We then define the target populations on which to focus and a quantitative outcome measure. Finally, we extract and filter features and risk factors to drive the modeling (Figure 1). The main outcome addressed in this study was a re-arrest, as experienced by individuals that had become involved in the criminal justice system. The study explored two main questions: (i) What are the modifiable and nonmodifiable risk factors associated with this outcome? (ii) How well can we predict the likelihood of re-arrest using such risk factors? Our approach employed an association analysis to explore the answers to the first question and a machine learning analysis to explore the answers to the second question. We further test the association of various risk factors to admission to acute service, namely crisis stabilization unit. We hypothesized that the relationship between mental health services and the criminal justice system may be bidirectional; to explore this hypothesis, we test associations between various risk factors including previous arrest records and the risk of admission to acute mental health treatment facilities (namely crisis stabilization unit). Data sources Datasets for mental health and substance abuse admissions and events were included which span the time window from October 1, 2010 to June 30, 2012. All mental health admissions before October 1, 2010 have been recorded with an admission date of October 1, 2010. Substance abuse admissions data span the period between July 1, 2009 and June 30, 2012. In this study we focus on Seriously Persistent Mentally Ill (SPMI) individuals that have one of the following diagnoses: Bipolar disorder (ICD9 codes: 296 to 296.19 and 296.40 to 296.99) Schizophrenic disorder (ICD9 codes: 295 to 295.99 and 297 to 298.99) Major depression (ICD9 codes: 296.20 to 296.39) The dataset includes data on an SPMI population of 29,558 individuals. Arrest data were supplied by the Department of Law Enforcement and was extracted from the Criminal Justice Information Services (CJIS). These data span a period from January 1, 2007 to September 6, 2012, and include records on 184,470 individuals. Out of these, 5,148 overlap with the SPMI population in the health ecosystem studied. The court runs a program that helps identify and divert detainees with a mental illness into a Jail Diversion Program (JDP), which seeks to reintroduce individuals into a sustained care environment, combining mental health and housing services as part of a structured year-long engagement. The court provided a list of participants for approximately ten years, overlapping the data contained in the other data sources.
528
Population Selection Criteria To analyze the relations between arrest and behavioral health service events we focused on a subset of the adult population having records both in the CJIS and mental health ecosystem datasets. We excluded 281 individuals from this cohort because of inaccurate and inconsistent timeline data. Out of the remaining individuals, a total of 3,274 were released from an arrest after October 1 st, 2010, which is the starting date of the mental health services recorded in the dataset. Of these, 3,171 were adults at the time of release a. In addition to viewing arrest as an outcome, we analyzed the association of past arrests with the outcome of admission to an acute mental health treatment using the SPMI cohort. We excluded individuals whose first recorded admission had ambiguous or unknown dates, creating a subset of 15,930 subjects. Of these, we further focused on the adult population (N=14,228). Statistical Analysis Association between non-modifiable risk factors, receiving services after release from jail, and the risk of re-arrest The initial association analysis examined non-modifiable risk factors including gender, age, race, mental health diagnosis and past arrests using a Cox proportional hazard model. The association of receiving different service types with the risk of re-arrest was evaluated, adjusting for these non-modifiable risk factors. The dataset contains thirty nine types of services, out of which fourteen service types were given to more than 20 patients in the cohort. Each of these service types was represented using an indicator covariate equal to one if an individual received the service at least once in the first quarter after release from jail and equal to zero otherwise. Continuous access to services Extended Cox models were used to examine the association of services given throughout the entire period after release from jail to the risk of re-arrest. More specifically, for each patient, all release dates after Oct 1, 2010 were listed and corresponding re-arrest dates were identified (or, if the patient was not re-arrested, the end-of-study date was determined). Starting from each such release date, the number of times each service was given to the patient in each consecutive 90 day time period was tabulated. Subsets of these time-varying covariates, in addition to the non-modifiable factors, were then used to infer the parameters of extended Cox models. In particular, models were constructed with the following features: 1. 2.
An indicator covariate identifying whether or not a specific service was given within the last ninety days (including non-modifiable risk factors) to predict re-arrest within the coming ninety days. An indicator covariate identifying whether or not a specific service was given since the last release from jail (including non-modifiable risk factors) to predict re-arrest within the coming ninety days.
Predictive modeling using elastic nets To test the predictability of the arrest outcome, data were partitioned into a training set which contained approximately 80% of the cohort and a test set which contained the remaining 20%. An elastic net regularized regression model was used where alpha was tuned to balance sparseness and stability on the training set. Because the goals of the analysis were set to predict re-arrest probability in the second quarter after release, the target population was similar to the one described in re-arrest risk factor analysis. However, individuals were excluded for which data were not available for two quarters. Applying this additional criterion, the cohort size was established as 1,679 individuals in the training set and 421 in the test set. We evaluated the predictive power of the model using a receiver operating characteristic (ROC) curve which compares the likelihood of correctly and incorrectly predicting re-arrest.
a
Due to the removal of exact birth dates, we use estimated ages at different time points. Here we include individuals with estimated age at release > 18.
529
Results Preliminary associations of demographic and historical factors with re-arrest The association between non-modifiable risk factors was estimated using Cox proportional hazard models. Past arrests factors were modeled as indicator variables whose value was one if the individual was arrested and released from jail between January 2007 (start date of the CJIS data) and October 2010, and zero otherwise. Preliminary associations (i.e., without adjusting for other variables) between these factors and the risks of re-arrest are summarized in Table 1. Associations with p-value < 1e-4, will remain significant at a 0.05 level after a Bonferroni correction for 500 hypotheses. In particular, schizophrenia, history of arrests, male gender, black race, and younger age are shown to be risk factors for increased likelihood of re-arrest, in agreement with previous studies4. Table 1. Preliminary associations between baseline characteristics and the risk of re-arrest Factor Gender (Female vs. Male) Race (Black vs. Other) Diagnosis (vs. Major Depression) Bipolar Disorder Schizophrenic Disorders Past arrests Age
P-value
Abstract Patients with a serious mental illness often receive care that is fragmented due to reduced availability of or access to resources, and inadequate, discontinuous, and uncoordinated care across health, social services, and criminal justice organizations. These gaps in care may lead to increased mental health disease burden and relapse, as well as repeated incarcerations. Further, the complex health, social service, and criminal justice ecosystem within which the patient may be embedded makes it difficult to examine the role of modifiable risk factors and delivered services on patient outcomes, particularly given that agencies often maintain isolated sets of relevant data. Here we describe an approach to creating a multisystem analysis that derives insights from an integrated data set including patient access to case management services, medical services, and interactions with the criminal justice system. We combined data from electronic systems within a US mental health ecosystem that included mental health and substance abuse services, as well as data from the criminal justice system. We applied Cox models to test the associations between delivery of services and re-incarceration. Using this approach, we found an association between arrests and crisis stabilization services in this population. We also found that delivery of case management or medical services provided after release from jail was associated with a reduced risk for re-arrest. Additionally, we used machine learning to train and validate a predictive model linking non-modifiable and modifiable risk factors and outcomes. A predictive model, constructed using elastic net regularized logistic regression, and considering age, past arrests, mental health diagnosis, as well as use of a jail diversion program, outpatient, medical and case management services predicted the probability of re-arrests with fair accuracy (AUC=.67). By modeling the complex interactions between risk factors, service delivery and outcomes, we may better enable systems of care to meet patient needs and improve outcomes. Introduction The mental healthcare system in the United States is fundamentally broken – it is fragmented, inconsistent, underfunded, and rapidly deteriorating. Dominated by a lack of both consistency and continuity of care, the system allows many clients to mentally decompensate, ultimately leading to three negative crisis outcomes – homelessness, emergency hospitalization and incarceration. This fragmentation in the treatment of the mentally ill is particularly evident when one considers that it has been estimated that between 44 and 61 percent of jail inmates in the United States have a mental health problem (James et al Mental health problems in prison). Many individuals with severe mental illness are released from prison every year in the United States and re-enter the community with a need to continue treatment for their mental health issues. Continuous mental health treatment of these individuals may help prevent relapse and recidivism. Lack of continuous care for adults with serious mental illness may not only result in more decompensation and crisis as the individual navigates the mental health, social, and criminal justice systems, but it also limits our understanding of the impact and interaction between modifiable risk factors and access to multiple services on patient outcomes such as re-arrest. Given that different agencies often maintain isolated datasets, the fragmentation of data systems and lack of access to continuous patient-level data means that it is difficult to collate data across health, social, and criminal justice agencies to evaluate the interplay between multiple services and outcomes. There is a critical need to evaluate patient-level and service-level data across multiple agencies in order to understand the mechanisms through which we may intervene to prevent or delay psychiatric crises.
526
Previous work evaluated data from US Medicaid claims files and arrest records and found a reduced risk of re-arrest with receipt of outpatient services1,2,3 and psychotropic medication possession3 in adults with mental illness. Other research using county and statewide criminal justice records and archival data from health and social services found that individual risk factors including being homeless, not having outpatient mental health treatment, and having involuntary psychiatric evaluation in the previous quarter, and being black, younger than 21 years and having a cooccurring substance abuse problem increased the odds of arrest 4. Recent studies for other medical applications have used electronic medical records data to establish predictive models for illness severity in various disease domains, including preterm infants5, congestive heart failure6, septic shock7 and HIV8. In this study, we describe an approach to modeling the interplay among services and outcomes across an ecosystem of medical and social services providers and the criminal justice system in which there is a constant flow of individuals with serious mental illness in and out of the criminal justice system. We explored associations between the occurrence of arrest and crisis services outcomes tested using hazard modeling with both fixed- and timedependent covariates. We demonstrate the utility of such a combined dataset for predictive modeling by training and testing a model for arrest prediction. Preliminaries In this study, the sets of covariates used for prediction include both basic risk factors as well as indicators of access to specific services. Cox models were applied to test the associations between access to services and outcomes. Predictive models were constructed using elastic net regularized logistic regression. Association analysis using Cox models Cox proportional hazard models were used for testing associations between risk factors and the expected time for failure events to occur9.This association is modeled using a hazard rate that represents the amount of risk as a function of time. The effect of each risk factor is assumed to be multiplicative with respect to the hazard rate. In addition to predicting the effect of services given immediately after release, we examine the effect of continuous access to services. These tests involve time dependent covariates such as access to services in every month after
Figure 1: Data preprocessing and feature extraction framework release from jail. Association tests with such covariates were performed using extended Cox models (see 11 for more details). Predictive modeling using elastic nets An elastic net is a method that allows classical regression models to deal with high dimensionality of observations.
527
This method performs data-driven variable selection and results in a sparse model that includes the most informative covariates. Learning in such models involves tuning of two parameters: (1) alpha – which controls the sparseness and stability of the model, where a higher alpha increases the tendency of the learning algorithm to filter out noninformative covariates; (2) beta – a regularization parameter that prevents over-fitting of the model to the data which is employed to obtain optimal generalization performance. These parameters are usually tuned using internal crossvalidation on a training data set. The accuracy of the model is assessed on a test set. For more details see12. Methods Framework – integrated view of patients ecosystem This study uses information extracted from electronic systems resident within a mental health ecosystem in the southern US. This included data used to support the claims process for medical and social services delivered to mental health and substance abuse patients, and data collected by the criminal justice system. All individual patient data used for the analysis were collected after obtaining appropriate consents and agreements, and personal information was removed to protect patient privacy. Patient data collected for this analysis span 21 months and describe the engagements that patients have with service providers in the ecosystem. An essential part of analyzing such longitudinal data is defining an index date, where data collected before that date serve as input to decision making and data from after that point in time define the outcome values and characterizes the treatment. We then define the target populations on which to focus and a quantitative outcome measure. Finally, we extract and filter features and risk factors to drive the modeling (Figure 1). The main outcome addressed in this study was a re-arrest, as experienced by individuals that had become involved in the criminal justice system. The study explored two main questions: (i) What are the modifiable and nonmodifiable risk factors associated with this outcome? (ii) How well can we predict the likelihood of re-arrest using such risk factors? Our approach employed an association analysis to explore the answers to the first question and a machine learning analysis to explore the answers to the second question. We further test the association of various risk factors to admission to acute service, namely crisis stabilization unit. We hypothesized that the relationship between mental health services and the criminal justice system may be bidirectional; to explore this hypothesis, we test associations between various risk factors including previous arrest records and the risk of admission to acute mental health treatment facilities (namely crisis stabilization unit). Data sources Datasets for mental health and substance abuse admissions and events were included which span the time window from October 1, 2010 to June 30, 2012. All mental health admissions before October 1, 2010 have been recorded with an admission date of October 1, 2010. Substance abuse admissions data span the period between July 1, 2009 and June 30, 2012. In this study we focus on Seriously Persistent Mentally Ill (SPMI) individuals that have one of the following diagnoses: Bipolar disorder (ICD9 codes: 296 to 296.19 and 296.40 to 296.99) Schizophrenic disorder (ICD9 codes: 295 to 295.99 and 297 to 298.99) Major depression (ICD9 codes: 296.20 to 296.39) The dataset includes data on an SPMI population of 29,558 individuals. Arrest data were supplied by the Department of Law Enforcement and was extracted from the Criminal Justice Information Services (CJIS). These data span a period from January 1, 2007 to September 6, 2012, and include records on 184,470 individuals. Out of these, 5,148 overlap with the SPMI population in the health ecosystem studied. The court runs a program that helps identify and divert detainees with a mental illness into a Jail Diversion Program (JDP), which seeks to reintroduce individuals into a sustained care environment, combining mental health and housing services as part of a structured year-long engagement. The court provided a list of participants for approximately ten years, overlapping the data contained in the other data sources.
528
Population Selection Criteria To analyze the relations between arrest and behavioral health service events we focused on a subset of the adult population having records both in the CJIS and mental health ecosystem datasets. We excluded 281 individuals from this cohort because of inaccurate and inconsistent timeline data. Out of the remaining individuals, a total of 3,274 were released from an arrest after October 1 st, 2010, which is the starting date of the mental health services recorded in the dataset. Of these, 3,171 were adults at the time of release a. In addition to viewing arrest as an outcome, we analyzed the association of past arrests with the outcome of admission to an acute mental health treatment using the SPMI cohort. We excluded individuals whose first recorded admission had ambiguous or unknown dates, creating a subset of 15,930 subjects. Of these, we further focused on the adult population (N=14,228). Statistical Analysis Association between non-modifiable risk factors, receiving services after release from jail, and the risk of re-arrest The initial association analysis examined non-modifiable risk factors including gender, age, race, mental health diagnosis and past arrests using a Cox proportional hazard model. The association of receiving different service types with the risk of re-arrest was evaluated, adjusting for these non-modifiable risk factors. The dataset contains thirty nine types of services, out of which fourteen service types were given to more than 20 patients in the cohort. Each of these service types was represented using an indicator covariate equal to one if an individual received the service at least once in the first quarter after release from jail and equal to zero otherwise. Continuous access to services Extended Cox models were used to examine the association of services given throughout the entire period after release from jail to the risk of re-arrest. More specifically, for each patient, all release dates after Oct 1, 2010 were listed and corresponding re-arrest dates were identified (or, if the patient was not re-arrested, the end-of-study date was determined). Starting from each such release date, the number of times each service was given to the patient in each consecutive 90 day time period was tabulated. Subsets of these time-varying covariates, in addition to the non-modifiable factors, were then used to infer the parameters of extended Cox models. In particular, models were constructed with the following features: 1. 2.
An indicator covariate identifying whether or not a specific service was given within the last ninety days (including non-modifiable risk factors) to predict re-arrest within the coming ninety days. An indicator covariate identifying whether or not a specific service was given since the last release from jail (including non-modifiable risk factors) to predict re-arrest within the coming ninety days.
Predictive modeling using elastic nets To test the predictability of the arrest outcome, data were partitioned into a training set which contained approximately 80% of the cohort and a test set which contained the remaining 20%. An elastic net regularized regression model was used where alpha was tuned to balance sparseness and stability on the training set. Because the goals of the analysis were set to predict re-arrest probability in the second quarter after release, the target population was similar to the one described in re-arrest risk factor analysis. However, individuals were excluded for which data were not available for two quarters. Applying this additional criterion, the cohort size was established as 1,679 individuals in the training set and 421 in the test set. We evaluated the predictive power of the model using a receiver operating characteristic (ROC) curve which compares the likelihood of correctly and incorrectly predicting re-arrest.
a
Due to the removal of exact birth dates, we use estimated ages at different time points. Here we include individuals with estimated age at release > 18.
529
Results Preliminary associations of demographic and historical factors with re-arrest The association between non-modifiable risk factors was estimated using Cox proportional hazard models. Past arrests factors were modeled as indicator variables whose value was one if the individual was arrested and released from jail between January 2007 (start date of the CJIS data) and October 2010, and zero otherwise. Preliminary associations (i.e., without adjusting for other variables) between these factors and the risks of re-arrest are summarized in Table 1. Associations with p-value < 1e-4, will remain significant at a 0.05 level after a Bonferroni correction for 500 hypotheses. In particular, schizophrenia, history of arrests, male gender, black race, and younger age are shown to be risk factors for increased likelihood of re-arrest, in agreement with previous studies4. Table 1. Preliminary associations between baseline characteristics and the risk of re-arrest Factor Gender (Female vs. Male) Race (Black vs. Other) Diagnosis (vs. Major Depression) Bipolar Disorder Schizophrenic Disorders Past arrests Age
P-value
