510579
2014
APY22110.1177/1039856213510579Australasian PsychiatryPaton et al.
AP
Debate
Editor’s note: The following debate took place at the Royal Australian and New Zealand College of Psychiatry (RANZCP) Congress in Sydney, in May 2013.
Australasian Psychiatry 2014, Vol 22(1) 10–12 © The Royal Australian and New Zealand College of Psychiatrists 2014 Reprints and permissions: sagepub.co.uk/journalsPermissions.nav DOI: 10.1177/1039856213510579 apy.sagepub.com
Debate: Clinical risk categorisation is valuable in the prevention of suicide and severe violence – No Michael B Paton Clinical Director, Mental Health Drug and Alcohol, Northern Sydney Local Health District, Macquarie Hospital, Sydney, NSW, Australia Matthew M Large Psychiatrist and Conjoint Senior Lecturer, School of Psychiatry, University of New South Wales, NSW, Sydney, Australia
Christopher J Ryan Psychiatrist and Senior Lecturer, Discipline of Psychiatry and Centre for Values Ethics and the Law in Medicine,) University of Sydney, Sydney, NSW, and; Department of Psychiatry, Westmead Hospital, Westmead, NSW, Australia
W
e agree with our opponents on three key points. First, patients should be given optimal care, appropriate to their individual needs, in safe environments. Second, it is not possible to predict whether or not a particular patient will commit suicide or serious violence. Third, it is possible to validly and reliably categorise patients into those at a relatively higher or lower probability for committing future suicide or severe violence. The only real point of dispute is this: our opponents believe that categorising patients by their level of likelihood of coming to, or committing, serious harm is valuable. That is, they believe that such categorisations bring about reductions in severe violence and suicide. We take the opposite view: we note that there is no evidence that risk categorisation does reduce severe violence or suicide. Moreover, a basic understanding of the mathematics of risk categorisation (whether that is done clinically or with the use of risk assessment tools) leads to an understanding that risk categorisation can never assist us in reducing severe violence or suicide. In this paper, we first briefly rebut some of the claims made by our opponents. Next, we review the empirical evidence around the value of risk assessment. Then we examine the mathematics of risk assessment and explore its implications and finally, we offer a way forward that leaves behind our profession’s recent obsession with ‘risk’.
A brief rebuttal Our opponents make a series of unsupported assertions about the value of risk categorisation. They state, for example, that risk categorisation is ‘an effective mode of clinical reasoning that reduces the chance of common diagnostic errors’, and that it ‘improves the likelihood of a safe patient outcome’; although neither claim carries a supporting citation. On the one occasion where they do appeal to the literature for support, they cite a study where risk categorisation was not directly examined and was merely one intervention among many.1 The only conclusion that they can draw from this reference is that risk categorisation might have contributed to the modest decrease in the suicide rates reported. Oddly, O’Connor et al. spend a good proportion of their paper critiquing the quality of a recent meta-analysis.2,3 Whilst this is not the forum to mount a rigorous defence of this work, it is worth noting that the meta-analysis in question was conducted using internationally-recognised techniques and was recently accorded the highest possible quality rating within a broad survey of studies of suicide risk assessment practices.4
Correspondence: Matthew Large, Mental Health Services, Prince of Wales Hospital, Barker Street, Randwick, NSW 2031, Australia. Email: [email protected]
10 Downloaded from apy.sagepub.com at KAI NAN UNIV on April 17, 2015
Paton et al.
The empirical evidence Since the 1980s, when articles about risk categorisation first began to appear, thousands of papers have been written about risk categorisation instruments and risk factors. Among all of these studies, only one directly examined whether or not risk assessment can reduce the incidence of serious harm committed by patients in the community.5 This large Dutch study compared patients whom had interventions as a result of a risk assessment and patients who had treatment as usual. The authors concluded they could not show that risk assessment embedded in shared decision-making gave rise to any preventive effect on violent or criminal behaviour. They noted, in fact, that the risk-assessed group were non-significantly more likely to commit a future violent act. Similarly, no study has ever demonstrated that risk categorisation can reduce the incidence of self-harm or suicide.6 This is not to say that some people who are categorised as high risk might not benefit from more intensive treatment. It is just to say that there is no evidence that a net reduction in harmful events can be achieved by shifting resources toward those patients categorised as ‘high-risk’ and away from patients categorised as ‘low-risk’. In other words, there is no evidence that risk categorisation is valuable. Far from being valuable, it is very possible that the shift in resources and clinical focus implied by categorising a patient as being low risk may lead to an increase in the harm befalling low-risk individuals and an increase in adverse events overall. For example, it may be that laws that demand risk categorisation are associated with an increase in the incidence of homicide. Segal compared homicide rates across the US.7 After controlling for many variables, he found that the states with narrowly defined criteria for civil commitment, based on obligatory dangerousness criteria, had a higher average of homicide rates than the states that had broader criteria for involuntary psychiatric treatment. Segal argued that this increased homicide rate was a result of the non-treatment of those categorised as at low risk.7 In a similar vein, we recently argued that if compulsory treatment under the New South Wales (NSW) Mental Health Act had depended upon the presence or absence of decisionmaking capacity, rather than a belief in the need for protection from serious harm, a local double homicide might have been prevented.8
The mathematics of risk categorisation involving rare harms Fortunately, suicide and serious violence are very rare events; however, as a direct consequence of this, categorisation based on the likelihood of these events will never be able to provide clinically useful information. The rarer an event (the lower its base rate), the more a high-risk group will contain false positives. This is
the essence of Bayes’ Theorem. This basic mathematical concept explains, among other things, why categorising people by their long-term risk for serious violence and suicide will always be more accurate than categorising people by their short-term risk. It is a simple fact that over the longer term, one can confidently expect more events; therefore, a higher base rate. The difficulties that a low base rate creates for risk categorisation have been demonstrated in numerous studies. In the eighties and nineties, two large prospective studies of suicide risk categorisation were published in the Archives of General Psychiatry.9,10 In neither study did a single ‘high-risk’ patient commit suicide. In the 1983 Pokorney study, all 63 suicides were among ‘low-risk’ patients and in the 1991 Goldstein study, all 46 suicides were categorised as ‘low-risk’. More recent, larger studies have hardly fared better. In 2012, Madsen and associates11 found that only 0.23% of their ‘high-risk’ inpatients died by suicide, and Steeg et al.12 found only 0.5% of suicide attempters whom were regarded as being of high risk completed suicide. Similar studies concerning homicide are rare; however, Fazel et al. performed a case control study of people whom committed a homicide after discharge from psychiatric hospitals in Sweden and found that the optimal risk categorisation had a sensitivity of 51% and a specificity of 86%.13 Combining these findings with the rate of homicide among patients beyond the first episode of their schizophrenia,14 one can calculate that only 3 in 10,000 so-called ‘high-risk’ patients will commit a homicide within the following year. The Steeg and Madsen papers are also informative with respect to the number of suicides that occur among those classed as low risk.11,12 In the Madsen study, 88% of suicides occurred in the low-risk groups and in the Steeg study, the suicide rate in the low-risk group was 14 times the national average. Similarly, the Fazel study suggests that one-half of homicides occur in the low-risk group.13
The lack of utility of risk categorisation Even though risk categorisation is able to validly and reliably divide patients, with statistical significance, into those at high and low risk for future suicide and serious violence, the low base rate of these events inevitably means that there are large numbers of patients being categorised as high risk whom will never go on to produce serious harm. Furthermore, the limited sensitivity of risk assessments means that a large percentage of those who do go on to serious harm will have been categorised as low risk. None of this is open to debate, it is simply the consequence of the characteristics of all known forms of risk assessment in mental health and the mathematics of probability. The only thing open to dispute is the value of risk categorisation: can risk categorisation guide our
11 Downloaded from apy.sagepub.com at KAI NAN UNIV on April 17, 2015
Australasian Psychiatry 22(1)
clinical interventions? We contend that there is no intervention that is so benign that we would deliver it to all the high-risk patients, knowing that most of them will never go on to serious harm, but is simultaneously so toxic or expensive that we would not deliver it to the low-risk patients, knowing that most of the serious harm will arise in this group. Once this is understood, it is obvious that an individual patient’s risk categorisation adds nothing of value to our clinical reasoning when we come to formulate their management.
Disclosure The authors report no conflict of interest. The authors alone are responsible for the content and writing of the paper.
References 1. While D, Bickley H, Roscoe A, et al. Implementation of mental health service recommendations in England and Wales and suicide rates, 1997–2006: A cross-sectional and before-and-after observational study. Lancet 2012; 379: 1005–1012. 2. Large M, Smith G, Sharma S, et al. Systematic review and meta-analysis of the clinical factors associated with the suicide of psychiatric inpatients. Act Psychiatr Scand 2011; 124: 18–29. 3. Large M, Ryan C and Nielssen O. The validity and utility of risk assessment for inpatient suicide. Australas Psychiatry 2011; 19: 507–512.
What is valuable? Psychiatrists should abandon risk categorisation: it adds nothing useful to our understanding of the patient before us. Suicide and serious violence can be prevented, but not by risk categorisation. In contrast to risk categorisation, there is good evidence that universal precautions (such as removing hanging points in wards, reducing access to firearms and placing barriers at jumping points) reduce the rates of either homicide or suicide, or both. We also know that people benefit from careful assessment and subsequent good clinical care, formulated in collaboration with the patient and their family. Our opponents ask, rhetorically, ‘Who would not place in involuntary care a person who has been threatening people in the street with serious aggression because he believes they are monitoring, controlling and poisoning him – even though actuarially [sic] the chances of this man killing a stranger is extremely remote?’ We answer that of course he should be treated, but not because of his risk. He should be given care because he is suffering and effective treatment will alleviate that. The care should be involuntary, because he has (presumably) lost the capacity to ask for it himself. We can’t usefully categorise people by level of risk, but we can help people recover. Mental health care should not be based on what we would like to be true, but must be guided by evidence and ethics.
4. Haney EM, O’Neil ME, Carson S, et al. Suicide risk factors and risk assessment tools: A systematic review. Report, Department of Veterans Affairs, Evidence-based Synthesis Program (ESP) Center, Portland VA Medical Center Washington (DC): Department of Veterans Affairs; March 2012. Available at: http://www.ncbi.nlm.nih.gov/books/ NBK92671/\ (accessed 21 October 2013). 5. Troquete NAC, Van den Brink RHS, Beintema H, et al. Risk assessment and shared care planning in out-patient forensic psychiatry: Cluster randomised controlled trial. Brit J Psychiatry 2013; 202: 365–371. 6. Wand T. Investigating the evidence for the effectiveness of risk assessment in mental health care. Iss Mental Health Nurs 2012; 33: 2–7. 7. Segal SP. Civil commitment law, mental health services, and US homicide rates. Social Psychiatry Psychiatr Epidemiol 2012; 47: 1449–1458. 8. Ryan CJ, Callaghan S and Large MM. Better laws for coercive psychiatric treatment: Lessons from the Waterlow case. Australas Psychiatry 2012; 20: 283–286. 9. Pokorny AD. Prediction of suicide in psychiatric patients. Report of a prospective study. Arch Gen Psychiatry 1983; 40: 249–257. 10. Goldstein RB, Black DW, Nasrallah A, et al. The prediction of suicide. Sensitivity, specificity, and predictive value of a multivariate model applied to suicide among 1906 patients with affective disorders. Arch Gen Psychiatry 1991; 48: 418–422. 11. Madsen T and Nordentoft M. Drs Madsen and Nordentoft reply. J Clin Psychiatry 2012; 73: 1034–1035. 12. Steeg S, Kapur N, Webb R, et al. The development of a population-level clinical screening tool for self-harm repetition and suicide: The ReACT Self-Harm Rule. Psychol Med 2012; 42: 2383–2394. 13. Fazel S, Buxrud P, Ruchkin V, et al. Homicide in discharged patients with schizophrenia and other psychoses: A national case-control study. Schizophr Res 2010; 123: 263–269. 14. Nielssen O and Large M. Rates of homicide during the first episode of psychosis and after treatment: A systematic review and meta-analysis. Schizophr Bull 2010; 36: 702–712.
12 Downloaded from apy.sagepub.com at KAI NAN UNIV on April 17, 2015
2014
APY22110.1177/1039856213510579Australasian PsychiatryPaton et al.
AP
Debate
Editor’s note: The following debate took place at the Royal Australian and New Zealand College of Psychiatry (RANZCP) Congress in Sydney, in May 2013.
Australasian Psychiatry 2014, Vol 22(1) 10–12 © The Royal Australian and New Zealand College of Psychiatrists 2014 Reprints and permissions: sagepub.co.uk/journalsPermissions.nav DOI: 10.1177/1039856213510579 apy.sagepub.com
Debate: Clinical risk categorisation is valuable in the prevention of suicide and severe violence – No Michael B Paton Clinical Director, Mental Health Drug and Alcohol, Northern Sydney Local Health District, Macquarie Hospital, Sydney, NSW, Australia Matthew M Large Psychiatrist and Conjoint Senior Lecturer, School of Psychiatry, University of New South Wales, NSW, Sydney, Australia
Christopher J Ryan Psychiatrist and Senior Lecturer, Discipline of Psychiatry and Centre for Values Ethics and the Law in Medicine,) University of Sydney, Sydney, NSW, and; Department of Psychiatry, Westmead Hospital, Westmead, NSW, Australia
W
e agree with our opponents on three key points. First, patients should be given optimal care, appropriate to their individual needs, in safe environments. Second, it is not possible to predict whether or not a particular patient will commit suicide or serious violence. Third, it is possible to validly and reliably categorise patients into those at a relatively higher or lower probability for committing future suicide or severe violence. The only real point of dispute is this: our opponents believe that categorising patients by their level of likelihood of coming to, or committing, serious harm is valuable. That is, they believe that such categorisations bring about reductions in severe violence and suicide. We take the opposite view: we note that there is no evidence that risk categorisation does reduce severe violence or suicide. Moreover, a basic understanding of the mathematics of risk categorisation (whether that is done clinically or with the use of risk assessment tools) leads to an understanding that risk categorisation can never assist us in reducing severe violence or suicide. In this paper, we first briefly rebut some of the claims made by our opponents. Next, we review the empirical evidence around the value of risk assessment. Then we examine the mathematics of risk assessment and explore its implications and finally, we offer a way forward that leaves behind our profession’s recent obsession with ‘risk’.
A brief rebuttal Our opponents make a series of unsupported assertions about the value of risk categorisation. They state, for example, that risk categorisation is ‘an effective mode of clinical reasoning that reduces the chance of common diagnostic errors’, and that it ‘improves the likelihood of a safe patient outcome’; although neither claim carries a supporting citation. On the one occasion where they do appeal to the literature for support, they cite a study where risk categorisation was not directly examined and was merely one intervention among many.1 The only conclusion that they can draw from this reference is that risk categorisation might have contributed to the modest decrease in the suicide rates reported. Oddly, O’Connor et al. spend a good proportion of their paper critiquing the quality of a recent meta-analysis.2,3 Whilst this is not the forum to mount a rigorous defence of this work, it is worth noting that the meta-analysis in question was conducted using internationally-recognised techniques and was recently accorded the highest possible quality rating within a broad survey of studies of suicide risk assessment practices.4
Correspondence: Matthew Large, Mental Health Services, Prince of Wales Hospital, Barker Street, Randwick, NSW 2031, Australia. Email: [email protected]
10 Downloaded from apy.sagepub.com at KAI NAN UNIV on April 17, 2015
Paton et al.
The empirical evidence Since the 1980s, when articles about risk categorisation first began to appear, thousands of papers have been written about risk categorisation instruments and risk factors. Among all of these studies, only one directly examined whether or not risk assessment can reduce the incidence of serious harm committed by patients in the community.5 This large Dutch study compared patients whom had interventions as a result of a risk assessment and patients who had treatment as usual. The authors concluded they could not show that risk assessment embedded in shared decision-making gave rise to any preventive effect on violent or criminal behaviour. They noted, in fact, that the risk-assessed group were non-significantly more likely to commit a future violent act. Similarly, no study has ever demonstrated that risk categorisation can reduce the incidence of self-harm or suicide.6 This is not to say that some people who are categorised as high risk might not benefit from more intensive treatment. It is just to say that there is no evidence that a net reduction in harmful events can be achieved by shifting resources toward those patients categorised as ‘high-risk’ and away from patients categorised as ‘low-risk’. In other words, there is no evidence that risk categorisation is valuable. Far from being valuable, it is very possible that the shift in resources and clinical focus implied by categorising a patient as being low risk may lead to an increase in the harm befalling low-risk individuals and an increase in adverse events overall. For example, it may be that laws that demand risk categorisation are associated with an increase in the incidence of homicide. Segal compared homicide rates across the US.7 After controlling for many variables, he found that the states with narrowly defined criteria for civil commitment, based on obligatory dangerousness criteria, had a higher average of homicide rates than the states that had broader criteria for involuntary psychiatric treatment. Segal argued that this increased homicide rate was a result of the non-treatment of those categorised as at low risk.7 In a similar vein, we recently argued that if compulsory treatment under the New South Wales (NSW) Mental Health Act had depended upon the presence or absence of decisionmaking capacity, rather than a belief in the need for protection from serious harm, a local double homicide might have been prevented.8
The mathematics of risk categorisation involving rare harms Fortunately, suicide and serious violence are very rare events; however, as a direct consequence of this, categorisation based on the likelihood of these events will never be able to provide clinically useful information. The rarer an event (the lower its base rate), the more a high-risk group will contain false positives. This is
the essence of Bayes’ Theorem. This basic mathematical concept explains, among other things, why categorising people by their long-term risk for serious violence and suicide will always be more accurate than categorising people by their short-term risk. It is a simple fact that over the longer term, one can confidently expect more events; therefore, a higher base rate. The difficulties that a low base rate creates for risk categorisation have been demonstrated in numerous studies. In the eighties and nineties, two large prospective studies of suicide risk categorisation were published in the Archives of General Psychiatry.9,10 In neither study did a single ‘high-risk’ patient commit suicide. In the 1983 Pokorney study, all 63 suicides were among ‘low-risk’ patients and in the 1991 Goldstein study, all 46 suicides were categorised as ‘low-risk’. More recent, larger studies have hardly fared better. In 2012, Madsen and associates11 found that only 0.23% of their ‘high-risk’ inpatients died by suicide, and Steeg et al.12 found only 0.5% of suicide attempters whom were regarded as being of high risk completed suicide. Similar studies concerning homicide are rare; however, Fazel et al. performed a case control study of people whom committed a homicide after discharge from psychiatric hospitals in Sweden and found that the optimal risk categorisation had a sensitivity of 51% and a specificity of 86%.13 Combining these findings with the rate of homicide among patients beyond the first episode of their schizophrenia,14 one can calculate that only 3 in 10,000 so-called ‘high-risk’ patients will commit a homicide within the following year. The Steeg and Madsen papers are also informative with respect to the number of suicides that occur among those classed as low risk.11,12 In the Madsen study, 88% of suicides occurred in the low-risk groups and in the Steeg study, the suicide rate in the low-risk group was 14 times the national average. Similarly, the Fazel study suggests that one-half of homicides occur in the low-risk group.13
The lack of utility of risk categorisation Even though risk categorisation is able to validly and reliably divide patients, with statistical significance, into those at high and low risk for future suicide and serious violence, the low base rate of these events inevitably means that there are large numbers of patients being categorised as high risk whom will never go on to produce serious harm. Furthermore, the limited sensitivity of risk assessments means that a large percentage of those who do go on to serious harm will have been categorised as low risk. None of this is open to debate, it is simply the consequence of the characteristics of all known forms of risk assessment in mental health and the mathematics of probability. The only thing open to dispute is the value of risk categorisation: can risk categorisation guide our
11 Downloaded from apy.sagepub.com at KAI NAN UNIV on April 17, 2015
Australasian Psychiatry 22(1)
clinical interventions? We contend that there is no intervention that is so benign that we would deliver it to all the high-risk patients, knowing that most of them will never go on to serious harm, but is simultaneously so toxic or expensive that we would not deliver it to the low-risk patients, knowing that most of the serious harm will arise in this group. Once this is understood, it is obvious that an individual patient’s risk categorisation adds nothing of value to our clinical reasoning when we come to formulate their management.
Disclosure The authors report no conflict of interest. The authors alone are responsible for the content and writing of the paper.
References 1. While D, Bickley H, Roscoe A, et al. Implementation of mental health service recommendations in England and Wales and suicide rates, 1997–2006: A cross-sectional and before-and-after observational study. Lancet 2012; 379: 1005–1012. 2. Large M, Smith G, Sharma S, et al. Systematic review and meta-analysis of the clinical factors associated with the suicide of psychiatric inpatients. Act Psychiatr Scand 2011; 124: 18–29. 3. Large M, Ryan C and Nielssen O. The validity and utility of risk assessment for inpatient suicide. Australas Psychiatry 2011; 19: 507–512.
What is valuable? Psychiatrists should abandon risk categorisation: it adds nothing useful to our understanding of the patient before us. Suicide and serious violence can be prevented, but not by risk categorisation. In contrast to risk categorisation, there is good evidence that universal precautions (such as removing hanging points in wards, reducing access to firearms and placing barriers at jumping points) reduce the rates of either homicide or suicide, or both. We also know that people benefit from careful assessment and subsequent good clinical care, formulated in collaboration with the patient and their family. Our opponents ask, rhetorically, ‘Who would not place in involuntary care a person who has been threatening people in the street with serious aggression because he believes they are monitoring, controlling and poisoning him – even though actuarially [sic] the chances of this man killing a stranger is extremely remote?’ We answer that of course he should be treated, but not because of his risk. He should be given care because he is suffering and effective treatment will alleviate that. The care should be involuntary, because he has (presumably) lost the capacity to ask for it himself. We can’t usefully categorise people by level of risk, but we can help people recover. Mental health care should not be based on what we would like to be true, but must be guided by evidence and ethics.
4. Haney EM, O’Neil ME, Carson S, et al. Suicide risk factors and risk assessment tools: A systematic review. Report, Department of Veterans Affairs, Evidence-based Synthesis Program (ESP) Center, Portland VA Medical Center Washington (DC): Department of Veterans Affairs; March 2012. Available at: http://www.ncbi.nlm.nih.gov/books/ NBK92671/\ (accessed 21 October 2013). 5. Troquete NAC, Van den Brink RHS, Beintema H, et al. Risk assessment and shared care planning in out-patient forensic psychiatry: Cluster randomised controlled trial. Brit J Psychiatry 2013; 202: 365–371. 6. Wand T. Investigating the evidence for the effectiveness of risk assessment in mental health care. Iss Mental Health Nurs 2012; 33: 2–7. 7. Segal SP. Civil commitment law, mental health services, and US homicide rates. Social Psychiatry Psychiatr Epidemiol 2012; 47: 1449–1458. 8. Ryan CJ, Callaghan S and Large MM. Better laws for coercive psychiatric treatment: Lessons from the Waterlow case. Australas Psychiatry 2012; 20: 283–286. 9. Pokorny AD. Prediction of suicide in psychiatric patients. Report of a prospective study. Arch Gen Psychiatry 1983; 40: 249–257. 10. Goldstein RB, Black DW, Nasrallah A, et al. The prediction of suicide. Sensitivity, specificity, and predictive value of a multivariate model applied to suicide among 1906 patients with affective disorders. Arch Gen Psychiatry 1991; 48: 418–422. 11. Madsen T and Nordentoft M. Drs Madsen and Nordentoft reply. J Clin Psychiatry 2012; 73: 1034–1035. 12. Steeg S, Kapur N, Webb R, et al. The development of a population-level clinical screening tool for self-harm repetition and suicide: The ReACT Self-Harm Rule. Psychol Med 2012; 42: 2383–2394. 13. Fazel S, Buxrud P, Ruchkin V, et al. Homicide in discharged patients with schizophrenia and other psychoses: A national case-control study. Schizophr Res 2010; 123: 263–269. 14. Nielssen O and Large M. Rates of homicide during the first episode of psychosis and after treatment: A systematic review and meta-analysis. Schizophr Bull 2010; 36: 702–712.
12 Downloaded from apy.sagepub.com at KAI NAN UNIV on April 17, 2015
