Evaluating Psychiatric General Hospital Multivariate

Prediction

Consultations

of Concordance

Frits J. Huyse, M.D., Ph.D., John S. Lyons, Ph.D., James J. Strain, M.D.

Abstract: A study of 316 consultation

cases was undertaken to examine the concordance between the psychiatric consultants’ recommendations and the consultees’ follow-through. Using a model of case concordance (defined as the proportion of all recommendations followed on each case), it was demonstrated that process variables, including who (the referring doctor p 5 0.07) recommended what (psychosocial ward management p < 0.02, psychosocial diagnostic action p < 0.02, biological diagnostic action p < 0.01, aftercare p 5 0.02, and number of recommendations p 5 0.03), and when (timing of consultation p 5 0.02), were the significant predictors of case concordance rather than clinical or demographic variables.

Introduction One direction taken to evaluate consultationliaison (C-L) psychiatrists’ work in the general hospital setting has been to study whether or not C-L recommendations are followed by nonpsychiatrist physicians and medical staff [1,2]. Unlike the majority of psychiatric interventions in which the psychiatrist is the primary provider of mental health services, the implementation of C-L interventions generally must involve others in the general hospital [3,4]. Psychiatric diagnostic workup and psychotherapy can be performed by medication consultants themselves; however,

From the Department of Psychiatry, Free Universitv, Amsterdam, The N’etherlands; the*North&estern University Medical School, the Mount Sinai Medical School Address reprint requests to: Dr. Huyse, Chief, Psychiatric C/L Service, The Free University Hospital, Amsterdam, The Netherlands. General Hospital Psychiatry 14, 363-369, 1992 0 1992 Elsevier Science Publishing Co., Inc. 655 Avenue of the Americas, New York, NY 10010

in the

and

changes or alterations of patient management and discharge planning on the medical/surgical unit require an effort by nonpsychiatrist physicians or nursing staff. Popkin et al. introduced the concept of concordance for the phenomenon of whether consultees comply with the recommendations of the C-L psychiatrist [l]. A subsequent series of studies demonstrated that ward staffs do not always do what consultants suggest [2,5-121. Consistently, it has been shown that concordance appears to vary by type of recommendation [5,7,12], but there is less evidence that it varies by patient characteristics [5,9,10]. In most studies, concordance has been defined and studied at the level of single recommendations, or single types of recommendations: medication and diagnostic action [1,6]. This is important ag it has demonstrated which recommendations are vulnerable to problems of nonconcordance. There are two primary limitations to this methodology, however. First, most consultations involve more than a single recommendation or single type of recommendation. As consultation cases may consist of one, two, or many recommendations [12], whether or not a consultee follows any of these recommendations should be considered within the context of the case. When concordance is analyzed using a single recommendation as the unit of analysis, the multiple recommendations of a case are masked. Even if specific calculations are used in case of multiple recommendations of a single type (such as three kinds of medications [1]), this reasoning still holds, as it ignores the outcome of other

363 ISSN 0163-8343192/$5.00

F. J. Huyse et al.

types of recommendations in the same case. An argument for the study of single types of recommendations has been to avoid the problem of how the concordance of different types of recommendations should be weighted. One could argue, however, whether recommending vitamins vs antidepressants, or sedimentation rates vs electroencephalogram are as homogeneous as suggested. By analyzing concordance at the level of the recommendation, one is unable to appropriately study characteristics of the case and the consultation process in a multivariate model. Variables that are significantly related to concordance at a univariate level may not be related when other factors are considered. Using multiple regression models assumes independent observations. The study of multiple recommendations from the same consultation case as separate observations violates this critical assumption [ 131. Therefore, for both clinical and statistical reasons, it is important to assess concordance at the level of the consultation case. To achieve this purpose, the present study examines, on a per-case basis, the concordance between psychiatric consultants’ recommendations in a series of consultation cases to determine what clinical and process factors predict medical staff follow-through with psychiatric recommendations for the hospitalized medically ill.

Methods Setting This study was accomplished through the Consultation-Liaison (C-L) Psychiatry Service at the Free University Hospital, located in Amsterdam, Holland, which has 614 beds and about 16,500 admissions per year. The average length of stay was 11.5 days. The C-L service employed a faculty of one full-time C-L psychiatrist (FJH), two PGY IV psychiatric consultation residents, and a full-time clinical liaison nurse specialist. The consultation procedures are comparable to those described in the U.S. literature [14,15]. During the course of the study, the consultation rate was 3.2% (2.6% if suicide attempts were excluded). The bed per capita size and the composition of the C-L service was comparable to other C-L services in Dutch university hospitals at that time [16,17]. As a result of a collaborative liaison psychiatry arrangement for the extensive psychiatric needs of the oral cavity tumor patients, there

364

was an overrepresentation of consultations the otolaryngology (ENT) department.

from

Sample During a period from January 1985 through May 1986, all consecutive patients referred to the Psychiatric Consultation Service from the inpatient units of the Free University Hospital of Amsterdam meeting a series of criteria were included in the study. Patients admitted for suicide attempts, patients staying less than 3 days after the first hospital consultation, and off-hours consultations not followed by the C-L service were excluded. More detailed information is available elsewhere [3,12]. Three hundred sixteen consultation cases were studied.

Criteria for Concordance In addition to Popkin’s strategy for the assessment of the concordance of physical diagnostic action and medication [2], specific operationalized criteria and a related method for the assessment of the concordance of additional biopsychosocial recommendations have been defined (Table 1). In order to remain consistent with previous work, it was decided to use qualitative criteria for the assessment of concordance: concordant, partially concordant, and nonconcordant [1,2]. However, as only 2.8% of recommendations were scored as partially concordant, this category was combined with nonconcordant.

Assessment

of Concordance

The assessment of the concordance was carried out by a research assistant (RA), working according to an observational hierarchy designed to minimize interference with the ward staff (Table 1). The RA visited the consultation patients’ ward after each consultation to determine concordance by examining doctor and nurse chart entries, and on indication, interviewing the staff.

Reliability of Assessment

of Concordance

During 1 week, the primary investigator (FJH) and the RA independently assessed the concordance in 13 consecutive cases. The consult notes contained 50 recommendations. Of these selected recommendations, 27 (FJH) vs 29 (RA) were considered to

Psychiatric

Table

1. Examples

of the assessment

Consultations

and Concordance

of concordance Observation levels”

Recommendation

Criterion

Concordance”

Term (days)

Put bars around bed

1 2 3

Observed Documented Interviewed Not confirmed

C C PC NC

1

Instructions on communication in case of disorientation and memory deficits; inform the patient in every contact where he is, what time it is, why he is here and what your intentions are

2

Documented: “orient the patient”

C

2

2

Not documented

NC

Approach

2 3 3 3 3

Documented Confirmed Tried, not reached No family Not documented nor confirmed

C C PC n.a. NC

4

2 3 3

Documented or confirmed Not documented

C

2

the family for a schedule

for visits

Information concerning the illness should be provided to the patient and/or to his family

“Observational nurse).

hierarchy as method of assessment:

bC = concordant,

PC = partially concordant,

1) direct observation;

NC = not concordant,

meet criteria for the assessment of concordance (Kappa 0.920). The Kappa for the interrater reliability for the assessment of the concordance itself was 0.84 (number of patients observed, N = 13; number of observed recommendations, N = 27; agreement on concordance, N = 24).

Analysis study, case concordance was defined as the proportion of recommendations made that were implemented by either the medical staff or the consultee. A histogram of the distribution of this proportion was somewhat negatively skewed because of a slightly higher than expected number of 1.0 concordance cases under the assumptions of a normal distribution. To allow for the parametric study of this proportion, a small positive constant (0.1) was added to the numerator. The equation for calculating case concordance was:

For the present

Case Concordance = # of recommendations implemented total # of recommendations

+ 0.1

This transformation normalized the distribution of case concordance but also tended to lower case

nor confirmed

NC

2) chart analysis (medical, nurse); 3) interview (consultee,

n.a.: does not apply.

concordance, with fewer recommendations at a slightly greater rate than cases with many recommendations. This effect is slight (“perfect” case concordance with two recommendations would be 0.95, for 10 recommendations it would be 0.99; half concordance with two cases becomes 0.476; for 10 recommendations it becomes 0.495), and does not have substantive impact on the subsequent analysis. Next, a hierarchical multiple regression was used to predict case concordance. In a hierarchical model, variables are entered based on a theoretical ordering of variables of interest. For this analysis variables were entered in the following order: demographic (income, age, sex); functional status diagnoses (DSM-III, (Karnofsky); psychiatric grouped in major domains, e.g., somatoform disorders); medical diagnoses (ICD-9, grouped into major domains, e.g., neoplasmata); major domains of recommendations (e.g., medication) [3,12]; consultant; timing of initiation of consultation (181; and the total number of recommendations rendered. The order of entry of variables was based on ease of measurement and parsimony: demographic variables are the easiest and most parsimonious, with diagnostic variables being next. Consultation process variables are the least parsimonious and 365

F. 1. Huyse et al.

most difficult to measure. Diagnoses (psychiatric and medical) and consultants were dummy-coded to allow inclusion in the model [13].

Results One-hundred-sixty-five (52.2%) of the cases were male, and 151 (47.8%) were female. The average age was 60.2 years (SD = 18.2). One hundred twenty-one (38.3%) had an infectious/parasitic disease; 72 (22.8%) were diagnosed with neoplasms; 61 (19.3%) had circulatory disorders; 23 (7.3%) had an ICD-9 diagnosis indicating an endocrine or metabolic disorder; 24 (7.6%) had a respiratory condition; 27 (8.5%) had a digestive disorder; 22 (7.0%) had diseases of the nervous system; and 13 (4.1%) had a genitourinary disease. Three (0.9%) cases involved pregnancy. The remaining ICD-9 conditions included 8 (2.5%) skin diseases, 15 (4.7%) connective tissue conditions, 1 (0.3%) congenital abnormality, 52 (16.5%) injuries/poisonings, 4 (1.3%) blood and blood organ disorders, and 21 (6.6%) ill-defined conditions. In terms of psychiatric diagnoses, 136 (43.0%) were diagnosed by the consultant as having organic brain disease; 48 (15.2%) were diagnosed as having adjustment disorders; 26 (8.2%) had substance abuse disorders; 30 (9.5%) were diagnosed with affective disorders; 15 (4.7%) were seen as somatization disorders; 37 (11.7%) were diagnosed as other psychiatri? conditions. For the 316 patients, there were 2,026 treatment recommendations by the psychiatric consultant. Eight percent of the recommendations had been effected by the consultant him-or herself, and 10% were not specified sufficiently to permit assessment as to whom would/should perform them and were consequently excluded from further study. With regard to the separate domains, each recommendation per domain could be multiple or single. Most cases had at least one medication recommendation (N = 242, 76.6%). A substantial number also were given psychosocial management recommendations (N = 234, 74.1%). A slight majority were given recommendations of biological diagnostic action [lab tests, medical consultations, and so forth (N = 165, 52.2%)]. One hundred ten (34.8%) were given recommendations of psychosocial diagnostic action (obtain additional information from the primary care physician or family). One hundred twenty (38.0%) were given recommendations regarding the timing of discharge.

366

Seventy-three (23.1%) were given aftercare recommendations. The full hierarchical regression model was significant (F(37,278) = 2.98, p < O.OOOl), indicating the successful prediction of case concordance with the above six classes of variables. The multiple R was 0.53, meaning that the model predicted 28% of the variation in case concordance; this is within acceptable limits for a multiple regression in an applied setting [13]. As the full model was significant, analysis of the contribution of individual variables could be undertaken to determine which variables significantly predicted concordance. The regression coefficients (beta) for each variable along with the T statistic testing, whether or not the variable was a significant predictor (beta significantly different from zero), are described (Table 2).

Discussion From the results, it seems both feasible and important to consider concordance at the case level. This methodology allows the identification of what factors predict whether C-L recommendations on a given case will be followed. It clearly appears that process variables (which psychiatrist recommends what and at what point in the hospital stay) are the more important ones. Demographic and clinical variables are relatively unimportant in explaining variation in case concordance. Consistent with earlier concordance studies which only analyzed single recommendations, it seems that the nature of the recommendation is an important consideration [1,2,6,8]. Aftercare recommendations occur in cases with the highest concordance, whereas recommendations requiring psychosocial management or any diagnostic action (whether biological or psychosocial) are aspects of less concordant cases. Earlier studies by Popkin et al., [6-111 and others [5,12] have already demonstrated that these types of recommendations were at risk for being effected by the consultant, although it was not known that these low concordance rates are independent of psychiatric or medical diagnoses and demographic characteristics. As these findings are consistent with earlier ones in the literature, this seems to be another argument in favor of the case concordance paradigm. Surprisingly, recommendations for medication were not associated with higher case concordance, even though these were the most concordant single recommendations [ 121. It may be that concordance

Psychiatric Consultations and Concordance

Table 2. Regression coefficients and their significance for predictors of case concordance Variable

Beta

T

P

0.063 - 0.033 0.061 0.077

1.11 -0.50 1.08 1.27

0.27 0.62 0.28 0.21

Adjustment disorder Affective disorder Substance use disorder Other mental disorder Organic mental disorder

0.022 0.093 0.058 0.103 0.130

0.25 1.16 0.77 1.43 1.15

0.80 0.25 0.44 0.15 0.25

Blood(forming) system Musculoskeletal system Neoplasms Central nervous system PregnancylBirthlPuerperium Trauma Congenital anomalies Genital/Urinary Circulatory EndocrinelMetabiImmune Digestive system Ill-defined conditions Infectious/parasitic Skin/subcutaneous

- 0.092 - 0.081 - 0.087 -0.025 0.019 - 0.060 0.044 0.041 0.028 0.001 - 0.056 - 0.078 - 0.066 -0.032

-1.67 -1.53 -1.16 -0.47 0.33 -1.05 0.74 0.73 0.48 0.02 -0.94 -1.27 -1.10 -0.54

0.96 0.13 0.25 0.64 0.74 0.30 0.46 0.47 0.63 0.99 0.35 0.21 0.27 0.59

Income Age Sex Karnofsky

Respiratory Deliberate self-harm

0.076 0.082

1.05 0.29 1.12 0.26

Psychosocial management Psychosocial Dx action Discharge management Biological Dx action Medication Aftercare

- 0.284 -0.178 - 0.030 -0.167 0.083 0.146

-4.09 - 3.02 -0.49 - 2.65 1.30 2.33

Consultant Consultant Consultant

0.036 0.109 -0.004

0.56 0.58 1.82 0.07” -0.06 0.95

-0.137 0.188

- 2.37 0.02” 2.29 0.03”

1 2 3

Timing of consultation Number of recommendations

0.0001” 0.003” 0.62 0.009” 0.20 0.02

“Tendencyto significance; “significant.

with medication recommendations is somewhat dependent on psychiatric or medical diagnoses, and therefore, in the present multivariate model, this type of recommendation does not explain unique variance in case concordance. Alternatively, as most cases (76%) involve at least one medication recommendation, there is little opportunity

for this variable to discriminate among cases. The same could be said of psychosocial management, however (74%), and these recommendations were associated with lower case concordance. As both psychological management on the ward and the obtaining of psychosocial information are behaviors that are not constituents of the normal repertoire of the nonpsychiatrist physician and nursing staff, low concordance rates for them are not surprising. On the other hand, aftercare recommendations are part of the modus operandi of ward staff and routinely assisted by social workers, discharge planners, and so forth. That is, whereas aftercare planning is a normal part of most stays in the general hospital, psychosocial management makes far more complex demands on the staff. It remains unclear why the number of recommendations is positively related to the outcome of the case concordance. It may be that cases with larger numbers of recommendations involve a psychiatric consultant who is more actively involved in the ongoing care of the patient, a factor that might provide an important boost to the credibility of the recommendations and add social pressure for conformity to the recommendations. Alternatively, medication recommendations were often multiple in a single case. Given the high overall concordance with this recommendation type, this effect might also partially explain the relationship between the number of recommendations and case concordance. Finally, it is also possible that this association is at least a partial artifact of the transformation of the case concordance variable. The finding that the timing of the consultation was related to case concordance is provocative. Consistent with findings of Callies et al. [7], recommendations in earlier consults had greater concordance. This association is independent of demographic variables, psychiatric diagnoses, medical diagnoses, and type of recommendations. Although the Callies et al. definition of timing differed somewhat among their studies (medication: first half of admission vs second half; diagnostic action, number of days after the consultation, and total length of admission), and from our definition, all studies to date report that the earlier the consultation the higher the rate of concordance. A number of alternative hypotheses might explain this finding: 1) early consults might reflect a higher awareness for psychosocial issues on the part of the consultee; 2) early consults allow more time for recommendations to be followed; and, 3) early

367

F. J. Huyse et al.

cases are not yet settled in terms of medical management, that is, the case is still “fluid’ and open as to how it should be approached. Finally, it appears that who provides the consultation is an important consideration in determining case concordance. The senior consultant on the consultation service had the highest overall concordance rate and this effect was nearly significant (p 5 0.10) even when entered at the end of the prediction equation.

Conclusion Cases that have a prolonged lag time in which recommendations are single and/or psychosocial or are made by a junior consultant are associated with decreased concordance. The present results have implications for future research in C-L psychiatry: 1) The method of measuring case concordance appears both reliable and valid, as findings regarding the variation associated with type of recommendation and timing replicate earlier reports that were based on recommendation-based analyses [1,2,5121. Concordance assessed at the level of a case has many advantages over recommendation-based assessment in outcome research. The most important among these advantages is that case concordance is a single variable for each case that can then be associated with clinical and outcome assessments for that case; and 2) The lack of consultee or ward staff compliance with the psychiatric consult recommendations leads to error measurement in intervention studies. Unless it is known how much of the recommendation(s) was implemented, outcome research results are in error by the factor of nonconcordance (noncompliance) [22]. The clinical implications of the present findings lie in the manifest importance of the consultation process variables in predicting concordance. For directors of C-L services and consulting psychiatrists, an awareness that what is said and when it is said might be the most important factors in determining whether a consultation recommendation is to be implemented has important implications for the outcome of the psychiatrist’s recommendations. Improving concordance most likely lies in altering the consultation process. Consistent with the present findings, Haynes et al. [19] have provided guidelines for enhancing compliance with recommendations that emphasize altering the consultation process as the best means of improving compliance. The Free University Hospital now uses nurse clinicians in the role of consultation auditors.

368

Thus, these nurse clinicians function somewhat like case managers, following up on patients to determine whether or not they are receiving recommended services even after discharge. In summary, psychiatrists need to be aware of the vulnerability of the implementations of their recommendations. Those recommendations particularly likely to be violated are those that involve diagnostic action, increased lag time, and fewer senior consultants. The authors would like to thank Mary A. Eichman for her assistance with the analysis of data and Arjen leninga for his assistance with data management.

References 1. Popkin MK, Mackenzie TB, Hall RCW, Garrard J: Physicians’ concordance with consultants’ recommendations for psychotropic medication. Arch Gen Psychiatry 36:386-389, 1979 2. Popkin MK, Mackenzie TB, Callies AL: Consultation-liaison outcome evaluation system. I. Consultant-consultee interaction. Arch Gen Psychiatry 40:215-219, 1983 in 3. Huyse FJ, Strain JJ, Hammer JS: Interventions consultation-liaison psychiatry. Part I. Patterns of recommendations. Gen Hosp Psychiatry 12:213-220, 1990 4. Lyons JS, Hammer JS, Petraitis J, Larson DB: Treatment opportunities in consultation psychiatry. Am J Psychiatry 145:1435-1437, 1988 5. Billowitz A, Friedson W: Are psychiatric consultants’ recommendations followed? Int J Psychiatry Med 9:179-189, 1978-79 6. Popkin MK, Mackenzie TB, Callies AL: Consultees’ concordance with consultants’ recommendations for diagnostic action. J Nervous Ment Dis 168:9-12,198O 7. Callies AL, Popkin MK, Mackenzie TB, Mitchell J: Consultees’ representations of consultants’ psychiatric diagnoses. Am J Psychiatry 137:1250-1253,198O 8. Popkin MK, Mackenzie TB, Callies AL, Cohn JN: An interdisciplinary comparison of consultation outcomes. Arch Gen Psychiatry 38:821-825, 1981 9. Popkin MK, Mackenzie TB, Callies AL, Hall RCW: Yield of psychiatric consultants’ recommendations for diagnostic action. Arch Gen Psychiatry 39:843845, 1982 10. Popkin MK, Mackenzie TB, Callies AL: Psychiatric consultation to geriatric medically ill inpatients in a university hospital. Arch Gen Psychiatry 41:703-707, 1984 11. Colon E, Popkin MK, Callies AL: Emergency psychiatric consultation to medical-surgical services. Psychosomatics 29:315-320, 1988 in 12. Huyse FJ, Strain JJ, Hammer JS: Interventions consultation-liaison psychiatry. Part II. Concordance. Gen Hosp Psychiatry 12:221-231, 1990 13. Tabachnick L, Fidel B: Using multivariate statistics, 2nd ed. New York, Harper & Row, 1989

Psychiatric

14. Lipowski ZJ: Review of consultation psychiatry and psychosomatic medicine I and II. Psychosomat Med 29:153-171, 201-224, 1967 15. Hackett TP, Cassem NH: Handbook of General Hospital Psychiatry. St Louis, CV Mosby, 1978 16. Huyse FJ, Hengeveld MW: The development of C-L psychiatry in the Netherlands: its social psychiatric heritage. Gen Hospital Psychiatry 11:9-15, 1988 17. Hengeveld MW: Consultatieve Psychiatric in Nederland: 1984. Internal publication. Netherlands Consortium for Consultation Liaison Psychiatry, 1987 18. Lyons JS, Hammer JS, Strain JJ, Fulop G: The timing of psychiatric consultation and hospital length of stay. Gen Hosp Psychiatry 8:159-162, 1984

Consultations

and Concordance

19. Haynes RB, Taylor DW, Sackett DL: Compliance in Health Care. Baltimore, Johns Hopkins University Press, 1979 20. Strain JJ, Strain JW: Liaison psychiatry. In Howell JA (ed), Clinical Psychiatry. Modern Pespectives in Clinical Psychiatry. Brunner/Mazel, New York. 1988. pp 76-101. . 21. Strain JJ, Taintor Z: Consultation-liaison psychiatry. In Kaplan H, Saddock B (eds), Comprehensive Textbook of Psychiatry, 5th ed. Williams & Wilkins, Philadelohia. 1989. DD 1272-1279 1 22. Schleifer’ SJ, Bh’aidway S, Lebovits A, Tanaka JS, Messe M, Strain JJ: Predictors of physician nonadherence to chemotherapy regimens. Cancer 67:945951, 1991

369