Validation of the Critical Care Pain Observation Tool in Critically Ill Patients With Delirium: A Prospective Cohort Study Salmaan Kanji, PharmD1; Heather MacPhee, BSc (Pharm), ACPR1; Avinder Singh, BSc (Pharm), ACPR1; Christel Johanson, BSc (Pharm), ACPR1; Jennifer Fairbairn, BSc (Pharm), ACPR1; Tammy Lloyd, BSc (Pharm), ACPR1; Robert MacLean, PharmD, BCPS1; Erin Rosenberg, MD, FRCPC2
Objectives: The 2013 clinical practice guidelines for the management of pain, agitation, and delirium in adult patients in the ICU suggest that pain be routinely assessed using a validated pain assessment tool. Currently available tools have only been evaluated in nondelirious critically ill patients, yet delirium can affect as many as 80% of ICU patients. The validated pain assessment tool adopted by our institution is the Critical Care Pain Observation Tool, and the objective of this study was to investigate the validity of this tool in patients with evidence of delirium. Design: Prospective cohort study. Setting: Two ICUs within a Canadian tertiary healthcare center. Patients: Forty consecutive adult patients deemed delirious on the day of enrollment using the Confusion Assessment Method for ICU. Measurements and Main Results: Serial Critical Care Pain Observation Tool assessments were conducted simultaneously by study personnel and objective nurses at baseline and after nonpainful and painful stimuli. Subjective opinions about pain and objective physical variables (including mean arterial pressure, heart rate, respiratory rate, and oxygen saturation) were collected at the same time points. Discriminant validity was described using paired t tests, whereas internal consistency was described using the Cronbach α statistic. Responsiveness of the Critical Care Pain Observation Tool was measured by effect size, and reliability was described as the agreement between raters. Comparisons between the Critical Care Pain Observation Tool and the subjective assessments Department of Pharmacy, The Ottawa Hospital, Ottawa, ON, Canada. Department of Critical Care, The Ottawa Hospital, Ottawa, ON, Canada. Supported, in part, by a research grant from the Canadian Society of Hospital Pharmacists Research Foundation. All authors’ institutions received grant support from the Canadian Society of Hospital Pharmacy. For information regarding this article, E-mail: [email protected] Copyright © 2016 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved. DOI: 10.1097/CCM.0000000000001522 1 2
Critical Care Medicine
and objective measurements were based on positive and negative percent agreement. Critical Care Pain Observation Tool demonstrated excellent discriminant validity as evidenced by a highly statistically and clinically significant change in mean Critical Care Pain Observation Tool scores between baseline and painful procedures (mean difference, 3.13 ± 1.56; p < 0.001; Cohen D, 2.0). Interrater agreement was also excellent (κ > 0.6), and scores between raters were highly correlated (r = 0.957). The Critical Care Pain Observation Tool possessed a high level of internal consistency (overall Cronbach α, 0.778). Percent agreement was found to be greater between the Critical Care Pain Observation Tool and the nurse’s subjective opinion of the presence or absence of pain when compared with that between the Critical Care Pain Observation Tool and physiologic variables (80.5% vs 67.5%, respectively). Conclusions: The Critical Care Pain Observation Tool is a valid pain assessment tool in noncomatose, delirious adult ICU patients who are unable to reliably self-report the presence or absence of pain. (Crit Care Med 2016; 44:943–947) Key Words: critical care pain observation tool; delirium; intensive care unit; pain assessment
T
he 2013 clinical practice guidelines for the management of pain, agitation, and delirium in adult patients in the ICU suggest that pain routinely be assessed using a validated pain assessment tool (1). The majority of critically ill patients will experience pain at some point during their ICU admission; however, the assessment of pain in many of these subjects is challenged by the patients’ potential inability to self-report the presence, absence, or intensity of pain because of reasons related to illness, pharmacotherapy, and mechanics. Tools for pain assessment in the critically ill patient with diminished ability to communicate have been developed and validated; however, most development and validation studies have consistently excluded patients with delirium (2, 3). As delirium in the ICU can affect upward of 80% of patients, a www.ccmjournal.org
Copyright © 2016 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.
943
MacPhee et al
significant barrier, therefore, exists in adequately assessing and managing pain in delirious patients, leaving these subjects vulnerable to underrecognition and subsequent undertreatment or overtreatment of pain (4, 5). The Critical Care Pain Observation Tool (CPOT), validated in critically ill, ventilated postoperative, trauma, and medical ICU adults who are unable to reliably self-report, has been deemed one of the most reliable behavioral pain assessment tools (1). Regular use of this tool has led to better pain management and improved clinical outcomes in ICU patients; however, its role in the delirious or cognitively impaired patient has not been established or tested (6). The aim of our study was to determine the discriminant validity, internal consistency, responsiveness, and reliability of CPOT in noncomatose, delirious adult ICU patients. Secondary objectives included describing the percent agreement between the CPOT and the nurses’ subjective assessment of pain and the percent agreement between the CPOT and objective physical criteria potentially indicative of pain in noncomatose, delirious adult ICU patients.
MATERIALS AND METHODS Study Setting This study was conducted in two ICUs of a Canadian tertiary healthcare center where the management of pain, agitation, and delirium protocolized in accordance with published guidelines using preprinted orders (1). Titration of sedation and analgesia is nurse driven using Richmond Agitation Sedation Scale (RASS) and the CPOT or visual analog scales, respectively (2, 7). Delirium is routinely assessed and documented using the confusion Assessment Method-ICU (CAM-ICU) every 4 hours as per local nursing policy (8). The two participating ICUs care for medical, surgical, neurocritical care, and trauma patients. Patient Population Subjects included English or French speaking ICU patients who are 18 years old and older and admitted to either of the two participating units (between March 2014 and June 2014) and exhibiting symptoms of delirium (CAM-ICU positive upon study enrollment and at the baseline assessment). Patients in whom physical response to pain could not be reliably assessed (e.g., comatose patients [RASS score, −4 or −5], quadriplegic patients, patients with limb or facial injuries, patients receiving neuromuscular blockers, patients with traumatic brain injury, patients with stroke-affecting limb mobility, patients requiring physical restraints, and patients with established cognitive deficits) were excluded from enrollment. Study Design A prospective cohort design was chosen to achieve the objectives of this study and local research, and ethics board approval was obtained prior to enrollment. The CPOT includes four behavioral categories: 1) facial expression, 2) body movements, 3) muscle tension, and 4) compliance with ventilator or vocalization (2). Each behavioral category is scored between 0 and 2, for a total score ranging from 0 to 8. Scores greater than 2 are suggestive of the 944
www.ccmjournal.org
presence of pain (2). CAM-ICU was used to assess delirium in either English or French (8). The RASS was used to assess sedation (7). Study personnel received the same (English or French) training for the CPOT and CAM-ICU as nurses in our ICU, which included review of electronic modules and real-life scenarios. The CPOT, CAM-ICU, and RASS are tools that are used in clinical practice in both participating ICUs by all nurses. Serial CPOT assessments were conducted simultaneously by one of two study personnel (H.M. and S.K.) and an independent nurse (who was not familiar with the patient) at three time points: 1) baseline, 2) during a nonpainful stimulus, and 3) during a painful stimulus. A minimum of three paired CPOT assessments were expected for each patient. The nonpainful stimulus was a noninvasive blood pressure measurement, and the painful stimuli were patient repositioning, endotracheal suctioning, or a dressing change (of a wound). Preemptive analgesia prior to painful stimuli was administered only if previously ordered and clinically indicated. Subjective opinions about pain and objective physical variables (including mean arterial pressure, heart rate, respiration rate, and Spo2) were collected at the same time points. The subjective opinion was obtained from the bedside nurse (caring for the patient on the day of study) regarding the patient’s response to both painful and nonpainful stimuli relative to their baseline. Using paper questionnaires and sealed envelopes, the bedside nurse was asked if he/she felt that the patient was experiencing an increase in pain in response to the stimulus and was given the options of “yes”, “no,” or “unsure”. All assessors were blinded to the assessments/opinions of others. All assessments were done during the day but not during a planned sedation interruption. Assessments were to be completed within 24 hours of enrollment, and the time between baseline assessments and response to stimuli were to be completed within 2 hours (Fig. 1). If a patient was deemed to be delirious at enrollment but not at the time of the baseline assessment, the assessment was delayed until the patient was assessed as delirious using the CAM-ICU. Similarly, if the patient was deemed too sedated (RASS score, –4 or –5) at baseline, the assessment was also delayed until the patient was awake enough to participate. Treatment of pain and all other aspects of clinical care during or after the study were left to the discretion of the clinical team. Statistical Analysis A sample of 40 patients was calculated to identify a one-point difference between scores assuming an sd of 1.5, power of 80%,
Figure 1. Study procedures. Serial measurements of Critical Care Pain Observation Tool, objective physiologic variables, and nurses’ subjective opinions about pain were collected at baseline and after nonnociceptive and nociceptive stimuli. All assessments were to be done within 2-hr time frame, and patients were required to be Confusion Assessment MethodICU positive at the time of all assessments. May 2016 • Volume 44 • Number 5
Copyright © 2016 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.
Clinical Investigations
and an α of 0.05. The sample size calculation was based on discriminant validity that is evaluated by comparing CPOT scores at baseline to CPOT scores after painful stimuli. Assumptions of sd and expected values were drawn from previous CPOT validation studies (2, 3, 9). Descriptive statistics were compiled for all variables using measures of central tendency and proportions as appropriate. Discriminant validity was described using paired t tests to compare baseline scores with those obtained during both nonpainful and painful stimuli. Discriminant validation was considered to be statistically significant (p < 0.05) between baseline and the painful comparison but not the nonpainful comparison. Responsiveness was also examined by calculating the effect size as reported as Cohen D coefficient where results of less than 0.2 were considered to be small, near 0.5 considered moderate, and more than 0.8 considered large. Internal consistency was described using the Cronbach α statistic with an α of greater than 0.6 considered acceptable. Reliability was described as the agreement between raters with weighted κ of greater than 0.6 deemed appropriate. Intraclass correlation coefficients (ICCs) were also calculated. In addition, reliability was also measured as the correlation between nurses’ CPOT assessment and that of the research personnel using the Spearman test and expressed as r values. Comparisons between the CPOT and subjective assessments and objective measurements were based on positive and negative percent agreement. Statistical analyses were conducted using SPSS version 20.0 (IBM Corp, Armonk, NY).
RESULTS A total of 40 patients were enrolled during the study period. The mean time from baseline assessment to study completion was 21 ± 13 minutes. Reasons for admission to the ICU were primarily medical, with sepsis syndromes predominating. The majority of patients were mechanically ventilated and men, with a mean Acute Physiology and Chronic Health EvaluationII score of 19.1 ± 4.7 (Table 1). Patients who were mechanically ventilated did not require or receive changes to their ventilation variables immediately prior to or during the study period. All patients were CAM-ICU positive at the time of assessment, with 75% of study subjects having received an antipsychotic within 24 hours prior to enrollment. The majority of patients were receiving analgesia and sedatives, and the median RASS score at the time of assessment was 0 (Table 2). Discriminant Validity and Effect Size Three paired CPOT assessments were conducted in each patient. Discriminant validity was observed for CPOT measurements between baseline and a painful stimulus but not between baseline and a nonpainful stimulus (Table 3). Patient repositioning was used as the painful stimulus in 35 of 40 patients (88%). Endotracheal suctioning and dressing change were used in four (10%) and one (3%) patient(s), respectively. Only one patient in whom the dressing changes were used as the painful stimulus received a single dose of preemptive opioid analgesia. The effect size was large with the mean difference between CPOT Critical Care Medicine
Table 1. Patients’ Demographics and Clinical Characteristics n = 40
Demographics
Age, yr, mean (sd)
67 (14)
Women, n (%)
16 (40)
Admitting service, n (%) Medical
31 (78)
Surgical
9 (23)
Reasons for ICU admission, n (%) Sepsis syndromes
17 (43)
Gastrointestinal bleed
2 (5)
Respiratory failure
3 (8)
Chronic obstructive pulmonary disease
5 (13)
Pulmonary embolism
1 (3)
Postoperative care
9 (23)
Other
3 (8)
Mechanically ventilated, n (%)
36 (90)
Acute Physiology and Chronic Health Evaluation-II, mean (sd)
19 (5)
Sedation, Analgesia, and Delirium at the Time of Assessment Table 2.
Assessment and Medication Usage
n = 40
Richmond Agitation Sedation Scale, median (range)
0 (−3 to +3)
Confusion assessment method-ICU positive, n (%)
40 (100)
Medications, n (%) Opioids
29 (72.3)
Propofol
28 (70.0)
Midazolam
7 (17.5)
Any antipsychotic
30 (75.0)
Regularly scheduled antipsychotic
26 (65.0)
Clonidine
7 (17.5)
Dexmedetomidine
1 (2.5)
measurements at baseline and after painful stimulus being three points on the eight-point scale (Cohen D, 2.0). Reliability The CPOT demonstrates a high level of internal consistency (reliability) with an overall Cronbach α of 0.778. Each item was found to contribute to the overall internal consistency of the CPOT tool as the Cronbach α score did not improve with the removal of any indicator. Indicator 1 (facial expression) www.ccmjournal.org
Copyright © 2016 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.
945
MacPhee et al
Table 3.
Discriminant Validity
Assessment Period
Mean Critical Care Pain Observation Tool Score (Mean ± sd)
Baseline
0.50 ± 0.82
Nonpainful stimulus
0.65 ± 0.74
Painful stimulus
3.63 ± 1.80
Comparison between baseline and nonpainful stimuli
Mean difference = 0.15 ± 0.53; p = 0.083 Comparison between baseline and painful stimuli
Mean difference = 3.13 ± 1.56; p < 0.001 Effect size: Cohen D, 2.0
Table 4.
Internal Consistency Cronbach α
Item
Overall
0.778
Without indicator 1 (facial expression)
0.681
Without indicator 2 (body movements)
0.701
Without indicator 3 (muscle tension)
0.739
Without indicator 4 (compliance with ventilator or vocalization)
0.760
contributed the most to the reliability of the CPOT, whereas indicator 4 (ventilator compliance or vocalization) contributed the least (Table 4). Interrater agreement was acceptable for each individual component of the tool and the overall CPOT score based on 120 paired assessments between study personnel and nurses. Scores between raters were also highly correlated based on high Pearson correlation r values and ICCs (Table 5). Agreement With Nurses’ Subjective Opinions and Physiologic Variables Based on 77 paired assessments, the percent agreement between CPOT and the nurses’ subjective opinion as to whether the patient was experiencing an increase in pain from baseline was 80.5%. Based on 80 paired assessments Table 5.
between the CPOT and physiologic variables, the percent agreement was 67.5%.
DISCUSSION This study is the first to establish that CPOT is a valid pain assessment tool in delirious adult ICU patients unable to reliably self-report the presence or absence of pain. Consistent findings have been previously obtained in studies where the CPOT was used in surgical (cardiac and neurosurgical), medical, and trauma ICU groups (2, 3, 6, 10, 11). Discriminant validity was supported by the finding that CPOT scores were higher during a painful stimulus than at those at baseline, whereas CPOT scores after a nonpainful stimulus were not statistically different from those at baseline. Similar discriminative validity was shown in prior CPOT studies (2, 3, 6, 10–12). Previous studies by Gélinas et al (2, 3, 9) have demonstrated that CPOT scores greater than 2 have a sensitivity of 86% and a specificity of 78% for predicting significant pain in critically ill medical, surgical, and trauma patients and in patients who were mechanically ventilated. A Cohen D coefficient of 2 (> 0.8) confirms that our effect size was large, and thus, the difference between mean CPOT scores at baseline and after the painful stimulus was likely clinically meaningful in addition to being statistically significant. These findings are similar to those of Chanques et al (11) where the effect size of CPOT was 1.55. Such results provide strength to the evidence that pain-related behaviors are observable even in the noncommunicative delirious critically ill patient. We describe a high level of internal consistency for the CPOT tool. The overall Cronbach α of 0.778 was similar to those obtained by Chanques et al (12) where the authors calculated an α of 0.76 for CPOT. All four domains of the CPOT tool were well correlated. Interrater reliability is essential in standardizing pain assessment in the ICU where multiple clinicians must reliably assess pain in noncommunicative patients. Our findings indicate that interrater reliability of CPOT between two trained raters was excellent as supported by high κ of greater than 0.6 for each individual component of the tool, as well as the overall score (Table 5). These findings are consistent with prior validation studies, where the CPOT also demonstrated moderate to high interrater reliability with κ of greater than
Interrater Reliability κ
se
p
Spearman Correlation, r
Intraclass Correlation Coefficient (95% CI)
Indicator 1 (facial expression)
0.669
0.058
< 0.001
0.836
0.910 (0.871–0.937)
Indicator 2 (body movement)
0.873
0.043
< 0.001
0.920
0.958 (0.939–0.970)
Indicator 3 (muscle tension)
0.886
0.045
< 0.001
0.922
0.959 (0.942–0.972)
Indicator 4 (compliance with ventilator or vocalization)
0.882
0.051
< 0.001
0.879
0.936 (0.908–0.955)
Total score
0.669
0.049
< 0.001
0.957
0.977 (0.967–0.984)
Item
946
www.ccmjournal.org
May 2016 • Volume 44 • Number 5
Copyright © 2016 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.
Clinical Investigations
0.6 or ICC of greater than 0.8 (2, 3, 6). In a recent study aimed at comparing psychometric properties of the CPOT, the reliability was near perfect with a weighted κ coefficient of 0.81 ± 0.03 (11). Percent agreement between the bedside nurses’ subjective assessment and CPOT scores (as assessed by the objective nurse and research personnel) was found to be greater than that between the CPOT score and physiologic variables. This is in keeping with the general consensus in the literature that supports the assessment of pain-related behaviors rather than using physiologic indicators alone for pain assessment in the ICU (1). Physiologic variables should only be considered a cue for further assessment of pain (1). There are some innate limitations related to our study design. As pain is complex and subjective, the patient’s selfreport of pain remains the gold standard (1). However, even in communicative patients within this study, the presence of delirium compromises the validity of self-reported pain. No other validated instrument has been developed to measure the presence or absence of pain in critically ill patients with delirium, and a comparison between the CPOT and a reference standard was not possible. In the absence of a gold standard, the usual measures of accuracy, such as sensitivity and specificity, do not apply. As such, discriminant validity was used as the primary objective. A second potential limitation to this study design is that painful and nonpainful stimuli must be standardized between patients. Increases in CPOT scores from baseline were observed in some patients after the nonpainful stimulus; however, the mean difference in CPOT scores between these assessments was not statistically significant, and the stimulus (noninvasive blood pressure cuff inflation) was standardized for all patients. We also did not evaluate the recovery period after the painful stimulus, which may have allowed for a better assessment of discriminant validity. Third, delirium typically fluctuates with periods of lucidity. Although it is possible that delirium waxed and waned throughout the study, we endeavored to minimize the impact of this by closely timing baseline, nonpainful and painful assessments and ensuring patients were CAM-ICU positive at both enrollment and at baseline. The mean time from baseline assessment to study completion was 21 ± 13 minutes. Finally, our cohort of patients with variable severities of delirium consisted of both untreated and treated patients. Most patients were receiving antipsychotics, sedatives, and opioids, but we did not collect the use of nonopioid analgesics. The confounding of these concomitant treatments was not measured, but we ensured that despite these treatments, all evaluated patients were delirious (as per the CAM-ICU) and rousable to voice (RASS score, ≥ –3) at the time of all assessments. It is still possible that the validity of the CPOT tool differs between those with variable severities of delirium and between those receiving treatment compared with those who are not. Despite these limitations, the CPOT demonstrated excellent discriminant validity, a large effect size, a high level of internal consistency, and excellent
Critical Care Medicine
interrater agreement in noncomatose, delirious adult ICU patients.
CONCLUSIONS The use of a valid pain assessment tool is important for pain management in all critically ill patients. Practice guidelines recommend the routine assessment of pain in all critically ill patients with a validated pain assessment tool (1). Our study findings indicate that the CPOT is a valid and reliable tool for the detection of pain in noncomatose, delirious adult ICU patients. The CPOT has now been validated in a broad range of critically ill patients and should be considered the single tool that can be applied to almost all ICU patients. Our study describes excellent discriminant validity, a large effect size, a high level of internal consistency (reliability), and excellent interrater agreement for the use of CPOT in assessing pain in delirious patients unable to reliably self-report. Further research is still warranted to assess pain in patients who are deeply sedated (RASS score, > –3) and those with limited or impaired mobility.
ACKNOWLEDGMENT We thank the intensive care nurses at the Ottawa Hospital’s Civic and General Campuses.
REFERENCES
1. Barr J, Fraser GL, Puntillo K, et al: Clinical practice guidelines for the management of pain, agitation, and delirium in adult patients in the intensive care unit. Crit Care Med 2006; 34:1691–1699 2. Gélinas C, Fillion L, Puntillo KA, et al: Validation of the critical-care pain observation tool in adult patients. Am J Crit Care 2006; 15:420–427 3. Gélinas C, Johnston C: Pain assessment in the critically ill ventilated adult: Validation of the critical-care pain observation tool and physiologic indicators. Clin J Pain 2007; 23:497–505 4. Cook AK, Niven CA, Downs MG: Assessing the pain of people with cognitive impairment. Int J Geriatr Psychiatry 1999; 14:421–425 5. Ely EW, Shintani A, Truman B, et al: Delirium as a predictor of mortality in mechanically ventilated patients in the intensive care unit. JAMA 2004; 291:1753–1762 6. Gélinas C, Arbour C, Michaud C, et al: Implementation of the criticalcare pain observation tool on pain assessment/management nursing practices in an intensive care unit with nonverbal critically ill adults: A before and after study. Int J Nurs Stud 2011; 48:1495–1504 7. Sessler CN, Gosnell MS, Grap MJ, et al: The Richmond AgitationSedation Scale: Validity and reliability in adult intensive care unit patients. Am J Respir Crit Care Med 2002; 166:1338–1344 8. Ely EW, Inouye SK, Bernard GR, et al: Delirium in mechanically ventilated patients: Validity and reliability of the confusion assessment method for the intensive care unit (CAM-ICU). JAMA 2001; 286:2703–2710 9. Gélinas C, Fillion L, Puntillo KA: Item selection and content validity of the critical-care pain observation tool for non-verbal adults. J Adv Nurs 2009; 65:203–216 10. Nürnberg Damström D, Saboonchi F, Sackey PV, et al: A preliminary validation of the Swedish version of the critical-care pain observation tool in adults. Acta Anaesthesiol Scand 2011; 55:379–386 11. Chanques G, Pohlman A, Kress JP, et al: Psychometric comparison of three behavioural scales for the assessment of pain in critically ill patients unable to self-report. Crit Care 2014; 18:R160 12. Chanques G, Payen JF, Mercier G, et al: Assessing pain in non-intubated critically ill patients unable to self report: An adaptation of the behavioral pain scale. Intensive Care Med 2009; 35:2060–2067
www.ccmjournal.org
Copyright © 2016 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.
947
Objectives: The 2013 clinical practice guidelines for the management of pain, agitation, and delirium in adult patients in the ICU suggest that pain be routinely assessed using a validated pain assessment tool. Currently available tools have only been evaluated in nondelirious critically ill patients, yet delirium can affect as many as 80% of ICU patients. The validated pain assessment tool adopted by our institution is the Critical Care Pain Observation Tool, and the objective of this study was to investigate the validity of this tool in patients with evidence of delirium. Design: Prospective cohort study. Setting: Two ICUs within a Canadian tertiary healthcare center. Patients: Forty consecutive adult patients deemed delirious on the day of enrollment using the Confusion Assessment Method for ICU. Measurements and Main Results: Serial Critical Care Pain Observation Tool assessments were conducted simultaneously by study personnel and objective nurses at baseline and after nonpainful and painful stimuli. Subjective opinions about pain and objective physical variables (including mean arterial pressure, heart rate, respiratory rate, and oxygen saturation) were collected at the same time points. Discriminant validity was described using paired t tests, whereas internal consistency was described using the Cronbach α statistic. Responsiveness of the Critical Care Pain Observation Tool was measured by effect size, and reliability was described as the agreement between raters. Comparisons between the Critical Care Pain Observation Tool and the subjective assessments Department of Pharmacy, The Ottawa Hospital, Ottawa, ON, Canada. Department of Critical Care, The Ottawa Hospital, Ottawa, ON, Canada. Supported, in part, by a research grant from the Canadian Society of Hospital Pharmacists Research Foundation. All authors’ institutions received grant support from the Canadian Society of Hospital Pharmacy. For information regarding this article, E-mail: [email protected] Copyright © 2016 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved. DOI: 10.1097/CCM.0000000000001522 1 2
Critical Care Medicine
and objective measurements were based on positive and negative percent agreement. Critical Care Pain Observation Tool demonstrated excellent discriminant validity as evidenced by a highly statistically and clinically significant change in mean Critical Care Pain Observation Tool scores between baseline and painful procedures (mean difference, 3.13 ± 1.56; p < 0.001; Cohen D, 2.0). Interrater agreement was also excellent (κ > 0.6), and scores between raters were highly correlated (r = 0.957). The Critical Care Pain Observation Tool possessed a high level of internal consistency (overall Cronbach α, 0.778). Percent agreement was found to be greater between the Critical Care Pain Observation Tool and the nurse’s subjective opinion of the presence or absence of pain when compared with that between the Critical Care Pain Observation Tool and physiologic variables (80.5% vs 67.5%, respectively). Conclusions: The Critical Care Pain Observation Tool is a valid pain assessment tool in noncomatose, delirious adult ICU patients who are unable to reliably self-report the presence or absence of pain. (Crit Care Med 2016; 44:943–947) Key Words: critical care pain observation tool; delirium; intensive care unit; pain assessment
T
he 2013 clinical practice guidelines for the management of pain, agitation, and delirium in adult patients in the ICU suggest that pain routinely be assessed using a validated pain assessment tool (1). The majority of critically ill patients will experience pain at some point during their ICU admission; however, the assessment of pain in many of these subjects is challenged by the patients’ potential inability to self-report the presence, absence, or intensity of pain because of reasons related to illness, pharmacotherapy, and mechanics. Tools for pain assessment in the critically ill patient with diminished ability to communicate have been developed and validated; however, most development and validation studies have consistently excluded patients with delirium (2, 3). As delirium in the ICU can affect upward of 80% of patients, a www.ccmjournal.org
Copyright © 2016 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.
943
MacPhee et al
significant barrier, therefore, exists in adequately assessing and managing pain in delirious patients, leaving these subjects vulnerable to underrecognition and subsequent undertreatment or overtreatment of pain (4, 5). The Critical Care Pain Observation Tool (CPOT), validated in critically ill, ventilated postoperative, trauma, and medical ICU adults who are unable to reliably self-report, has been deemed one of the most reliable behavioral pain assessment tools (1). Regular use of this tool has led to better pain management and improved clinical outcomes in ICU patients; however, its role in the delirious or cognitively impaired patient has not been established or tested (6). The aim of our study was to determine the discriminant validity, internal consistency, responsiveness, and reliability of CPOT in noncomatose, delirious adult ICU patients. Secondary objectives included describing the percent agreement between the CPOT and the nurses’ subjective assessment of pain and the percent agreement between the CPOT and objective physical criteria potentially indicative of pain in noncomatose, delirious adult ICU patients.
MATERIALS AND METHODS Study Setting This study was conducted in two ICUs of a Canadian tertiary healthcare center where the management of pain, agitation, and delirium protocolized in accordance with published guidelines using preprinted orders (1). Titration of sedation and analgesia is nurse driven using Richmond Agitation Sedation Scale (RASS) and the CPOT or visual analog scales, respectively (2, 7). Delirium is routinely assessed and documented using the confusion Assessment Method-ICU (CAM-ICU) every 4 hours as per local nursing policy (8). The two participating ICUs care for medical, surgical, neurocritical care, and trauma patients. Patient Population Subjects included English or French speaking ICU patients who are 18 years old and older and admitted to either of the two participating units (between March 2014 and June 2014) and exhibiting symptoms of delirium (CAM-ICU positive upon study enrollment and at the baseline assessment). Patients in whom physical response to pain could not be reliably assessed (e.g., comatose patients [RASS score, −4 or −5], quadriplegic patients, patients with limb or facial injuries, patients receiving neuromuscular blockers, patients with traumatic brain injury, patients with stroke-affecting limb mobility, patients requiring physical restraints, and patients with established cognitive deficits) were excluded from enrollment. Study Design A prospective cohort design was chosen to achieve the objectives of this study and local research, and ethics board approval was obtained prior to enrollment. The CPOT includes four behavioral categories: 1) facial expression, 2) body movements, 3) muscle tension, and 4) compliance with ventilator or vocalization (2). Each behavioral category is scored between 0 and 2, for a total score ranging from 0 to 8. Scores greater than 2 are suggestive of the 944
www.ccmjournal.org
presence of pain (2). CAM-ICU was used to assess delirium in either English or French (8). The RASS was used to assess sedation (7). Study personnel received the same (English or French) training for the CPOT and CAM-ICU as nurses in our ICU, which included review of electronic modules and real-life scenarios. The CPOT, CAM-ICU, and RASS are tools that are used in clinical practice in both participating ICUs by all nurses. Serial CPOT assessments were conducted simultaneously by one of two study personnel (H.M. and S.K.) and an independent nurse (who was not familiar with the patient) at three time points: 1) baseline, 2) during a nonpainful stimulus, and 3) during a painful stimulus. A minimum of three paired CPOT assessments were expected for each patient. The nonpainful stimulus was a noninvasive blood pressure measurement, and the painful stimuli were patient repositioning, endotracheal suctioning, or a dressing change (of a wound). Preemptive analgesia prior to painful stimuli was administered only if previously ordered and clinically indicated. Subjective opinions about pain and objective physical variables (including mean arterial pressure, heart rate, respiration rate, and Spo2) were collected at the same time points. The subjective opinion was obtained from the bedside nurse (caring for the patient on the day of study) regarding the patient’s response to both painful and nonpainful stimuli relative to their baseline. Using paper questionnaires and sealed envelopes, the bedside nurse was asked if he/she felt that the patient was experiencing an increase in pain in response to the stimulus and was given the options of “yes”, “no,” or “unsure”. All assessors were blinded to the assessments/opinions of others. All assessments were done during the day but not during a planned sedation interruption. Assessments were to be completed within 24 hours of enrollment, and the time between baseline assessments and response to stimuli were to be completed within 2 hours (Fig. 1). If a patient was deemed to be delirious at enrollment but not at the time of the baseline assessment, the assessment was delayed until the patient was assessed as delirious using the CAM-ICU. Similarly, if the patient was deemed too sedated (RASS score, –4 or –5) at baseline, the assessment was also delayed until the patient was awake enough to participate. Treatment of pain and all other aspects of clinical care during or after the study were left to the discretion of the clinical team. Statistical Analysis A sample of 40 patients was calculated to identify a one-point difference between scores assuming an sd of 1.5, power of 80%,
Figure 1. Study procedures. Serial measurements of Critical Care Pain Observation Tool, objective physiologic variables, and nurses’ subjective opinions about pain were collected at baseline and after nonnociceptive and nociceptive stimuli. All assessments were to be done within 2-hr time frame, and patients were required to be Confusion Assessment MethodICU positive at the time of all assessments. May 2016 • Volume 44 • Number 5
Copyright © 2016 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.
Clinical Investigations
and an α of 0.05. The sample size calculation was based on discriminant validity that is evaluated by comparing CPOT scores at baseline to CPOT scores after painful stimuli. Assumptions of sd and expected values were drawn from previous CPOT validation studies (2, 3, 9). Descriptive statistics were compiled for all variables using measures of central tendency and proportions as appropriate. Discriminant validity was described using paired t tests to compare baseline scores with those obtained during both nonpainful and painful stimuli. Discriminant validation was considered to be statistically significant (p < 0.05) between baseline and the painful comparison but not the nonpainful comparison. Responsiveness was also examined by calculating the effect size as reported as Cohen D coefficient where results of less than 0.2 were considered to be small, near 0.5 considered moderate, and more than 0.8 considered large. Internal consistency was described using the Cronbach α statistic with an α of greater than 0.6 considered acceptable. Reliability was described as the agreement between raters with weighted κ of greater than 0.6 deemed appropriate. Intraclass correlation coefficients (ICCs) were also calculated. In addition, reliability was also measured as the correlation between nurses’ CPOT assessment and that of the research personnel using the Spearman test and expressed as r values. Comparisons between the CPOT and subjective assessments and objective measurements were based on positive and negative percent agreement. Statistical analyses were conducted using SPSS version 20.0 (IBM Corp, Armonk, NY).
RESULTS A total of 40 patients were enrolled during the study period. The mean time from baseline assessment to study completion was 21 ± 13 minutes. Reasons for admission to the ICU were primarily medical, with sepsis syndromes predominating. The majority of patients were mechanically ventilated and men, with a mean Acute Physiology and Chronic Health EvaluationII score of 19.1 ± 4.7 (Table 1). Patients who were mechanically ventilated did not require or receive changes to their ventilation variables immediately prior to or during the study period. All patients were CAM-ICU positive at the time of assessment, with 75% of study subjects having received an antipsychotic within 24 hours prior to enrollment. The majority of patients were receiving analgesia and sedatives, and the median RASS score at the time of assessment was 0 (Table 2). Discriminant Validity and Effect Size Three paired CPOT assessments were conducted in each patient. Discriminant validity was observed for CPOT measurements between baseline and a painful stimulus but not between baseline and a nonpainful stimulus (Table 3). Patient repositioning was used as the painful stimulus in 35 of 40 patients (88%). Endotracheal suctioning and dressing change were used in four (10%) and one (3%) patient(s), respectively. Only one patient in whom the dressing changes were used as the painful stimulus received a single dose of preemptive opioid analgesia. The effect size was large with the mean difference between CPOT Critical Care Medicine
Table 1. Patients’ Demographics and Clinical Characteristics n = 40
Demographics
Age, yr, mean (sd)
67 (14)
Women, n (%)
16 (40)
Admitting service, n (%) Medical
31 (78)
Surgical
9 (23)
Reasons for ICU admission, n (%) Sepsis syndromes
17 (43)
Gastrointestinal bleed
2 (5)
Respiratory failure
3 (8)
Chronic obstructive pulmonary disease
5 (13)
Pulmonary embolism
1 (3)
Postoperative care
9 (23)
Other
3 (8)
Mechanically ventilated, n (%)
36 (90)
Acute Physiology and Chronic Health Evaluation-II, mean (sd)
19 (5)
Sedation, Analgesia, and Delirium at the Time of Assessment Table 2.
Assessment and Medication Usage
n = 40
Richmond Agitation Sedation Scale, median (range)
0 (−3 to +3)
Confusion assessment method-ICU positive, n (%)
40 (100)
Medications, n (%) Opioids
29 (72.3)
Propofol
28 (70.0)
Midazolam
7 (17.5)
Any antipsychotic
30 (75.0)
Regularly scheduled antipsychotic
26 (65.0)
Clonidine
7 (17.5)
Dexmedetomidine
1 (2.5)
measurements at baseline and after painful stimulus being three points on the eight-point scale (Cohen D, 2.0). Reliability The CPOT demonstrates a high level of internal consistency (reliability) with an overall Cronbach α of 0.778. Each item was found to contribute to the overall internal consistency of the CPOT tool as the Cronbach α score did not improve with the removal of any indicator. Indicator 1 (facial expression) www.ccmjournal.org
Copyright © 2016 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.
945
MacPhee et al
Table 3.
Discriminant Validity
Assessment Period
Mean Critical Care Pain Observation Tool Score (Mean ± sd)
Baseline
0.50 ± 0.82
Nonpainful stimulus
0.65 ± 0.74
Painful stimulus
3.63 ± 1.80
Comparison between baseline and nonpainful stimuli
Mean difference = 0.15 ± 0.53; p = 0.083 Comparison between baseline and painful stimuli
Mean difference = 3.13 ± 1.56; p < 0.001 Effect size: Cohen D, 2.0
Table 4.
Internal Consistency Cronbach α
Item
Overall
0.778
Without indicator 1 (facial expression)
0.681
Without indicator 2 (body movements)
0.701
Without indicator 3 (muscle tension)
0.739
Without indicator 4 (compliance with ventilator or vocalization)
0.760
contributed the most to the reliability of the CPOT, whereas indicator 4 (ventilator compliance or vocalization) contributed the least (Table 4). Interrater agreement was acceptable for each individual component of the tool and the overall CPOT score based on 120 paired assessments between study personnel and nurses. Scores between raters were also highly correlated based on high Pearson correlation r values and ICCs (Table 5). Agreement With Nurses’ Subjective Opinions and Physiologic Variables Based on 77 paired assessments, the percent agreement between CPOT and the nurses’ subjective opinion as to whether the patient was experiencing an increase in pain from baseline was 80.5%. Based on 80 paired assessments Table 5.
between the CPOT and physiologic variables, the percent agreement was 67.5%.
DISCUSSION This study is the first to establish that CPOT is a valid pain assessment tool in delirious adult ICU patients unable to reliably self-report the presence or absence of pain. Consistent findings have been previously obtained in studies where the CPOT was used in surgical (cardiac and neurosurgical), medical, and trauma ICU groups (2, 3, 6, 10, 11). Discriminant validity was supported by the finding that CPOT scores were higher during a painful stimulus than at those at baseline, whereas CPOT scores after a nonpainful stimulus were not statistically different from those at baseline. Similar discriminative validity was shown in prior CPOT studies (2, 3, 6, 10–12). Previous studies by Gélinas et al (2, 3, 9) have demonstrated that CPOT scores greater than 2 have a sensitivity of 86% and a specificity of 78% for predicting significant pain in critically ill medical, surgical, and trauma patients and in patients who were mechanically ventilated. A Cohen D coefficient of 2 (> 0.8) confirms that our effect size was large, and thus, the difference between mean CPOT scores at baseline and after the painful stimulus was likely clinically meaningful in addition to being statistically significant. These findings are similar to those of Chanques et al (11) where the effect size of CPOT was 1.55. Such results provide strength to the evidence that pain-related behaviors are observable even in the noncommunicative delirious critically ill patient. We describe a high level of internal consistency for the CPOT tool. The overall Cronbach α of 0.778 was similar to those obtained by Chanques et al (12) where the authors calculated an α of 0.76 for CPOT. All four domains of the CPOT tool were well correlated. Interrater reliability is essential in standardizing pain assessment in the ICU where multiple clinicians must reliably assess pain in noncommunicative patients. Our findings indicate that interrater reliability of CPOT between two trained raters was excellent as supported by high κ of greater than 0.6 for each individual component of the tool, as well as the overall score (Table 5). These findings are consistent with prior validation studies, where the CPOT also demonstrated moderate to high interrater reliability with κ of greater than
Interrater Reliability κ
se
p
Spearman Correlation, r
Intraclass Correlation Coefficient (95% CI)
Indicator 1 (facial expression)
0.669
0.058
< 0.001
0.836
0.910 (0.871–0.937)
Indicator 2 (body movement)
0.873
0.043
< 0.001
0.920
0.958 (0.939–0.970)
Indicator 3 (muscle tension)
0.886
0.045
< 0.001
0.922
0.959 (0.942–0.972)
Indicator 4 (compliance with ventilator or vocalization)
0.882
0.051
< 0.001
0.879
0.936 (0.908–0.955)
Total score
0.669
0.049
< 0.001
0.957
0.977 (0.967–0.984)
Item
946
www.ccmjournal.org
May 2016 • Volume 44 • Number 5
Copyright © 2016 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.
Clinical Investigations
0.6 or ICC of greater than 0.8 (2, 3, 6). In a recent study aimed at comparing psychometric properties of the CPOT, the reliability was near perfect with a weighted κ coefficient of 0.81 ± 0.03 (11). Percent agreement between the bedside nurses’ subjective assessment and CPOT scores (as assessed by the objective nurse and research personnel) was found to be greater than that between the CPOT score and physiologic variables. This is in keeping with the general consensus in the literature that supports the assessment of pain-related behaviors rather than using physiologic indicators alone for pain assessment in the ICU (1). Physiologic variables should only be considered a cue for further assessment of pain (1). There are some innate limitations related to our study design. As pain is complex and subjective, the patient’s selfreport of pain remains the gold standard (1). However, even in communicative patients within this study, the presence of delirium compromises the validity of self-reported pain. No other validated instrument has been developed to measure the presence or absence of pain in critically ill patients with delirium, and a comparison between the CPOT and a reference standard was not possible. In the absence of a gold standard, the usual measures of accuracy, such as sensitivity and specificity, do not apply. As such, discriminant validity was used as the primary objective. A second potential limitation to this study design is that painful and nonpainful stimuli must be standardized between patients. Increases in CPOT scores from baseline were observed in some patients after the nonpainful stimulus; however, the mean difference in CPOT scores between these assessments was not statistically significant, and the stimulus (noninvasive blood pressure cuff inflation) was standardized for all patients. We also did not evaluate the recovery period after the painful stimulus, which may have allowed for a better assessment of discriminant validity. Third, delirium typically fluctuates with periods of lucidity. Although it is possible that delirium waxed and waned throughout the study, we endeavored to minimize the impact of this by closely timing baseline, nonpainful and painful assessments and ensuring patients were CAM-ICU positive at both enrollment and at baseline. The mean time from baseline assessment to study completion was 21 ± 13 minutes. Finally, our cohort of patients with variable severities of delirium consisted of both untreated and treated patients. Most patients were receiving antipsychotics, sedatives, and opioids, but we did not collect the use of nonopioid analgesics. The confounding of these concomitant treatments was not measured, but we ensured that despite these treatments, all evaluated patients were delirious (as per the CAM-ICU) and rousable to voice (RASS score, ≥ –3) at the time of all assessments. It is still possible that the validity of the CPOT tool differs between those with variable severities of delirium and between those receiving treatment compared with those who are not. Despite these limitations, the CPOT demonstrated excellent discriminant validity, a large effect size, a high level of internal consistency, and excellent
Critical Care Medicine
interrater agreement in noncomatose, delirious adult ICU patients.
CONCLUSIONS The use of a valid pain assessment tool is important for pain management in all critically ill patients. Practice guidelines recommend the routine assessment of pain in all critically ill patients with a validated pain assessment tool (1). Our study findings indicate that the CPOT is a valid and reliable tool for the detection of pain in noncomatose, delirious adult ICU patients. The CPOT has now been validated in a broad range of critically ill patients and should be considered the single tool that can be applied to almost all ICU patients. Our study describes excellent discriminant validity, a large effect size, a high level of internal consistency (reliability), and excellent interrater agreement for the use of CPOT in assessing pain in delirious patients unable to reliably self-report. Further research is still warranted to assess pain in patients who are deeply sedated (RASS score, > –3) and those with limited or impaired mobility.
ACKNOWLEDGMENT We thank the intensive care nurses at the Ottawa Hospital’s Civic and General Campuses.
REFERENCES
1. Barr J, Fraser GL, Puntillo K, et al: Clinical practice guidelines for the management of pain, agitation, and delirium in adult patients in the intensive care unit. Crit Care Med 2006; 34:1691–1699 2. Gélinas C, Fillion L, Puntillo KA, et al: Validation of the critical-care pain observation tool in adult patients. Am J Crit Care 2006; 15:420–427 3. Gélinas C, Johnston C: Pain assessment in the critically ill ventilated adult: Validation of the critical-care pain observation tool and physiologic indicators. Clin J Pain 2007; 23:497–505 4. Cook AK, Niven CA, Downs MG: Assessing the pain of people with cognitive impairment. Int J Geriatr Psychiatry 1999; 14:421–425 5. Ely EW, Shintani A, Truman B, et al: Delirium as a predictor of mortality in mechanically ventilated patients in the intensive care unit. JAMA 2004; 291:1753–1762 6. Gélinas C, Arbour C, Michaud C, et al: Implementation of the criticalcare pain observation tool on pain assessment/management nursing practices in an intensive care unit with nonverbal critically ill adults: A before and after study. Int J Nurs Stud 2011; 48:1495–1504 7. Sessler CN, Gosnell MS, Grap MJ, et al: The Richmond AgitationSedation Scale: Validity and reliability in adult intensive care unit patients. Am J Respir Crit Care Med 2002; 166:1338–1344 8. Ely EW, Inouye SK, Bernard GR, et al: Delirium in mechanically ventilated patients: Validity and reliability of the confusion assessment method for the intensive care unit (CAM-ICU). JAMA 2001; 286:2703–2710 9. Gélinas C, Fillion L, Puntillo KA: Item selection and content validity of the critical-care pain observation tool for non-verbal adults. J Adv Nurs 2009; 65:203–216 10. Nürnberg Damström D, Saboonchi F, Sackey PV, et al: A preliminary validation of the Swedish version of the critical-care pain observation tool in adults. Acta Anaesthesiol Scand 2011; 55:379–386 11. Chanques G, Pohlman A, Kress JP, et al: Psychometric comparison of three behavioural scales for the assessment of pain in critically ill patients unable to self-report. Crit Care 2014; 18:R160 12. Chanques G, Payen JF, Mercier G, et al: Assessing pain in non-intubated critically ill patients unable to self report: An adaptation of the behavioral pain scale. Intensive Care Med 2009; 35:2060–2067
www.ccmjournal.org
Copyright © 2016 by the Society of Critical Care Medicine and Wolters Kluwer Health, Inc. All Rights Reserved.
947
