Validation of the Italian version of the Critical Pain Observation Tool in brain-injured critically ill adults.

BACKGROUND AND AIM: Pain in intensive care units (ICUs) is a frequent and often undermanaged problem. Brain-injured patients are often unable to reliably self-report their pain, calling forth the need to use behavioural scales such as the Critical-Care Pain Observation Tool (CPOT). This study aimed to test the reliability and validity of the Italian CPOT use with brain-injured ICU adults.
METHOD: A sample of 50 adults critical care patients was included. Each patient was assessed by two independent observers at three predefined times - at rest; during mobilization for hygiene; 20 minutes later - using the CPOT, PAINAD, and NRS.
RESULTS: A good correlation was found between independent observers scores during painful procedure, establishing interrater reliability of CPOT. Criterion validation was supported by a strong correlation between CPOT and PAINAD scores, and a moderate relation between CPOT and NRS scores. The CPOT was able to discriminate between patients undergoing painful versus non-painful procedures. However, PAINAD performed better in this sample, as revealed by the comparison between the two AUC of ROC curves.
CONCLUSIONS: The Italian CPOT use was found reliable and valid in this patient group.

Introduction

The International Association for the Study of Pain (IASP) defines pain as “an unpleasant sensory and emotional experience associated with actual or potential tissue damage or described in terms of such damage” (1). The definition emphasises the subjective nature of pain and suggests that its intensity can be assessed only by someone experiencing it. It is obvious that many patients treated in intensive care units (ICUs), particularly those intubated and mechanically ventilated, do not fit this definition as they cannot self-report pain sensations or assess their intensity. The assessment of pain in ICU patients is a daily challenge for therapeutic teams, especially in patients who are endotracheally intubated, mechanically ventilated or analgo-sedated. This is the case of patients reporting spinal cord injury (SCI). Chronic pain is an important problem following SCI and is a major impediment to effective rehabilitation. The reported prevalence of chronic SCI pain is variable but averages 65% with around one third of these people rating their pain as severe, and such pain represents a major burden to the patients (2, 3). Patients with brain injury are often not able to self-report due to an alteration of loss of consciousness (LOC), the anatomic localization of the lesion, disorientation, or other cognitive deficits, leading to a smaller number of self-reports. Nursing staff is usually unaware of their pain and it is likely that pain remains untreated (4). Low level of consciousness and being on ventila-tor or receiving tranquilizers are some examples of this condition (5). Untreated and prolonged pain can affect endocrine, cardiovascular, immune, nervous and mus-culoskeletal systems and result in chronic pain syn-drome, prolonged hospitalization (6). Furthermore, it can increase cat-echolamine and stress hormones release, which in turn would cause increased blood pressure, increased heart rate, and increased oxygen demand, and decreased per-fusion (7).

Nurses play a vital role in pain assessment (8), and de-termine pain intensity and perform medical and non-medical treatment for these patients. Therefore, it is necessary for nurses to be fully aware of a standard and systematic pain evaluation protocol (9). In response to these needs, methods were offered to evaluate pain in patients more effectively: as, for example, behavioural pain assessment tools.

The Critical-Care Pain Observation Tool (C-POT) is currently considered one of the best scales, both for psychometric properties and clinical feasibility. In 2006, the American pain management nursing associa-tion recommended the CPOT for evaluating pain in tracheal intubated and unconscious patients (5). The observational studies have demonstrated that CPOT has good psychometric indices as for the inter-observer agreement of assessments in medical, surgical and trauma patients; yet without cerebral stroke (e.g., 10; 11; 7). The CPOT was developed by Gélinas et al. (2006) in French and shortly afterward translated into and validated in other languages. The CPOT has also been translated into Italian and validated in the Italian population. The Italian validation has been made in 2001 by Stefani, Nardon, Bonato, Modenese, Novello, and Ferrari. In their study, 50 nursing staff members from three different critical care settings of Vicenza Hospital administered the C-POT to 121 in patients, at rest and after usual nursing care activities. In addition, NOPPAIN forms were completed during care activities and communicative patients were asked to rate their pain using numerical rating scale 0-10. A good internal consistency and good levels of agreement between independent raters were observed (ρ=0.55 at rest and 0.66 during activity). Moderate correlations between C-POT and numerical rating scale 0-10, and between C-POT and NOPPAIN were found. Moreover, C-POT scores varied from rest to activities, and from non-painful to painful procedures. Their results, added to positive nurses’ evaluations, support the scale utility and use in the clinical setting (12).

Aim

A total of eight observational assessment tools were developed for ICU adults unable to self-report (11), and two of them: the Behavioral Pain Scale (BPS; 13), and the Critical-Care Pain Observation Tool (CPOT; 10) were recommended for clinical use in medical, surgical, and non-brain trauma ICU patients (14). Studies are still required to validate their use in brain-injured ICU patients. This study specifically aimed to test the reliability and validity of the Italian CPOT in a sample of brain-injured ICU adults. The CPOT was compared to the pain scales (NRS and PAINAD) already in use in the ICU taken into consideration in the current study.

Methods

Design and sample

A descriptive, comparative design was used. Approval from the Institutional Review Board (IRB) at the University of Parma and written consent and assent from patients and/or caregiver were obtained before study recruitment. A convenience sample of adult, both verbal and nonverbal, critical care patients were included. Participants were >18 years old, reported traumatic spinal cord and brain injuries, 33 of whom required mechanical ventilation, whilst 17 were self-ventilating. Patients excluded from the study were those who were discharged from the unit before 48 hours. An a priori power analysis, conducted with the software G*Power (15), indicated that we needed to have at least 10 subjects to have 80% power for detecting a medium-sized effect when employing the traditional .05 criterion of statistical significance (16). We collected 300 observations from 50 patients, 41 men and 9 women (mean age 42; SD=17.31).

Instruments

Critical-Care Pain Observation Tool (CPOT). Initially developed in French, the CPOT was translated into English using a back-to-back translation method. For the current study, we utilised the Italian version, that was validated by Stefani et al. (12).

The CPOT is based on four domains: facial expression, body movements, muscle tension, and compliance with the ventilation for intubated patients and vocalization for extubated patients (10). Patients are scored a 0, 1, or 2 in each of the four sections, giving an overall score of 0 (no pain) to 8 (maximum pain). According to the data reported by Gelinas et al. (10), the cut-off point is 2-3, while a score of >2 indicates the occurrence of pain. Descriptions are given to explain the expected behaviours for each increment, enabling consistent scoring within each domain. The CPOT has demonstrated interrater reliability with k coefficients ranging between 0.52 and 0.80 (10, 17). Content validity of the CPOT was ascertained with four physicians and 13 critical care nurses (10). Discrimination validity has been documented by higher CPOT scores recorded during noxious procedures, compared with lower scores during baseline (18).

Pain Assessment in Advanced Dementia (PAINAD). The PAINAD is a behaviour-observation tool developed for use in patients whose dementia is so advanced that they cannot verbally communicate the fact that they are in pain (19). The PAINAD has been found to be reliable and valid in hospitalized post orthopedic surgery older adults who are cognitively intact (20). The PAINAD was designed to assess pain by looking at five specific indicators: breathing, vocalization, facial expression, body language, and consolability. Total scores range from 0 to 10, with higher scores indicating more severe pain. Interrater reliability of the PAINAD indicates strong reliability across five studies (21). Pearson r ranges from 0.75 to 0.97, with most ranges over 0.80 (21). Discriminant validity of the PAINAD is documented by Wilcoxon signed ranks test revealing scores considerably higher during periods of likely pain than episodes of unlikely pain (20). Validity was also demonstrated by using quality improvement data to capture pain and improvement in pain scores. PAINAD scores preceding as-needed pain medication (6.7±1.8) and 30 minutes after administration of pain medication (1.8±2.2) were significant (p<.001) (21).

Numeric Rating Scale (NRS). Patients who were able to communicate (not only verbally), were also asked to rate their pain ‘‘on a scale from 0 to 10, with 0 equal to no pain and 10 equal to worst possible pain’’ using the NRS. The NRS is an 11-point scale for patient self-reporting of pain. It is for adults and children 10 years old or older. The NRS is a verbally administered scale that measures pain intensity (“how much pain do you feel right now?”). The NRS can also be used to measure pain unpleasantness (“how unpleasant horrible/yucky is the pain right now?”). The end points represent the extremes of the pain experience (for ex., 22).

Procedure

This study was conducted over a 6-months period at Montecatone Rehabilitation Institute, a hub in the north of Italy. Montecatone Rehabilitation Institute is a highly specialized hospital dedicated to the intensive rehabilitation of people affected by severe bone marrow failure or cerebral lesions with traumatic or atraumatic origin.

Each patient was assessed at three predefined times, starting usually 24 hrs since ICU admission, by a pair of evaluators: a nurse from the research group and a nurse from the institute. The nursing staff was informed on the CPOT scoring via a Power Point by one of the nurses from the research group two weeks before the beginning of the data collection period. Pairs of evaluators were not assigned or randomized but were established on a convenience basis. They were asked to assess the patient at rest (time 0), during mobilization for hygiene (nociceptive procedure) (time 1), and 20 minutes after these procedures (time 2). Results were recorded on a data collection form that included, in addition to the CPOT, the PAINAD, and the Numerical Rating Scale.

Data analysis

All data were analysed with SPSS version 20 (SPSS, Chicago, IL). Internal consistency was assessed with Cronbach’s coefficient α using the scores during hygiene procedures when the patient was most likely to be experiencing pain. Inter-rater reliability was assessed using Spearman coefficient. Spearman correlation coefficient was also used to examine the relationship between the CPOT, PAINAD and NRS scores in order to test construct validity. We hypothesized that a significant correlation would be found between the three scales scores seeing that they were supposed to measure the same concept (pain). The discriminant validation was examined by calculating within-patient differences in scores between the assessments on T0, T1, and T2, using a t-test. We hypothesised that if the CPOT really measures pain, the CPOT scores should be much higher during painful procedures than while the patient is at rest. Receiver operating characteristic (ROC) curves and the area under the ROC curves were calculated to illustrate the relationship between sensitivity and specificity of the CPOT and - as a further measure of discriminatory validity - to evaluate the probability of the CPOT in correctly identifying patients with controlled and non-controlled pain, as defined by the NRS score.

Results

The range of CPOT scores was 0 to 7 (Mean=1.65; SD=1.14) - scores can range from 0 to 8. Internal consistency reliability for the CPOT was .78 in the version for intubated patients and .86 in the version for extubated patients. The item that contributed less to the reliability in the version for intubated patients was the item 4 (compliance with the ventilation; α=.64). The item that contributed less to the reliability in the version for extubated patients was the item 3 (muscle tension; α=.71).

Table 1 shows correlation coefficients (Spearman) between the scores attributed by the pairs of evaluators during the three measurements. The correlation coefficients for each indicator of the CPOT were almost always significant and vary from ρ=.42 (p<.05) to ρ=.99 (p<.001). No significant correlations were found between the observers in the version for intubated patients at time 0 (at rest).

Table 1.

Inter-rater reliability (ρ di Spearman) for each CPOT measurement

	T0ρ (Spearman)	T1ρ (Spearman)	T2ρ (Spearman)
Total Score (intubated version)	n.s.	.90***	.42*
Facial Expression	n.s.	.68**	.46*
Body Movements	n.s.	.99***	.69**
Muscle Tension	n.s.	.62**	n.s.
Compliance with the ventilator	n.s.	.67**	n.s.
Total Score (extubated version)	.78**	.66**	.49*
Facial Expression	.55**	.58**	n.s.
Body Movements	.51*	n.s.	.51*
Muscle Tension	.63**	n.s.	.51*
Vocalization	n.s.	.50*	n.s.

0.001;

0.01;

0.05 (all 2-tails)

The range of PAINAD scores was 0 to 6 (Mean=.76; SD=.91), even though scores can range from 0 to 10. Low scores may be due to the fact that 22 patients (44%) out of the entire sample were unconscious. Ten patients (20%) out of the entire sample had been sedated in the last 1-hour. Twenty-three patients (46%) out of the entire sample received pain therapy with strong (18%), weak (14%) opioids, whilst the remainder received no opioid treatment (receiving instead different antalgic therapy). The correlation between the PAINAD and the CPOT was .85 (p<.01). The range of NRS scores was 0 to 10 (Mean=3.29; SD=2.74). The correlation between the NRS and the CPOT was moderate (ρ=.38, p<.01). However, 90% of patients that reported pain intensity >4, received a CPOT score >0. The correlation between the NRS and the PAINAD was .67 (p<.001).

As evident from Figure 1, there was a statistically significant difference between CPOT scores when the patients were at rest (time 0: Mean=.33; SD=.62), and during mobilization for hygiene (nociceptive procedure) (time 1: Mean=2.65; SD=1.84) (t=-13.12, p<.001). Furthermore, a statistically significant difference was found between time 1 and time 2: 20 minutes after the nociceptive procedures (Mean=.60; SD=.95) (t=12.63, p>.001).

Figure 1.

Within-patient differences in CPOT scores between the assessments on T0, T1, and T2 (t-test)

The discriminant validation was further examined by calculating Receiver Operating Characteristic (ROC) curves. The area under the ROC curve was .86 (p<.001; SE.=.054; 95% CI [.76; .97]), including moderate diagnostic accuracy of the CPOT for the identification of critical pain (Figure 2) – an AUC range between .8 and .9 is classified as “Good” (23). For a CPOT cut-off score equal or greater than 5, sensitivity and specificity were 1 and 0.750, respectively.

Figure 2.

Receiver operating characteristic curves for CPOT scores

We wanted to know whether the PAINAD – currently used at Montecatone Rehabilitation Institute - could have been substituted by the CPOT in the use with patients reporting traumatic spinal cord and brain injuries, seeing that the PAINAD was specifically developed for use in patients with dementia. However, in our sample, the PAINAD resulted more accurate: its ROC curve, rising high above the diagonal to the left more than the one designated by the CPOT scores, is indicative of higher discrimination (AUC = .98; p = 0.001; SE=.020; 95% CI [.94;.1.02]) (Figure 3) – an AUC range between .9 and 1.0 is classified as “Excellent” (23). Moreover, the PAINAD confidence interval included the CPOT confidence interval.

Figure 3.

Receiver operating characteristic curves for PAINAD scores

The AUCs were compared via a z test (Hanley and McNeil method; 24). Z value (6.08) far exceeding the critical value of 1.96 required for statistical significance at an alpha of 0.05. Hence, the PAINAD performed better than the CPOT (p<.01) in this sample.

Discussion

While a previous study has established the reliability and validity of the Italian CPOT to assess pain in critically ill patients (12), it has not been validated for use in brain-injured ICU adults, which was the purpose of this study.

Internal consistency and reliability for the CPOT were high in both the version for intubated and extubated patients. The item that contributed less to the reliability in the version for intubated patients was the item 4: compliance with the ventilation; the item that contributed less to the reliability in the version for extubated patients was the item 3: muscle tension. These results are similar to those of Stefani et al. (12). These results are reasonably due to the fact that muscle rigidity may be less useful for the assessment of pain in this kind of patients, which often report paraplegia and tetraplegia.

Good correlation was obtained between independent observers during the painful procedure (mobilization for hygiene), establishing interrater reliability of the CPOT scores between trained observers who were previously new to the tool use. On the other hand, a non-existent or moderate relation was found during the rest period, in the version for intubated patients. This may suggest that the CPOT is more reliable when assessing pain in patients during procedures when they are more likely to express pain-related behaviours.

Criterion validation was supported by a strong correlation between CPOT and PAINAD scores. Furthermore, it was confirmed by moderate positive associations between the CPOT scores and the patients’ self-reports of pain - as expressed by NRS scores. Considering that a behavioural scale such as the CPOT and the self-report of pain are measuring two distinct dimensions of the pain experience (i.e., behavioural and sensorial dimensions) (25), moderate rather than high correlations were expected. These results confirm the results of a recent study by Boitor, Lachance Fiola, Gelinas (26).

This study revealed significant changes in behaviours during painful procedure versus rest as reflected by the differences in CPOT scores during each condition. This supports the hypothesis that the CPOT is readily able to discriminate between patients undergoing painful versus non-painful procedures in brain-injured ICU adults. However, PAINAD performed better in this sample, as revealed by the comparison between the two AUC of ROC curves. This result may be due to the fact that the PAINAD was already in use in the ICU in exam, hence the nurses might have performed better in the scoring. A more accurate training on the use of the CPOT and the comparison with data from different ICUs might confirm or disconfirm this hypothesis.

Other limitations to this study may be the fact that the observers were not blinded to the painful procedure and may have expected and, ultimately, perceived more intensive behavioural reactions. Moreover, observers were not blinded to patients’ self-reports of pain at the time of CPOT scoring.

Although anecdotal data indicated that nurses had found the CPOT easy to utilise, in this study, a feedback questionnaire from the nurses who served as observers was not contemplated. Finally, although results are based on a good sample size (as suggested by the result of the power analysis), future larger scale studies in different hospitals are needed to strengthen and confirm the conclusions of this study.

In conclusion, the CPOT was found to be highly sensitive for and correlated positively with self-reported pain, but was not found to be very specific in this patient population. Muscle rigidity appeared to be less useful for the assessment of pain in this group and may require further scale revision and testing for optimal use. Overall, the CPOT was found to be reliable and valid for use in this patient group, confirming the results of a recent study by Joffe, McNulty, Boitor, Marsh, and Gélinas (27), which tested the validity and reliability of the CPOT in a similar sample. These results have provided new evidence fulfilling an important gap highlighted in the pain practice guidelines of the Society of Critical Care Medicine (14).