Validity of the Rotterdam Elderly Pain Observation Scale for institutionalised cognitively impaired Dutch adults.

BACKGROUND: The Rotterdam Elderly Pain Observation Scale (REPOS) has not yet been validated for institutionalised cognitively impaired adults. To fill this gap of knowledge, we tested psychometric properties of the REPOS when used for pain assessment in this population.
METHODS: In this multicentre observational study, residents were filmed during a possibly painful moment and at rest. Healthcare professionals were asked to rate residents' pain by means of a Numeric Rating Scale (NRS)-proxy. Two researchers assessed pain with the REPOS and the Chronic Pain Scale for Non Verbal Adults with Intellectual Disabilities (CPS-NAID) from video-recordings.
RESULTS: In total, 168 observations from 84 residents were assessed. Inter-observer reliability between the two researchers was good, with Cohen's kappa 0.72 [95% confidence interval (CI) 0.64 to 0.79]. Correlation between the REPOS and CPS-NAID for a possibly painful moment was 0.73 (95% CI 0.65 to 0.79). Sensitivity (85%) and specificity (61%) for the detection of pain were calculated with REPOS ≥ 3 and NRS ≥ 4 as a reference value. Item response theory analysis shows that the item grimace displayed perfect discrimination between residents with and without pain.
CONCLUSION: The REPOS is a reliable and valid instrument to assess pain in cognitively impaired individuals.

Introduction

Pain management in institutionalised adults with cognitive impairments receives little attention internationally. Chronic pain is the biggest concern and its assessment deserves to be prioritised. Many people with a cognitive impairment also suffer from painful conditions such as musculoskeletal disorders, arthritis or severe spasticity associated with contractures or joint dislocations (de Knegt and Scherder 2010; van der Putten and Vlaskamp 2011; Boerlage et al. 2013). Due to their intellectual and communication incapability, they are often unable to communicate possible pain (Defrin et al. 2015; Barney et al. 2020).

The reported chronic pain prevalence for cognitively impaired adults are 13% and 18% (de Knegt and Scherder 2010; van der Putten and Vlaskamp 2011; Walsh et al. 2011; Boerlage et al. 2013) versus 85% for cognitively impaired children (Breau et al. 2003; Massaro et al. 2013). A possible explanation for this discrepancy is that studies in adults rely even more than in children on proxy report of health providers, which tends to be unreliable (Horgas and Dunn 2001).

Observational pain assessment is considered a valuable alternative to self‐report (Herr et al. 2019). A number of validated pain observation scales have been developed for children (Voepel‐Lewis et al. 2002; Breau et al. 2002a; Breau et al. 2002b; Hunt et al. 2004; Terstegen et al. 2004; Duivenvoorden et al. 2006), and adults with a cognitive impairment (Zwakhalen et al. 2006; Burkitt et al. 2009; Van Herk et al. 2009), most of which, however, target short procedural pain, for example, influenza vaccination (Meir et al. 2012; de Knegt et al. 2013). Assessment of chronic pain is complicated by the fact that the associated behaviour differs between individuals and also differs from acute pain behaviour.

In 2008, we developed and validated the Rotterdam Elderly Pain Observation Scale (REPOS; see Appendix S1) for chronic and sub‐acute pain assessment of nursing homes residents and hospital patients who were unable to communicate pain by self‐report (Van Herk et al. 2009; Boerlage et al. 2019).

Individuals with a cognitive impairment may differ from nursing home residents in many aspects though. They are in general younger and show different behaviour to painful stimuli. Many suffer from behavioural problems and painful co‐morbidities such as spasticity and gastro‐intestinal reflux (van der Putten and Vlaskamp 2011). Some seem indifferent to pain, whereas others will react with laughing where crying was expected. Pain assessment becomes even more problematic if they suffer from autism spectrum disorder (ASD) as well (Allely 2013). Individuals with ASD suffer from social communication deficits and often show inflexible repetitive behaviour, which is in 70% of all cases associated with sensory‐perceptual anomalies (American Psychiatric Association 2013). As a result, they may seem insensitive to pain. Still, studies that included monitoring of heart rate and endorphin levels showed increases in these parameters suggestive of pain sensitivity (Allely 2013). Measuring these parameters is hardly feasible in clinical practice – which leaves us with pain observation scales.

Caregivers of Dutch institutions for adults with an intellectual disability have started to use the REPOS, although its validity and reliability for this group have yet not been established. The primary aim of this study was therefore to test the reliability (internal consistency, reliability and measurement error), validity (construct validity, structural validity and cross‐cultural validity) and responsiveness of the REPOS as a pain observation scale for institutionalised adults with an intellectual disability. The secondary aims were to calculate the optimal cut‐off value of the REPOS for the detection of pain and to verify that the REPOS is also valid and reliable in intellectually disabled individuals with ASD.

Participants and methods

Design

This was a multicentre prospective observational study running from April 2016 until December 2018. The study was initiated and supervised by the principal investigator (A. B.) from the Erasmus University Medical Center in Rotterdam. Subjects were recruited at 10 locations of eight long‐term institutions for cognitively impaired adults located in five different parts of the Netherlands. This study had been approved by the Erasmus MC ethics review board (MEC‐2015‐588) and by the local boards of directors of the eight institutions.

The study protocol followed the consensus‐based standards for the selection of health based measurements instruments (COSMIN) checklist (Mokkink et al. 2010).

Residents

All institutionalised adult residents of the participating centres suspected to suffer from chronic or sub‐acute pain were suitable for inclusion. The COSMIN guidelines for validation studies indicated that we needed to include at least between 50 and 99 subjects in order to assure that conclusions about the reliability of the instrument under investigation are justly drawn.

Procedure

For each location, a health professional with an interest in the project was appointed as contact person. These contact persons received all available written information about the study including instructions for the process if video recording. They informed their colleagues, searched for residents that met the inclusion criteria, asked the legal representatives for informed consent and took care of the video recording.

After informed consent had been obtained from a resident's legal representative, the resident was filmed during one possibly painful activity (i.e. physiotherapy, transfer, dressing or bathing) and during one restful situation when pain was not likely to be present or less intense. For both occasions, the caregiver assigned a Numeric Rating Scale (NRS) score as a proxy score. If someone who knew the resident well was present (e.g. a relative), he or she completed the individualised NRS (Solodiuk et al. 2010). These results will be presented elsewhere.

To determine the construct validity, two observers (A. B. and L. S.) independently from one another viewed the video recordings and in alternating order applied the REPOS and the Chronic Pain Scale for Non Verbal Adults with Intellectual Disabilities (CPS‐NAID). Intra‐observer reliability was established for one observer (A. B.) by comparing her present assessments of the video‐recordings with assessment of the same video‐recordings at least 12 months later.

Data collection

Information about the participants' age, sex, aetiology of the cognitive impairment (if known), aetiology of pain, presence of ASD, presence of dementia, possible palliative care trajectory, medical history, analgesic prescription and co‐medication prescription was collected from the medical records.

Instruments

The NRS is a validated global pain rating scale from 0 = no pain to 10 = worst pain possible (Jensen and McFarland 1993). NRS scores of 4 and higher indicate substantial pain that should be treated. The NRS can be used for self‐report of pain and for proxy report (Jensen 2003; von Baeyer and Spagrud 2007; Hadjistavropoulos et al. 2014). In the current study, the NRS was applied as a proxy instrument only.

The REPOS has been validated for chronic and sub‐acute pain in non‐communicative adults and cognitive impaired elderly (Van Herk et al. 2009). It consists of 10 items describing a behaviour (relating to facial expression, emotional status, motor behaviour and vocalisation), which an observer scores as absent (0) or present (1) after a 2 min observation period. Total scores range from 0 to 10. The REPOS is always used in combination with the expert opinion of an NRS‐observer. This expert opinion will consider patient‐related and environmental characteristics with the help of the step‐by‐step REPOS decision tree (see supporting information). The validation study found significant differences in REPOS scores between painful and rest situations and found a large correlation between the REPOS and the Pain Assessment In Advanced Dementia scale (PAINAD) (r = 0.75), indicating good validity (Cohen 1988). For nursing home residents, a cut‐off score of 3 had the highest differential qualities with a good sensitivity (0.85) and specificity (0.83). A REPOS score of 3 or higher combined with an observer NRS rating of 4 or higher suggests moderate to severe pain (Van Herk et al. 2009). Two further studies confirmed good validity of the REPOS for palliative care patients (Masman et al. 2018) and for non‐communicative hospital patients (Boerlage et al. 2019).

The CPS‐NAID (Burkitt et al. 2009) has been validated for adults with a severe or profound intellectual disability. It consists of 24 behavioural items, scored from not present (0) up to very often present (3), with a total score between zero and 72. The observation period is 5 min, and a score of 10 or higher indicates that a person is in pain (Burkitt et al. 2009).

Data analysis

Normally distributed data are presented as mean and standard deviation (SD) and non‐normally distributed data as median and interquartile range (IQR). An independent samples t‐test was used to compare outcomes between the two possible orders of observation (REPOS first and CPS‐NAID second or vice versa), to evaluate whether the order of observation affected the outcome results. The test was not statistically significant (P = 0.16), which suggests that using an alternating order of the two pain observation scales has no confounding effect on the results. Inter‐observer and intra‐observer reliability of the REPOS items was assessed with the unweighted Cohen's kappa, and a value of 0.65 or higher (Cohen 1988) was considered to indicate good reliability. Inter‐observer and intra‐observer reliability for the REPOS total score was assessed using the intraclass correlation coefficient with 95% confidence intervals, using a two‐way mixed effects model based on absolute agreement, single measures. Internal consistency of the REPOS was calculated with Cronbach's alpha. A value of Cronbach's alpha 0.70 or higher indicates good internal consistency (De Vet et al. 2011).

The REPOS was previously shown to be a one‐dimensional scale for the non‐communicative adults and cognitive impaired elderly (Van Herk et al. 2009). To verify the unidimensionality of the REPOS for institutionalised cognitively impaired adults, we performed a principal component analysis of a tetrachoric correlation matrix (i.e. a matrix of correlations that takes into account the dichotomous nature of the items) of the REPOS items. The dimensionality of the scale was assessed by inspecting a scree plot. The calculations were performed using the functions ‘tetrachoric’ and ‘principal’ in the ‘psych’ package in R.

A two‐parameter logistic item response theory (2PL‐IRT) model was used to describe the measurement performance over all measurements and of each item of the REPOS for measuring pain, where presence of pain was defined as an NRS score of 4 or greater. Item response theory is a class of statistical methods that places patients and items on the same latent scale, which in this case corresponds with the severity of pain (Terwee et al. 2007). For each item, this statistical method estimates a threshold parameter and a discrimination parameter. The threshold parameter describes the severity of pain associated with an item, with higher values implying that an item is observed only in patients with more severe pain. The discrimination parameter describes the ability of an item to differentiate between patients who have pain scores below and above this threshold. To verify that the 2PL‐IRT model was appropriate for our data, a one‐parameter IRT model was compared with the 2PL‐IRT model using a likelihood ratio test.

Due to the lack of a gold standard, concurrent validity between the REPOS and the NRS‐proxy and between the REPOS and the CPS‐NAID was calculated with the use of Pearson correlation coefficients with 95% confidence intervals (95% CI). According to the COSMIN method, construct validity relies on testing hypotheses on the expected correlations between instruments when a gold standard is missing. We hypothesised that the responsiveness determined with the help of the Pearson correlation coefficients between change scores (scores in pain minus scores in rest in the same patients) of REPOS, CPS‐NAID and NRS‐proxy scores should be at least 0.70.

The sensitivity and specificity of all REPOS assessments were calculated with receiver operating characteristic (ROC) curve analysis, with NRS‐proxy equal to 4 or higher as reference value. The optimal cut‐off value of the REPOS was chosen by optimising the Youden index (i.e. sensitivity plus specificity minus 1). In case of repeated measurements, the ROC curve and 2PL‐IRT analyses were performed without correction for within‐subject dependence among observations and without reporting measures for uncertainty.

Because we were interested to find out whether a REPOS observation can help to establish the presence of pain in individuals with ASD or Down syndrome, we checked the reliability and validity of our observations separately in the subgroup of 15 residents with ASD and 12 with Down syndrome.

Item response analysis was performed with Stata version 15.64 and principal components analysis with the package psych in R, and all other statistical analyses were carried out with IBM SPSS version 25. All statistical tests were two‐sided, with a P‐value of 0.05 considered statistically significant.

Results

Ninety‐one legal representatives gave consent and we included 84 individuals with an ID of whom video recordings were available. Sixty‐one were male (73%), and the median age was 58 (IQR 45 to 68) years. Of 23 residents (27.4%), the aetiology of the intellectual disability was unknown; 15 (17.9%) suffered from (perinatal) hypoxic ischemia, and 12 had Down syndrome (14.3%) (see further details in Table 1). Co‐morbidities were the following: 15 residents (17.9%) suffered from ASD; nine others (10.7%) had been diagnosed as having dementia. Seven (8.3%) residents received palliative care.

Table 1

Study population characteristics (n = 84)

Male N (%)	61 (72.6)
Median age (IQR) in years	58 (45 to 67.8)
Aetiology of intellectual disability	N (%)
Unknown	23 (27.4)
(Perinatal) hypoxic ischemia	15 (17.9)
Down syndrome	12 (14.3)
Posttraumatic brain injury	9 (10.7)
Developmental anomalies of the CNS	8 (9.5)
Chromosomal anomalies	8 (9.5)
Post infectious encephalopathy	6 (7.1)
Metabolic disorder	2 (2.4)
Neurodegenerative disorder	1 (1.2)
Aetiology of pain	N (%)
Unknown	24 (28.6)
Skeletal pain	28 (33.3)
Physical decline	6 (7.1)
Intestinal pain	4 (4.8)
Contractures	3 (3.6)
Neurodegenerative	3 (3.6)
Malign pain	2 (2.4)
Urogenital pain	2 (2.4)
Auto‐mutilation	2 (2.4)
Other causes	10 (11.9)

The health professionals did not know what caused the pain for 24 (28.6%) residents. Table 1 gives an overview of the known causes for pain.

In total, 168 REPOS observations combined with a NRS‐proxy score were available for analysis, of which 84 during a possibly painful moment and 84 in rest.

The median NRS‐proxy during a painful moment for all residents was 4.0 (IQR 2.0 to 6.0), and in rest 0 (0 to 1.0). The REPOS score during a painful moment was 5.5 (IQR 4.0 to 6.0), and in rest 1.0 (IQR 0 to 2.0) (see Table 2). Based on a REPOS ≥ 3 and NRS‐proxy ≥ 4, 41 (48.8%) residents suffered from pain during the possibly painful moment in question.

Table 2

Pain scores, total group and subgroup residents with ASD

	Total group	ASD
	N = 84	N = 15
	Median (IQR)	Medan (IQR)
NRS‐proxy painful moment	4.0 (2.0 to 6.0)	4.0 (2.0 to 7.0)
NRS‐proxy rest	0 (0 to 1.0)	0 (0 to 1.0)
REPOS total score painful moment	5.5 (4.0 to 6.0)	6.0 (4.0 to 6.0)
REPOS total score rest	1.0 (0 to 2.0)	2.0 (0 to 2.0)
CPS‐NAID painful moment	8.0 (4.3 to 10)	8.0 (5.0 to 11.0)
CPS‐NAID rest	1.0 (0 to 3.0)	2.0 (0 to 5.0)

ASD, autism spectrum disorder.

Of these 41 residents with pain, 145 (37%) had no analgesic prescription, and 22 (54%) received one or more non‐opioids prescribed for pain treatment, that is, paracetamol (acetaminophen) and non‐steroidal anti‐inflammatory drugs (NSAIDs). Data S1 gives an overview of all analgesics and co‐medications prescribed to all residents and those with pain specifically. More than one prescription per resident is possible.

Reliability

Inter‐observer reliability for the REPOS between the researchers was good, with Cohen's kappa 0.72 (95% CI 0.64 to 0.79). The intraclass correlation (ICC) for the total REPOS score (A. B. and L. S.) was 0.96 (95% CI 0.95 to 0.97) and for the NRS‐proxy 0.72 (95% CI 0.63 to 0.78). Intra‐observer reliability (REPOS scored by A. B.) was good with Cohen's kappa 0.89 (95% CI 0.85 to 0.92).

Item response analysis

The prevalence of items scored as present and the results of the IRT analysis are given in Table 3.

Table 3

REPOS items scored as present and scored as present, discriminative parameter and threshold parameter from item response theory analysis

REPOS items	N = 168	2PL‐IRT analysis N = 168
	N scored (%) as present	Discrimination parameter †	Threshold parameter ‡
Tense face	125 (74.4)	2.81	−0.82
Raising upper lip	90 (53.6)	4.60	−0.15
Moving body parts	76 (45.2)	1.57	0.12
Eyes (almost) squeezed	74 (44.0)	8.79	0.09
Sounds of restlessness/verbal expressions	64 (38.1)	1.34	0.43
Grimace	58 (34.5)	62.01	0.38
Breath holding/faltering respiration	29 (17.3)	1.41	1.42
Panicky, panics attack	10 (6.0)	0.77	3.34
Moaning/groaning	9 (5.4)	0.64	4.73
Frightened, fearful look	2 (1.2)	2.54	2.71

Discrimination parameter: severity of pain associated with item, that is, higher values imply that an item is observed only in patients with more severe pain.

Threshold parameter: negative values indicate that these items are seen frequently in patients without pain as well.

2PL‐IRT, two parameter item response theory model.

The item ‘tense face’ was scored as present 125 (74.4%) times over all 168 observations. The item ‘raising upper lip’ was scored 90 (53.6%) times, whereas the item ‘frightened fearful look’ was scored only twice (1.2%) (see Table 3).

The unidimensionality of the REPOS was confirmed by means of principal component analysis of a tetrachoric correlation matrix of the REPOS items. The scree plot shows that the first dimension explains 54% of the variance, while the loading plot shows that all items scored positive on the first dimension. These results were considered a verification of the unidimensionality of the REPOS for institutionalised cognitively impaired adults.

A comparison between a one parameter IRT‐model and the 2PL‐IRT test using a likelihood ratio test gave a chi‐square value of 88.31 (df = 9) and P < 0.001, which shows that the two‐parameter model is preferred over the one‐parameter model.

The highest discrimination parameter of the 2PL‐IRT analysis was seen for the item ‘grimace’ displayed perfect discrimination between residents with and without pain in the observed data, so that no estimate of the discrimination parameter could be calculated. The discrimination parameters for the items ‘eyes (almost) squeezed’ and ‘raising upper lip’ as well as ‘frightened fearful look’, ‘moving body parts’, ‘breath holding/faltering respiration’ and ‘sounds of restlessness’ were also found discriminative and ranged from 1.34 to 8.79. The items ‘tense face’ and the item ‘raising upper lip’ had the lowest threshold parameters, respectively −0.82 and −0.15, indicating that these items were also seen when it was not likely that the resident was in pain. The items ‘moaning/groaning’ (4.73), ‘panicky/panics attack’ (3.34) and ‘frightened fearful look’ (2.71) had the highest threshold parameters, which indicates that these items were seen when the resident was in pain.

The internal consistency of the REPOS over all 168 measurements was good with a Cronbach's alpha of 0.81.

Figure 1 shows the item‐characteristic curves of all measurements. Each curve represents a behaviour of the REPOS and items with steeply ascending lines that are closest to the median of the probability (theta) have the highest discriminative value.

Figure 1

Item characteristic curves of the two‐parameter logistic item response theory analysis for all measurements. The value on the y‐axis is the probability (Pr) that a behavioural item will be scored, and theta represents the latent scale (severity of pain). REPOS per item. [Colour figure can be viewed at wileyonlinelibrary.com]

Concurrent validity and responsiveness

The correlation between the total REPOS score assigned through video observation and the NRS‐proxy was 0.55. The correlation between the total REPOS score and the CPS‐NAID during a painful moment was 0.73 (95% CI 0.65 to 0.79) and for rest 0.77 (95% CI 0.66 to 0.84).

The ability of the REPOS to detect change over time (responsiveness) expressed by the correlation between the change scores of REPOS total and CPS‐NAID was 0.66 (95% CI 0.57 to 0.74).

The area under the curve (AUC) of the ROC curve of the REPOS total, with NRS‐proxy as reference, for all measurements was 0.79. The sensitivity and specificity of the REPOS total, using an optimal cut‐off of the REPOS of ≥3, were respectively 85% and 61%. The positive predictive value was 47%, and the negative predictive value 91%. Figure 2 shows the ROC curves for all measurements.

Figure 2

ROC curve for all measurements and ROC curve for the measurements in ASD residents. [Colour figure can be viewed at wileyonlinelibrary.com]

Residents diagnosed with autism spectrum disorder

Eight of these 15 residents diagnosed with ASD suffered from pain (based on an REPOS ≥ 3 and NRS proxy ≥ 4). There are small (statistically not significant) differences in the REPOS scores of the total group and the residents with ASD. The median score for painful moment is 0.5, and the median score during rest is 1 point higher. For the median of the CPS‐NAID scores, the difference is only seen during rest with a 1‐point higher score in the residents with ASD. Their pain scores are presented in Table 2.

Item response analysis in residents with autism spectrum disorder

Subgroup analysis of the 30 observations of residents with ASD showed similar percentages of positively scored items, that is, ‘tense face’ 24 times scored (80%), ‘raising upper lip’ 18 times (60.0%), and ‘frightened fearful look’, ‘panicky, panic attack’ and ‘moaning, groaning’ scored as present only once (3.3%).

The outcomes of the 2PL‐IRT analysis (i.e. discrimination and threshold parameters) were similar between the 30 observations of residents with ASD and the 168 observations in the entire sample. Table 4 presents the results of the subgroup analyses based on the prevalence per item scored as present, the threshold and discrimination parameters estimated with the 2PL‐IRT analysis. Figure 3 shows the item‐characteristic curves of all measurements of the ASD residents.

Table 4

Residents with ASD: REPOS items scored as present and scored as present, discriminative parameter and threshold parameter from item response theory analysis

Behavioural items	N = 30	2PL‐IRT analysis N = 30
	N positively scored (%)	Discrimination parameter †	Threshold parameter ‡
Tense face	24 (80.0)	2.81	−0.82
Raising upper lip	18 (60.0)	4.60	−0.15
Moving body parts	16 (53.3)	1.57	0.12
Sounds of restlessness/verbal expressions	14 (46.7)	1.34	0.43
Eyes (almost) squeezed	12 (40.0)	8.79	0.09
Grimace	12 (40.0)	62.01	0.38
Breath holding/faltering respiration	4 (13.3)	1.41	1.42
Panicky, panics attack	1 (3.3)	0.77	3.84
Moaning/groaning	1 (3.3)	0.64	4.73
Frightened, fearful look	1 (3.3)	2.54	2.71

Discrimination parameter: severity of pain associated with item, that is, higher values imply that an item is observed only in patients with more severe pain.

Threshold parameter: negative values indicate that these items are seen frequently in patients without pain as well.

ASD, autism spectrum disorder.

Figure 3

Item characteristic curves of the two‐parameter logistic item response theory analysis for the 30 measurements in residents with ASD. The value on the y‐axis is the probability (Pr) that a behavioural item will be scored, and theta represents the latent scale (severity of pain). REPOS per item. [Colour figure can be viewed at wileyonlinelibrary.com]

Concurrent validity and responsiveness in residents with autism spectrum disorder

The correlation between the REPOS and the NRS‐proxy was 0.64 (95% CI 0.37 to 0.81). The correlation between the total REPOS and the CPS‐NAID for pain was 0.85 (95% CI 0.71 to 0.93).

Responsiveness in terms of the correlation between the change scores of REPOS total and CPS‐NAID was 0.78 (95% CI 0.45 to 0.92).

The AUC of the ROC curve of the REPOS total score for all measurements was 0.79. Based on Youden's index a cut‐off for the REPOS total score of 3 or higher was chosen, for which the sensitivity was 80%, the specificity 65%, the positive predicted value 53% and the negative predictive value 87%. Figure 2 shows the ROC curves for the measurements of the residents with ASD.

The internal consistency of the REPOS over all 30 measurements was good with a Cronbach's alpha 0.79.

Residents diagnosed with Down syndrome

Five of the 12 residents diagnosed with Down syndrome suffered from pain (based on an REPOS ≥ 3 and NRS proxy ≥ 4). The median NRS‐proxy score for these five residents during a painful moment was 5.0 (IQR 4.5 to 6.5) and during rest 0 (0 to 2.0); the corresponding REPOS scores were respectively 5.0 (IQR 5.0 to 7.0 and 1.0 (IQR 1.0 to 2.5).

The correlation between the REPOS and the NRS‐proxy for 24 measurements was 0.66 (95% CI 0.35 to 0.84). The correlation between the total REPOS and the CPS‐NAID for pain was 0.91 (95% CI 0.80 to 0.96).

The responsiveness of the REPOS, as reflected by the correlation between the change scores of the REPOS total score and CPS‐NAID, was 0.93 (95% CI 0.84 to 0.97).

The AUC of the ROC curve of the REPOS total score was 0.91. Based on Youden's index, for the REPOS total score, we chose a cut‐off of 3 or higher, for which the sensitivity was 100%, the specificity 63%, the positive predicted value 42% and the negative predictive value 100%.

The internal consistency of the REPOS for all 24 measurements was good, as reflected by a Cronbach's alpha 0.75.

Discussion

This multicentre, prospective observational study confirms that the REPOS is a reliable and valid pain observation scale for application in institutionalised adults with a cognitive impairment. Reliability analysis revealed a good interrater and intra‐rater reliability and internal consistency. The REPOS was able to distinguish between pain and no pain and had a good correlation with CPS‐NAID scores. Yet the correlation with NRS‐proxy scores was moderate (see discussion below). ROC curve analysis confirmed that a REPOS score of 3 or higher indicates the presence of pain. The cut‐off value of 3 also holds for nursing home residents, hospital patients and palliative care patients alike (Van Herk et al. 2009; Masman et al. 2018; Boerlage et al. 2019).

For this study population, the unidimensionality was of the REPOS was confirmed. The item response analysis revealed that, like in hospital patients, grimacing most probably indicates pain (Boerlage et al. 2019) in adults with cognitive impairment. The behaviour ‘eyes are (almost) squeezed’ was often seen when a resident was not in pain as well. The fact that behaviours ‘tense face’ and ‘raising upper lip’ were seen during possibly painful as well as non‐painful situations might be explained by the increased facial activity that many cognitively impaired adults display, such as tics. If these are the only two positively scored items, it is most likely that the resident is not in pain.

The accuracy of the REPOS cut‐off score 3 or higher was confirmed by the good sensitivity (80%) and moderate specificity (61%).

We hypothesised that acceptable responsiveness – defined as ‘the ability of an instrument to detect change over time in the construct to be measured’ – was set at a correlation of at least 0.70. The Pearson correlation coefficient for both the difference between pain and no pain (change over time) of the REPOS, and that between pain and no pain of the CPS‐NAID was 0.66, which was somewhat lower than hypothesised, but is still considered acceptable.

The lower than expected correlation we found between the REPOS and the NRS‐proxy necessitates some discussion. A possible explanation is that most caregivers found it hard to assign an NRS‐proxy score. Most of the caregivers for cognitively impaired individuals in the Netherlands have a social pedagogic background and not a medical or nursing background (Jackson 2006; Boerlage et al. 2013). They have probably never been instructed on behaviours that might be indicative for the presence of pain. Nevertheless, even for caregivers with a medical or nursing training it is difficult to establish whether or not a cognitively impaired person is in pain. Parents or siblings might be able to help because they have known the resident for a longer time and under different circumstances. With ageing, the risk of painful conditions increases, and parents or siblings are less available, which makes it even more difficult to establish the presence of pain (McGuire and Kennedy 2013; Findlay et al. 2015). Some professional care providers might even be less perceptive for pain in the residents they care for because they believe that non‐communication of pain suggests the absence of pain (Breau et al. 2003; Rothschild et al. 2019). These are possible explanations for the moderate correlation between the estimated pain intensity by the caregivers and the more objective observation with the use of the REPOS by the principal investigator. In general, it is recommended to educate health professionals of cognitively impaired adults and children about pain, pain behaviour, pain treatment, and how to use pain observation scales. To support future REPOS users to achieve good interrater reliability we have developed an e‐module with information about how to score video recordings of patients with cognitive impairments (www.comfortassessment.nl/reposscale).

Individuals with autism spectrum disorder

Knowledge about the pain perception and pain behaviour of individuals with ASD is scarce, and professional opinions on this subject differ. Some of these professionals doubted whether individuals with ASD are sensitive to pain at all, while others suggest a higher pain threshold (Allely 2013). There are indications that self‐injurious behaviour, which is often seen in individuals with CI who suffer from ASD as well, might be related to untreated pain (Summers et al. 2017). Allely et al. found that endorphin levels and physiologic reactions increased as a reaction to painful stimuli in individuals with ASD (Allely 2013). We wondered whether a REPOS observation can help to establish the presence of pain in these individuals. We therefore calculated the reliability and validity coefficients in the subgroup of 15 residents with ASD separately.

The results of these subgroup analyses were comparable to the results from the total group although we need to be cautious because of the low sample size.

We performed a subgroup analysis of the measurement properties of the REPOS in residents with Down syndrome (n = 12). The analyses revealed that the properties were comparable with those of the total group. To our knowledge, adult residents with Down syndrome have not been studied before in the context of observational pain instruments.

Strengths and limitations

The fact that this study ran in many different institutes in the Netherlands increases the generalisability; that is, based on these results, the REPOS can be used in all Dutch institutes for cognitively impaired adults.

A possible limitation could be that the principal investigator did not receive any information about residents that might have been eligible but were missed or considered too fragile by the health professional. Although we assume that, because the study ran in many institutes, this has little influence on the results, there is a small possibility that by excluding fragile residents might have resulted in selection bias.

Another possible limitation is that data were not collected by researchers with expertise in pain assessment, which could have impaired uniformity of the video recordings. The analyses in the small subgroup of residents with ASD are promising.

A major problem with pain observations is the lack of a solid gold standard. We compared the REPOS with the NRS‐proxy as well as the CPS‐NAID (Horgas and Dunn 2001; Seers et al. 2018). Although the NRS‐proxy score carries a risk for underestimation or overestimation of a patient's pain, we considered it the best instrument to estimate pain intensity in a research setting.

Conclusion

The REPOS is a reliable and valid instrument to assess pain in cognitively impaired individuals and promising in cognitively impaired individuals who suffer from ASD as well. Further study in a larger group is necessary to confirm the validity and reliability of the REPOS for the assessment of cognitively impaired residents with ASD.

Source of Funding

This study was conducted with financial support of the Foundation for the promotion of Social Pedagogical Care (Stichting SPZ).

Conflict of Interest

None of the authors have a conflict of interest to declare.

Data S1. Supporting Information

Click here for additional data file.