The DEMS-DOSS study: validating a delirium monitoring tool in hospitalised older adults.

OBJECTIVE: to evaluate the sensitivity, specificity and test-retest reliability of the Delirium Early Monitoring System-Delirium Observation Screening Scale (DEMS-DOSS).
DESIGN: prospective diagnostic accuracy study of a convenience sample of admitted older adults with DEMS-DOSS and reference standard assessments.
SETTING: 60-bed aged care precinct at a metropolitan hospital in Sydney, Australia. PARTICIPANTS: 156 patients (aged ≥65 years old) were recruited to participate between April 2018 and March 2020. One hundred participants were included in the analysis. MEASUREMENTS: Participants were scored on the DEMS-DOSS. Trained senior aged care nurses conducted a standardised clinical interview based on the Diagnostic and Statistical Manual of Mental Disorder (DSM)-IV delirium criteria, within two hours of DEMS-DOSS completion. The senior aged care nurse undertaking the DSM-IV interview was blinded to the results of the DEMS-DOSS.
RESULTS: Participants' mean age was 84 (SD ±7.3) years and 39% (n = 39) had a documented diagnosis of dementia. Delirium was detected in 38% (n = 38) according to the reference standard. The DEMS-DOSS had a sensitivity of 76.3% and a specificity of 75.8% for delirium. The area under the receiver operating characteristics curve for delirium was 0.76. The test-retest reliability of the DEMS-DOSS was found to be high (r = 0.915).
CONCLUSION: DEMS-DOSS is a sensitive and specific tool to assist with monitoring new onset and established delirium in hospitalised older adults. Further studies are required to evaluate the impact of the monitoring tool on health outcomes.

Delirium monitoring aims to embed delirium as a vital sign for ongoing assessment to improve detection of delirium.

Delirium monitoring tools serve as practice-based mediators of work-based learning to upskill ward staff.

There is a lack of validated delirium monitoring tools, this paper validates the Delirium Early Monitoring System-Delirium Observation Screening Scale (DEMS-DOSS) as a monitoring tool.

Introduction

Delirium has a high prevalence in older hospitalised adults and is associated with negative healthcare and economic outcomes [1, 2]. Delirium continues to be under-recognised and missed in up to two-thirds of cases, therefore, opportunities to improve the care of hospitalised older adults are also missed [3, 4]. Over the past two decades, the development and introduction of practice guidelines and validated screening tools such as 4AT [5, 6], Nu-DESC [7] and Confusion Assessment Method (CAM) [8] have highlighted the importance of preventing, recognising and treating delirium in clinical settings.

More recently, there is the suggestion that more is needed to improve patient care beyond the initial delirium screening, and delirium monitoring may fill this gap [9]. Delirium monitoring is defined by Marra and colleagues [10] as the ‘initial detection and ongoing evaluation of a patient’s delirium status during a hospital admission by using predefined criteria’. As such it entails both delirium screening and also ongoing assessment of progress and recovery. Monitoring embeds delirium as a vital sign for ongoing assessment and not only improves detection but importantly also understanding of delirium and communication between patients, families and clinicians [10]. Further, a monitoring tool may allow staff to quantify subtle changes in cognition to prompt further assessment for the new onset of delirium. However, there is ongoing debate on the utility of delirium monitoring and challenges in their application in clinical settings [10]. As such in practice, delirium monitoring is rarely undertaken with a lack of validated tools. To date, one specific delirium monitoring tool, Recognising Delirium As part of your Routine, has been validated [11].

The Delirium Early Monitoring System (DEMS) was developed as a monitoring tool to promote ongoing delirium detection and assessment [12]. Crucially, DEMS has a contemporary educational theoretical underpinning and has been designed specifically to facilitate learning through practice [13]. Similar to the Modified Early Warning Scale, it links the scores on monitoring scales directly to actions [14]. This is particularly important as evidence suggests that simple teaching about delirium alone fails to lead to practice change and there is a need for tools such as DEMS to support ongoing learning by initiating actions and, thereby, informing practice changes [13]. Actions are determined according to resources within healthcare setting but may include reporting the finding to the shift coordinator for a medical review within four hours if Delirium is detected [12].

The DEMS was piloted in the North East of England using a modified screening tool [12], Delirium Observation Screening Scale (DOSS) [15, 16]. DOSS was selected because it is an observational delirium scale rather than a subject-based cognitive assessment that may not lend itself to repeated application in a clinical setting. The DEMS-DOSS has been well received and acceptable in practice and noted to facilitate initiation of action triggers as needed [12]. However, the validity and reliability of the DEMS-DOSS as a tool for monitoring delirium has yet to be established. This is an important step that is necessary before further evaluating the impact of DEMS-DOSS on work-based learning, ward culture and healthcare outcomes.

Aims

The objectives of this study were to evaluate the sensitivity and specificity of DEMS-DOSS as a delirium monitoring tool as well as its test–retest reliability.

Methods

Prospective diagnostic accuracy study to validate DEMS-DOSS as a delirium monitoring tool in patients aged ≥65 years admitted to aged care wards at a metropolitan hospital in Sydney, Australia, from April 2018 to March 2020. Patients were excluded if they had severe hearing impairment, unable to verbally communicate and/or unable to provide consent or without a recognised substitute decision-maker.

Participants’ recruitment and sample size

Within 72 hours of hospital admission, patients were screened for eligibility by senior aged care nurses Supplementary Appendix 1. Convenience sampling was used. Informed consent to participate was provided and data were collected for each participant over a 24-hour period. Demographic information including gender, primary language, country of origin, number of comorbidities and documented evidence of dementia or other neurodegenerative disease was obtained from the medical records. The target sample size was 120 (i.e. minimum power of 90% to detect a respective sensitivity and specificity values of 0.8–0.9 and 0.5–0.9 of a screening test at α = 0.05) [17].

Instruments

DEMS-DOSS

A modified version of the 13 item DEMS-DOSS, consisting of 12 items (Figure 1) was completed. Each item was rated as 0 (not observed) or 1 (observed) and a score of >3 on the DEMS-DOSS indicated the presence of possible delirium requiring further investigation. Nursing staff were trained in the use of DEMS-DOSS. Training consisted of a seven-minute video, crib sheets and bedside training with the SACNs.

Figure 1

DEMS-DOSS scoring sheet.

Diagnostic and Statistical Manual of Mental Disorders criteria for delirium

The reference standard assessment was the Diagnostic and Statistical Manual of Mental Disorder (DSM-IV) criteria for delirium [18]. This assessment incorporated a clinical interview which focused on alertness, orientation and attention. As there is a variation of opinion among delirium experts as to how the cut-offs of the DSM-IV criteria should be applied [19], the research team determined that a participant positive for two or more criteria would be rated to have likely delirium. The senior aged care nurses with expertise in delirium recognition and diagnosis completed the DSM-IV criteria. The senior aged care nurses were trained and supervised by an aged care nurse practitioner. Prior to the commencement of the study, the senior aged care nurses independently assessed a subgroup of patients (n = 3) to check inter-rater reliability using the DSM-IV criteria. A level of 100% agreement was reported for each item.

Procedures

Participants were scored once on DEMS-DOSS between 8 and 10 am, the morning after recruitment to the study (Time A) (Figure 1). To determine test–retest reliability, at least 25% of participants were rated on DEMS-DOSS by a second nurse within a two-hour timeframe (Time B). A trained senior aged care nurse, blinded to the DEMS-DOSS results but not blinded to clinical information, conducted a standardised clinical interview based on the DSM-IV delirium criteria, within two hours of DEMS-DOSS (Time A) completion Supplementary Appendix 1.

Statistical analysis

Statistical analysis was conducted using IBM SPSS Version 26 [20]. Descriptive statistics provided an overview of participants’ demographics. Pearson’s r and Cronbach’s α were calculated to evaluate the test–retest and internal reliability of the DEMS-DOSS. Receiver operating characteristic (ROC) was computed to ascertain the diagnostic accuracy of DEMS-DOSS to yield sensitivity, specificity and area under the ROC curve. Data from participants who completed both DEMS-DOSS and DSM-IV assessments were included for analysis. Approval was received from the Human Research Ethics Committees of Griffith University and South Eastern Sydney Local Health District (17/218).

Results

In total, 156 older hospitalised adults consented to participate in the study; however, only 100 participants were included in the analysis. The reasons participant data were excluded from the study were; discharged before ‘DEM-DOSS competed’ (n = 18), ‘DEMS-DOSS’ not completed within 48 hours (n = 24), DSM-IV not completed within correct timeframe (n = 1), incomplete or missing data (n = 2) and others reasons (transferred to another ward or decreased) (n = 11) (Supplementary Appendix S2). Supplementary Appendix S3 shows the demographic characteristics of participants. The mean age was 84 years (SD ± 7.3), 39% (n = 39) had an existing dementia diagnosis and 49.0% (n = 49) had seven or more comorbidities. The study included culturally and linguistically diverse older adults, with 55% (n = 55) reporting English as their first language and the remaining as, Greek (20%), Italian (10%) and others (16%). Around 38% were evaluated as being positive for delirium based on the DSM-IV.

The sensitivity and specificity of DEMS-DOSS for delirium were 76.3 and 75.8%, respectively. The ROC is depicted in Figure 2. The area under the curve was 0.76 (95% CI: 0.66–0.86) which is deemed as fair/acceptable [21]. Cronbach’s α for internal consistency was good; 0.84. Overall, the test–retest reliability of DEM-DOSS was found to be high (r = 0.92).

Figure 2

Receiver operating characteristic (ROC) curve for the DEMS-DOSS versus reference standard the DSM-IV delirium criteria.

Discussion

Previously, we have demonstrated that DEMS-DOSS as a monitoring tool is easy to use in practice [12]. We now report evidence that DEMS-DOSS is a valid and reliable tool that can, therefore, be considered for introduction into practice for the monitoring of delirium in hospitalised older adults. This is an important finding as, currently, there is a lack of validated monitoring tools and a greater need for delirium monitoring to guide effective delirium practice beyond screening alone. The added value of monitoring tools is that they have the ability to become more embedded within the practice and therefore hold the promise of greater yield in terms of driving work-based learning and practice change [13].

The specificity of 76% for DEMS-DOSS is less than that reported for diagnostic tools such as CAM and 4AT [5, 6, 8]. However, it is important to note that delirium monitoring tools serve a different purpose than screening tools. Their value is derived from ease of use and educational impact to sustain changes in clinical practice. In this regard, monitoring tools serve as practice-based mediators of work-based learning to upskill ward staff and develop a new discourse, in clinical spaces [22].

This study has several strengths, including being pragmatic, real world and with limited exclusion criteria. The senior aged care nurses conducting the DSM-IV were experts in delirium recognition and were blinded to DEMS-DOSS scores. The delirium rate of 37.3% aligns with the estimated prevalence of delirium among older inpatients [23]. We acknowledge the following limitations of this study. While we aimed to ascertain dementia status from the participants’ medical history, this was likely an underestimate as dementia—like delirium—is underdiagnosed [24]. There was a smaller than expected sample size. This sampling reflects the challenges of conducting research in an acute care setting, as 18.5% of recruited participants were discharged, transferred to another ward or died prior to data collection being completed (Supplementary Appendix S2). However, delirium research is known to be challenging and the recruitment difficulties encountered are not unexpected [25].

This study is an important step leading to the implementation into the practice of a tool that could change delirium monitoring practices. Future research should investigate the impact of DEMS-DOSS on healthcare outcomes, ward culture and developing effective delirium practice within hospital settings.

Click here for additional data file.