Sleep-deprived residents and rapid picture naming performance using the Mobile Universal Lexicon Evaluation System (MULES) test.

OBJECTIVE: The Mobile Universal Lexicon Evaluation System (MULES) is a rapid picture naming task that captures extensive brain networks involving neurocognitive, afferent/efferent visual, and language pathways. Many of the factors captured by MULES may be abnormal in sleep-deprived residents. This study investigates the effect of sleep deprivation in post-call residents on MULES performance.
METHODS: MULES, consisting of 54 color photographs, was administered to a cohort of neurology residents taking 24-hour in-hospital call (n = 18) and a group of similar-aged controls not taking call (n = 18). Differences in times between baseline and follow-up MULES scores were compared between the two groups.
RESULTS: MULES time change in call residents was significantly worse (slower) from baseline (mean 1.2 s slower) compared to non-call controls (mean 11.2 s faster) (P < 0.001, Wilcoxon rank sum test). The change in MULES time from baseline was significantly correlated to the change in subjective level of sleepiness for call residents and to the amount of sleep obtained in the 24 h prior to follow-up testing for the entire cohort. For call residents, the duration of sleep obtained during call did not significantly correlate with change in MULES scores. There was no significant correlation between MULES change and sleep quality questionnaire score for the entire cohort.
CONCLUSION: The MULES is a novel test for effects of sleep deprivation on neurocognition and vision pathways. Sleep deprivation significantly worsens MULES performance. Subjective sleepiness may also affect MULES performance. MULES may serve as a useful performance assessment tool for sleep deprivation in residents.

Introduction

Sleep deprivation impairs several critical components of neurocognition, particularly memory, attention, processing speed, and higher-level processes, as well as ocular motor responsiveness [[1], [2], [3], [4]]. Residents often work 24-h or longer call shifts which predisposes them to work in a sleep-deprived state. Recently, the Accreditation Council for Graduate Medical Education (ACGME) formally extended the number of maximum hours that a resident can work to 28 h [5]. With this increase in maximal work hours, it is important to recognize the possible effects of sleep deprivation on resident physicians.

A 2005 study found that residents working a heavy call schedule compared to colleagues working a lighter call schedule experienced decreased attention and reaction times similar to the effects of approximately two alcoholic drinks [6]. These effects can translate to serious clinical errors. A 2005 meta-analysis showed that a physician's clinical performance dropped by 1.5 standard deviations with sleep loss after 24 to 30 h of wakefulness [7]. In one prospective interventional study, medical residents were separated into two groups: one traditional work schedule group (containing 24-h plus shifts, maximum 28 consecutive hours) and one interventional group (reducing maximum consecutive hours to 16 h). When compared to the intervention group, the residents working a traditional schedule made 35.9% more serious medical errors, including significantly more medication and diagnostic errors [8]. Additionally, the residents in the intervention group were found to obtain significantly more sleep prior to their shifts and made over 50% fewer attentional failures while on-call [9]. Working in a sleep-deprived state not only poses a direct risk to patient safety, but also poses a risk to the residents themselves. One study found that for every one extended work shift added to a residents monthly schedule, their risk of getting into a car crash during their commute from work increased by 16.2% [10].

Post-concussion testing methods have been increasingly used in order to assess sleep deprivation associated cognitive deficits [11,12]. One post-concussion test used for this purpose is the King Devick test (KDT). Utilizing a rapid number-naming task, this test requires the participant to perform coordinated saccadic eye movements, process and interpret the visual information, and then produce speech in order to assess multiple neurocognitive functions [[13], [14], [15]]. A 2012 study found that on-call neurology residents demonstrated significantly worse performance on the KDT than non-call residents [16]. Another 2019 study confirmed these findings in surgical and medical residents [17].

In the area of post-concussion testing, the Mobile Universal Lexicon Evaluation System (MULES) has been a major development because it is able to evaluate an extensive neurocognitive network including but not limited to vision pathways, attention, perception, memory, language, speech, and executive functioning [18]. The MULES test primarily differs from the KDT in its use of a rapid picture-naming task as opposed to number-naming. The use of pictures should theoretically increase the learning effect of the test as images are more information rich and meaningful, allowing for greater neural encoding. In addition, rapid picture naming likely captures more extensive visual networks. For instance, rapid picture naming requires color vision processing, an intricate process involving area V1 along with the extra-striate ventral-stream pathway (including V2, V4, and the dorsal posterior inferior temporal cortex) [[19], [20], [21], [22]]. In one study comparing rapid picture naming via MULES to rapid number naming via the KDT, it was found that participants required twice as long on average to name each picture in the MULES versus each number on the KDT (0.72 s vs. 0.33 s, respectively) [18]. The MULES test has already been studied in other neurologic conditions including concussion, multiple sclerosis, and Parkinson's disease in which it has been able to identify impairment [[23], [24], [25]]. Given its evaluation of multiple neurocognitive networks, the MULES test may be useful in determining the effect of sleep deprivation in residents. The goal of the present study is to investigate the effect of sleep deprivation on MULES performance in residents taking 24-h call (call group) vs. controls not taking call (non-call group).

Subjects and methods

Participants

Study protocols were approved under the New York University (NYU) Grossman School of Medicine Institution Review Board (IRB) and written informed consent was obtained from all participants. Neurology residents and staff from NYU Grossman School of Medicine were enrolled in this study. Subjects comprised a convenience sample of individuals who were taking overnight call with a 24-h shift (n = 18) and a control group of individuals who were not taking call with an ~12 h shift (n = 18).

Performance testing

Before testing, each participant completed a survey to self-report the number of hours of sleep obtained in the prior 24 h, timing of last caffeine consumption, subjective level of sleepiness using the Karolinska Sleepiness Scale (KSS), and sleep quality using the Pittsburgh Sleep Quality Index (PSQI). The KSS is a validated measure of sleepiness, particularly useful for the assessment of occupational sleepiness, and ranges from 1 (extremely alert) to 10 (extremely sleepy, cannot keep awake) [26,27]. The KSS was administered prior to both baseline and follow-up testing. The Pittsburgh Sleep Quality Index (PSQI) was administered prior to baseline testing to evaluate sleep quality measures over the past month on average, such as amount of sleep obtained, time to fall asleep, disruptions during sleep, sleep deprivation symptoms, enthusiasm for getting things done, and overall subjective rating of sleep quality [28]. The PSQI score ranges from 0 to 21 with a value of 5 or less associated with good sleep quality and a value of greater than 5 associated with poor sleep quality.

Each participant performed the MULES at baseline (prior to 24-h call or 13-h day shift for controls) and follow-up (post-24-h call or prior to next 13-h day shift for controls). For each cohort, the MULES test was administered between 6:30 AM to 7:30 AM for both baseline and follow-up. The MULES consists of a double-sided 8.5 × 11-in. laminated sheet with 54 colored pictures comprising food, animals, and random objects (Fig. 1). MULES score is the time in seconds required to name all the pictures from left to right and top to bottom on both sides of the sheet.

Fig. 1

The Mobile Universal Lexicon Evaluation System (MULES) is a rapid picture naming test (MULES Test © New York University, text and photographs, registration number TXu002026665, all rights reserved). The MULES is printed two-sided on an 8.5 × 11-in. laminated sheet of paper and includes 54 original photographs of animals, food, and random objects. The participant names the pictures from left to right as rapidly as possible. The MULES score is the time in seconds required to name all pictures on both sides (participant flips the laminated sheet during the test). Figure adapted from [23].

Statistical analyses

Statistical analyses were performed using R software. Preliminary assessment indicated that MULES scores did not follow a normal distribution and thus non-parametric statistical tests were conducted. Wilcoxon rank-sum tests were used to determine statistical significance between MULES times and subject groups. Linear regression analysis was used to assess MULES change between groups adjusting for potential confounding factors. Spearman rank correlations were used to test the significance of associations between baseline MULES and sleep obtained in 24 h prior to testing as well between the change in MULES performance and variables collected by the questionnaire (duration of sleep obtained in 24 h prior to testing, KSS subjective level of sleepiness score, and PSQI sleep quality questionnaire score). For all statistical tests, type I error for statistical significance was set at P < 0.05.

Results

MULES testing data for participants is summarized in the table (Table 1). There were 39% (7/18) males and 61% (11/18) females in the on-call group. The control group comprised 44% (8/18) males and 56% (10/18) females. The participants taking call did not have less sleep over the prior 24 h at baseline (P = 0.95) but did have significantly less sleep when post-call relative to controls (P < 0.001, Wilcoxon rank sum test).

Table 1

Table of sleep, sleepiness, and MULES testing data for participants taking call and controls not taking call.

	Residents taking call (n = 18)	Controls not taking call (n = 18)	Effect size (mean difference)	P value
Duration of sleep in past 24 h at baseline, h, mean (range)	6.7 (5.0–8.5)	6.7 (5.0–8.5)	0.0	0.949
Duration of sleep in past 24 h at follow-up, h, mean (range)	0.8 (0.0–3.0)	6.2 (3.0–8.0)	5.4	< 0.001
Change in sleep obtained in 24 h prior to testing (follow-up – baseline), h, mean (range)	−5.9 (−8.5 - -2.0)	−0.4 (−2.0–0.5)	5.5	< 0.001
KSS at baseline, mean (range)	3.6 (1.0–7.0)	3.7 (1.0–8.0)	0.1	0.620
KSS at follow-up, mean (range)	6.7 (2.0–10.0)	4.3 (2.0–7.0)	2.4	0.003
Change in KSS, mean (range)	3.2 (0.0–6.0)	0.7 (−3.0–2.0)	2.5	< 0.001
PSQI score, mean (range)	5.8 (3.0–11.0)	5.3 (2.0–11.0)	0.6	0.569
Baseline MULES time score, s, mean (range)	42.0 (32.6–60.1)	47.7 (38.6–64.8)	5.7	0.013
Follow-up MULES time score, s, mean (range)	43.2 (31.2–61.3)	36.5 (28.1–49.1)	6.7	0.023
Change in MULES time score from baseline, s, mean (range)	1.2 (−6.3–11.5)	−11.2 (−28.4 - -3.5)	12.4	< 0.001

Legend: Bolded = statistically significant (P < 0.05).

Abbreviations: KSS = Karolinska Sleepiness Scale; PSQI = Pittsburgh Sleep Quality Index; MULES = Mobile Universal Lexicon Evaluation System.

Subjective sleepiness as measured by KSS scores were similar at baseline between groups (P = 0.62, Wilcoxon rank sum test). At follow-up testing, participants taking call reported significantly greater subjective sleepiness relative to controls (P = 0.003). Sleep quality as measured by PSQI scores were similar between groups (P = 0.57). The variability in reporting caffeine consumption precluded proper analysis of this variable.

MULES times were significantly slower for controls at baseline (P = 0.01) and residents taking call at follow-up (P = 0.02). Less improvement from baseline MULES times were observed in residents taking call (mean 1.2 s slower, range − 6.3 to 11.5) compared to controls not taking call (mean 11.2 s faster, range − 28.4 to −3.5) (P < 0.001, Wilcoxon rank sum test; Fig. 2). This difference remained significant after adjusting for baseline MULES scores (P < 0.001, linear regression).

Fig. 2

Box plot demonstrating MULES scores change in seconds for participants taking call (n = 18) vs. controls not taking call (n = 18, P < 0.001, Wilcoxon rank-sum test). The lines within the boxes represent the medians and the boxes delineate the interquartile range (25th to 75th percentiles). The whiskers represent the range of observations minus outliers (represented by dots).

There was a significant correlation between MULES score at baseline and sleep obtained in the 24 h prior to baseline for call residents, with less sleep associated with slower MULES score (r = −0.52, P = 0.03). In addition, change in MULES time scores from baseline were significantly correlated to the amount of sleep obtained in the 24 h prior to follow-up testing for the entire cohort, with less sleep associated with less improvement in MULES score (r = −0.70, P < 0.001). When adjusting for baseline MULES scores for the entire cohort, less sleep obtained in the 24 h prior to follow-up testing was predictive of worsening (slower) MULES (P < 0.001, linear regression). Within the post-call resident group, the change in MULES time scores did not significantly correlate with duration of sleep obtained during call (r = 0.32, P = 0.13).

There was also a significant correlation between changes in MULES time scores from baseline and the change in KSS subjective level of sleepiness score for residents taking call, with greater worsening from baseline MULES among those who reported more sleepiness (r = 0.46, P = 0.02). Being on call was predictive of having a higher (more tired) KSS score at follow-up when adjusting for pre-call sleepiness for call residents (P < 0.001, linear regression).

For all participants, there was no significant relation between changes in MULES scores from baseline and sleep quality questionnaire score as measured by the PSQI (r = 0.18, P = 0.31).

Discussion

The MULES is a novel test for evaluating the effects of sleep deprivation on attention, memory, language, rapid eye movements, and complex visual processing. In the current study, the MULES was administered to residents taking 24-h-call before and after their shifts and the change in performance was compared to that of their non-call colleagues. One primary finding is that sleep deprivation significantly worsens MULES performance. The sleep deprivation resulting from 24 h of wakefulness was enough to cause significant worsening in MULES performance among call versus non-call residents. The results indicate a learning effect among the control group (11.2 s faster on follow-up), while the call residents exhibited extinguishment of this phenomenon (1.2 s slower on follow-up). Further research is needed to help determine to what degree this sleep-loss related decrease in performance is due to impaired higher order executive functions (e.g. learning or image processing) or lower order functions (e.g. attention or arousal). [29] Prior studies found similar results using the KDT test in call versus non-call residents [16,17]. Similar findings between the present MULES study and the prior KDT studies provides additional support for the ability of post-concussion testing methods (e.g. MULES and KDT) to identify the neurocognitive impact of sleep deprivation.

It was noted that residents taking call had significantly faster MULES scores at baseline compared to controls not taking call. It is possible that the control group may have been recovering from a recent call or night-float shifts, which may have contributed to slower baseline MULES scores. When accounting for differences in baseline MULES scores, the change in MULES scores continued to be statistically significant, with call residents exhibiting worsening (slower) MULES scores.

Less sleep in the 24 h prior to follow-up testing was significantly associated with less improvement in MULES scores for the entire cohort. However, among residents taking call, no association was found between duration of sleep obtained during call and change in MULES score. Regardless, performance on MULES was impaired in this group who also obtained significantly less sleep in the 24-h prior to follow-up testing compared to controls (average 0.8 h for residents taking call versus 6.2 h for controls). Although when a sub-analysis of the residents taking call was performed, there was no correlation between MULES change and sleep obtained. This same finding was seen in a prior study of the KDT in post-call residents and may be due to variability in individual vulnerability to the effects of sleep deprivation [16]. A few residents who obtained no sleep during call were not as impaired in MULES performance as may be expected, while others were more substantially impaired. Possible confounding factors may potentially include substances taken during call which were not analyzed.

In addition, subjective sleepiness as measured by KSS score was significantly associated with MULES performance in residents taking call. In other words, being more tired as represented by having a higher KSS score was predictive of performing more slowly on the MULES. As such, subjective sleepiness may also affect MULES performance. This finding is consistent with a prior study of the KDT in post-call residents in which self-report of increased sleepiness using the KSS was associated with less improvement in KDT time [16] and to a study that found slower peak eye saccade velocities with increased sleepiness [30]. Given the findings that sleep and sleepiness can impact one's MULES score, it would be helpful to assess one's sleep and sleepiness status along with performing a MULES test to detect concussion in order to mitigate potential confounders.

There were several limitations of this study. One limitation is the small sample size. Additionally, masking of the groups was also not possible because the residents and test administrator knew whether the residents were taking call or not. Another limitation is that substances taken in the 24 h prior to testing (e.g., alcohol, caffeine, and other stimulants) were not analyzed. These substances may affect neurocognition and thus MULES performance. Furthermore, participants did not report whether they were recovering from a recent call or night-float shifts which may have affected baseline MULES scores. Future studies with a larger sample size and addressing these limitations would be useful. It would also be interesting to examine the effects of night-float in residents on MULES performance in future studies as circadian rhythm disruptions may also affect MULES scores.

Another future research investigation could be to evaluate how much sleep is needed to prevent a significant decrease in MULES score. This could be used to help set sleep recommendation guidelines for residents. Additionally, because the outcomes of a resident working in a sleep-deprived state can cause tremendous harm to patients and themselves in the form of serious medical errors or driving errors, MULES testing could serve as a potential personal/patient safety assessment tool [8,10]. Athletes with a suspected concussion who perform significantly worse on MULES testing (compared to their baseline) can be diagnosed with a concussion and, depending on the sport and severity, may be restricted from continued play [23]. This concept could be translated to the medical profession. Practically, this would mean sleep-deprived residents who perform significantly worse on MULES testing could receive additional oversight or shift-relief when safely possible. Just as concussed athletes are prevented from playing for their personal safety, residents with significantly impaired neurocognitive performance might benefit from added safety measures to protect their patients and themselves.

This study showed that the MULES test has the potential to serve as a useful performance assessment tool for sleep deprivation in residents and may also be useful for those from other populations for whom sleep cycles are a variable aspect of occupational duties.

Disclosure statements

The authors have no financial interests.

Additional disclosure: Dr. Laura J. Balcer is editor of the Journal of Neuro-Ophthalmology.

Financial support

This study was supported in part by the NYU Grossman School of Medicine.