The sensitivity and specificity of statistical rules for diagnosing delayed neurocognitive recovery with Montreal cognitive assessment in elderly surgical patients: A cohort study.

Delayed neurocognitive recovery (DNR) is common in elderly patients after major noncardiac surgery. This study was designed to investigate the best statistical rule in diagnosing DNR with the Montreal cognitive assessment (MoCA) in elderly surgical patients.This was a cohort study. One hundred seventy-five elderly (60 years or over) patients who were scheduled to undergo major noncardiac surgery were enrolled. A battery of neuropsychological tests and the MoCA were employed to test cognitive function at the day before and on fifth day after surgery. Fifty-three age- and education-matched nonsurgical control subjects completed cognitive assessment with the same instruments at the same time interval. The definition of the international study of postoperative cognitive dysfunction (ISPOCD 1) was adopted as the standard reference for diagnosing DNR. With the MoCA, the following rules were used to diagnose DNR: the cut-off point of ≤26; the 1 standard deviation decline from baseline; the 2 scores decline from baseline; and the Z score of ≥1.96. The sensitivity and specificity as well as the area under receiver operating characteristic curve for the above rules in diagnosis of DNR were calculated.The incidence of DNR was 13.1% (23/175) according to the ISPOCD1 definition. When compared with the standard reference, the 2 scores rule showed the best combination of sensitivity (82.6%, 95% confidence interval [CI] 67.1%-98.1%) and specificity (82.2%, 95% CI 76.2%-88.3%); it also had the largest area under receiver operating characteristic curve (0.824, 95% CI 0.728-0.921, P < .001). The cut-off point rule showed high sensitivity (95.7%) and low specificity (37.5%), whereas the 1 standard deviation and the Z score rules showed low sensitivity (47.8% and 21.7%, respectively) and high specificity (93.4% and 97.3%, respectively).Compared with the ISPOCD1 definition, the 2 scores rule with MoCA had the best combination of sensitivity and specificity to diagnose DNR.

MoCA is a widely used brief instrument for cognitive function. However, there is lack of consistence in the statistical method of diagnosing delayed neurocognitive recovery with MoCA in surgical patients. Present study compared 4 common statistical rules in literatures. Our result showed that 2 scores rule had the best combination of sensitivity and specificity to diagnose delayed neurocognitive recovery in surgical patients.

Introduction

According to the 2018 nomenclature recommendations, newly occurred cognitive disorder within 30 days after surgery was named as delayed neurocognitive recovery (DNR).[ DNR can be further classified into mild neurocognitive disorder (NCD) or major NCD according to the severity of cognitive dysfunction.[ It is recommended that diagnosis of DNR should be performed in line with the criteria of NCD in the Diagnostic and Statistical Manual for Mental Disorders, fifth edition.[ Key point of this criteria is the statistical method to evaluate impairment or change in cognition.[

Currently, the “standard” method is to perform assessment with a battery of neuropsychological tests and calculate the reliable change index.[ Major deficit of this method is vast difference in the number and context of tests.[ This significantly increases the discrepancy of results amongst different studies.[ This method also requires more than half an hour to be completed, which limits its popular use during daily clinical practice. Brief cognitive assessment instruments, such as the Montreal cognitive assessment (MoCA) and mini-mental state examination, have formative content and can be finished in several minutes.[

It is reported that MoCA performs better than mini-mental state examination in patients over 60 years old.[ The performance of MoCA has been validated in Chinese community-based population.[ In surgical patients, MoCA has also been used to detect the changes of perioperative cognitive function.[ In these studies, the patient's cognitive function was assessed at preoperative and postoperative period, respectively. And then relative change of the 2 values was calculated to reflect the variation of cognitive function.[

Several statistical rules have been used to calculate variations of cognitive function, such as “cut-off point,”[ “1 standard deviation (SD) decrease from baseline,”[ “2 scores decrease from baseline,”[ and “Z score.”[ It is definite that the rules of calculation will significantly influence the accuracy of diagnosis.[ However, until now, it is still unclear which calculation rule is the most appropriate one in diagnosing DNR. The purpose of this study was to define the best statistical rule to diagnose DNR with MoCA in elderly patients after surgery.

Method

This cohort study was a predefined secondary analysis of our previous dataset.[ The study protocol was approved by the Clinical Research Ethics Committee of Peking University First Hospital (2015-987) and registered at Chinese Clinical Trial Registry on December 1, 2015 (www.chictr.org.cn, registration number ChiCTR-IPR-15007654). Written informed consents were obtained from all patients and control subjects.

Patients

Potential participants were screened the day before surgery (or on Friday for those who underwent surgery the next Monday). The inclusion criteria in the previous study were elderly (age ≥60 years) patients who were scheduled to undergo elective major noncardiac surgery with expected duration ≥2 hours under general anesthesia. Those who met any of the following criteria were excluded:

refused to participate;

history of schizophrenia, epilepsy or Parkinson disease;

visual, hearing, language, or other barrier that impeded communication and neuropsychological assessment;

history of traumatic brain injury or neurosurgery;

severe bradycardia (heart rate less than 40 beats per minute), sick sinus syndrome, or atrioventricular block of degree 2 or above without pacemaker;

severe hepatic dysfunction (Child-Pugh grade C);

renal failure (requirement of renal replacement therapy).

As a secondary analysis, patients who met one of the following criteria were also excluded from present study:

lack of preoperative or postoperative MoCA tests;

lack of preoperative or postoperative neuropsychological tests.

Anesthesia and perioperative management

Anesthesia was induced with intravenous propofol and sufentanil, and maintained with intravenous propofol and sufentanil as well as inhalation of a 1:1 nitrous oxide-oxygen mixture. The target anesthesia depth was to maintain bispectral index value between 40 and 60. Rocuronium and/or cisatracurium were administered for muscle relaxation. Fluid infusion and blood transfusion were performed according to routine practice. Blood pressure was maintained within 20% from baseline. Nasopharyngeal temperature was maintained between 36.0 and 37.0°C.

All patients were transferred to the post-anesthesia care unit or the intensive care unit before being sent back to general wards. Patient-controlled intravenous analgesia was provided for postoperative analgesia, which was established with 0.5 mg/mL morphine and programmed to deliver a background infusion rate of 0.5 mg/h and a 1 mg bolus with a lock-out interval of 8 minutes. For patients with a numeric rating scale pain score (an 11-point scale where 0 indicates no pain and 10 the worst pain) ≥4, supplemental morphine at doses from 2 to 4 mg could be administered at a 10-minute interval. Intravenous nonsteroidal anti-inflammatory drugs and/or oral tramadol could also be administered.

Control subjects

To define practice effects of repeated cognitive assessment, 53 control subjects were enrolled from patient's family members who had comparable age and education. The inclusion and exclusion criteria were the same as those of patients except that they did not undergo surgery (Table 1).

Table 1

Demographic characteristic.

Administration of neuropsychological tests and MoCA

A battery of neuropsychological tests was employed to test cognitive function in patients the day before and on the 5th day after surgery (or the same time interval for control subjects). The test battery included 7 tests with 9 subscales which had been used in our previous studies.[ Specific tests were as the following: mental control, digit span (forward and backward), visual retention, paired associate verbal learning, digit symbol, Halstead-Reitan trail making Test (Part A), grooved pegboard test (dominant and nondominant hand). Cognitive assessment with MoCA-Beijing version (MoCA-BJ) was also repeatedly performed at the same timepoints.

Investigators who performed cognitive assessment received special training on psychometric test administration and relevant interview techniques by a psychiatrist before start of study. During the study phase, testing was performed and scored in a standardized manner in order to minimize inter-examiner difference. Repeat assessments for each patient were conducted by the same examiner. Parallel forms of tests were used in a randomized manner in sequential testing in order to minimize practice effect.

Standard reference for diagnosing DNR

Reliable change index as used in the International Study of Post-Operative Cognitive Dysfunction [ISPOCD 1]) was calculated to diagnose cognitive impairment and the result was considered as standard reference.[ To quantify practice effect, baseline scores were compared with subsequent test results 5 days later in control subjects. For patients, preoperative scores were compared with postoperative test results, subtracted the average practice effect from these changes, and then divided the result by the control-subject SD to obtain a Z score for each test. The test results were adjusted so that a positive Z score indicated deterioration from the baseline test. The Z scores of all tests in an individual patient were then summarized and divided by the SD for this sum of Z scores in the control subjects, creating a combined Z score. A patient was defined as having DNR when 2 Z scores in individual tests or the combined Z score were 1.96 or more.

Diagnosis of DNR with MoCA

The following calculation methods were used to diagnose cognitive disorder:

Cut-off point rule: postoperative MoCA score ≤26 was considered as DNR.[

1 SD rule: DNR was established if postoperative MoCA score was 1 SD lower than preoperative baseline.[

2 scores rule: DNR was established if postoperative score was 2 scores lower than preoperative baseline.[

Z score rule: DNR was defined as a Z score of the change in MoCA scores of 1.96 or more.[ According to this method, Z score was calculated by subtracting the average learning effect (ie, mean change in MoCA scores of control subjects) from the individual changes, and dividing the result by the SD for the MoCA score changes of the control group.[

Primary outcome

The primary outcome was the incidence of DNR diagnosed according to the ISPOCD1 definition.

Secondary outcome

Secondary outcome was the incidence of DNR diagnosed by MoCA at postoperative 5 days.

Statistical analysis

Sample size

The incidence of DNR in elderly patients after major noncardiac surgery was about 25%.[ The width of confidence interval was set at 0.2 and statistical significance at 0.05. Assumed both the sensitivity and specificity were at 90%, 138 patients and 82 patients were required for sensitivity and specificity analysis respectively.

Outcome analysis

Normal continuous data were presented as mean ± SD and compared by independent sample t test. Categorical variables were presented as number of patients (percentage). Sensitivity and specificity were analyzed to test the diagnostic characteristic of different calculation methods. The receiver operating characteristic curve, along with the area under the curve and 95% confidence interval (CI), was utilized to assess the ability of different calculation methods to diagnose DNR.

P-value less than .05 was considered as statistical significance. All statistical analyses were performed with the SPSS statistical package version 14.0 (SPSS Inc, Chicago, Ill).

Result

A total of 175 surgical patients and 53 control subjects completed both pre- and postoperative (or at the same time interval) neuropsychological tests and MoCA assessment. The mean age and education level of control group were comparable with surgical patients (Table 1 and Table 2).

Table 2

Results of MoCA and neuropsychological tests.

The incidence of DNR

The incidence of DNR was 13.1% (23/175) when diagnosed according to the ISPOCD1 definition. When MoCA was used, the incidence of DNR was 66.9% (117/175) by cut-off point rule, 12.0% (21/175) by 1 SD rule, 26.3% (46/175) by 2 scores rule and 5.1% (9/175) by Z score rule (Table 3).

Table 3

The sensitivity and specificity of MoCA by different rules.

Sensitivity and specificity analysis

When compared with the standard reference, the 2 scores rule showed the best combination of sensitivity (82.6%, 95% CI 67.1%–98.1%) and specificity (82.2%, 95% CI 76.2%–88.3%); it also had the largest area under receiver operating characteristic curve (0.824, 95% CI 0.728–0.921, P < .001), Table 3 and Figure 1. The cut-off point rule showed high sensitivity (95.7%) and low specificity (37.5%), whereas the 1 SD and Z score rules showed low sensitivity (47.8% and 21.7%, respectively) and high specificity (93.4% and 97.3%, respectively), Table 3.

Figure 1

The areas under ROC curve of different statistical rules in diagnosing DNR. The 2 scores rule showed the best ability in diagnosing of DNR than other 3 methods. Blue line indicated the cut-off point rule. Red line indicated the 1 SD rule. Green line indicated the 2 scores rule. Orange line indicated the Z score rule. CI = confidence interval, DNR = delayed neurocognitive recovery, ROC = receiver operating characteristic, SD = standard deviation.

Discussion

In present study, we demonstrated that, when compared with the ISPOCD1 defination, the 2 scores rule had the best combination of sensitivity and specificity to diagnose DNR when using MoCA as cognitive assessment instruments in elderly surgical patients.

The incidence of DNR varies from 5% to 60% in patients after major noncardiac surgery and is highly related with patients poor outcome such as increased complications and mortality.[ Accurate diagnosis is the key point to prevent and treat DNR.[ The main gap between DNR diagnosis and clinical practice includes lack of standard criteria and cross-cultural consideration of instruments.[

DNR refers to subtle disorder of cortical function and may influence isolated domains such as thought processes, verbal memory, visual memory, language comprehension, attention, and concentration.[ Several assessment instruments have been recommended to assess the function of core domains including Rey Auditory Verbal Learning Test (for memory), trail making A and B (for hand-eye coordination, attention, and concentration), and Grooved pegboard (for performance speed).[ However, the number and content of neuropsychological tests varies greatly amongst different studies.[ This increases several concerns:

the number of examined domains affects the incidence of DNR[;

the content of examination has significant impact on accuracy of diagnosis which is the so-called floor effect[;

the discrepancy in diagnosis criteria impedes the possibility to compare different study result.[

MoCA is a 30-score instrument which covers major domains of cognition including short term memory, visuospatial abilities, executive functions, attention, concentration and working memory, language and orientation to time and place.[ The first advantage of MoCA is that it has a formative content with predefined items.[ Second, it can be executed by nonpsychiatric healthcare staffs after training.[ Third, it can be finished in 5 to 10 minutes.[

Chinese version of MoCA, that is, MoCA-BJ, a direct translation from its original version, was used in present study.[ Considering culture difference, it is not surprise to see that the performance of MoCA-BJ was not as good as its original version in American population (sensitivity 90%–100% and specificity 91%–100%).[ Even in Chinese community-based population, it was also recommended that adaptation of its context was needed to improve its accuracy.[

Up to now, there is no consensus on the methods of MoCA score calculation in perioperative situation.[ But the methodology of calculation significantly affects its accuracy.[ The “cut-off point rule” is widely used to detect NCD or dementia in community-based population study.[ In surgical patients, this method cannot reflect the new-onset acute change of cognition during perioperative period. For example, 106 patients had a preoperative MoCA score less than 26 and 88 of them was diagnosed as DNR in present study. In fact, only 35 of the 88 DNR patients experienced postoperative cognition decline (postoperative MoCA was 1 score lower than preoperative baseline value). This would significantly lower the specificity of “cut-off point rule.” The accuracy of “1 SD rule” is affected by distribution characteristic of preoperative data. For example, the data of small sample size study trends to be non-normative and it is hard to calculate SD. Even in community-based population study, the SD of MoCA varies from 1.3 to 5.6.[ The advantage of “Z score rule” is that it minimizes the practice effect especially in multiple assessment.[ However, this method has high specificity but low sensitivity.[ Although the criteria of 2 scores decline from baseline value was arbitrarily defined, the “2 scores rule” presented best performance in diagnosing DNR.[ When the criteria of DNR was defined as 3 scores decline from baseline value in present study, the incidence of DNR would be 18.3% (32/175). The sensitivity and specificity of MoCA was 78.3% (95% CI 61.4%–95.1%) and 88.1% (95% CI 83.0%–93.3%), respectively.

Limitation of present study included single-center study design and a relatively small sample size. Further large sample size study is needed to validate the best method of MoCA calculation to diagnose DNR in elderly surgical patients.

Conclusions

Results of the present study showed that, in elderly patients after major noncardiac surgery, the “2 scores rule” had the best combination of sensitivity and specificity to diagnose DNR with MoCA according to the ISPOCD1 definition.

Author contributions

JH participated in study administration and manuscript draft. CJL and BJW participated in study administration. XYL participated statistical analysis. DLM participated in study design, manuscript revision and final approval of publication. DXW participated in study design and manuscript revision.