Is the analogue cognitive test from the ISPOCD equivalent to the digital cognitive test Mindmore? A protocol for a randomised cross-over study including qualitative interviews with self-reported healthy seniors.

INTRODUCTION: Postoperative cognitive decline affects cognitive domains such as executive functions, memory, concentration and information processing. The analogue neuropsychological test developed by the International Study Group of Postoperative Cognitive Dysfunction (ISPOCD) is a well-established test for assessing cognitive performance. However, analogue tests are time-consuming, rarely cost-effective and can be at risk of administration bias. Digital solutions are comparable to analogue ones, have higher degrees of compliance and enable more standardised execution than analogue tests. Currently, there is a lack of recommendations for clinical evaluation of the patient's cognition in the perioperative setting, standard care usually means no cognitive assessments prior or after the surgery. There is a need to find an equivalent neuropsychological test to the ISPOCD to make it accessible and easier to implement in a clinical context for perioperative patients. This study aims to examine how healthy seniors perform on two neuropsychological tests, analogue versus digital and measure equivalency between tests with correlation analysis. METHODS AND ANALYSIS: This study will use a randomised cross-over design, including qualitative interviews regarding test experiences. Healthy participants ≥60 years of age will be eligible to participate in the study. Cognitive function will be measured by using the ISPOCD test and the Mindmore digital test. The participants will self-report depressive symptoms with the Geriatric Depression Scale-15, user experience of the digital test using a modified version of the System Usability Scale and answer questionnaires targeting their experiences after the tests. Furthermore, according to the Swedish Quality of Recovery Scale, self-reported concentration difficulties will also be measured. ETHICS AND DISSEMINATION: The study has been approved by the Swedish Ethical Review Authority (Dnr 2021-05486-01) and will follow the principles outlined in the 1964 Helsinki Declaration and its later amendments. Results from this study will be disseminated in peer-reviewed journals, at scientific conferences, and in social media. TRIAL REGISTRATION NUMBER: 2021-01095; ClinicalTrials.gov. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

The analogue and digital cognitive tests included measures verbal episodic memory, executive functions, selective attention and visuospatial function.

This is an equivalence study, which combines quantitative and qualitative measures of participant experiences.

The small sample size of the study in Sweden may not be generalisable to other contexts.

Introduction

Population ageing is increasing globally, and in Europe, it is predicted that the number of people over 75 years and older will be expanded by 60% in year 2050.1 In western countries, it is estimated that almost 40% of all surgical procedures are performed on persons 65 years or older.2 Advances in surgical and anaesthesia methods, together with better perioperative nursing care, have resulted in safer surgeries and lower rates of serious complications.3 However, the care at the hospital is short, thus the major part of the postoperative recovery occurs after discharge from the hospital.3 4 Preoperative clinical evaluation of cognition is today not a routine part of preoperative and postoperative evaluation.5 6 As a result, memory impairment may go undiagnosed, putting impaired patient brains at danger of further cognitive decline. Closer preoperative coordination among geriatric specialists and the perioperative team would allow for a more sophisticated assessments of at-risk patients.6 At present, it is often the patient or a relative who is the first to detect a change in cognitive capacity after surgery, typically when resuming daily activities, leading to the older person ‘never being the same’ after surgery.2 5 7

Postoperative neurocognitive decline (previously termed postoperative cognitive dysfunction; POCD) is one of the most common complications after otherwise uneventful surgery and affects multiple cognitive domains such as memory, executive function, information processing speed and attention8–12 with subsequently impaired day-to-day memory, language skills, attention and learning compared with levels demonstrated preoperatively.13 POCD is diagnosed up to 30 days postoperatively (delayed neurocognitive recovery)14 and is a subtler deterioration in cognition than postoperative delirium because it is not characterised by obvious clinical symptoms such as a change in the level of consciousness.15 Preoperative risk factors for POCD include increasing age, preoperative cognitive impairment (undiagnosed mild cognitive impairment, vascular dementia, Alzheimer’s disease or other dementia), lower level of functional independence, educational attainment, history of coronary artery bypass surgery and pulmonary disease, alcohol abuse, electrolyte disturbances, psychoactive medications and preoperative delirium.6 9 16 The incidence of cognitive dysfunction in older adult patients 1 week after surgery is approximately 25% and remains at 10% at 3 months.6 9 17 18 Intraoperative factors have been hypothesised to play a role in the occurrence of POCD. The choice of anaesthesia (general vs regional) has not been found to influence the development of POCD,15 19 20 nor have the choice of anaesthetic agent,21 depth of anaesthesia22 or intraoperative hypotension.23

A well-established test for assessment of postoperative cognitive function in patients is the neuropsychological test developed by the International Study Group of POCD (ISPOCD).9 24 The ISPOCD tests assess cognitive performance using four different tests—the Visual Verbal Learning Test, the Concept Shifting Test, the Stroop Colour-Word Test and the Letter-Digit Coding Test9—that provide seven variables for analysis and that have been validated in the perioperative setting over the past two decades.13 25 26 The tests must be administered in the same sequence at each test session and by the same test leader following a standardised instruction manual to ensure as uniform a test situation as possible. The tests have to be carried out in quiet rooms with only the patient and the test leader present.9 Thus, the ISPOCD tests need to be performed by a trained investigator and take about 20 min to perform. Consequently, no consensus exists concerning the duration of the intervals between surgery and test sessions.24 It is also relevant to screen for depression when conducting neuropsychological tests, since depression is linked to cognitive impairment and lack of motivation to complete tests.24

The implementation of digital questionnaires has become more common as digital devices in daily living have become more widespread.27 Recent research suggests that older adults are open to and think that digital devices positively impact their everyday lives.25 However, a few obstacles are reported, such as not understanding potential technical issues.1 The advantages of using digitalised patient-reported outcomes rather than analogue administration are well documented. Digitalised outcomes are also found to be equivalent to analogue versions27 28 but with a higher degree of compliance.28 Earlier research report that digital cognitive tests are in general comparable to the original analogue versions of such tests and that digitised cognitive tests provide standardised execution without the testers’ influence, a high measure of reliability and automated data collection.29 These studies, thus, indicate the preferability of digitalised cognitive tests compared with analogue tests. Earlier POCD research have used different digital cognitive tests such as the Cogstate detection reaction time (psychomotor function) and the identification test (attention),30 one card learning task (visuospatial).31

Furthermore, only 14% of the included studies in a review from 2021 used digital cognitive tests or a mix of both tests, meaning that 86% were analogue tests carried out with a test leader.32 Nevertheless, no digital version of ISPOCD test is available in Swedish or in any other language. However, there is a digital version of a similar cognitive test in Swedish called Mindmore. Mindmore is a collection of several cognitive tests aiming to measure different cognitive domains (ie, processing speed, attention, verbal and visual memory and executive functions). The digital tests have been found to be comparable to the analogue version administered by trained test leaders.33

Aim

The aim of this study is threefold: (1) to evaluate whether an analogue test conducted by a trained test leader using the ISPOCD test battery is comparable with the self-administered Mindmore digital cognitive test and to study the acceptability, feasibility, usability and costs for performing the tests, (2) to assess whether there are any associations between the test results and self-reported concentration difficulties, depression, age or level of education and (3) to explore experiences of being assessed for neurocognitive performance and the pros and cons with the different testing methods.

Methods and analysis

Study design

This will be a randomised cross-over design including qualitative interviews regarding the testing procedures.

Participants

Healthy participants (n=50) comprising men and women >60 years of age are included. The participants will self-report whether they meet any of the following exclusion criteria: inability to read and speak Swedish or suffering from a nervous system disease, severe psychiatric disorder, alcoholism, drug dependence or severe visual or auditory disorder. The definition of healthy seniors in this study is persons who will not suffer from any of the exclusion criteria meaning that they could be suffering from other medical disorders such as heart failure. Sample size estimation is guided by earlier cross-over studies with older adults assessing cognition digitally and by analogue methods.34 35

Cognitive tests

Verbal episodic memory

ISPOCD

The Visual Verbal Learning Test consists of 15 words that are individually presented over a series of three consecutive presentations. Each word is presented visually for 2 s. The participant is asked to recall as many words as possible after each trial as well as after 20 min without further presentation of the words, that is, delayed recall (table 1).

Table 1

Included tests in International Study Group of Postoperative Cognitive Dysfunction (ISPOCD) and the represented test in the Mindmore digital test

Cognitive domain	ISPOCD	Mindmore	Differences
Verbal episodic memory	Visual Verbal Learning Test (VLT) including delayed recall	Consortium to Establish a Registry for Alzheimer’s Disease (CERAD)	VLT: 15 words×3 trialsCERAD: 10 words×3 trials
Executive, visuospatial	Concept Shifting Task (CST)	Trail Making Test (TMT—A & B)	CST: 16 circles×3TMT: 25 circles×2
Executive, selective attention	Stroop Colour-Word test (SCW)	Stroop Colour-Word Test (SCW)	ISPOCD 40 words×3Mindmore 24 words×1
Executive, visuospatial	Letter Digit Coding Test (LDC)	Symbol Digits Processing Test (SDPT)	LDS: 60 sSDPT: 90 s

Mindmore

CERAD includes a 10-word verbal learning test with three learning trials, a recall trial after ~7 min, and a recognition trial. Words are read and visually presented in writing to the patient, and the patient replies in speech.

Executive, visuospatial (assess processing speed)

The Concept Shifting Test is a trail-making test where the participant is required to alternate between letters and digits. Each part consists of 16 circles on a paper. In part A, the circles are numbered from 1 to 16. The test subject is instructed to draw a short line over each number in ascending order from 1 to 16, as rapidly as possible. Part B consists of letters from A to P. Part C consists of eight digits and eight letters to be paired, and the test person is instructed to draw a short line over each and letter order (1-A, 2-B, 3-C and so on).

Trail Making Test A & B. Each part consists of 25 circles on a paper. In part A, the circles are numbered from 1 to 25. The test subject is instructed to draw a line between them in ascending order of numbers, from 1 to 25, as rapidly as possible. Part B consists of 13 digits and 12 letters to be paired, and the test subject is instructed to draw a line between them in ascending number and letter order (1-A, 2-B, 3-C and so on). The digitised versions of the trail making test have shown strong correlation with the results on the traditional test versus the digital version in healthy individuals.33

Executive, selective attention (assess the ability to concentrate and ignore distracting stimuli)

In the Stroop Colour-Word Test, 40 words spelling out a colour are printed in contrasting ink colours (eg, the word ‘green’ printed with red ink), and the participant is asked to say the printed colour of the word rather than the actual meaning of the word. The test is repeated three times with different words.

In the Stroop Colour-Word test, 24 words spelling out a colour are printed in contrasting ink colours (eg, the word ‘green’ printed with red ink), and the participant is asked to tell the printed colour of the word rather than the actual meaning of the word. The digitised versions of the Stroop Colour-Word Test have shown moderate to strong effect size corresponding correlation with the results on the traditional test versus the digital version in healthy individuals.33

Executive, visuospatial (assess mental processing speed and concentration)

In the Letter Digit Coding Test, the participant is asked to match letters with a specific digit for 60 s.

The Symbol Digits Processing Test (SDPT) is based on the original Symbols Digits Modalities Test (SDMT). The symbols in the Mindmore SDPT test are randomised to minimise practice effects. Presented are an array of nine symbol–digit pairings at the top of the screen, one symbol in the middle of the screen and a 3×3 number grid at the bottom of the screen. The participant is required to associate the symbol in the middle to one of the digits on the grid below. As soon as a choice is made a new symbol appears. The final score is the number of correct matches in 90 s. The digitalised SDMT shows comparable sensitivity and specificity to the traditional paper-based SDMT in detecting overall cognitive impairment on a neuropsychological test battery in adult patients suffering from multiple sclerosis36

Acceptability, feasibility and usability

The time taken by participants to complete the tasks will be recorded. After each test, that is, ISPOCD or Mindmore, the participants will be asked to rate on a 5‐point scale, from not at all to extremely, how hard, stressful and acceptable they considered the cognitive assessments to be. The participants will answer yes or no whether they considered the tests difficult and, if so, will describe which parts of the tests. After all tests are conducted, the participants will be asked which tests they thought were preferable and why. Furthermore, they will be asked if they were undergoing surgery if they think it would be possible to take the digital test to detect POCD and, if yes, how often.

The usability of the digital test (table 2) will be assessed with a questionnaire consisting of six items from the System Usability Scale (SUS)37 to be answered on a 5-point scale from ‘strongly agree’ to ‘strongly disagree’.

Table 2

Definitions of feasibility, usability and acceptability in the context of neurocognitive assessment

Term	Definition
Acceptability	Factors that affect and influence the participant’s willingness to assess their cognitive performance and to use the digital self-administered cognitive assessment
Feasibility	The extent to which cognitive assessment can be effectively implemented in clinical settings
Usability	The extent to which the digital self-administered cognitive assessment can be used by the targeted population

I think it is possible to use the system for repeated measurements.

I think the system was easy to use.

I think I would need the support of a technical competent person to be able to use this system.

I think that most people would learn to use this system quickly.

I think the system is cumbersome to use.

I felt confident using this system.

The original SUS questionnaire consists of 10 items,37 but only those items of the SUS whose answers are considered in terms of usability will be used after this short test period.

To further explore the feasibility and acceptability (table 2) for future implementation of the cognitive tests, the participants’ experience of being assessed for neurocognitive performance will be studied. Semistructured one-on-one interviews with 20 of the participants will be conducted within 1 month after undergoing the two tests. The participants will be strategically sampled, thus there will be a mix of educational levels, ages and gender. The interviews will take place online via a phone call. The person who will conduct the interviews will not be involved in the test procedures.

Concentration difficulties

The symptom concentration difficulties in the past 24 hours (one of the symptoms included in the Swedish web version of Quality of Recover38) will also be measured on an 11-point numerical scale from 0 (‘none of the time’) to 10 (‘all of the time’).

Depression screening

The Geriatric Depression Scale (GDS) is a widely used instrument to screen for depression in the older adult population. The original scale is based on 30 items, and a shorter 15-item scale was developed to be used in settings lacking time and where patients might experience fatigue. The 15-item GDS is shown to have sufficient reliability, validity and efficacy39 and to have high diagnostic performance, pool sensitivity and pool specificity.40 41 The cut-off score is standardised with a score of 541. The scale is composed of items with binary (yes/no) answers to questions such as ‘Do you feel helpless?’ or ‘Are you in good spirits most of the time?’.42 Every participant will answer the Swedish version of the GDS-1543 after each test.

Demographic data such as age, gender and education level will also be collected.

Procedure

Participants will be recruited from Swedish non-governmental organisations serving older adults, and activity centres for seniors as well as Karolinska Institutet’s web page ‘Research subjects wanted’ (https://ki.se/en/research/research-subjects-wanted). Planned start date of study is March 2022 and planned end date is April 2022.

The participants will be randomised by computer software to one of two conditions—(1) The ISPOCD tests followed by the Mindmore digital test or (2) the Mindmore digital test followed by the ISPOCD test—with 2 weeks passing between the cognitive tests for both groups. The participants will not know on beforehand which test they will start with, only the test leaders will know. The chosen interval was guided by earlier studies comparing digitalised versus analogue tests of cognitive function in healthy participants people with a wash out period ranging from 30 min to 4 weeks between the studies.33 35 44 Data will be both electronically and manually, and the data forms will be kept at the university site in a locked cabinet. Participant files are stored in a secure and accessible place and manner and access to the data is restricted.

The ISPOCD tests will be conducted by a test leader for this purpose. The digital test will be self-administered on a touch screen tablet (10.1″ Windows). Both tests will be carried out in the same location, with a duration of approximately 20–30 min each in a quiet environment. The amount of time the test leader spends on instructions will be counted as well as the time for conducting the ISPOCD tests.

Analysis

Descriptive statistics of demographic data will be presented with numbers, percentages and mean (SD) or min–max, as appropriate. Non-parametric tests will be used to analyse differences in feasibility and usability between the analogue versus the digital tests. Comparability between the analogue and digital tests will be measured with correlation coefficients and intraclass correlation coefficient (ICC). We will follow Cohen’s guidelines of interpreting the correlation, for example, 0.1, 0.3 and 0.5 as cut-off scores for small, medium and large correlation.45 For ICC, we will use Kline’s criteria, for example, 0.7 low, 0.7–0.89 adequate, values over 0.9 or above are high.46

Exploratory multiple linear regression models will be used to analyse cognitive performance with each cognitive domain of each test (ISPOCD and Mindmore) as the dependent variables and age and level of education as the independent variables due to their potential impact as risk factors for POCD.

The analysis considers the costs for the digital screening tests, including the application software, licence, web administrator interface, data storage, analysis, security and IT support (obtained from Mindmore) as well as time the test leader spends instructing the participant. The cost for the analogue test will include time the test leader spends instructing the participant and the time for conducting the tests, documentation and analysis. All cost estimates will include social fees and overheads and will be converted from Swedish krona to euro using an approximate exchange rate.

All data will be analysed using IBM SPSS V.27 (IBM), and a two-tailed p value <0.05 will be considered significant.

In order to further describe the participants’ experiences of using the analogue ISPOCD test battery conducted by a trained test leader and the self-administered digital test, a qualitative descriptive study will be conducted and analysed by Krippendorff’s content analysis.47 The data will be collected through phone interviews with 20 participants. All interviews will be audio recorded and transcribed verbatim. The transcribed interviews will be, according to Krippendorff’s technique, clustered into groups in order to identify repetitive themes and to allow the researchers to make inferences from the grouped data and to intuitively analyse the similarities in the data.47

Patient and public involvement

Patients or the public were not involved in the design, or conduct, or reporting or dissemination plans of this research.

Discussion

The present study will investigate whether digital tests require fewer resources, both monetarily and timewise as well as exploring the experiences of undergoing neurocognitive tests. The need to monitor symptoms at a distance using digital solutions has never been as important or acute as currently due to the outbreak of the COVID-19 pandemic. Digital tests offer advantages in terms of standardising test instructions, data processing and automation and reduces administration bias.32 Digital tests can also facilitate the implementation of postoperative cognitive follow-up because testing for postoperative cognition is mostly carried out when performing research and with a test leader and paper and pencil. To suffer from cognitive decline is associated with stigma and can affect people in seeking help.48 It is, therefore, of great importance to find a screening tool that is easy to use, valid and reliable in a busy preoperative setting5 as well as in the postoperative follow-up after discharge. Cognitive tests can capture those individuals suffering preoperatively from mild cognitive impairment and from POCD, and such screening tools are preferably digital.32

The present study is novel in measuring the costs for tests and in the further exploration of the feasibility and acceptability of tests by interviewing the participants about their experience undergoing the two neurocognitive test procedures. It is hard to find any research describing this, especially the experiences of different test procedures. An important question to answer is whether digital tests are cost-effective and less stressful because no test leader is needed, and this can lead to important evidence when implementing cognitive tests in clinical practice.

Ethics and dissemination

As the study contains cognitive tests, every study participant will receive detailed information about duration and extent of the tests by the involved research group. Every participant will obtain information about consent, confidentiality and the voluntary basis of the study, the participant can refuse participation at any time. All study-related information will be stored securely electronically or physically on the university site, the local databases will be secured with password-protected access systems. The trial investigators will have the right to access the data, at present, there is no plan to grant public access to participant-level data set and statistical code after the study is finished. Written consent will be obtained. After the participants have finished both tests, each participant will receive their results on their digital cognitive performance. If any of the participants show an abnormal result, they will be contacted personally by one of researchers and go through the results. The information about the test result will not be communicated to anyone other than the participant himself. This is to protect the participants’ personal integrity and their private lives. The project has been approved by the Swedish Ethical Review Authority (Dnr 2021-05486-01) and will follow the principles outlined in the 1964 Helsinki Declaration and its later amendments. Results from this study will be disseminated in peer-reviewed journals, at scientific conferences, and in social media.