An Exploratory Study of the Development and Pilot Testing of an Interactive Visual Tool of Neuropsychological Test Results in Memory Clinics.

BACKGROUND: Neuropsychological feedback is an important part of the neuropsychological assessment process. However, patients have difficulties remembering this information.
OBJECTIVE: The aim of this study was to develop a web-based visual tool to improve the understanding of neuropsychological results, information retention, and psychologist-patient communication.
METHODS: The visual tool was developed and optimized using an iterative three-phase stepwise approach to determine its usability, technology acceptance, and feasibility in a memory clinic population. Feedback from different user perspectives (patients, family members, and psychologists) was obtained in each phase using a multimethod approach (e.g. a multidisciplinary brainstorm session, think-aloud sessions, focus groups). The prototype was subsequently tested in a pilot study.
RESULTS: The first phases offered insights that led to optimization of the prototype. On a scale ranging from 0 to 100, psychologists evaluated the usability as high [88.1±7.6,70-87]. During the pilot study, both patients and significant others gave positive feedback, but information retention in patients remained low. All participants thought the benefits of the visual tool included seeing cognitive strengths and weaknesses with a translation to daily life all at one glance and receiving feedback on paper to take home. Important barriers were mentioned by psychologists, such as a limited set of tests included and no integration with hospital systems.
CONCLUSION: Overall, patients, family members, and psychologists reported that a visual display of the cognitive profile with insights into daily life had added value to clinical practice. Feedback from the pilot study was adopted in the tool for future implementation purposes.

INTRODUCTION

A dementia diagnosis has a huge impact on patients and their significant others. Understanding the consequences of this diagnosis may, however, enable them to make positive adjustments and cope with the experienced loss [1-3]. Therefore, accurate and clear communication during the diagnostic process and disclosure are required. A common assessment tool in supporting the diagnosis of dementia is a neuropsychological assessment. A recent systematic review showed the added value of a neuropsychological assessment in improving the accuracy of diagnosis, predicting patient outcomes, and increasing referrer satisfaction [4]. The majority of patients also perceived the assessment as helpful [4].

Neuropsychological feedback is an important co-mponent of a neuropsychological assessment, which offers the opportunity to discuss cognitive strengths and weaknesses and provides recommendations for daily life functioning [5, 6]. Furthermore, neuropsychological feedback increases a patient’s quality of life, understanding of a disease, and coping abilities [7]. Previous studies showed that patients were more satisfied with a neuropsychological assessment when they received such feedback and that they perceived this feedback as useful [8-11]. However, a standardized approach in communication techniques for providing feedback is lacking [12]. Giving feedback may be challenging due to the complex nature of neuropsychological test results [5, 6]. In addition, information retention of medical information is low in general [13-15]. Written information is more easily remembered compared to verbal information, and the former was found to improve treatment adherence and enhance free recall of neuropsychological recommendations [13, 16, 17]. However, neuropsychological reports sometimes face challenges related to readability and complexity for laypersons. Understanding written reports and remembering neuropsychological feedback might be even more challenging for people with cognitive impairments or low literacy [18]. Adapting these reports is often necessary to make findings more understandable for patients and family [19].

In other medical fields it has been shown that visual aids enhance the recall of spoken medical instructions [20, 21]. Visual aids also improved the understanding of medical instructions, especially when a combination of visual, written, and verbal instructions was used [22]. To our knowledge, no studies have explored the use of visual aids in providing neuropsychological feedback. A patient-friendly tool to visualize neuropsychological test performances could enable professionals to deliver patient-tailored information and may improve understanding of cognitive strengths and weaknesses and patient satisfaction.

The aim of this study was to develop a web-based visual tool for neuropsychological test results to increase the understanding of neuropsychological performances, to improve information retention, and to support communication during feedback sessions. The second aim of this study was to determine the usability, technology acceptance, and feasibility of the tool.

METHODS

Iterative development process

An iterative process was used to develop the web- based interactive visual tool during the “Innovations in NeuroPsychological assessment in the timely Diagnosis of Dementia” (INPAD) project. The visual tool was co-created with patients, family members, and neuropsychology experts using a stepwise approach including three consecutive phases spread over a 3-year time period (2017–2020) (Fig. 1). All sessions in each phase were audiotaped and analyzed by summarizing and grouping the feedback.

Fig. 1

Overview of the iterative development process of the visual tool.

Phase 1: Exploration potential user views

Methods and participants

During the first phase, the content and design of the visual tool was developed using an iterative approach during which adaptions were made to concept versions of the visual tool based on the feedback collected. The following steps were carried out:

Three multidisciplinary brainstorming sessions of 1.5 hours with different disciplines (researchers, neuropsychologists, software developers, interaction designer) to decide which cognitive tests to include and what visualization should be used.

Meeting between the researchers of the INPAD project and client panel members (patients with dementia, informal caregivers) from the Alzheimer Center Limburg to inquire what type of visualization should be used.

Showing concept versions of the visual tool using the think-aloud method [23] in neuropsychology experts (n = 18) working at memory clinics across twelve Dutch hospitals. These potential users were asked to think out loud while inspecting the different versions of both the patient and clinician modules. They were asked to rank the versions according to their preference.

Showing concept versions of the visual tool during four focus group sessions with fourteen patients and thirteen family members across three Dutch MCs. Characteristics of patients and family members are represented in Table 1. Participants were asked to participate if they were referred to the memory clinic for the evaluation of cognitive complaints and if they had had a neuropsychological assessment in the past year. More information about this procedure is described elsewhere [24]. The participants were asked for their opinions regarding these concept versions and to rank the versions according to their preference.

Table 1

Characteristics of patients (n = 14) and family members (n = 13)

	Focus group participants
	(n = 27)
	Patients	Family members
	(n = 14)	(n = 13)
Age (mean±SD	66.1±7.9	62.9±15.1
[min-max])	[49–76]	[30–82]
Women n (%)	5 (36%)	8 (62%)
Diagnosis MC visitor
NCI	6 (43%)	5 (39%)
MCI	3 (21%)	4 (30%)
AD dementia	3 (21%)	2 (15%)
CBS	1 (7.5%)	1 (8%)
FTD	1 (7.5%)	1 (8%)
Relationship to MC visitor
Spouse	11 (85%)
Child	2 (15%)

Demographics are represented in n (%) unless stated otherwise. SD, standard deviation; NCI, no cognitive impairment; MCI, mild cognitive impairment; AD, Alzheimer’s disease; CBS, corticobasal syndrome; FTD, frontotemporal dementia.

Results

Following the exploration of potential user views, a consensus was reached on a visual tool with two modules (separate modules for psychologists and patients/family members) and on the cognitive tests to be included (Concept Shifting Test [25], Trail Making Test [26], Letter Digit Substitution Test [27], Stroop Color-Word Test [28], Digit Span from the Wechsler Adult Intelligence Scale IV-NL [29], category fluency [30], Auditory Verbal Learning Test [31], and Visual Association Test [32]). In the prototype, the standardized age and education adjusted scores would be implemented based on the available extensive normative data for the Dutch versions of the commonly used tests in Dutch MCs [33]. The module for the psychologist should improve the diagnostic workflow and should contain detailed information regarding neuropsychological results. Psychologists preferred the incorporation of a reliable change index [34, 35]. The module for the patient should be easy to read and include recommendations for daily life functioning. Patients expressed a need for more information on how to deal with their deficits. Feedback was given on the content (e.g., missing cognitive tests, questionnaires and observations, missing “language” as relevant cognitive domain, descriptions of daily life consequences too brief) and design (e.g., larger font size, different colors, and better contrasts). A line graph was preferred by all users, as this was easiest to interpret.

Phase 2: Development of prototype

The web-based prototype was developed using all feedback from the first phase. Implemented in the prototype of the tool was the automatic transformation of raw scores into age and education adjusted standard scores [33], as well as the computation of reliable change indices according to Jacobson and Truax [34] or Chelune et al. [35]. The cognitive test battery was adapted to the recently published monodisciplinary guideline for neuropsychological assessment in the diagnostic work up for mild cognitive impairment and dementia [36]. The usability of the prototype was evaluated by the same neuropsychological experts (n = 18) who participated in phase 1. Additional experts working at seven different institutions were recruited to maximize variation (total n = 25). The think-aloud procedure was repeated, but this time, experts had to fill in raw test results from a dummy patient into the web-based prototype. They also completed a survey about the design of the tool with eleven items scored on a 7-point Likert scale (1 = totally disagree, 7 = totally agree). Experts were also asked to list three positive and negative aspects of the tool. Usability was evaluated using the System Usability Scale (SUS), which consists of ten items scored on a 5-point Likert scale (1 = strongly disagree, 5 = strongly agree) with a maximum score of 100. This scale has been shown to have good psychometric properties [37, 38].

Overall, the experts were satisfied with the adaptations in the tool (e.g., more detailed description of daily life implications, easy data entry). Some additional remarks to optimize usability were given (e.g., the need for more specific fields for observations during the neuropsychological assessment, clearer warnings for data entry, choice in type of standard scores). On the survey, experts reported being very satisfied with the design (6.1 ± 0.8, range: 5–7), and the usability was rated as excellent (88.1 ± 7.6, range: 70–87) [39]. The most commonly listed positive aspects were user-friendliness, clear information to give to patients, and opportunity to improve harmonization between MCs. The most frequently mentioned negative aspects were the lack of other commonly used tests, and fonts being too small.

Phase 3: Pilot study of feasibility and technology acceptance

Methods

All feedback from phase 2 was implemented, resulting in an optimized prototype of the visual tool (Figs. 2 and 3). An essential adaptation consisted of extending the section on everyday life implications. Here, psychologists could select implications of the neuropsychological assessment results for the patient’s daily life and provide corresponding tips per cognitive domain (e.g., ‘you can switch your attention very well when doing multiple things’, ‘learning new information costs a lot of effort’, ‘writing things down could help you to remember to do the things you have planned’, ‘try and use pictures as a memory aid’). Next to standard sentences, psychologists could add their own interpretations to individualize these recommendations. This prototype was subsequently used in the pilot evaluation in which two groups were compared (one receiving neuropsychological feedback without the tool and one with the visual tool) in three hospital-based centers in the Netherlands (Maastricht UMC+, Catharina Hospital Eindhoven, Radboudumc Nijmegen) to ascertain diversity in diagnostic disclosure procedures.

Fig. 2

Patient view (of a fictional patient) of the visual tool. This figure shows the results of the most recent assessment on the domains of mental speed, complex attention, memory, and language. Each dot in the bar above the cognitive domains represents the score on a subtest. An orange dot indicates a performance lower than minus 2 standard deviations. Below the figure the translation to daily life functioning can be found.

Fig. 3

Psychologist view (of a fictional patient) of the visual tool. LDST, Letter Digit Substitution Test; TMT, Trail Making Test; int, interference; VLT, Verbal Learning Test; VAT, Visual Association Test; BNT, Boston Naming Test; Flu, Fluency.

Pilot study participants

In each of three centers, ten patients from the memory clinic (with a family member, if available) were recruited (30 dyads). Patients were recruited when they were referred for a neuropsychological assessment. Patients were excluded if they had a Mini-Mental State Exam score <20, impaired vision (i.e., would not be able to perceive the visual tool), an intellectual disability (Full Scale IQ ≤70), or a history of (<12 months) or current treatment for psychiatric disorders (e.g., major depressive disorder, bipolar disorder, psychotic disorder). Between February and December 2019, 37 patients and 35 family members were recruited for this pilot study. During the intake with the psychologist, patients, and family members received written information about the study. Two weeks before the consultation with the psychologist, they were contacted by phone to inquire if they wanted to participate. Four patients and two family members did not want to participate, and three patients were excluded. In total, 30 patients and 28 family members participated in the study. One dyad participated in the study but did not return the surveys, despite several reminders by phone. Furthermore, the eight psychologists who used the visual tool participated in the pilot study.

Procedures

During the pilot study, patients (n = 15) and family members (n = 15) received individual neuropsychological feedback without the visual tool. Feedback given by the psychologists was not standardized but could include use of white boards or highlighting results on test scoring sheets. Next, the visual tool was implemented in the participating centers and used by the psychologists during individual feedback sessions with patients (n = 15) and family members (n = 13). After the feedback session, patients and family members of both groups completed several questionnaires. Furthermore, in both groups, six dyads (patient –family members) were randomly selected for in-depth interviews. A semi-structured guide was used with questions about their experience with the consultation and, if applicable, the visual tool. The psychologists also completed a questionnaire after each consultation and participated in an in-depth interview using a semi-structured guide with questions about the experience with the visual tool during neuropsychological feedback.

Questionnaire for the patients and family members

To assess whether patients and their family mem-bers understood the explained findings of the neuropsychological assessment, three items (“I understand my neuropsychological assessment results”, “The explanation gave insight into cognitive weaknesses”, “The explanation gave insight into cognitive strengths”) were constructed by the authors. Items could be scored on a 5-point Likert scale (1 = fully disagree, 5 = fully agree). To evaluate the interaction between participants and psychologists, questions from the Patient Provider Interaction questionnaire [40] were adapted and translated into Dutch. Items were scored on a 4-point Likert scale (1 = no, 2 = possibly no, 3 = possibly yes, 4 = yes). The total score ranged from 0 to 52. Several other questionnaires were also administered after they were translated into Dutch. First, the Patient Satisfaction Questionnaire (PSQ) was used to measure the satisfaction following the consultation using a visual analogue scale ranging from 0 (not satisfied at all) to 100 (extremely satisfied) [41]. The Patient-Doctor Relationship Questionnaire (PDRQ-9) evaluated the perceived relationship (e.g., helpfulness, trustworthiness, understanding) with the psychologist [42], and items could be scored on a 5-point Likert scale (1 = not at all appropriate, 5 = totally appropriate). The total score ranged from 0 to 45. The participants who received feedback with the visual tool were asked additional questions regarding the added value (“The graph made me understand the information better”, “The graph made it easier to follow the consultation”, “The graph gave me more insight into the cognitive complaints”, “Receiving the graph to take home had added value”), layout, and application of the visual tool (e.g., whether or not they inspected it at home or discussed it with others). These items were constructed by the authors and could be scored on a 5-point Likert scale (1 = totally disagree, 5 = totally agree).

Questionnaire for the psychologists

To investigate the level of acceptance, the psychologists completed the Technology Acceptance Model questionnaire [43]. The questions addressed the following domains: perceived usefulness, perceived ease of use, intention to use, and attitude toward using the technology. The higher the scores on these domains, the more likely someone is to use the technology in question (Fig. 4) [43]. Additional questions on feasibility and satisfaction with the visual tool were constructed by the authors. All items (technology acceptance, feasibility, and satisfaction) were scored on a 7-point Likert scale (1 = totally disagree, 7 = totally agree).

Fig. 4

Relation between domains of Technology Acceptance Model and actual system use.

Data analysis

Statistical analyses were conducted using version 24 of the Statistical Package for Social Sciences (SPSS). To examine group differences between the groups (with versus without visual tool), χ2-tests or independent samples t-tests were conducted. Furthermore, descriptive statistics were reported. All interviews were transcribed verbatim and analyzed independently by two raters using ATLAS.ti version 8.4.4. One of the authors was not involved in previous phases of the INPAD study and became involved at a later stage to ascertain an independent view when conducting the analysis. First, an inductive approach was applied by adding open codes in the transcript. Then, themes were based on both the open codes and the semi-structured guide: expectations, experiences during the consultation, facilitators and barriers in remembering the information, and adjustment after the consultation. The categories and codes were discussed with two other authors to improve credibility [44]. To gain insight into differences between the centers, the interviews were analyzed separately. The quotes used in the results section were selected and translated from Dutch into English.

RESULTS

Results among patients and family members

Background characteristics

Table 2 describes the characteristics of 30 patients and 28 family members. The majority of family members were spouses and had daily contact with the patient. No significant differences regarding age, sex, and education were found between participants in the group with and without the tool.

Table 2

Characteristics of patients (n = 30) and family members (n = 28)

	Without visual tool (n = 30)		With visual tool (n = 28)
	Patients	Family members	Patients	Family members
	(n = 15)	(n = 15)	(n = 15)	(n = 13)
Age, y (mean±SD	72.5±8.7 [49–85]	67.2±12.2 [44–77]	71.1±7.9 [59–88]	65.3±13.3 [37–89]
[min-max])
Women	6 (40%)	11 (73%)	6 (43%)	9 (75%)
Education
Low	0 (0%)	0 (0%)	0 (0%)	0 (0%)
Middle	10 (67%)	9 (60%)	10 (71%)	5 (42%)
High	5 (33%)	6 (40%	4 (29%)	7 (58%)
Diagnosis
NCI	6 (40%)		7 (47%)
MCI	6 (40%)		6 (40%)
Dementia	3 (20%)		2 (13%)
Relationship to patient
Spouse		13 (87%)		10 (83%)
Child		2 (13%)		2 (17%)
Contact with patient
Daily		13 (87%)		10 (83%)
4–6 times a week		0 (0%)		0 (0%)
1–3 times a week		2 (13%)		2 (17%)
1–3 times a month		0 (0%)		0 (0%)

Demographics are represented in n (%) unless stated otherwise. SD, standard deviation; NCI, no cognitive impairment; MCI, mild cognitive impairment.

Expectations before the consultation

Both patients and family members with and without the visual tool expressed that they did not know what to expect from the consultation. This was present in two centers where patients received the diagnostic disclosure together with neuropsychological feedback. These patients were worried and experienced feelings of uncertainty. Furthermore, they were all focused on the question of whether they, or their family member, had dementia or not. In one center where neuropsychological feedback was given after two weeks, these feelings were not present.

“Those are all things you take along  ...   There is a signal in your head that is dormant, but present. Then I think, my God, what will be waiting for me if that is the case?” (patient with visual tool)

Experiences during the consultation

Overall, patients and family members in both groups were very positive regarding the consultation with the psychologist. They liked the psychologists’ tranquility, clear explanation, and attitude. They also appreciated the opportunity to ask all the questions they wanted, and they felt no time constraints. On the questionnaire, no significant differences were found between the groups concerning interaction, patient-psychologist relationship, and satisfaction with the consultation (Table 3).

Table 3

Comparison between groups without and with the visual tool concerning communication, satisfaction and patient-psychologist relationship

	Without visual tool		With visual tool
	(n = 30)		(n = 28)
	Patients	Family members	Patients	Family members
	(n = 15)	(n = 15)	(n = 15)	(n = 13)
Communication	46.7 ± 4.7 [23–43]	49.9 ± 3.9 [39–52]	47.6 ± 3.6 [41–52]	49.8 ± 2.3 [44–52]
Satisfaction with
Needs addressed	80.7 ± 14.9 [60–100]	83.3 ± 9.8 [70–100]	74.3 ± 22.1 [20–100]	80.0 ± 12.1 [50–100]
Patient’s involvement	83.3 ± 12.9 [60–100]	86.0 ± 9.9 [70–100]	79.3 ± 23.0 [20–100]	84.2 ± 10.0 [70–100]
Information given	84.0 ± 12.4 [60–100]	86.0 ± 11.8 [60–100]	76.4 ± 22.4 [20–100]	84.2 ± 9.9 [70–100]
Emotional support	78.7 ± 24.8 [0–100]	84.0 ± 11.8 [70–100]	77.9 ± 19.7 [40–100]	80.8 ± 14.4 [50–100]
Interaction in general	85.33 ± 11.9 [70–100]	86.7 ± 10.5 [70–100]	81.4 ± 16.1 [50–100]	84.2 ± 10.0 [70–100]
Overall satisfaction	82.4 ± 11.1 [66–100]	85.2 ± 9.3 [70–100]	77.9 ± 19.6 [32–100]	82.7 ± 9.9 [64–100]
Relationship	39.7 ± 4.9 [32–45]	38.7± 4.9 [32–45]	37.9 ± 6.2 [27–45]	39.3 ± 4.2 [33–45]

All scores are represented as the mean ± SD [range]. The higher the score, the more patients or family members agreed. The communication scale ranged from 0 to 52, the satisfaction scales from 0 to 100 and the relationship scale from 0 to 45.

“I went home feeling relieved. Perhaps due to receiving attention for the first time. Those people know what they are doing. So many people do not understand it at all. You get the feeling that you are stupid.” (center 3, patient without visual tool)

Regardless of the visual tool, information retention was reported to be low, which was experienced more by patients than by family members. They often remembered the diagnosis (or only that it was not dementia) but forgot other details. In case of no dementia this was accompanied by feelings of relief; in case of a dementia diagnosis, this was a confirmation of their suspicions. Participants needed some time during the consultation to process the information about the diagnosis.

“It lets you know where this is coming from. This gives some peace of mind even though it is a disease that is not really nice. Then you know what caused it and that you are not going crazy.” (center 1, patient with visual tool)

Patients and family members especially perceived benefits due to receiving information and concrete recommendations about the implications of the neuropsychological assessment results for daily life. The group in which the visual tool was used, appreciated that they could take home the concrete tips on paper. In both groups, the consultation mainly focused on cognitive weaknesses, and participants would have liked to hear more about cognitive strengths as well.

Facilitators and barriers in remembering information

Over 60% of the patients and 80% of the family members in the group without the visual tool indicated that they would have liked to receive something on paper to take home and thought that a visual aid would have been helpful. During the interviews, they explained that receiving information on paper meant that they could read it at home at their own pace. When the visual tool was used, more than 90% of the patients and 80% of the family members liked that a graph was used and perceived the ability to take it home on paper as having added value. Both groups indicated that taking someone with you to the consultation facilitated information retention. Barriers in information retention were related to the emotions that accompanied receiving the diagnosis, cognitive impairment of the patient, and the setting of the consultation (e.g., when there were too many people present).

“Just put everything on paper. If you cannot recall it anymore you can check it again. If you then still do not recall it, you check it again. Paper is very patient.” (center 3, family member without visual tool)

“The psychologist drew a figure of the cognitive profile of my mother. And then I thought it wasn’t too bad  ...   A picture says more than a thousand words. Then you can see it visually.” (center 2, family member without visual tool)

Adjustment after the consultation

During the interviews, patients and family members from both groups mentioned that they discussed the results of the neuropsychological assessment within their social network. They also experienced that they were able to better cope with the cognitive complaints after the consultation (e.g., being more honest, more understanding toward the cognitive complaints). However, it remained difficult for some family members to adapt to the changed behaviors of the patient.

Opinion of visual tool

Patients and family members indicated that visual information was easier to understand and easier to remember compared to only receiving verbal information. One patient with dementia found the graph difficult to understand. Other aspects that added value included the ability to see their cognitive profile at one glance, the ability to receive implications of the neuropsychological assessment results for daily life, and an easier interpretation of their own performance by comparing it to a reference group. They also highly appreciated that they received something to take home (Fig. 5). Furthermore, they emphasized that the visual tool is only helpful when explained by psychologists during a face-to-face consultation.

Fig. 5

Level of agreement regarding the added value of the visual tool in patients (n = 15) and family members (n = 13).

Results among psychologists

The work experience of the psychologists involved (n = 8) was on average (mean±SD) 14.3±6.6 years [range: 2.5–22]. Seven psychologists followed basic or advanced post-master’s professional clinical training, and one psychologist was still in training for a clinical license.

Added value of visual tool

In line with the patients and family members, psychologists reported seeing the cognitive strengths and weaknesses at one glance and giving something to take home as advantages of the visual tool. Having the reliable change index implemented in the tool was also considered a benefit. The psychologists argued that using multiple modalities (auditory, written, and visual) might lead to a better understanding and retention of the information. The visual tool was evaluated as user-friendly and easy to use. Furthermore, some psychologists considered the visual tool as a way to standardize neuropsychological feedback sessions across psychologists.

“I think we say a lot to patients who sometimes only remember if they have dementia or not. By making it visual you can take them by the hand, and I think they will remember it much better  ...   It really helps because you when you explain this in words, that makes it easily gets more complicated. To keep the language simple, a visual aid is very supportive.” (psychologist 8)

“I think the added value is that you can see the profile of strengths and weakness in one image. With repeated measurements you can also see it at one glance, what went better and what takes more effort. You have to use less words to explain this, and the words you use are supported by the visual tool.” (psychologist 2)

Changes in the consultation

Using the visual tool led to a different way of explaining neuropsychological assessment results. It was considered valuable for explaining these results because psychologists could use the tool to summarize their explanation, and it felt more professional as they did not have to search through papers or draw their own graphs on a board or piece of paper. According to the psychologists, the use of the visual tool did not necessarily lead to more interaction. However, they indicated that the structure of the interaction did change, as patients and family members were prompted by the tool to ask questions right away. All psychologists said they would use the tool in their clinical work, but using the tool was something they had to get used to. Furthermore, psychologists in two centers said they would only use it if barriers and concerns were solved. This was particularly related to the integration of more tests in the tool to fit better with routine practice.

“I had to get used to it, because I have a routine in how I say and explain things.  ...   I was searching for what I was actually seeing on my screen and how I could explain that in small steps to the person sitting in front of me.” (psychologist 2)

“It makes it professional in a way. It appeals more to the imagination more than when I only have a folder full of loose sheets and papers” (psychologist 4)

Barriers and concerns

As mentioned above, a major concern was the limited number of tests available in the visual tool, which made it difficult to explain the full assessment. A few practical concerns were also raised, such as better integration with the software of the electronic patient files so that test results do not have to be entered twice, a lack of clarity with respect to the graphs due to grayscale printers at the clinics, and the low availability of computers in some consultation rooms.

“Just imagine that you administered other cognitive tests which are now missing in the tool. This can truly change the visualization of the results. Sometimes it seemed as if the domain memory had a very low performance, but we administered other memory tests that went well. Then, I have to adapt that with a pen and that creates confusion.” (psychologist 7)

Technology acceptance

All psychologists indicated that the visual tool was helpful for explaining neuropsychological assessment results and that the ability to select the imp-lications of the results for daily life was an important feature of the system. They were satisfied with the system (mean±SD, range; 5.6±0.7, 5–7) and indicated that it had a good design (5.9±0.6, 5–7), attractive layout (6.0±0.5, 5–7), and readable text (5.8±0.7, 5–7). Furthermore, high levels of agreement were present in the domains of perceived usefulness (5.4±0.8, 4.0–6.2) and attitude toward using (6.0±0.8, 4.5–7.0). Average agreement was reported on perceived ease of use (3.5±0.6, 2.6–4.3).

DISCUSSION

The aim of this study was to develop a web-based visual tool to improve the explanation and understanding of neuropsychological test results, information retention, and psychologist-patient communication. The second aim was to determine the usability, technology acceptance, and feasibility of the tool. An iterative process incorporating perspectives of different end-users (patients, family members, neuropsychology experts) was used to develop the tool. During the early development, already all users reported positive feedback about the visual tool to explain the results of a neuropsychological assessment. The usability of the tool was evaluated as excellent by psychologists. The pilot study indicated that regardless of the visual tool, patients and family members were positive regarding the consultation with the psychologist. The visual tool led to an increase in understanding of neuropsychological assessment results. All users reported high levels of satisfaction with the visual tool. In general, these high levels could be explained by having something available to aid explanation and to improve awareness of receiving extra explanation. In addition, the qualitative data showed that patients, family members, and psychologists specifically indicated that the overview of cognitive strengths and weaknesses, the implications of the results for daily life, and receiving something to take home was an added value of the visual tool. However, the information retention in patients remained low. Psychologists indicated that the tool was useful and intuitive, and technology acceptance was high. Nevertheless, some barriers were mentioned that have to be addressed to increase feasibility in clinical practice.

Experiences with neuropsychological feedback

Patients and family members from both groups were positive regarding the consultation with the psychologist, which might be related to patients receiving useful feedback [8, 11]. Neuropsychological feedback is increasingly part of care as usual and used as a therapeutic intervention [8, 45]. However, in Dutch MCs, neuropsychological feedback is not part of standard practice. This is sometimes related to financial barriers but may also be due to the multidisciplinary nature of the MC [46], in which the medical specialist often communicates the diagnosis. Based on the small body of literature on experiences with a neuropsychological assessment in MC settings, a need for a follow-up consultation with the neuropsychologist has been reported [8, 24, 47, 48]. Furthermore, providing neuropsychological feedback has been related to positive outcomes in other disorders [7]. For example, in patients with traumatic brain injury, feedback helped both patients and family cope with the consequences [49], and in patients with a mood disorder, reductions in psychiatric and cognitive symptoms were observed [50].

Information retention during neuropsychological feedback

In our pilot study, the information retention of patients, with and without the visual tool, remained low. No other studies have investigated whether retention of neuropsychological feedback improved by offering visual aids, but other fields showed improved retention and understanding [21, 51]. Fewer worries were reported among patients and families in the center who had already received the diagnostic disclosure two weeks before receiving neuropsychological feedback. While low retention is likely to be at least partly due to memory problems, other factors may play a role as well. For example, the low level of information retention might be due to the emotional impact of receiving a diagnosis. Emotional information might be prioritized over other competing information, and neutral information has been found to decay over time more rapidly than emotionally loaded information [5, 52]. Other explanations for low medical information retention could be clinician-related factors (e.g., using difficult terminology), patient-related factors (e.g., education, experienced stress), or a focus on diagnosis-related information [13, 15]. All participants indicated that receiving both oral, written, and visual information would facilitate information retention. To our knowledge, only two studies showed that written information increased free recall of neuropsychological recommendations, but no differences were found in the recall of diagnostic information [16, 17]. Offering tailored and high-quality information remains important because it can give patients the opportunity to take more responsibility for their health, respond better to health-related problems [53], feel more in control [54], and participate in a more active manner in shared decision making [55]. It is also important to provide neuropsychological feedback after patients have received the diagnostic disclosure and not at the same time. Furthermore, interventions that support information provision improved quality of life in patients with dementia [56].

Strengths and limitations

A prominent strength of the current study is co-creation with different users of the tool, which might have resulted in the high rates of usability in our visual tool. The feedback of patients, family members, and neuropsychology experts on the content, design, and layout of the tool led to an adapted version that better fits the needs of the end-users. Co-creation involves different user perspectives because every individual brings their own knowledge, which can lead to collective creativity [57], a shared understanding of the product [58], innovative generation of ideas, critical thinking, reciprocal knowledge gain, and long-term involvement of the potential users [59]. From a software development perspective, high usability (also referred to as “quality in use”) prevents the occurrence of errors and results in higher user satisfaction, reduced training and documentation costs, and a better cost-benefit ratio [60]. Another strength is the involvement of three different centers to take the variability between centers into account. The variability in the sample was further increased by using a purposive sampling approach to account for the heterogeneity present in memory clinic populations. Furthermore, multiple types of triangulation in our methodological approach have led to an enrichment of the data.

However, some limitations have to be considered as well. Due to the variability between the centers, it is difficult to generalize our findings to a broader memory clinic population. Methodological problems with questionnaire data should also be considered. For example, respondents often score equally high or low on domains of outcome measures, which can lead to an underestimation [61]. Another important note is that the visual tool might not be suitable for all participants. In our study, one patient with dementia found the visual tool too difficult to comprehend. However, the caregiver of this patient was very grateful for the information because it gave her more tools to cope with the diagnosis of her husband.

CONCLUSION

A standardized visual tool could be used as a com-munication aid during neuropsychological feedback sessions by giving the psychologist a frame of relevant feedback aspects, including the translation to daily life recommendations. The visual tool developed in this paper provides such a framework and can be used as a starting point. Future research should optimize the visual tool. More cognitive tests and normative data should be included. Additionally, integration with the hospital-based patient system would be ideal. Moreover, adaptation of the tool for other patient groups, such as patients with traumatic brain injury, stroke or psychiatric disorders, should be explored, as well as longer evaluation periods in larger samples.