Qualitative study of affective identities in dementia patients for the design of cognitive assistive technologies.

Our overall aim is to develop an emotionally intelligent cognitive assistant (ICA) to help older adults with Alzheimer's disease (AD) to complete activities of daily living more independently. For improved adoption, such a system should take into account how individuals feel about who they are. This paper investigates different affective identities found in older care home residents with AD, leading to a computational characterization of these aspects and, thus, tailored prompts to each specific individual's identity in a way that potentially ensures smoother and more effective uptake and response. We report on a set of qualitative interviews with 12 older adult care home residents and caregivers. The interview covered life domains (family, origin, occupation, etc.), and feelings related to the ICA. All interviews were transcribed and analyzed to extract a set of affective identities, coded according to the social-psychological principles of affect control theory (ACT). Preliminary results show that a set of identities can be extracted for each participant (e.g. father, husband). Furthermore, our results provide support for the proposition that, while identities grounded in memories fade as a person loses their memory, habitual aspects of identity that reflect the overall "persona" may persist longer, even without situational context.

Introduction

There are nearly 900 million people aged 60 years and over living worldwide, and this rise in life expectancy is associated with increased prevalence of chronic diseases like dementia; today 46 million people live with dementia worldwide (World Alzheimer Report)[1]. Dementia is characterized by the progressive deterioration of both cognitive and functional abilities, leading gradually to a loss of the ability to perform daily living activities (ADLs, e.g. domestic activities, medication).[2]

Cognitive assistive technologies have been put forward as a solution to support older adults with dementia in living independently at home for longer while not increasing the burden on family caregivers.[3] Such technologies, often coupled with a form of artificial intelligence, facilitate compensation for lost functions by prompting the user through different steps of certain ADLs, thereby enhancing the user's autonomy.

Previously, an artificially intelligent cognitive assistant (ICA) entitled the COACH was developed to assist older adults with Alzheimer's disease (AD) during ADL through the monitoring of the person and the provision of audio-visual cues from a fixed pre-recorded (“canned”) set when the person stops making progress.[3-5] The COACH has been found to work very well when tested in a real-world setting, reducing the need for caregiver assistance by up to 100% in some cases.[3,5]

However, for some persons, the COACH failed to provide appropriate assistance, leading to confusion or agitation in the older adult end-user. This may be due to an affective (emotional) misalignment of the COACH with the specific needs of the individual. Considering the heterogeneity in socio-cultural and personal affective identities, a primary reason for lack of effectiveness may be the static, non-adaptive nature of the “canned” prompts. While significant effort has been made to design prompts based on the methods and styles of human caregivers,[6] a simple “one size fits all” style of prompting may be limiting. For example, one person may find certain prompts to be too imperious, and would respond better to a more servile approach, while some may prefer a more imperative prompting style. When developing technology assistants for older adults, it is critical to consider that each person comes from a different background, has a different sense of self and identity, and has different emotional responses to prompts, whether given by human or machine.

Affective identity is considered a powerful tool for reasoning about illness in general.[7] Studies of identity in AD have found that identity changes dramatically over the course of the disease[8] and that persons with AD often have more vague or abstract notions of their self-identity[9] than do non-afflicted persons. Previous research on engagement with AD patients has shown that, for instance, interventions that involve tasks or objects with specific meaning to the person, and thus support their sense of self, will be more likely to engage that person.[10] Thus, explicit models of emotional identity offer an attractive mechanism for developing more appropriate and effective assistive technologies.

Emotion, identity, and socio-cultural civility and normative behavior are missing from most current ICAs. In order to be fully effective, usable, and desirable to older adults, an ICA must be able to reason dynamically about the affective nature of an interaction. The work we describe in this paper unites two main streams of our research: work on the COACH ICA, and work on a basic theory of emotional interaction called Affect Control Theory.

Affect Control Theory (ACT) is a comprehensive social psychological theory of social interaction.[11] ACT proposes that people's social perceptions, actions, and emotional experiences are governed by a psychological need to maintain consistency between culturally shared fundamental sentiments about social situations and transient impressions resulting from the dynamic behaviors of people in those situations. Humans seek and create situations that confirm or are consistent with, and avoid and suppress situations that disconfirm or are inconsistent with, their culturally shared affective sentiments. This “affect control principle” is a powerful predictor of human behavior[12] in many social domains, including health and illness.[7] Fundamental sentiments are represented as social objects (such as people's identities and behaviors or environmental settings) as vectors in a three-dimensional affective space. The basic vectors of the affective space are called Evaluation/valence, Potency/control, and Activity/arousal (EPA). The EPA space is hypothesized to be a universal organizing principle of human socio-emotional experience, based on the discovery that these dimensions structure the semantic relations of linguistic concepts across languages and cultures.[13,14] EPA profiles of concepts can be measured with the semantic differential, a survey technique where respondents rate affective meanings of concepts on numerical scales. In general, within-cultural agreement about EPA meanings of social concepts is high even across subgroups of society, and cultural-average EPA ratings from as few as a few dozen survey participants have been shown to be extremely stable over extended periods of time.[13]

Identities in ACT refer to situational roles that a person takes on when interacting with others. The associated sentiments are known as identity sentiments. Thus, in an academic institutional context, when teaching a class of students, a person may select an identity of “professor,” with an associated identity sentiment that is somewhat good (approximately 1.5) somewhat powerful (approximately 1.5) and not active or inactive (approximately 0.5). ACT also explains a person's overall sense of self as a synthesis or combination of many different identities that are regularly enacted.[15] This combination is referred to as the “persona,” and the associated sentiment as the “fundamental self-sentiment” or “persona sentiment.” Thus, the “professor” may also be a “father,” a “husband,” and a “musician,” for example. The persona is a combination of these four identities (with possibly different weights), and the self-sentiment is either an averaged single value,[15] or a probabilistic mixture.[16]

The current research aims to extract identities based on ACT from qualitative interviews (N = 21) with end-users and to further our understanding of how they interact with the aim of producing more affectively aligned reactions of a virtual assistant. Findings will in turn inform the improvement of the overall (cross-individual) effectiveness and potential uptake of such systems. We believe that this emerging emotional or affective reasoning technology will surmount one of the last major hurdles before ICAs can be used in a widespread manner for effective self-managed aging-in-place. The first analysis and outcomes of the interviews were coded using affect control theory (ACT), which will build the basis for a novel emotionally intelligent ICA, as described in the next section.

System architecture

The system for assisting residents with dementia with the hand-washing task has three different input channels, as shown in Figure 1. A camera mounted on the ceiling, above a sink, captures a video, which is used to track hand locations. The hand-tracking system is used to estimate the progress of the user in the task. The estimated stage of the user during the task based on the hand positions is used as a belief state for the next step in the system. The second input is captured via another camera mounted near the sink or tap to record the facial attributes. The face analysis system analyses the facial expressions and predicts the current affective or emotional state of the user. There has been a significant amount of research on automatic facial expression recognition.[17] A facial expression analysis pipeline similar to that described in Joshi et al.[18] is adapted to construct the face analysis component. The final input to the system is based on an in-person interview conducted with the user before their interaction with the system, during which different identity profiles and affective identities associated with the user are determined. This interview is currently being developed based on the work presented in this paper.

Figure 1.

Schematic of the system.

The outputs from the three components are fed into a BayesACT[19] engine. BayesACT is a probabilistic and decision-theoretic generalization of ACT. It can maintain multiple hypotheses about sentiments simultaneously as a probability distribution, and can make use of an explicit utility function to make value-directed action choices. This allows the model to generate affectively intelligent interactions with people by learning about their identity, predicting their behaviors using the ACT principles of affective alignment, and taking actions that are simultaneously goal-directed (get the person's hands washed) and affect-sensitive (do so in a way that is emotionally aligned). This engine is able to choose an appropriate action with an EPA output that minimizes deflection according to ACT principles. Identity and emotionally aware prompting instructions are passed on and are delivered by a virtual assistant, an example of which (developed by researchers at the University of Colorado) is shown in Figure 1.

Methods

Participants

The study was approved by the University of Waterloo Office of Research Ethics, and by the Schlegel/UW Research Institute for Aging ethics and research committee, and subjects gave informed consent to participate. In any cases of incapacity to give informed consent, the current surrogate decision maker, often a family member, was asked to give consent for study participation.

Twelve older adult care home residents and nine associated caregivers were enrolled and interviewed for this study. Each interview session lasted approximately 45 minutes, was audio recorded upon obtaining the participant's consent, and was carried out by either a trained psychologist or sociologist. Inclusion criteria for the residents were: (1) over the age of 50 years; (2) fluent in English; (3) normal or corrected hearing; (4) diagnosis of dementia; (5) cognitive impairment; and (6) impairment in initiating and performing sequences of ADL steps (as reported by their caregivers). The inclusion criteria for the caregivers was that they must be familiar with the resident they were paired with. No technology proficiency was required from either the end-user or the caregiver.

Questionnaires

Based on the previously described principles of ACT, a semi-structured interview tool was designed for older adult residents of a care home as well as their caregivers. All questions were open-ended and phrased conversationally, to allow maximum flexibility in accommodating the needs of respondents. The questionnaire was divided into two sections: (1) demographic and role-related parameters; and (2) evaluation of the digital virtual assistant (see Figure 2) developed by the University of Colorado.20 For this evaluation, different short videos of the virtual assistant providing prompts were shown to the participants who were asked to comment on its design and on how they perceived the different prompts (see Table 1). This section of the interview also sought to determine the likelihood that the virtual assistant face or voice would increase rather than decrease confusion, or be rejected and disliked due to an emotional misalignment with their identities.

Figure 2.

Screenshot of ICA.

Table 1.

Overview of interview tool.

Identity-domains	Residents	Caregivers
Family roles	With whom did you live? Are you married? Do you have children? How many? What are their names/ages? Do they visit you? …	Do you know what kinds of family roles he/she has or has had? Does he/she ever go back to those roles when confused? How do you know when he/she has taken on that past role? Are there any particular triggers to bringing out these roles? How do you respond when he/she thinks he/she is in this role?
Occupational role	What kind of work did you do before you retired? Did you always do that job? Did you enjoy the work that you did?…	Do you know what kinds of occupational roles he/she has or has had? …
Historical roles	Were you born in Canada? Why did you immigrate to Canada? Were you ever in the military, or involved in a war?…	Do you know what kinds of historical roles he/she has or has had? …
Relational identities  with emotional content	Who or what makes you feel happy? When he or she is doing/saying what? Who or what makes you feel angry or upset?…	Do any of these identity changes cause strong emotions (fear, anxiety, anger, etc.)? What helps give him/her reassurance when confused? …
Feedback on virtual  assistant	If you saw this virtual assistant on a screen by the sink in the washroom, would it bother you? How would you feel having the virtual assistant talk to you? Overall, what do you like about this virtual assistant? What do you not like? Do you think it can be improved? How?	In your opinion, do you think XY would be willing and able to respond to prompts from an avatar like this? Overall, what do you like about this computer assistant? What do you not like? Do you think it can be improved? How?

To characterize the participants' identities, the questionnaire asked questions about life domains (family and origin, occupation/vocation, personal history such as immigration or military service, and relationships associated with strong feelings such as traumatic or deeply romantic ones). The categories of identities were based on structural approaches to identity,[21] using some of the most common domains of identity: family, work, and networks (see Table 1). The interview also drew on identity work related to ACT,[22] which argues that a person's identities represent that individual's historical biography, and thus are subject to change and development as the person's situation changes. Thus the questions were kept general, with a historical focus to encourage better recall among elders with dementia, but also allowing for current changes.

Data analysis

All interview recordings were transcribed and then coded and analyzed using HyperResearch (version 3.7.3, Copyright ©1988–2014 Researchware, Inc.), a qualitative text analysis program, in order to identify sets of affective identities for each participant as well as their emotional content. After an extensive discussion between the two researchers who performed the interviews, agreement was reached on a preliminary coding system that was organized mainly around the three themes: biographical identity, current identity, and loss of identity or confusion. In this paper, we focus mainly on the biographical identities. Furthermore, once the main identities were extracted, a numerical code was attributed to each identity using its “EPA rating” (described below). This rating will serve as the basis for the ICA to learn to provide the appropriate response.

EPA ratings are constructed from scales presenting adjectives at each end point in order to describe the negative and positive poles of the dimension (Evaluation, Potency and Activation). Nine marking positions are between the end points, and adverbs at the bottom characterize the meaning of each marking position (See Figure 3). Positive units measure goodness, powerfulness, and liveliness; negative units indicate badness, powerlessness, or quiescence. Ratings are converted into numbers depending on which position is marked. This specific EPA rating scale used is referred to as a “semantic differential.” For example, something that a rater views as “quite good, nice” would lead to a score of +2 on Evaluation.

Figure 3.

Rating scales for measuring EPA—a “semantic differential.”

These EPA ratings correspond to values that have been established previously through surveys conducted in multiple countries using batteries of scales. The resulting data sets have been organized in “dictionaries”: EPA ratings for thousands of identities, behaviors, and traits in five languages across six countries. For instance, the EPA rating in the Canadian ACT dictionary for the identity “nurse” is perceived (Figure 3) among the survey respondents as E: 1.65 (quite positive); P: 0.93 (slightly powerful) and A: 0.34 (neutral active). For our study, we used mainly the Canadian EPA values[23] that were gathered and collected from surveys of undergraduates in Ontario in 2001. If a certain identity was not found in the Canadian dictionary, we looked it up in the American dictionary constructed from surveys of undergraduates in Indiana in 2002–2004.

Results

In total N = 12 care home residents were interviewed (n = 7 females, n = 5 males), and N = 9 caregivers (n = 7 females, n = 2 males). The mean age of the residents was 84.5 years with a range of 63–96 years. All residents had been diagnosed with AD and showed cognitive and functional impairment to an extent that affected their autonomy in performing certain complex ADLs.

Identity and persona

Based on the interviews with older adults and caregivers, only the most important identities that seem to shape the overall persona were extracted and their related EPA value drawn from the dictionary attributed. Characteristics and extracted identities for each participant are presented in Table 2.

Table 2.

Overview of characteristics of interviewed residents and their caregivers and their set of affective identities with associated EPA values.

Participants	Gender	Age	Clinical information	Biographical identities	ACT dictionary EPA values of persona
R01	Nurse in the care home	63	Young-onset moderate dementia	From Waterloo, father, husband, brother, Canadian, lawyer	Father E: 1.84—quite positive P: 1.78—slightly to quite powerful A: 0.02—neutral active Lawyer E: 0.21—neutral P: 2.09—quite powerful A: 0.12—neutral Husband E: 1.15—slightly positive P: 1.04—slightly powerful A: 0.10—neutral
C01	Female			Lawyer, father, husband, exerciser
R02	Male	82	Moderate dementia	Many different jobs (cars, grocery store), manager, lost father in war, brother, husband–lover, widow, father	Father: E: 1.84/P: 1.78/A: 0.02 Salesman: E: 0.33/P: 0.18/A: 0.73 Husband: E: 1.15/P: 1.04/A: 0.10 Gentleman: E: 2.21/P: 1.21/A: 0.60
C02	N/A			N/A
R03	Female	93	Mild to moderate dementia	English, immigrant, housewife, mother, sister	Briton: E: 1.25/P: 0.42/A: 0.27 Mother: E: 2.19/P: 1.75/A: 0.32 Housewife: E: 1.46/P: −0.24/A: 0.27
C03	N/A			N/A
R04	Female	90	Moderate dementia	Athlete, feminist, wife, mother, nurse	Athlete: E: 1.00/P: 1.58/A: 2.04 Wife: E: 2.38/P: 0.81/A: 1.22 Mother: E: 2.19/P: 1.75/A: 0.32 Feminist: E: 0.05/P: 0.14/A: 1.05
C04	N/A			N/A
R05	Female	84	Moderate dementia	Wife, mother, sister, home maker, church goer, fortunate woman	Mother: E: 2.19/P: 1.75/A: 0.32 Homemaker: E: 1.46/P: −0.04/A: 0.01 Wife: E: 2.38/P: 0.81/A: 1.22 Christian: E: 1.10/P: 0.26/A: −0.07 Sister: E: 1.84/P: 0.47/A: 1.54
C05	Daughter			Wife, mother, homemaker, career, German background, older sister, peacekeeper, protector of sister
R06	Male	84	Moderate dementia	Chaplain, father, pastor, husband, Dutchman	Dutchman: E: 0.83/P: 0.47/A: 0.20 Father: E: 1.84/P: 1.78 /A: 0.02 Priest: E: 2.20/P: 1.09/A: −1.50
C06	Daughter			Dutchman (immigrant), father, brother, Sunday school teacher, husband, likes to be in control, respects more men, proud person
R07	Male	76	Moderate dementia	Teacher, PhD student, businessman, bisexual	Teacher: E: 2.08/P: 1.60/A: 0.07 Academic: E: 1.54/P: 1.02/A: 0.30 Bisexual: E: 0.39/P: −0.38/A: 1.33 Husband: E: 1.15/P: 1.04 /A: 0.10
C07	Wife			Bipolar, husband, happy childhood, good student, good teacher, grandson, close to mother, conflict with father, environmental worker, businessman, helper
R08	Male	92	Severe dementia	Worked in customs, stock keeper, father	Brother: E: 1.66/P: 1.28/A: 1.35 Father: E: 1.84/P: 1.78 /A: 0.02 Slob: E: −0.93/P: −0.92/A: 0.00 Sports fan: E: 1.36/P: 0.97/A: 2.23 Army officer: E: 0.72/P: 1.87/A: 1.41
C08	Son			Brother, father, custom and excise officer, sports fan, uncle, self-centered, loner, worked for government, military (soldier), alienates himself from family, slob (doing nothing), husband, golfer
R09	Female	96	Severe dementia	Teacher, wife, mother	Teacher: E: 2.08/P: 1.60/A: 0.07 Mother: E: 2.19/P: 1.75/A: 0.32 Sister: E: 1.84/P: 0.47/A: 1.54 Wife: E: 2.38/P: 0.81/A: 1.22
C09	Daughter			Good historian, abandoned, sister, aunt, mother, librarian, teacher, worked in sales, worked Red Cross during war, wife, engineer's wife, divorced, controlling
R10	Female	83	Mild to moderate dementia	Wife, mother, sister, secretary, homemaker, tailor	Bookkeeper: E: 0.79/P: 0.02/A: −0.92 Mother: E: 2.19/P: 1.75/A: 0.32 Homemaker: E: 1.46/P: −0.04/A: −0.01 Wife: E: 2.38/P: 0.81/A: 1.22
C10	Daughter			Tailor, wife, mother, bookkeeper, daughter, sister, loves kids, homemaker, secretary, worked at a soda company, likes to take care of others, sister, daughter, steady, looks on bright side
R11	Male	90	Moderate dementia	Biology teacher, lived in Africa, father, adventurer, live alone, brother, military airforce (ground crew)	Teacher: E: 2.08/P: 1.60/A: 0.07 Father: E: 1.84/P: 1.78 /A: 0.02 Airforce officer: E: 1.36/P: 1.74/A: 1.66 Traveler: E: 1.38/P: 0.59/A: 0.78 Divorcee: E: −0.52/P: −0.19/A: −0.18
C11	Daughter			Adventurer, father, teacher, lived in Africa, airforce (military) husband, divorced, big brother, grandchild (closer to grandfather than his parents), principal and superintendent, school administrator and a town administrator, beer drinker
R12	Male	82	Moderate dementia	Husband, widower, father, carer of sick wife, “fixer,” brother, steady job was in a distillery for 40 years, proud	Smart aleck: E: 0.99/P: −0.40/A: 0.79 Father: E: 1.84/P: 1.78 /A: 0.02 Farmer: E: 1.55/P: 0.76/P: −0.80 Brother: E: 1.66/P: 1.28/A: 1.35 Husband: E: 1.15/P: 1.04 /A: 0.10
C12	Son			Father, smart aleck, alcoholic, easy going, helper/career, brother, submissive—no argue, truck driver, farmer/gardener, winemaker, dancer, horseshoe player, provider, family-centered

R01, resident; C01, caregiver.

Interviews with individuals who were most impaired highlighted a greater loss of identity; they were unable to remember details about large periods of their lives. Yet each retained a core persona, which can be defined as a synthesis of many different identities that are regularly enacted over the course of a life and, thus, shape their interactions and their responses to our questions, as exemplified by Resident 03 (R03), a woman in her late 80s with moderately advanced AD, who had difficulty answering simple questions. However, distinctive characteristics of her persona emerged in her interview, despite her difficulty in remembering her biographical identities.For R03, we attributed the three following main identities and extracted the associated EPA rating value from the ACT dictionary: Briton (E: 1.25/P: 0.42/A: 0.27); Mother (E: 2.19/P: 1.75/A: 0.32) and Housewife (E: 1.46/P: −0.24/A: 0.27). Overall, this participant seems to have a “persona” that is slightly to quite positive, neutral to slightly powerful, and neutral active. The main identities of the persona were chosen based on confirmation by the caregiver or on an emphasis placed by the resident.

– So where did you live before you came here to [facility]?

R03: I was in England. Before….

– Before coming here. And with whom did you live?

R03: In England, umm… let me see…. It's hard to say (laughs).

– Were you married?

R03: Oh yes. Yes… I had a… had children. (pause for 3 seconds)

– What were the names of your children?

R03: I don't know (laughs).

– (Laughs along) That's okay, that's okay. But generally you have a good relationship with your family members, you get along?

R03: Oh yes… yes… I do.

– Did you grow up in England?

R03: Yes. North of…. where… I, where I where… it's hard to say. (pause for 8 seconds)

– Doesn't matter…

R03: What I'd like to know (laughs) because I don't it… know [unclear] I had to [group this?] was in Canada and I try to get them together, but they had some little ones when I get them, got them together and that uhh…

– Do you know when that was?

R03: No… I don't… honestly… I honestly cannot tell….

– It doesn't matter.

For men, the identity of “breadwinner” remained salient; even when they had been retired for some time, or could not remember details of their occupation. This is exemplified in a quote from R06, who insisted on bringing up his work despite attempts to discuss different topics.Both R06 and his caregiver both frequently mentioned that he was originally from the Netherlands and that he cares a lot for his children, which led to the assignment of the following three biographical identities: Dutchman (E: 0.83/P: 0.47/A: 0.20), Father (E: 1.84/P: 1.78/A: 0.02) and Priest (E: 2.20/P: 1.09/A: −1.50). When discussing potential sources of agitation or conflict with the caregiver, it was suggested to treat him according to his past identities and not current ones such as “patient,” thus in a suggestive rather than commanding manner in order to avoid making him feel inferior. This type of information is critical to inform the design of the prompting styles to be provided by the ICA.For older adults who were less impaired, illness-driven identity changes were more challenging to uncover. Such respondents were able to bring a stronger focus to the situation, were generally able to provide more details about their pasts, and responded more directly to the questions asked. In these cases, interviews with those who knew them, such as family members or staff, were frequently much more informative in describing how AD had impacted the participant's identity.Generally, higher EPA values for the identities informing the overall persona were associated with indications that ICAs exhibiting more dominating/directing prompting style would not be effective.In-depth analysis of interview transcripts revealed that residents retained their personas despite circumstances, such as AD, that weakened their identity. R12, for instance, was described by his son as someone who has always been selfish, a smart aleck, and unwilling to put himself out for anyone. These traits were present despite the impact of dementia of R12's identity.

– So I am a psychologist but I work together with the Computer Science department at the University of Waterloo.

R06: Okay?

– So and they are developing a system that's supposed to help people, elderly people in their activities of daily living by prompting, they say, by guiding them once for example you don't know any more what you were doing in the activity, to help you to find your way back in the activity….

R06: I have worked as a chaplain in a large hospital in Toronto.

– Ahh okay, okay… so yea so… so just to say, we would like to find out something about people for who we are designing the system you know, so that's why I would like to ask you actually a lot of questions actually about yourself. Like for example to start, umm… a little bit about you. Where do you live before you came to [care home]?

R06: Uhh… (pause for 5 seconds) In… (pause for 4 seconds) Burlington, Ontario.

– Okay…

R06: And… I… worked as a chaplain….

[…]

– Umhm… Did you enjoy the work?

R06: Certainly. Yes. Working with people, trying to help them to come to acceptance of their situa… situations. And not trying to control them but allowing them to live their own lives. And to give them all the support that I could possible give to them. Walking to them.

C06: Also in general, the conflict comes at times with the other residents or with the staff in general…. I guess what you tell me is that he might have a problem with other people yeah treat him like a child, or tell him what to do, make him feel inferior you know.

C09: It was more the cooking I think. She was more a teacher, she was not the nurturing type person at all. But she loved cooking, she was an amazing gardener, amazing seamstress all those types of skills. And the teaching was full time. Yup until she was married because back then you couldn't teach while you were married…

or

C11: So I don't know when dementia progresses if that changes, but where he is now, he still identifies with… who he is still that person along with who he is now…

C09: I don't know… I think that… I believe that she may be open to it. Let's put it that way. She doesn't like to be told what to do however. So it would have to be done in a clever way so that she doesn't feel like she is being told what to do…

C12: He will sit there and talk to you all day, but if you ever say no… like if he says oh look at this green bottle, and you say oh that's clear, he gets defensive, he gets lost…

[…]

C12: Because he has been the father and the husband, he took care of everybody, so he doesn't want people to try and think they are going to use something to help him.

– Like for example she could say by washing your hands, you help other people not getting sick something like that

C12: Yeah… like phrase it totally different where it doesn't pinpoint him that it is there to help him.

Virtual human findings

To summarize the overall feedback provided directly by the interviewed target end-users, we underline the importance of the challenges of successfully integrating the technology into daily life routines. Each individual end-user brings to the experience of technology a different persona and set of affective states. Therefore, an important obstacle to overcome is to respond equally to the needs of each individual. One issue raised by participants was the difficulty of accepting a technology that reminds the user of being “disabled,” “different,” “dependent,” and that highlights the loss of cognitive function. For example, devices designed to support a user in simple tasks such as hand washing were perceived as condescending and insulting, particularly for highly educated users with previous experience in positions of power (used to being the “provider” for others rather than receiving help from others), thus usually with high power (P) values. Analysis of interview data further revealed that it is critical to avoid labeling end-users as “demented,” “old,” or “in the need of support for even the simplest tasks.” The challenge is to frame technology as useful in prolonging independent living, and relieving the burden on family caregivers. Interview outcomes further show that virtual assistants must not be perceived as meant to replace highly valued interpersonal relationships between older adults and their caregiver and furthering isolation, illustrated by the following quote: “I would rather have a ‘real’ person helping me than a machine.”

With regards to prompting styles, residents (and caregivers thereof) with identities that were positive and powerful reported preferring to be in control over what happens to them, thus preferring more subtle prompts. For example, as presented in Figure 4, a user with the EPA code E: + 1/P: +1/A: −1 would receive a prompt of a slightly smiling female virtual assistant dressed in pastel colors suggesting in a mellow tone “Hey John, you may want to use the soap… how about turning off the water?….” Alternatively, C01 suggested hiding the prompting among other types of information relevant to hand washing to make it appear more suggestive.On the other hand, residents with neutral valence and weaker identity profiles prefer to be less in control and therefore did not mind a dominating prompting style. For example (see Figure 4) a user with the EPA code: E: 0/P: −1/A: + 1, would receive a prompt from a male virtual assistant with a neutral facial expression dressed in white or light grey saying in a stronger voice tone “Hey Maria, please use the soap,… now please turn off the water….”

Figure 4.

Prompting styles overview based on EPA values.

C01: If it says you're fighting infection by washing your hands… if you're, you know gearing it to something else, you're preventing yourself from getting sick, or preventing spread of germs. Residents can understand, (R01) can understand that process then right, and it could appreciate it more, why he's being told to wash his hands.

As another example, a user with the EPA code: E: −1/P: +1/A: 0, would receive a prompt from a male virtual assistant with a more serious less smiling facial expression, dressed in dark blue saying in a neutral voice tone “Hey John, let's wash our hands. Now we take the soap,… don't forget the towel on your left….” However, if the identity would have an even higher value on Potency (P), such for example with the code: E: −1/P: +2/ A: 0, the look and the tone of the voice would be the same but only the wording would change into an even more submissive prompt: “Hi John, we've got some really nice soap here, it might make your hands feel good…. If you would like to use the towel, it's on your left… are you still using the water, or are you ready to turn it off?… Don't forget that it's flu season, washing your hands is a good way to keep from getting sick!”

Validation through ACT simulations

We can replicate these findings in ACT by simulating interactions between identities for the user and for the virtual assistant. We are primarily interested in the affective interpretations of the prompts as given by the virtual assistant from the perspective of the user, as that is a critical component for the uptake of prompts. Let us first consider whether the user thinks of himself as either a “patient” (current identity with EPA: [0.9, −0.7, −1.0], or neutral valence, slightly powerless, and somewhat inactive), or a “boss” (biographical identity with EPA: [0.48, 2.16, 0.94], or slightly good, quite powerful, and somewhat active). We consider further that the user is interpreting the virtual assistant as a “nurse” (EPA: [1.65, 0.93, 0.34]). Using ACT, we can predict the optimal behavior for the assistant to maximize uptake of the directives. This is an affective signature, and we will seek labels in the ACT dictionary that correspond to this signature, but are denotatively related to prompting (i.e. we select the closest label in the Indiana 2002–2004 ACT dictionary that has the same denotative meaning as giving an instruction or prompt).

In Table 3, we can see that ACT predictions are in line with interview results: more powerful identities call for less powerful, more deferential behaviors (prompts).

Table 3.

ACT behavior predictions (Simulation 1).

Identities		Simulated best behavio
VA	User	EPA	Label	Example
Nurse	Patient	2.0, 1.5, 0.2	Advise	“I think you should use the soap now”
Nurse	Boss	1.9, 0.6, −0.2	Ask about something	“Do you think you should use the soap now?”

VA, virtual assistant.

Another possibility is that users may only ascribe negativity in Evaluation to the nurse virtual assistant, which can be simulated with an “impatient nurse” (Table 4).

Table 4.

ACT behavior predictions (Simulation 2).

Identities		Simulated best behavior
VA	User	EPA	Label	Example
Impatient nurse	Patient	−0.5, 0.7, 0.2	Lecture	“Washing your hands is important—please do so now”
Impatient nurse	Boss	−0.1, −0.1, 0.5	Prod	“Please use the soap, it's really important”

VA, virtual assistant.

The outcomes of this simulation yield a difference in predicted behaviors, with the patient expecting a lecture and the boss expecting a prodding type of behavior.

In some instances, we found residents to be more negative about the virtual assistant, stating they didn't need help from the device and didn't like it. We therefore simulate with the virtual assistant having the identity of “do-nothing” (EPA: [−1.9, −2.0, −2.1], or quite negative, powerless, and inactive), and find the results described in Table 5.

Table 5.

ACT behavior predictions (Simulation 3).

Identities		Simulated best behavior
VA	User	EPA	Label	Example
Do-nothing	Patient	−1.2 −1.2, −1.4	Ignore	No prompt given
Do-nothing	Boss	−0.6, −1.7, −0.8	Beg	“Please use the soap, I beg of you”

VA, virtual assistant.

In this case, we see a significant difference in predicted behaviors. A patient would expect no help from such an assistant, whereas a “boss” would expect the virtual assistant to beg them to do something.

Conversely, it is also possible to simulate what type of identity would be responsible for a prompt. Using “suggest something to” (EPA: [1.8, 1.4, 0.8]), we ask ACT the question “who would suggest something to a boss/patient.” For the scenario involving a “boss,” this would be an identity with an EPA of [1.3, 1.1, 0.8], or a “right-hand man” or “businesswoman.” For the scenario involving a “patient,” the identity would differ only in Evaluation (“more good”) (EPA: [1.8, 1.2, 0.8]), corresponding to a “bride” or “colleague” or “intimate.” We could use this type of simulation to tailor the “look” of the virtual assistant for a particular user dependent on what type of behavior/prompt we want to provide.

Implications for the COACH system

Based on these findings, we propose that Evaluation, Potency, and Activity in the virtual assistant are each handled by different communication modalities, as shown in Figure 4. Evaluation is communicated primarily by the facial expressions of the virtual assistant, as well as the colors of clothing and background. In previous work, we proposed a method for mapping E, P, and A to the facial expressions of a virtual human.[24] This method was based on Ekman's theory on how the facial muscles of mouth, brows and eyes contribute towards emotion display on the face,[25-27] as was used in the Interact software.[28] Formulas for adjusting lips, brows, and eyes were carefully defined based on input EPA profile values so that they appeared to be realistic. Expressions for six EPA vectors, along with the closest emotion label in the ACT attribute dictionary, are shown in Figure 5.

Figure 5.

Sample facial expressions according to EPA.

Potency is handled through a gender choice, and by the content of the prompts (i.e. the words used to deliver the prompt). We selected various wordings for each prompt that correspond to more or less powerful phrasings of the same content. Based on the comments of the participants, the psychologist and computer scientist developed some wording suggestions that could map different values on the Potency dimensions, going from a weak, suggestive prompting style (−2) to a very dominating/commanding prompting style (+2):Activity is expressed through the strength and tone of the voice. We propose to modulate the strength and loudness of the voice, making more active prompts louder with a strong tone, and inactive prompts quieter with a softer tone.

−2: “Hey John, it's Tuesday… we've got some really nice soap here, it might make your hands feel good…. If you would like to use the towel, it's on your left… are you still using the water, or are you ready to turn it off?… Don't forget that it's flu season, washing your hands is a good way to keep from getting sick!”

−1: “Hey John, you may want to use the soap… how about turning off the water?…”

0: “Hey John, let's wash our hands. Now we take the soap… don't forget the towel on your left….”

+1: “Hey John, please use the soap,… now please turn off the water….”

+2: “Take the soap now… rinse your hands… take the towel now… now dry your hands.”

Discussion and conclusions

To summarize our findings, interviews with end-users of a virtual assistant and their caregivers show that: (1) a set of identities grounded in the past can be extracted for each participant, which supports the conclusion that even though these identities gradually fade away with time and memory loss due to AD, an overall “persona” that has been shaped by these identities remains preserved; (2) simulations using ACT for a variety of identities reflect similar dynamics to those seen in interviews; (3) consultation with end-users together with ACT simulations can inform the delivery of prompts from technology aimed at individuals with AD to maximize uptake.

Our results are in line with other studies,[27] such as those of Caddell and Clare,[29,29] suggesting that there is a persistence of self (persona) throughout the progression of dementia, although many studies record some degree of deterioration in aspects of self or identity. There is a growing body of literature suggesting that a sense of self can indeed survive through late stages of dementia and that this should be taken into consideration when designing interventions for meaningful engagement[31,32] with both human and virtual assistants. These lines of evidence lend support to our recommendation that prompts should be tailored to the overall “persona” of the user.

Our results provide support for the proposition of a lasting persona despite cognitive impairment. Figure 6 conceptualizes and illustrates this concept.

Figure 6.

Persistence of persona shaped by different affective identities adopted over time, which fade upon the onset of Alzheimer's disease.

Our results will inform further development of the hand-washing system and other assistive technologies. For each set of affective identities found in this sample of the population, a specific personalized prompting style will be programmed that reaffirms the user in his or her overall persona and situational identity. To achieve this goal, different modalities will be modified and enhanced (e.g. look and facial expression of virtual assistant, wording of prompts, tone of voice). These improvements to existing and emerging assistive technologies will ensure that their goal of increasing users' autonomy and independence and decreasing caregiver burden are maximized while the challenges of acceptability and adoption are minimized.

Our findings further underscore the importance of understanding past and current identities of persons with impaired cognitive abilities in technology-based efforts to provide individualized care and enhance quality of life. The results can be seen as validating care practices that are already in use,[33] but also as emphasizing the need for recognition among caregivers of the persistence of identity roles among individuals with dementia. Understanding the specifics of self-identity may help caregivers communicate with persons with dementia and develop individualized and meaningful activities that are based on the person's overall persona. This understanding has been demonstrated in previous research to facilitate a significant improvement in mood, and a reduction of agitated behavior.[33,34]

When developing new technologies for individuals with dementia and their caregivers, it is critical to include the perspective of end-users. Our study highlights the importance of participatory design as we were able to gain new insights into the acceptability and adoption of a virtual assistant through in-person interviews. Future work in this area should consider stakeholder consultations to inform design, delivery, and format of innovative interventions to maximize uptake and adoption.

We appreciate the limitations of the present study. The sample size (N = 21) is small for an interview study, and results must be considered as preliminary. Furthermore, it can be argued that EPA values that were gathered and collected from surveys of undergraduates in Ontario in 2001 may be quite different from EPA ratings that elderly persons in Ontario in 2016 would give. However, usually, these ratings representing sentiments on these different identities are quite stable over time and culturally shared, even across age groups. Despite these limitations, our findings have immediate relevance for the development of emotionally aware assistive technology and will inform a larger-scale effort that will capture insights from a broader segment of the older adult population. Future work may benefit from updated, more representative dictionaries for special populations, such as older adults with and without dementia.

Despite these limitations, the technology and findings from this research will further progress towards achieving the long-term goal of meaningful and emotionally responsive assistive technology to support older adults with dementia. This will be accomplished through the expansion of the system to other ADL, permanent in-home installations, and longer term studies of technology usage. Our results provide key insights in answering foundational questions related to emotional interactions between humans and machines.