Situation Model for Situation-Aware Assistance of Dementia Patients in Outdoor Mobility.

BACKGROUND: Dementia impairs spatial orientation and route planning, thus often affecting the patient's ability to move outdoors and maintain social activities. Situation-aware deliberative assistive technology devices (ATD) can substitute impaired cognitive function in order to maintain one's level of social activity. To build such a system, one needs domain knowledge about the patient's situation and needs. We call this collection of knowledge situation model.
OBJECTIVE: To construct a situation model for the outdoor mobility of people with dementia (PwD). The model serves two purposes: 1) as a knowledge base from which to build an ATD describing the mobility of PwD; and 2) as a codebook for the annotation of the recorded behavior.
METHODS: We perform systematic knowledge elicitation to obtain the relevant knowledge. The OBO Edit tool is used for implementing and validating the situation model. The model is evaluated by using it as a codebook for annotating the behavior of PwD during a mobility study and interrater agreement is computed. In addition, clinical experts perform manual evaluation and curation of the model.
RESULTS: The situation model consists of 101 concepts with 11 relation types between them. The results from the annotation showed substantial overlapping between two annotators (Cohen's kappa of 0.61).
CONCLUSION: The situation model is a first attempt to systematically collect and organize information related to the outdoor mobility of PwD for the purposes of situation-aware assistance. The model is the base for building an ATD able to provide situation-aware assistance and to potentially improve the quality of life of PwD.

INTRODUCTION

People with dementia have difficulties in performing everyday activities [1] and assistive technology devices (ATDs) have the potential to help them maintain their independent social life by supporting their everyday mobility. On the one hand, such systems should only intervene if needed so that the user still relies on his or her own cognitive resources. On the other hand, they should be able to also recognize normal behavior and to potentially reinforce positive behaviors. Such systems have been termed situation aware deliberative ATDs [2]. To be operational, such a device needs to track the user’s actions, recognize the goal, detect the errors in behavior, and decide about the best way of assisting the user [3]. To achieve that the ATD needs domain knowledge in machine understandable format, which describes the actions a user is able to execute, the errors in behavior that may occur, and the reasons for these errors as well as the situation in which the person is. We call this collection of knowledge a situation model. To our knowledge there are no existing situation models for ATDs supporting the outdoor mobility of people with dementia. To address this problem, in this work we present our attempt at building such a situation model for people with dementia. The situation model and the ATD are aimed at assisting people with mild dementia who can still independently interact in society but who sometimes exhibit cognitive problems and disorientation.

In the following, we discuss in more detail the problems associated with mobility in dementia and the potential of ATDs to improve this mobility. Later, we discuss the idea of situation-aware assistance and the concept of situation models for situation-aware assistance.

Mobility in dementia

An active and socially integrated lifestyle protects against cognitive decline in aging [4]. People with mild cognitive impairments have a reduced risk of conversion to dementia when engaged in outdoor and social activities [5, 6]. Furthermore, it allows people with mild cognitive impairments to continue their independent social life and to retain their dignity and self-esteem. In dementia, cognitive rehabilitation shows promising results in delaying further cognitive decline [7]. Outdoor mobility gains importance in this context, both as a prerequisite for an active lifestyle and also as a cognitively demanding activity in itself. Decreased mobility in dementia results from progressive cognitive decline. The complexity of highly dynamic outdoor environments poses an additional obstacle for cognitively impaired people. Indeed, people with dementia report decreased access to socially salient public places and socially stimulating activities [8] due to lack of motivation, previous experiences of errors in complex situations, and feeling of insecurity. The interplay between cognitive decline and decreasing mobility forms a vicious circle: decreased cognitive abilities reduce social activities which in turn impairs cognitive abilities, already affected by the disease process. In this perspective, ATDs that respect the remaining cognitive abilities and provide help for the user when needed might have the potential to break this vicious circle and preserve the autonomy of persons with dementia.

Such an assistive system needs to be able to reason about the user’s situation, as the situation influences the intervention strategy the ATD will follow[9, 10]. To be able to provide situation awareness, however, the system should have access to the knowledge base that represents the various situations related to outdoor mobility. Additionally, a group of individuals with dementia is typically heterogeneous with respect to declined mental abilities, and above that they are very heterogeneous with respect to premorbid characteristics as environmental exposure, desired activities, preferred range and variation of activities, and subjective meaning of particular activities. Thus, an ATD that can make use of situation knowledge has the advantage of providing a standard platform/instrument to quickly develop an individual prediction model or activity profile that is maximally useful for the individual user.

Situation models for situation-aware ATDs

Situation-aware assistance means that the ATD plays the role of a mediator between the person and the environment by observing the person through sensors, predicting his goals, and, only if needed, providing the missing information required to achieve the goal [3, 11, 12]. In other words, the assistance is not limiting the person’s outdoor activities unless necessary. A core element of a situation-aware assistance system is the situation model. It defines the domain knowledge needed to reason about the person’s situation. This knowledge could include personal preferences, social background, external and internal factors influencing the behavior of the person with dementia, elements of the environment he is acting in, etc. [13]. The situation model together with sensor observations is then used to reason about the person’s current situation and goals as well as about possible assistance strategies [3, 14]. It is also used as a codebook for annotating the observed behavior.

Situation models can be obtained in two ways, either through specification-based approaches, or through learning-based approaches [15]. Specifica-tion-based approaches rely on manually incorporating expert knowledge into logic rules that allow reasoning about the situation [15]. On the other hand, learning-based methods can rely on the sensor data to learn the situation [16] or on textual sources to extract the situation related information and its semantic structure [17-20]. In this work, we use specification-based approach to build the situation model. The situation model is often encoded in the form of ontologies. In this work, we consider a situation model to be a special case of ontology that incorporates the domain knowledge required by an assistive system to reason about the user behavior and the possible assistance strategies.

Objective

The objective of this paper is to provide a situation model for situation-aware assistance of people with dementia during their outdoor mobility. This model was developed as part of the SiNDeM project (Funded by the German Federal Ministry of Education and Research (BMBF, reference number: 16SV7091), aiming to enable technology support for outdoor mobility of people with dementia. The model has two purposes: 1) to provide the knowledge base in a situation-aware assistive technology device, namely a device that recognizes situations of disorientation and provides individually tailored guidance in real time; and 2) to be used as a codebook for annotating the data, collected during the outdoor mobility of people with dementia.

As a result, this work provides the basic structure of a situation model of outdoor mobility behavior and the associated cognitive factors that form the basis of knowledge-based situation-aware assistance and that can be used to code (transcribe) behavior traces. The situation model is implemented in the form of an OWL ontology.

METHODS

Development process for situation models

To build the situation model, we followed a development process based on the general ontology process proposed in [21]. It consists of four phases; Fig. 1 shows the process. The proposed process consists of domain analysis, conceptualization, implementation, and maintenance. The domain analysis involves analysis of different scenarios, discussion of relevant problems, observation of use cases, analysis of the requirements of the system, and review of existing solutions. The second step of the process is the conceptualization. It involves the conceptualization of the model and the integration of existing ontologies or relevant information. The last step in the development process is the model implementation. It deals with implementing the formal model in a concrete representation language. In addition, it has the explicit phase “Maintenance” which involves adapting and extending the developedmodel. In parallel to these phases, there are two additional processes. These are the evaluation of the results of each phase and their documentation.

Fig.1

The proposed development process for situation models. Figure adapted from [21]. The process consists of domain analysis, conceptualization, implementation, and maintenance. In parallel to these phases, there are two additional processes. These are the evaluation of the results of each phase and their documentation.

Domain analysis

System requirements examination

System examination was performed by first reviewing the literature with respect to the problems associated with outdoor mobility in people with dementia. Moreover, we reviewed the requirements identified in existing works addressing the problem of assisting people with cognitive impairments during their outdoor mobility. The literature review followed the principles proposed in [22, 23]. We focused on high-quality peer-reviewed articles and conference papers based on recognized ranking systems [e.g., the Core conference ranking (http://www.core.edu.au/index.php/conference-portal), Scientific Journal Rankings (http://www.scimagojr.com/)]. Furthermore, as the topic of assisting mobility of people with dementia is an emerging one, we also explored scholarly databases (e.g., Elsevier, ACM, IEEE, Google Scholar, PubMed). Our search string was defined using a brainstorming process with the ATD designers and medical experts to elaborate a keyword matrix. The matrix with the search strings contained the following strings: “assistance”, “dementia”, “outdoor mobility”, “assistive system”, “assistive device”, “Alzheimer”, “problems”.

Review of existing ontologies

To identify existing ontologies of interest, a literature review was conducted. We followed the same procedure as above. Brainstorming with experts, we categorized the ontologies into four categories: ontologies describing the behavior of people with dementia, ontologies describing outdoor mobility, ontologies describing the body motion, and ontologies for activity recognition. The following reasoning was behind our choice. Ontologies and studies describing the behavior of people with dementia represent our target group. Outdoor mobility ontologies represent the use case, namely outdoor mobility. Movement ontologies describe the body position and movement. We addressed these ontologies as the body position and motion can be an indicator for a person’s disorientation. Finally, we decided that ontologies for activity recognition are worth looking into, as ATD relies on activity recognition to identify the person’s situation. Furthermore, we looked into the ontology repositories, OBO Foundry [24] and NCBO BioPortal [25]. We used the same strings as during the literature review only omitting the strings “situation model” and “ontology”.

Outdoor mobility study

People with dementia were observed during their outdoor mobility. Ethical approval was granted by the ethics commission at the medical faculty of the University of Rostock (Approval #A2014-0154). The study had two purposes: 1) to collect sensor data that is used for training a model of human behavior during outdoor mobility; and 2) to collect behavior traces for the construction of a situation model. During the initial study, the psychologist followed the participants in 20-min walks, observed their behavior, and later discussed the observed problems with the ontology developer. Finally, the psychologist, together with a computer scientist responsible for the later situation model and ATD development, analyzed the video log of the walks to identify the observed behavior.

Additionally, a life-space assessment was conducted (see Fig. 2). There, the study participants took part in an at least 4-week long assessment of their outdoor mobility. It was recorded through sensors and mobility diaries. The data from the long-term assessment has not been fully analyzed. However, initial analysis provided some insights about the behavior patterns and user preferences.

Fig.2

Life-space levels as proposed in [29]. Life-space of 0 is limited to the bedroom; Life-space of 1 is limited to the home of the person; Life-space of 2 is limited to the immediate outside of the home; Life-space of 3 is limited to the neighborhood; Life-space of 4 is limited to the town where the person is living; Life-space of 5 is unlimited.

Study participants. Our study participants were patients from the local memory clinic who were diagnosed with amnestic mild cognitive impairment or clinically probable Alzheimer’s disease dementia, according to the national guidelines [26]. Cognitive testing had been performed using the CERAD cognitive battery [27]. Severity of dementia was graded using the Mini-Mental State Exam (MMSE) [28]. We only included people with an MMSE score of 18 and above (mild to moderate dementia). Participants were interviewed about their current level of mobility. Patients who reported to have independently reached a Life-space level [29] of at least three (Level 3 indicates that the person has been outside the house, visited places in the neighborhood) within the last month (see Fig. 2 for details on the levels of Life-space) were eligible to further participate in the mobility assessment. Patients were only included after oral and written information on the study procedures, and after they had given their written informed consent. The study had been approved by the local institutional review board of the University Medicine Rostock.

The mobility assessment study comprises two separate setups, a guided walk and a long-term assessment.

Guided walk. The fifteen participants that answered the above criteria were invited to a 20-min guided walk from which thirteen completed the walk. The sensor system consisted of sensors to measure electrodermal activity, electrocardiography, GPS, and accelerometric data. Data about electrodermal activity and electrocardiography were intended to infer the stress level and possible states of disorientation during the walk. For later analysis and annotation, the walk was videotaped and events were protocoled by a trained psychologist. The motion sensors were attached to the left ankle, the chest, and the left wrist. The sensor data was annotated according to the schema identified in the situation model and it was shown that the data conforms with the behavioral findings in the situation model. Detailed information about the sensors and their placement can be found in Supplementary Material B.

Prior to the recorded and videotaped part of the walk, the participant and the psychologist met at a defined public location (see Fig. 3). The psychologist guided the participant to a tram station approximately one kilometer away. The route was designed to be cognitively challenging for patients with dementia. Aside from the usually crowded tram station, the route contained a four-way crossing of two main streets, as well as narrow streets in a residential neighborhood. At the tram station, the participant was asked to walk back on his/her own. On the way back, the psychologist stayed in the background, observed and protocoled the behavior, and only interacted with the participant in case of unresolvable disorientation or potentially endangering situations. The study contained two potential risks for the participants: increased stress level and risk associated with the surrounding traffic. To address the first risk, the participants were briefed before the study and the route was clearly explained. It was also made clear that the psychologist will be following the participants at all times, so that they will not be left alone. This also addresses the second problem, where to avoid any risks from the traffic, the psychologist stayed near the participants at all times, so that in case of risk she could physically or verbally intervene.

Fig.3

The guided walk. The person with dementia is guided by a psychologist to a tram station, which is 1 km away. The person is then asked to find the way back to the start point on their own.

Life-space assessment. The guided walk was followed by a long-term assessment over at least 4 weeks, called life-space assessment. Here, daily outdoor mobility activities of 11 out of the initial 15 study participants were recorded. The sensor system for the long-term assessment was only GPS andaccelerometer. In addition, the participant kept a mobility diary. It was developed based on experts’ brainstorming. After the first prototype was developed, it was tested on two of the study participants and optimized before being used in the study itself. We asked the participants to wear a bandage containing GPS and accelerometer and to add an entry to their mobility diary every time they left their home. The diary was filled in twice for each outing, once for the way to their goal, and once for the way back home. Each entry in the diary had a standardized form and contained date, time, destination, stopovers, means of transportation, incidents, and eventually company (e.g., relatives, friends, or a dog).

Interviews with people with dementia

To collect information about behavior patterns, preferences, and social and medical background, a psychologist conducted structured interviews with people with dementia. Furthermore, the participants were asked questions aiming at identifying the affinity of the target user group to technology, and any individual system requirements. The interviews followed a semi-structured form and were conducted with 14 participants (6 of which were women), age from 58 to 86 years old. Cognitive status of participants ranged from mild cognitive impairment to middle stages of Alzheimer’s disease dementia (the average MMSE value of the participants was 24.8 with a minimum of 17 and a maximum of 28). The interviews were digitally recorded, transcribed, and evaluated based on the Mayring method for content analysis [30]. The procedure was performed by the same psychologist that conducted the interviews.

The design of the interviews was based on the technology acceptance model (TAM) [31]. The choice of TAM was based on the assumption that in order to provide assistance to people with dementia, they have to be able to accept such ATD. The interviews consisted of questions divided into the following five categories.

General questions and social background: these included education, current or former occupation, social status.

General questions concerning the usage of technology: these included usage of technology at home and work, frequency of usage of technology, problems during usage of technology, and readiness to use technology.

Questions regarding the need of assistance during outdoor mobility: these included types of outdoor activities, escorts during outdoor activities, frequency of outdoor activities, importance of independence during outdoor mobility, type of transportation mode, radius of the outdoor activities according to the life-space levels; problems during outdoor mobility, support during mobility, changes in outdoor activities based on the disease.

Questions regarding devices and assistance through navigation: these included the usage of mobile phones, types of functions used from the mobile phone, problems during usage of mobile phones, experience with navigation systems, problems during usage of navigation systems.

Questions regarding the concept of an ATD for outdoor mobility: these included dimensions and functions of such device.

Conceptualization

Interaction unit analysis

During the conceptualization phase, the identified relevant knowledge was conceptualized through interaction unit analysis (IUA) as proposed by Hoey et al. [32] All relevant information gathered during the domain analysis was categorized. 17 categories were identified during the requirements analysis and literature review (the complete list of categories can be found in Table 1). The list of possible behaviors and their relations to the rest of the elements in the different categories were identified (e.g., the weather influences the type of transportation a person will take). For each type of challenging behavior also the possible causes for observing it were identified [i.e., possible types of errors that occurred (for more details see Table 4, Errors due to disorientation)]. The result of this process was a list of possible behaviors with their challenging counterparts and the possible causes of challenging behavior. An example of the interaction unit analysis can be seen in Table 2.

Table 1

Results from the literature review with respect to the identified system requirements. Black box indicates the requirement was met in the corresponding work

Concepts from system	Concepts present in existing ontologies
requirements analysis	Behavior of People						Mobility					Motion				Activity
	with Dementia															Recognition
	[39]	[40]	[41]	[42]	[43]	[44]	[45]	[46]	[47]	[48]	[49]	[50]	[51]	[52]	[53]	[54]	[55]	[16]	[56]	[57]
Interaction with Environment
Action	+	+	+	+	+	+	–	–	+	–	–	+	+	–	+	+	+	+	+	+
Interaction	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	+	+	+	+	+
Communication	–	–	–	–	–	–	–	–	+	–	–	–	–	–	–	–	–	–	–	–
Extrinsic Factors
Area	–	–	–	–	–	–	+	+	+	+	+	–	–	–	–	–	–	–	–	–
Transportation mode	–	–	–	–	–	–	+	+	–	+	+	–	–	–	–	–	–	–	–	–
Location	+	+	–	–	–	–	–	+	+	+	–	–	–	–	–	+	+	+	+	+
Local conditions	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–
Objects	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	+	+	+	+	+
Intrinsic Factors
Motion	+	–	–	–	–	–	–	–	–	–	–	+	+	+	+	+	+	–	+	–
Physiological state	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–
Intention / Goal	+	–	–	–	+	–	–	–	–	–	–	–	–	–	–	+	–	+	–	+
Long-term Intrinsic Factors
Routines	+	–	–	–	–	–	–	–	–	–	–	–	–	–	–	+	+	–	–	–
Transportation preferences	–	+	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–
Social activities	–	+	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–
Social background	–	+	–	–	–	–	–	–	–	–	–	–	–	–	–	–	+	–	–	–
Medical history	–	+	–	–	–	–	+	–	–	–	–	–	–	–	–	+	–	–	–	–
Technology affinity	–	+	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–	–

+ concept included –concept not included.

Table 4

Identified concepts and subconcepts based on the system requirements, literature review, mobility study, and interviews

Concept			Example	Source
Interaction with Environment	Challenging behavior	See Table 3		Attempting to orient based on signs.	Mobility study, IUA, [58]
	Action	See Table 3		Crossing a street.	Mobility study, IUA, [16, 39, 40, 43, 44, 47, 54–57, 59]
	Interaction / Manipulation	See Table 3		Touching an object.	Mobility study, IUA, [16, 57]
	Communication	See Table 3		Speaking to a person.	Mobility study, IUA, [47]
Extrinsic Factors	Area	Rural		The person is in a field.	[45, 46], Interviews
		Urban		The person is in the city.	[45, 46], Interviews
	Mode of Transportation	Non-motorized	Walking	Walking on foot.	[46, 49], Interviews
			Cycling	Using bicycle.	[46, 49], Interviews
		Motorized	Public transportation	The person takes a bus.	[45, 46, 49], Interviews
			Private transportation	The person drives a car.	[45, 46, 49], Interviews
	Location	Building		The person is in a public building.	[40, 46, 47]
		Street		The person drives on the street.	[46, 47]
		Off-road		The person walks in the forest.	Mobility study
		Crossing		The person waits at the crossing.	[46, 47]
		Sidewalk		The person walks on the sidewalk.	[46, 47]
	Local conditions	Meteorological state		It is raining.	[2], Interviews
		Public locations state		Opening hours of public buildings.	[2], Interviews
		Transportation state		Working hours of public transportation.	[2, 9], Interviews
	Objects	Animate	Human	Human other than the user.	Mobility study, [57], Interviews
			Animal	A dog accompanying the user.	Mobility study, [57], Interviews
		Inanimate	Signs, doors, bags, etc.	[16, 54–57] Interviews
Intrinsic Factors	Motion	Rotational motion	The person turns around.	[39, 50–56, 59, 60]
		Translation motion		The person walks forward.	[50–56, 59, 60]
	Errors due to disorientation	Initiation error		The person does not leave the start position.	[41–44]
		Organization error		The person does not use landmarks to orient in the environment	[41–44]
		Realization error		The person is unable to make an appropriate decision at an intersection.	[41–44]
		Sequence error		The person starts walking before planning the route.	[41–44]
		Judgment error		The person is unable to recognize potential errors and safety issues during task execution.	[41–44]
		Completion error		The person crosses the street without attention to traffic.	[41–44]
	Physiological state			The person’s heart-rate is very high.	[2]
	Intention / Goal	See Table 3		The person wants to visit a particular place.	[9, 16, 39 43, 55, 57]
Long -term	Routines			The person goes to the bakery every morning.	[2, 39, 40, 54, 55], Interviews
Intrinsic Factors	Transportation preferences			The person usually travels with the tram.	[2, 9], Interviews
(User Profile)	Social activities			The person meets with other elderly people once a week.	[2, 40], Interviews
	Social background			The person worked as an engineer before retiring.	[2, 40], Interviews
	Medical history			The person has a cardio-vascular disease.	[2, 40, 45, 54], Interviews
	Technical affinity	Type 1 users		Technical understanding and interest in technology based on professional experiences.	Interviews
		Type 2 users		Technical understanding and interest in technology based on personal interests and hobbies.	Interviews
		Type 3 users		None or small interest in modern technology.	Interviews

“Concept” contains the concepts and their subconcepts, “Example” gives an example of the concept, and “Source” gives the source from which the concept was identified.

Table 2

Example output from the IUA for the action walk. It shows the normal action, its challenging counterpart, the possible causes behind the challenging behavior, and an example of this behavior

Action	Challenging	Cause	Example
	action
walk	wander	initiation error	The person cannot remember the destination. The person does not start walking.
		organization error	The person cannot remember the way to the destination.
		realization error	The person cannot realize that walking in a certain direction is necessary for reaching the destination. The person walks without using landmarks for orientation.
		sequence error	The person knows the destination but cannot perform the necessary steps for reaching it. The person starts walking before he is oriented.
		judgment error	The person knows the destination and the way to it but is unaware of obstacles or dangers on the street.
		completion error	The person does not realize that he has reached the destination.

Integration of existing models

As no suitable existing models to integrate into the situation model were found, we concentrated on including the relevant knowledge we have found from existing ontologies and models (from Step Review of existing ontologies from the domain analysis).

Implementation

During the implementation phase, the Web Ontology Language (OWL) [33] and OBO Edit (http://oboedit.org/) were used to encode the ontology. OWL was chosen, as it is a semantic web language that can represent rich and complex knowledge about things, groups of things, and the relations between things. Moreover, it is a widely-used language that allows the use of the ontology across different applications. That also allows the easy conversion of the ontology in another targeted language.

Evaluation

To validate the situation model, the OBO Edit tool and its reasoner were used:

Verification tests on the situation model in OBO Edit were run;

The OBO Edit reasoner was used to identify redundant links in the ontology.

Furthermore, domain experts were asked to revise the situation model. The experts’ revision of the semantic structure is considered to be genuine evaluation for ontologies [34] and as the situation model is a special case of ontology, we also use the experts’ revision for evaluation. Apart from that, two annotators were asked to annotate the video logs from the guided walk in the mobility study using the same codebook extracted from the situation model. The annotation serves as a ground truth for evaluating the performance of the ATD when recognizing the user behavior. In this case, it was also used to evaluate whether the concepts, identified in the situation model, are recognizable in the video log of the mobility study. This provides information about the objectiveness of the situation model. In order to measure this, the interrater reliability in terms of Cohen’s kappa between the two annotators was calculated. Cohen’s kappa between 0.00–0.20 indicates slight agreements between annotators, between 0.21–0.40 fair agreement, between 0.41–0.60 moderate agreement, between 0.61–0.80 substantial agreement, and between 0.81–1.00 almost perfect agreement[35, 36]. We have to mention that there are no established benchmarks for this methodology. Landis and Koch [35] addresses the annotation of dance movement, while [36] comes from the field of radiology. Regardless, to our knowledge, existing benchmarks from different fields use the same scales to measure the annotation reliability with Cohen’s kappa. As the aim was to evaluate whether the concepts in the situation model can be recognized in the observed behavior by different annotators, a training phase was not conducted with the annotators. This is opposed to other approaches aiming at training the annotators to always code a given behavior with the same label. In the latter case, the name of the label could be irrelevant, as the annotators are already trained to associate the label with the corresponding behavior, regardless of the fact that the label could trigger other behavior associations in untrained annotators.

RESULTS

Model structure and contents

The system requirements evaluation and the review of existing ontologies yielded a knowledge base divided into four main groups describing the factors influencing the behavior of people with dementia during their outdoor mobility. These are “interaction with the environment”, “extrinsic factors”, “intrinsic factors”, and “long-term intrinsic factors”. The results from the requirements evaluation and the literature review can be found in Table 1. More details about the rationale behind the obtained results can be found in Supplementary Material A.

During the outdoor mobility study, four challenging behaviors, associated with disorientation during outdoor mobility were identified. These are “spatial reasoning”, “topological reasoning”, “communication”, and “wandering”. Apart from them, different normal actions were identified. The list of identified behaviors can be found in Table 3. The outdoor mobility study also produced a set of goals the participants were following as well as different transportation modes. These are listed in Table 4 under “Intention / Goal” and “Transportation preferences”. More details about the outdoor mobility study can be found in Supplementary Material B.

Table 3

Identified interaction concepts during the outdoor mobility study and the IUA as well as their subconcepts

Challenging behavior	Rational (compensatory behavior)	Communication action
		Spatial action
		Topological action
	Irrational	Wandering
Action	Walk	–
	Evade obstacle	–
	Cross street	–
	Wait	–
	Drive a vehicle	–
	Take public transportation	–
	Enter a building	–
Interaction / Manipulation	Haptic interaction	Buy an object
		Take an object
		Put an object
		Manipulate an object
		Open object
		Close object
		Touch a person / object
	Visual interaction	Read a sign
		Search for landmarks
Communication	Direct communication	Make eye contact with a person
		Speak to a person
	Remote communication	Use device to communicate
Goal	Do sport	–
	Visit the doctor	–
	Visit the garden	–
	Do shopping	–
	Take a walk	–
	Visit friends / family	–

The interviews with people with dementia showed three different types of technology users: Type 1: people that have technical understanding and interest in technology based on their professional experiences; Type 2: people that have technical understanding and interest in technology based on their personal interests and hobbies; Type 3: none or small interest in modern technology. The type of users affects the way in which the ATD can provide assistance. More details about the outdoor mobility study can be found in Supplementary Material B.

The knowledge extracted from the domain analysis and IUA was divided into different sub-categories and the relations between the different elements were identified. This resulted in 101 concepts categorized in a hierarchical structure where the uppermost categories are those identified from the system requirements examination, while the middle and bottom layers are based on the literature review, the interviews, the mobility study, and the interaction unit analysis. The concepts and their hierarchical structure can be seen in Table 4. More details about the definition of each concept can be found in Supplementary Table 1.

Apart from the concepts, 11 relation types were identified, divided into five categories. These are “relations describing the abstraction hierarchy of concepts”, “relations describing the interaction of the user (person) with the environment”, “relations describing locational dependencies”, and “relations describing the causal dependencies between concepts”. The complete list of relations can be seen in Table 5. Some additional details and examples about the relations can be found in Supplementary Material C.

Table 5

Relations identified between the different concepts (left) and their meanings (right). For more details and examples of the relations, see Supplementary Material C

Relations describing the abstraction hierarchy of concepts
is a	indicates that the given concept is a sub-concept of another concept
consists of	indicates that a given concept is divided into several sub-concepts and that only the collection of all these concepts represents the upper concept
has	indicates that a given concept has certain properties
Relations describing the interaction of the user (person) with other concepts
uses	indicates that a given concept (i.e., the user) makes use of other concept
executes	indicates that a given concept (i.e., the user) can execute another concept
is applied on	indicates that one concept can be applied to another
Relations describing locational dependencies
is at	indicates that a given concept is located at another concept that describes a type of location
is executed at	indicates that a given concept is executed at a given other concept that describes a type of location
Relations describing the causal dependencies between concepts
affects	indicates that a given concept is influenced by another concept
causes	indicates that the presence of a given concept causes the presence of another concept
is achieved through	indicates that a given concept is the effect of another concept

Figure 4 shows a conceptualization of the situation model with the corresponding relations between the different concepts. The situation model implemented in OWL is publicly available for download at the Library of the University of Rostock [37]. The concepts’ definitions can be found in Supplementary Table 1.

Fig.4

The identified concepts and the relations between them. Orange indicates the person with dementia, green the interaction the person can execute. The behavior can be normal or challenging (both in dark grey). The rest of the concepts are in light grey, while their subconcepts or concepts identified during the mobility study and interviews are in white. The full list of concepts can be found in the Supplementary Material C and Supplementary Table 1. The implemented situation model can be found in [37] (https://doi.org/10.18453/rosdok_id00000150).

Validation of the situation model

A verification test was run on the situation model. It produced 14 non-critical warnings. All of them addressed misspelling of some concepts and were appropriately corrected.

Furthermore, the reasoner detected 5 redundant relations in the situation model. The redundant relations and the explanation for their redundancy can be found in Table 6. They were also removed from the final situation model. The resulting model can be downloaded from [37]. The list of concepts and their hierarchical structure can be found in Table 4.

Table 6

Redundant relations in the situation model

Child name	Parent name	Derived by the reasoner
Abnormal behavior	Interaction with environment	from link Abnormal behavior → is_a → Normal behavior. Normal behavior → is_a → Interaction with environment
Communication action	Abnormal behavior	from link Communication action → is_a → Rational (compensatory behavior). Rational (compensatory behavior) → is_a → Abnormal behavior
Spatial action	Abnormal behavior	from link Spatial action → is_a → Rational (compensatory behavior). Rational (compensatory behavior) → is_a → Abnormal behavior
Topological action	Abnormal behavior	from link Topological action → is_a → Rational (compensatory behavior). Rational (compensatory behavior) → is_a → Abnormal behavior
Wandering	Abnormal behavior	from link Wandering → is_a → Irrational. Irrational → is_a → Abnormal behavior

“Child name” and “Parent name” indicate the manually modeled “is-a” relations between the two concepts. “Derived by reasoner” shows the alternative way through which the relations were derived, making the manually modeled ones redundant. The relations were discovered by validating the model with the OBO Edit tool. They were later removed from the final version of the model.

Evaluation by experts

A team of clinician experts in the field (also coauthors in the manuscript) revised the situation model and added certain clinically relevant concepts and relations to the model, increasing its pragmatic usability. The new concepts and relations mainly addressed factors describing the disease and the associated errors in behavior. Table 4 contains the errors in behavior (errors due to disorientation).

Data annotation

The video logs from the guided walk were annotated by two annotators. The annotators concentrated on annotating the normal and challenging behavior of the study participants. The annotation of disorientation showed an interrater reliability of k = 0.61 where k indicates Cohen’s kappa. k of 0.61 indicates a substantial overlapping between the two annotations [35, 36]. We consider the overlapping of the annotation to be satisfactory, as the two annotators were not trained to recognize the observed behavior. This means that they were able to assign the same concept from the situation model to the video log without previously learning the mapping between a given concept and the observed behavior. The difference in the annotation of the two annotators is explained by the fact that the first annotator more often interpreted the behavior of the study participants as compensatory behavior rather than as normal behavior. Interestingly enough, it was observed that the overlapping of the annotation of the directly observed physical behavior was lower than that of the indirectly observed cognitive processes (e.g., topological reasoning). This indicates that the annotators were able to agree on the cognitive processes but had different interpretations of the physical actions.

Nevertheless, the results from the annotation showed that both annotators were able to observe the selected from the situation model concepts in the video log. This stands to show that the selected concepts are suitable for annotating the behavior of the targeted group. Furthermore, it shows that non-trained annotators are able to label the observed behavior with the same labels, which shows that the selected labels are identifiable also for untrainedpersons.

DISCUSSION

The ability to be mobile outdoors allows people with dementia to continue their social life. Therefore, people with dementia can benefit from systems that non-invasively assist their outdoor mobility. To realize such system, however, one needs the knowledge base necessary for recognizing the user’s actions, goals, and needs.

In this work, we proposed a situation model for the outdoor mobility of people with dementia. We first investigated the requirements for the knowledge base based on the user needs and system requirements. We then conducted an ontology analysis based on relevant state of the art ontologies. Furthermore, we analyzed the behavior of people with dementia during their outdoor mobility. We also conducted semi-structured interviews with people with dementia to identify behavior patterns, preferences, and problems during outdoor mobility. We then refined the knowledge base through interaction unit analysis and realized it in the form of OWL ontology. The resulting situation model serves two purposes: 1) to provide the code raster for annotating the outdoor mobility of people with dementia; and 2) to serve as a knowledge base in an ATD aiming to assist people with dementia during their outdoor mobility. To validate the model, two annotators independently used selected concepts from the situation model to annotate the video log collected during a mobility study. The results showed substantial overlapping between the two annotations (Cohen’s kappaof 0.61).

Below we discuss the advantages and limitations of the proposed model as well as the clinicalimplications.

Advantages

The situation model provides the code raster needed for annotating the user behavior. It also provides the semantic meaning behind the annotated behavior, such as reasons for the observed behavior and relation to other annotated behaviors. The situation model is also essential for detecting the user situation and reasoning about the appropriate intervention in case of abnormal behavior. It is the first systematically developed situation model for assistance during outdoor mobility for people with dementia. Existing situation models concentrate on assisting people with dementia in indoorsettings.

Our work is also the first to conduct a mobility study for people with dementia in order to obtain the sensor data and the context information necessary for developing assistive systems for outdoor mobility for people with dementia. It is also specifically tailored to address the needs of people with dementia. For that reason, it contains detailed information about errors in behavior during outdoor mobility and the corresponding causes of these errors.

The annotation of the guided walk also showed that using the code raster from our situation model produces substantial [35] agreement between two different annotators (Cohen’s kappa of 0.61). This stands to show that the proposed model is not only a subjective interpretation but that it also reflects the perception of people who did not take part in the model development. Finally, the situation model incorporates the knowledge from the literature, from personal experiences of people with dementia, and from the requirements needed to develop an ATD for outdoor mobility. This makes it a good starting point for any assistive system that aims at outdoor mobility for people with dementia.

Limitations

Apart from the advantages listed above, there are some limitations associated with the proposed model.

The model does not integrate the knowledge from the long-term assessment and the mobility diary as at the time of development they were not analyzed. We were, however, able to conduct an initial analysis of the long-term assessment study and to integrate our first impressions in the situation model. In the future, we plan to annotate the long-term study with the code raster from the situation model, which will provide additional evaluation of the model’s functionality.

Clinical implications

The situation model for outdoor mobility for people with dementia has two clinical application fields.

1) It tests hypotheses on disoriented behavior in real world settings with sensor based behaviorassessment.

The phenomenology of functional decline in dementia is poorly described. The majority of studies use rating instruments by proxy or self-rating instruments to determine impairments in (instrumental) activities of daily living including way-finding, e.g., the Bayer ADL scale. Such scales are usefully employed to determine the effect on aggregated functional outcomes in clinical trials, but even here they have been criticized for limited sensitivity to functional change and limited validity in respect to underlying behavior features [38]. Another approach is to assess the degree of impairment of cognitive functions that are involved in the maintenance of navigation abilities using psychometric scales. This approach cannot directly assess the effect of cognitive change of everyday function in a given patient, but is based on inference from a stochastic model on the association between cognitive ability and everyday function in groups of patients. In acknowledgment of this fact, major stakeholders, such as the Alzheimer’s Association, the Food and Drug Administration, pharmaceutical industry, and the European Research Council (ERC) have adopted the concept of real world evidence. In 2015, the ERC asked for the “collection and analysis of high quality real world evidence (RWE)”, stating that “many factors prevent the reliable prediction of the natural course of [Alzheimer’s disease] AD in real-world settings... [Including]... lack of clarity on how to best model the natural history of the disease using real world data, particularly as disease stages are difficult to define outside of highly qualified assessment centers or research cohorts.” (http://www.imi.europa.eu/sites/default/files/uploads/documents/IMI2Call6/IMI2_Call6_FINAL.pdf, p. 29). Sensor based automated assessment of disorientation using an adequate situation model can be regarded a major patient-powered real world endpoint to directly and objectively assess the effect of disease progression or interventions on patient function.

2) An adequate model of disorientation in dementia can serve to inform a situation-aware ATD to select an intervention that meets the actual need of a person in a given phase of disorientation. The inferred type of orientation error would drive the selection of the intervention allowing for a user-tailored set of intervention. This in turn provides the user with as little information as necessary to find his/her way back again rather than a generic maximal intervention, such as calling the caregiver every time the user leaves a predefined route, irrespective of potentially underlying decisions or needs of the user.

In the future, we plan to integrate the situation model into a situation detection system. We will evaluate its performance by applying it to the sensor data collected during the mobility study. This will provide us with evidence of whether the system is able to detect cognitive problems and disorientation based on state of the art sensors.

Later, a new study with people with dementia will be conducted where the aim is to test the system’s ability to provide adequate intervention strategies and assistance cues.

Click here for additional data file.