Wearable multimodal sensors for the detection of behavioral and psychological symptoms of dementia using personalized machine learning models.

Introduction: Behavioral and psychological symptoms of dementia (BPSD) signal distress or unmet needs and present a risk to people with dementia and their caregivers. Variability in the expression of these symptoms is a barrier to the performance of digital biomarkers. The aim of this study was to use wearable multimodal sensors to develop personalized machine learning models capable of detecting individual patterns of BPSD.
Methods: Older adults with dementia and BPSD (n = 17) on a dementia care unit wore a wristband during waking hours for up to 8 weeks. The wristband captured motion (accelerometer) and physiological indicators (blood volume pulse, electrodermal activity, and skin temperature). Agitation or aggression events were tracked, and research staff reviewed videos to precisely annotate the sensor data. Personalized machine learning models were developed using 1-minute intervals and classifying the presence of behavioral symptoms, and behavioral symptoms by type (motor agitation, verbal aggression, or physical aggression).
Results: Behavioral events were rare, representing 3.4% of the total data. Personalized models classified behavioral symptoms with a median area under the receiver operating curve (AUC) of 0.87 (range 0.64-0.95). The relative importance of the different sensor features to the predictive models varied both by individual and behavior type. Discussion: Patterns of sensor data associated with BPSD are highly individualized, and future studies of the digital phenotyping of these behaviors would benefit from personalization.

INTRODUCTION

People living with dementia can experience behavioral and psychological symptoms of dementia (BPSD). These symptoms are related to the neuropsychiatric manifestations of their disease, can arise in the context of unmet needs, and are influenced by interactions with the environment and caregivers. While motor agitation and verbal or physical aggression are commonly used categories to describe these behaviors, the specific expression of these behaviors varies greatly depending on the individual. , The clinical significance of these symptoms is tied to the distress they cause to people with dementia and their caregivers, the risk that they can present to the safety of the individual and those around them, and their impact on the health‐care system. ,

With advances in wearable technologies, there has been interest in developing digital biomarkers of behavioral symptoms using data collected from a variety of wearable sensors. , To date, most studies in this area have focused on using wearables to assess the severity of symptoms by summarizing wearable data over periods from days to months and finding correlations with observational rating scales. , , For example, studies have found a moderate correlation between activity levels (accelerometers) and clinical measures of wandering or agitation, , , , , and some association between sensor measures of stress (electrodermal activity, galvanic skin response) and ratings of agitation. These studies have been limited by small sample sizes, brief periods of data collection (< 1 month), and reliance on observational clinical rating scales with moderate reliability.

The next frontier is the use of wearable sensor data to develop algorithms that can detect the onset and presence of the behaviors in real time. This has been termed digital phenotyping, or the moment‐by‐moment quantification of the individual‐level human phenotype in situ using data from personal digital devices. The value of a digital phenotype for behavioral symptoms in dementia lies in its uses for clinical care and research for monitoring symptoms over time and supporting intervention and treatment. As a first step toward this goal, we have previously demonstrated that wearable sensor data can be used to detect behavioral symptoms minute‐to‐minute with an area under the receiver operating curve (AUC) of 0.82 and that incorporating data from multiple sensors improves algorithm performance over a single sensor. While these algorithms achieved a fair performance in classifying the presence of agitation, there were several important limitations. These generic models combined the sensor data from all participants and combined all subtypes of behaviors under the label of “agitation”: these generic models thus are missing important information about the individual expression of behaviors. Given the large inter‐individual differences in the expression of BPSD, the digital biomarkers for these symptoms are also likely to be heterogeneous, as are the biomarkers for different types of behaviors.

RESEARCH IN CONTEXT

Systematic Review: There are few systematic studies that examine the accuracy of digital biomarkers for detecting behavioral symptoms. The most commonly examined behaviors are motor agitation or apathy and use a single sensor (accelerometer). Sensor data are often summarized over days to weeks and across individuals, and correlated with clinical scales rather than directly observed behaviors. These limitations have impacted the accuracy of digital biomarker studies thus far and are an important barrier to advances in this field.

Interpretation: Using a multisensor wearable device we can accurately detect behavioral symptoms minute‐to‐minute. Personalization significantly improves the accuracy of the model to detect behavioral symptoms. Feature ranking is valuable to demonstrate the variability across individuals.

Future Directions: Our findings lend optimism to the search for digital biomarkers for behavioral symptoms in dementia and will help drive future studies in this area toward tuning or active learning to improve the personalization of machine learning models.

Toward an understanding of inter‐individual differences in the digital biomarkers of agitation, the aims of this study are to: (1) develop personalized models able to detect the presence or absence of behavioral symptoms, (2) examine behavioral symptom–specific personalized models, and (3) explore which sensor features are most important for modeling agitation within individuals and for specific behaviors. To achieve this, we used wearable multimodal sensor data collected from people with dementia with BPSD admitted to a tertiary dementia care unit to develop and evaluate personalized models that classify each minute as “agitation” or “non‐agitation” behavior. We then developed separate personalized models for each of three categories of behavior (motor agitation, verbal aggression, physical aggression), and examined the most important features in each of these models.

METHODS

Study overview

An overview of the study is provided in Figure 1. The study took place on the Specialized Dementia Unit (SDU) at Toronto Rehabilitation Institute (TRI), University Health Network, Toronto, Ontario, Canada, and was approved by the designated research ethics boards at University Health Network and Ryerson University, Toronto, Ontario, Canada. Details about the study and setting can be found in Ye et al.

FIGURE 1

Study overview. BVP, blood volume pulse; EDA, electrodermal activity

Study participants

Participants were 20 older adults admitted to the SDU from long‐term care homes for the assessment and treatment of BPSD in the context of advanced dementia. Inclusion criteria were age greater than 55 years old, diagnosis of dementia, and history of agitation or aggressive behaviors. Consent was provided by the substitute decision‐makers in all cases, and individuals were withdrawn from the study if they showed dissent to wearing the device. Data collection was stopped when one of these conditions was met: no behavioral symptoms documented for 1 week, repeated removal of the wristband, on discharge from the unit, or after 2 months of data collection.

Data collection

The wearable device used was the Empatica E4 (Empatica) which collects motion data (through accelerometer [ACC]) and physiological indicators (blood volume pulse [BVP], electrodermal activity [EDA], and skin temperature [TEMP]). The Empatica E4 device has previously been evaluated in stress detection studies among other applications. , , , The E4 was applied to the dominant wrist each morning after the individual received their morning care (usually between 0700 and 1000), and was removed when preparing for bed at night (usually between 1700 and 2000), which was necessary for daily device charging and data download. Fifteen video cameras were installed throughout the SDU‐TRI in the common areas, such as hallways, dining, and recreational areas.

Nursing staff were provided training on the documentation of behavioral events, and asked to note the start and end times, the location, and the context behind the event. Each event was marked with a green sticker in the margin of the chart. Research staff would review the charts weekly and gather information about each event. The video recordings corresponding to 30 minutes before and after the charted period of agitation were then watched by clinical research staff to fine‐tune the start and end times of the events and to gather further descriptive information about the event. Finally, the description of each event was used to divide them into one of three categories (motor agitation, verbal aggression, or physical aggression) using the definitions as provided by the Behavioral Supports Ontario Dementia Observation System (BSO‐DOS). , If at any point during the event, an aggressive act such as hitting, pushing, or kicking was noted, the entire event was categorized as physical aggression. Where there was a combination of verbal aggression and motor agitation, the event was categorized based on the behavior that was predominant. To be included in an analysis by behavior category, a participant needed to have at least two agitation events in the given category. Any data which fell outside of an labeled agitation event was categorized as non‐agitation, and this could include the participant at rest, engaged in activity, or walking.

Data processing and feature extraction

Agitation events are relatively infrequent compared to non‐agitation, thus the data included in this analysis was restricted to only include days in which at least one agitation event took place and was able to be fully labeled with start and end time (a total of 125 days of data collection). We extracted 37 features from filtered sensor readings after applying non‐overlapping windows of 1‐minute length. Chosen features represent the most commonly used time and frequency‐domain characteristics of each sensor signal. , , , , , , , , The extracted features are listed in Table S1 in supporting information along with their definitions.

Data analysis

Personalized models were developed to classify agitation versus non‐agitation minutes. To directly compare generic versus personalized models, we analyzed our dataset using two different approaches. The first was a leave‐one‐subject‐out approach, where the model was trained with data from all participants but one and tested on the last participant. The second approach involved using the data from one participant at a time, with half of the participant's data used for training and half for testing. We thus built two models for each participant—a generic one based on the leave‐one‐subject‐out approach and a personalized model. To compare these two approaches, for each participant, the models are tested on the same dataset. For example, the personalized model for participant 1 was trained on half of its data and tested on the second half. The leave‐one‐subject‐out model for participant 1 is trained on data from the other 16 participants (excluding participant 1) and tested the same half of the data that was used for testing of the personalized model. Personalized classification models were also developed separately for each participant to classify non‐agitation versus each separate category of behavioral symptom.

In our previous study, we identified that the random forest (RF) classifier with cost had the best performance among all tested classifiers; therefore, we chose to use it in this study to classify using 1‐minute data windows. The cost matrix was introduced into the RF model to address data imbalance by assigning a higher cost to the classification of agitation event as non‐agitation than the classification of non‐agitation as an agitation event.

Classifiers were built based on the 37 time and frequency‐domain features extracted from sensor data using window approach (non‐overlapping windows with 1‐minute length) as described in Spasojevic et al. Classifier parameters were tuned using internal two‐fold cross‐validation on the training set. There were two parameters for RF classifier—the number of trees with the range 10, 30, 50, 70, 90 and the number of predictors with values in the range [f/5, 2f/5, 3f/5, 4f/5], where f is the number of features in the feature set (f = 37). After inner cross‐validation, the selected parameters were used to retrain the classifiers on the training set and performance was evaluated based on test set. We performed 2‐fold cross‐validation for all classifiers to evaluate the data sets. AUC was used as a performance metric.

To examine the relative importance of different sensors and features for agitation detection, we applied rankfeatures() function in Matlab that ranks features by class separability (agitation vs. non‐agitation) using different criteria to assess the significance of every feature for separating two labeled groups. We used the “roc” criterion as a non‐parametric test due to non‐normal distributed classes. This criterion represents the area between the empirical receiver operating characteristic curve and the random classifier slope. Feature ranking was performed for each mode,l which provided weights reflecting the feature importance in the model. These weights were then normalized within each category in the range of 0 to 1 (where 1 is the highest weighted feature) by dividing all weights by the maximum weight within each category.

RESULTS

Description of participants and behaviors

Twenty participants consented to the study, but three did not exhibit any behavioral events during the data collection period and were excluded from this analysis. The included 17 participants’ demographic details are listed in Table 1. Over the 125 days of data included in this analysis, there were 305 fully labeled agitation events. The duration of agitation events varied from 1 minute up to 3 hours, with a mean duration of 14.9 ± 23.5 minutes and a median duration of 6 minutes (interquartile range 2–20).

TABLE 1

Description of participants and data

	Total N = 17
Demographics
Age (years), mean (SD)	78.9 (8.9)
Age (years), range	65–93
Sex (% female)	58.8%
Clinical characteristics
Clinical Dementia Rating score: number (%)	0: 0
	1: 0
	2: 3 (18%)
	3: 14 (82%)
Neuropsychiatric Inventory total score, mean (SD)	55.1 (23.7)
Neuropsychiatric Inventory agitation subscale, mean (SD)	8.2 (4.4)
Neuropsychiatric Inventory aggression subscale, mean (SD)	7.8 (3.7)
Neuropsychiatric Inventory motor disturbance subscale, mean (SD)	8.9 (3.9)
Data characteristics
Days of data included per participant, median (interquartile range)	4 (2–11)
Hours of data per day included per participant, median (interquartile range)	10.2 (8.8–11.3)
Labeled agitation events per participant, median (IQR range)	9 (5–30)

Abbreviations: IQR, interquartile range; SD, standard deviation.

Personalized models: all agitation

Combining all events as “agitation,” 17 personalized models were built to classify agitation minutes versus non‐agitation minutes within each individual participant. Model performance was variable across participants, with a median AUC of 0.87, and a range of AUC values from 0.64 to 0.95 (Figure 2A). Compared to generic models, personalized models significantly outperformed the generic models for all but one participant (Figure 3). The mean absolute difference in AUC score between the personalized and “leave‐one‐out” models was 0.28 ± 0.19.

FIGURE 2

Receiver operating characteristic (ROC) curves demonstrating the predictive power of the personalized models by plotting the true positive rate (sensitivity) versus false positive rate (1‐specificity) at different thresholds for classifying agitation/non‐agitation. Each curve represents a different participant (see legend). The dashed gray line represents an ROC curve for a random classifier. ROC curves are shown for (A) all agitation events, (B) motor agitation, (C) verbal aggression, and (D) physical aggression

FIGURE 3

Comparison by participant of the performance of personalized models to generic leave‐one‐subject‐out models for the classification of agitation minutes versus non‐agitation minutes. AUC, area under the receiver operating characteristic curve

Personalized models: behavior categories

Personalized models were built for each participant to distinguish each separate category of behavior (motor, verbal, physical) and non‐agitation minutes. In total, there were 32 sets of data analyzed across 17 participants as not all participants had the minimum required two behavioral events in each category (Table 2). The median AUC for the motor agitation model was 0.90 (range 0.49–0.98), for verbal agitation 0.86 (0.53–0.98) and for physical aggression 0.82 (0.71–1.0). Three personalized models across the three behavior categories had poor performance (AUC < 0.7), all of which had small number of agitation events on which to train the models (< 7 agitation events each), while 11 models had AUC values > 0.9 (Figure 2B, C, D).

TABLE 2

Number of non‐agitation and agitation 1‐minute windows by agitation type across participants

Participant	Non‐agitation minutes	Total agitation minutes	Motor	Verbal	Physical	Ratio non‐agitation: agitation	Participant
1	2470	2429	41		18	23	59:1
2	2004	1991	13	6	6	1	153:1
3	9977	9910	67	2		65	148:1
4	13393	13196	197	15	1	181	67:1
5	507	501	6	6			84:1
6	1944	1938	6			6	323:1
7	560	544	16	16			34:1
8	5619	5437	182	61		121	30:1
9	10485	10094	391	215	36	140	26:1
10	1138	1124	14	13		1	80:1
11	8868	8049	819	601	124	94	10:1
12	4245	3951	294		160	134	13:1
13	1463	1421	42	42			34:1
14	2821	2698	123	47	76		22:1
15	2858	2775	83	7	13	63	33:1
16	1338	1330	8		6	2	166:1
17	634	582	52	52			11:1
Total	70324	67970	2354	1083	440	831	29:1

Feature ranking

To determine which features and sensors were important markers for agitation behaviors, feature ranking was performed for each personalized model. The relative weight of each feature in each personalized model is shown visually in heatmaps in Figure 4 and numerically in the supporting information. There was considerable variability between participants in the most important sensor features for classifying agitation versus non‐agitation (Figure 4A). Looking across participants, the accelerometer features were most heavily weighted, in particular the mean, minimum, and standard deviation of the ACC signal, as well as the spectral entropy of the ACC signal. Phasic EDA measures, maximum heart rate, and mean temperature were also weighted as important in a subset of participants. Examining the differences between participants in the top 10 most heavily weighted features, participants appeared in two groupings: 8 participants had a median of 6 (range 5–8) ACC features in the top 10, while in the remaining 9 participants, there was a median of 1 (range 0–2) ACC features in their top 10. These nine participants had a combination of EDA features (median 3, range 1–6), BVP features (median 4, range 3–7), and TEMP features (median 1, range 0–2) in their top 10.

Feature ranking for personalized models for (A) all agitation events, (B) motor agitation, (C) verbal aggression, and (D) physical aggression. Refer to legend for the label for each feature extracted from the accelerometer (ACC), electrodermal activity (EDA), blood volume pulse (BVP), and temperature (TEMP) sensors

Examining the personalized models by behavior category, features from across all four sensors were associated with motor agitation including the standard deviation, mode, and entropy of the EDA signal, possibly representing motion artifacts in the EDA signals (Figure 4B). Interestingly, EDA features were weighted similarly to the ACC movement features and BVP features, with participants having a median of 3 (range 0–7) EDA features in their top 10, a median of 2 (range 0–7) ACC features, and a median of 3 (range 0–7) of BVP features. In six participants, the mean temperature was heavily weighted, while the maximum heart rate was heavily weighted in three participants.

Events consisting primarily of verbal aggression were most strongly associated with BVP features including the maximum and standard deviation of the heart rate, and the mean temperature (Figure 4C), although there was considerable individual variability in the feature pattern. For example, six of the eight participants with verbal aggression had few ACC features (median of 1, range 0–2), while two had at least seven ACC features in their top 10.

Overall, the ACC features were most important for classifying events involving physical aggression (Figure 4D), including the standard deviation and several Teager energy features of the signal. Across all participants, there were a median of 6 (range 0–8) ACC features in their top 10, although two groupings are again seen, with six participants having a median of seven ACC features (range 5–8) and four participants having a median of one (range 0–2). Those four participants with fewer ACC features had a predominance of BVP features (median 6, range 4–8) such as heart rate variability and interquartile range.

DISCUSSION

In this study, we make use of rigorously annotated multimodal sensor data to detect episodes of agitation in people with dementia. Rather than looking for common signals across all participants, we have developed models that are personalized based on the individual and that are based upon the type of behavior being exhibited. Our results demonstrate that it is possible to classify behaviors as agitated or not agitated with fair to good performance in 1‐minute windows. We have confirmed that personalized models significantly improve upon the performance of generic models. To further support the value of personalized and behavior‐specific models, we confirm that the sensor features weighted by the machine learning models varies both by individual and by type of behavior.

These findings are an important step to move beyond correlation and toward predictive analytics for behavioral symptoms in dementia. While there are discernable patterns in wearable sensor signals moment to moment that are characteristic of agitated behaviors, these patterns are largely individual. Unlike studies that have used commercial‐grade wearable devices with proprietary data processing algorithms, we used machine learning to delve into a large and complex raw sensor dataset in 1‐minute time frames. This approach has the dual advantages of making use of the full richness and detail of the data and of improving signal recognition within the data. , , One limitation of this approach was that it required a research‐grade multimodal sensor device with a short battery life. We were thus unable to examine behaviors occurring at night. Wearable technologies are rapidly advancing and affordable, clinical‐grade multi‐modal devices with advanced data processing and synchronization capabilities and week‐long battery life are on the horizon.

By using interpretable models in this study, our results provide some insights as to which features of the sensor data are most important in predicting the presence of agitation. As expected, many of the most strongly weighted features were related to movement. Physical activity in people with severe dementia has previously been shown to be correlated with degree of agitation. However, any algorithms that use motor activity alone to detect agitation risks either misclassifying healthy physical activity, such as dancing, or pathologizing wandering behaviors. Similarly, a focus on motor activity alone may miss other types of agitation, such as verbal aggression, or agitation in people with dementia who are non‐ambulatory. There is also a need to tease apart “non‐agitation,” a category in this study which encompasses positive behaviors (social engagement, pleasurable activities), neuropsychiatric symptoms such as apathy, and periods of rest or sleep. Identifying patterns consistent with these behaviors would allow us to go beyond markers of agitation or distress to track positive engagement. In this study, we have not specifically proposed a threshold or cut‐off for defining agitation or non‐agitation. Striking a balance between the sensitivity of the classification algorithm to correctly detect agitation and the risk of false positives depends largely on the application of the algorithm and an examination of the costs of false positives versus false negatives.

Our findings suggest that features derived from multiple different sensors are important for the development of digital biomarkers for behavioral symptoms. Vulnerability to stress is considered a determinant of BPSD and expression of behavioral symptoms can be conceptualized as a stress response. As such, digital biomarkers of stress are also relevant in identifying and monitoring BPSD. Previous studies have demonstrated that electrodermal activity and heart rate are reliable digital biomarkers of stress in young, healthy populations, and that there is a large inter‐individual variability in expression of stress response. In people with dementia, studies have found a correlation between heart rate and agitation, and electrodermal activity and agitation. , In further support of the need of multi‐modal sensors, we have previously shown that incorporating data from several different sensors outperforms a single sensor for detecting agitation.

The development and clinical validation of algorithms to classify behaviors are challenging and require that data be accurately annotated moment to moment as to the presence or absence of the behavior of interest. An important strength of this study is a well‐annotated sensor dataset. Clinically important episodes of agitation were flagged by staff and reviewed in videos to establish as best as possible the start and end time. However, one limitation is that it is possible that agitation events took place which were not captured on video or observed by staff. Future work can make use of positive‐unlabeled learning algorithms, an approach in which models are trained on both positive data (agitation events) and unlabeled data (combination of normal and unreported agitation events) to address this limitation.

It is important to note that most of the poorly performing personalized models were due to an inadequate amount of agitation events for training the supervised classification models. Even in this cohort with severe BPSD, episodes of clinically important agitation are relatively rare events. Given the degree of imbalance in the data between agitation and normal events, future studies should consider alternatives to supervised classification, such as the use of unsupervised learning methods, such as anomaly detection and/or one‐class classification approaches. , The small sample size in this study is an important limitation. Our sample size was limited by the large data storage requirements for the videos and the resource‐intensive annotation process. Larger data sets are needed to validate and ensure the generalizability of the algorithms. It is possible that generic models using a larger dataset may approach the performance of personalized models. In this study, models were built using sensor data alone, but we found that participants could be grouped based on their patterns of sensor features associated with agitation. There may be clinical or demographic variables, such as dementia subtype or severity, that distinguishes these groups. Including these variables in the models would be expected to help generic model performance. To address these issues, future large studies are required in clinically well‐characterized cohorts, and using efficient methods for labeling of the sensor data with behavioral symptoms in real time.

With the revolution in digital medicine underway comes the need for appropriately validated digital biomarkers. There is increasing recognition of the value of personalized metrics for health and well‐being based on longitudinal monitoring of symptoms. The most likely clinical use of BPSD digital biomarkers is for n = 1 monitoring, such as tracking symptoms or behaviors over time and in response to treatment, thus there is a clear need for agitation detection algorithms to be accurate within individuals over time. Our results support that there are individualized patterns of digital biomarkers for agitation in dementia, and that to improve the accuracy of agitation detection algorithms, future studies are needed to characterize these individual patterns.

CONFLICTS OF INTEREST

Outside of this work, AI has received research grants from the Canadian Institutes of Health Research; AGE‐WELL NCE; Province of Ontario; and the Centre for Aging + Brain Health Innovation and an Academic Scholars Award, Department of Psychiatry, University of Toronto. SS has received postdoctoral funding from the University Health Network, AGE‐WELL NCE, and Royal Bank of Canada. KN has received grants from the Social Sciences and Humanities Research Council. AW received a research grant from the Nursing Research Interest Group, Ryerson University. LSM is president of the not‐for‐profit Canadian Gerontological Nursing Association. SSK has received a research grant from the Alzehimer's Society and a honorarium from the International Conference on Gerontechnology, Hong Kong. BY and AM have no disclosures.

Supporting Information

Click here for additional data file.

Supporting Information

Click here for additional data file.

ACC	1	Teager Energy Maximum	BVP	18	Heart Rate Variability
	2	Teager Energy Minimum		19	Mean Inter‐Beat‐Interval
	3	Teager Energy Mean		20	Power Spectral Density
	4	Teager Energy Simple Square Integral		21	Spectral Energy Ratio
	5	Mean		22	Tachogram Power Low
	6	Minimum		23	Tachogram Power Medium
	7	Standard Deviation		24	Tachogram Power High
	8	Inter‐Quartile Range		25	Tachogram Energy Ratio
	9	Spectral Entropy		26	Mean
	10	DC Power		27	Maximum
EDA	11	Tonic Max of the Signal Derivative		28	Minimum
	12	Tonic Mode of the Signal		29	Standard Deviation
	13	Phasic Area under the Signal		30	Inter‐Quartile Range
	14	Phasic Number of Peaks		31	Total Average Power
	15	Standard Deviation		32	Spectral Entropy
	16	Spectral Entropy		33	Energy
	17	Entropy		34	DC power
			TEMP	35	Slope Angle
				36	Mean
				37	Standard Deviation