ULTEMAT: A mobile framework for smart ecological momentary assessments and interventions.

In this paper we introduce a new Android library, called ULTEMAT, for the delivery of ecological momentary assessments (EMAs) on mobile devices and we present its use in the MoodBuster app developed in the H2020 E-COMPARED project. We discuss context-aware, or event-based, triggers for the presentation of EMAs and discuss the potential they have to improve the effectiveness of mobile provision of mental health interventions as they allow for the delivery of assessments to the patients when and where these are most appropriate. Following this, we present the abilities of ULTEMAT to use such context-aware triggers to schedule EMAs and we discuss how a similar approach can be used for Ecological Momentary Interventions (EMIs).

Introduction

Ecological Momentary Assessment (EMA), also known as in the moment assessment or experience sampling, is widely accepted as an important means of eliciting information about a person's physical and mental state in the moment as opposed to retrospective assessment. EMA approaches were developed in various fields starting in the early 20th century and any given approach strongly reflects the technological capabilities of that time. The first studies were conducted in the 1920s and consisted of daily diary entries using pen and paper by participants on their experiences, symptoms and behaviour in studies lasting several days or weeks (Robbins, 2014). One of the main disadvantages of this approach was that diaries were typically completed once per day at the end of the day, resulting in issues around memory problems and recall bias (Stone, 2007). Furthermore, a missed diary entry typically represented missing data for a full day and often constituted a high proportion of missing data. As mobile technology became available, research studies started making use of reports that were requested in a more ecologically valid way by prompting users for such reports with watches or pagers at specified time points throughout the day (Czikszentmihalyi, 1987). An example of early use of this approach is for the capturing of thoughts in the natural environment (Hurlburt, 1979) but this pen and paper approach with electronic prompts for input was still in use in the mid to late 2000s (Wichers et al., 2007) for the capturing of appraisals of stress and positive affect. In a study assessing self-reported substance abuse behaviour in a group of college-age marijuana users, SMS messages were used as the medium to perform EMAs with an overall response rate of 89%. Moreover, the SMS-based EMAs showed significantly higher usage than reports obtained through timeline follow back assessments. However, some studies show that the use of electronic prompts does not necessarily result in satisfactory compliance to the self-report requests (Broderick et al., 2003).

Notwithstanding the varying degrees of success in improving adherence to the requests, various publications have demonstrated the potentially beneficial use of EMAs as part of treatments for depression, anxiety disorders, substance abuse and schizophrenia. For example, Walz et al. (2014) performed a review of EMA use for anxiety disorder studies between 1997 and 2014 in studies using paper diaries, dedicated apps on personal digital assistants and phones (voice calls) for the collection of the EMAs. They concluded that EMAs may provide a better insight in temporal variability of symptoms in anxiety disorders and the relationship of symptoms with situational cues. Wenze et al. (Wenze and Miller, 2010) suggested that EMA's capability to assess patients in those settings that are most problematic, such as particular social contexts, may result in better insights in the aetiology of mood disorders. Furthermore, Pfeiffer et al. concluded that the information gleaned from weekly EMAs captured through automated telephone-based interactive voice response calls can be predictive of future depression outcomes (Pfeiffer et al., 2015).

With the advent of smart phones, a technology that allows very natural gathering of EMA data has become available. In a very short time, smart phones have become a dominant interface between users and the ICT infrastructure that is all around us. As they already present, and allow users to respond to short messages from other users, social media or online services, the smart phones are an appealing alternative to dedicated devices (such as the now almost obsolete personal digital assistants) for the delivery of EMAs.

The current paper describes a software framework for the delivery of EMAs on smart phones called ULTEMAT (University of Limerick Tool for Ecological Momentary Assessment and intervention Triggering). In Section 2, we introduce the ULTEMAT framework and its use for traditional time-triggered EMAs briefly. This is followed in Section 3 by a practical example of use of the ULTEMAT platform in a mobile application to be used as an adjunct in depression interventions. In Section 4 we discuss potential directions for future delivery of EMAs based on context awareness and in Section 5 we discuss how this is possible within the ULTEMAT platform. A discussion around context-aware delivery of EMAs and the extension to Ecological Momentary Interventions (EMIs) is presented in Section 6 and conclusions are presented in Section 7.

The ULTEMAT framework for time-based EMA triggering

ULTEMAT is a newly developed platform that strives to provide a flexible approach to the scheduling and delivery of EMAs.

In its simplest form, ULTEMAT controls the delivery of time-based EMAs to the user as graphically represented in Fig. 1.

Fig. 1

Time-based delivery of EMAs.

From left to right Fig. 1 shows the generation of messages by either a human operator or an automated system. The minimum information provided in the EMA is: the question, the means of assessment (slider, multiple choice…) and the time that the EMA should be shown to the user. These EMAs are sent to a server where they are stored for retrieval by the user's mobile device. Upon retrieval, EMAs are shown to the user at the time prescribed by the EMA. The ULTEMAT platform was created such that EMAs can consist of traditional single or multiple item questionnaires responded to on Likert (1932) or visual analogue scales, but EMAs can also consist of automated monitoring of the user based on the device's sensors (such as accelerometers to assess activity levels, GPS to assess location, microphone to assess environmental sounds). This is also referred to as unobtrusive or objective assessment (Asselbergs et al., 2016). EMAs can also take the form of a more extensive interaction with the user, such as a short game (Tong et al., 2016).

Use of ULTEMAT within ECOMPARED

ULTEMAT is used in the H2020 E-COMPARED project, which investigates the cost-effectiveness of blended cognitive behavioral therapy (CBT) depression treatments in which face-to-face treatments are complemented by ICT based intervention adjuncts (Kooijstra et al., 2014). These are compared to face-to-face treatments as usual in primary and specialised outpatient mental healthcare services in 9 EU countries (NL, DE, FR, CH, SP, UK, PL, SE, DK) (Kleiboer, 2016). One of these ICT adjuncts is a mobile EMA app, called MoodBuster, which makes use of ULTEMAT for Android.

The MoodBuster app

The role of MoodBuster in the E-COMPARED project is to:

Present the user with ratings on various aspects of their physical and mental health.

Provide the user with motivational feedback to improve treatment adherence.

Provide the user with a graphical interpretation of their mood levels over time.

Allow therapists to communicate with their patients.

The above functionalities were translated to five different types of messages:

Rating requests/assessments are scheduled throughout the day to obtain information from the user on mood, sleep quality, rumination, self-esteem, enjoyment of activities, social contacts and level of (pleasant) activities. Once received by the phone, an alarm is generated at a time determined by the rating request after which a notification appears. When the user selects this notification, the screen depicted in Fig. 2 is opened, in which the user can perform the rating. As the daily rating of all above-mentioned aspects would result in a significant burden on the user, on most days only a sub-set of the measures are rated. On other days, a full questionnaire is presented to the user.

Fig. 2

Mood rating.

Users can obtain an insight in their mood fluctuations using the graphical representation in Fig. 3. These mood graphs are scrollable and zoom-able such that users can get insights in both long-term and short-term changes in mood fluctuations.

Fig. 3

Graph of mood for one week period.

Questionnaire requests are a special type of rating request that specify that all ratings in the request should be presented to the user consecutively at a predefined time.

Feedback messages provide the patient with automated feedback on their therapy progression based on the user's ratings and their adherence to these ratings. Users are notified of new feedback messages through the phones notification manager at which point the messages are available in the Inbox type view shown in Fig. 4. Selecting any of these messages the user has access to the message details as illustrated in Fig. 5.

Fig. 4

Inbox with feedback and reminder messages.

Fig. 5

Example feedback message.

Reminder messages prompt the user to adhere to various elements of the therapy. For example, in the MoodBuster system an automatic reminder is generated if a user does not respond to rating requests. The presentation of Reminder messages is similar to that of Feedback messages (see Fig. 6).

Fig. 6

Example reminder message.

Therapist messages, should they be required, are messages generated by the therapist to provide users with any further information deemed to be of use by the therapist. These messages are very similar to emails, will create a notification and will appear in the user's Inbox. For the E-COMPARED project, we chose not to provide users with an app facility to respond to these therapist generated messages. Rather, users were encouraged to use a web interface for this purpose. These messages can be used to reinforce content discussed during a previous therapy session, provide specific personalised motivation or tips, or any other content deemed useful by the therapist.

Healthcare provider interface

Although strictly speaking not part of ULTEMAT, to illustrate the potential for integration of ULTEMAT in healthcare platforms, a short description is provided of the interface used by therapists to interact with their patient This interface allows therapists to view the patient's progress in their therapy, provide the aforementioned patient specific ‘Therapist messages’, schedule meetings with the patient and obtain information on the EMA ratings provided by the patient through the MoodBuster App. Traces of varying EMA types can be plotted in one graph, as illustrated in Fig. 7, to provide an insight into symptom levels, trends therein and potential relationships between the various EMAs and other data captured.

Fig. 7

EMA chart showing the relationship between Activities performed and levels of worrying.

Future directions: ‘smart’ EMAs

Whilst EMAs are an increasingly popular component of mobile interventions and smart phones are an appealing medium for EMA delivery, time-triggered scheduling of these components may not allow EMAs to be used to their full potential. However, smart phones offer rich possibilities for so-called event-triggered EMAs (Intille, 2007) that may extend the benefits of EMAs significantly by allowing scheduling of these EMAs to coincide with events of interest.

Smart phones typically contain a plethora of sensors and other data sources that can inform aspects of a user’s wellbeing, activities or intentions and thus smart phones can be used to schedule EMAs based on this information. Although it may be interpreted as an invasion in a user's privacy, the use of such data has been reported as acceptable to users in various studies. For example, Ramanathan et al. (2013) found that HIV positive individuals expressed a desire for automated location based medication reminders, which would require access to sensitive geo-location data. Based on focus groups with individuals experiencing binge eating, Juarascio et al. (2015) reported that participants accepted the automated collection of time stamps, geo-location, physical activity and weight measurements with some respondents expressing the wish to be able to control whether this information was shared with others.

The potential of smart phones goes far beyond the monitoring of geo-location and physical activity. Smart phones allow for the unobtrusive monitoring of user behaviour and the resulting data can be used to make inferences about the user's context, activities, behaviours and, to some extent, mental status (Burns et al., 2011). Burns et al. used machine learning algorithms and a large set of mobile phone derived measures (a minimum of 38) to predict aspects of patients’ mental state, activities and social contexts. Although the prediction of some states, such as mood, was poor, contextual states, such as location, conversational state and the vicinity of friends and other people, were predicted with accuracy rates of between 60% and 91%. This means that some EMAs can be activated in a context aware fashion based on the inferred user physical activity, environment, and smart phone interactions. However, these results also show that these predictions should be treated with care and that further assessment of the appropriateness of an EMA may be required.

A simple example of such a context or event-based schedule is the triggering of a sleep rating just after the user has gotten out of bed. Conversely, context awareness can also be used to prevent the presentation of EMAs when the user is expected not to respond to these, for example when driving a car or whilst giving a presentation. We predict that context aware triggering of EMAs, which we call smart EMAs, will yield various beneficial effects for several reasons listed below:

Adherence to the smart EMAs may be higher as they can be scheduled when it suits the user best to respond or when the smart EMA is likely to be most appropriate.

Information gleaned from the smart EMA may provide new insights as the context in which the EMA was gathered can also be captured. For example, a period of bad weather may result in reduced physical activity (the information of interest). In this case, recording of actual weather in the user's vicinity could help by attributing the low level of activity to bad weather instead of to the user's status. Moreover, it may be possible to reduce the recall bias generally associated with self-reports, as the smart EMAs may allow for the presentation of the self-report as close as possible to the moment of interest.

Assessment Attrition may be reduced as the timing and frequency of smart EMAs can be adjusted to the user's reaction to the smart EMAs. For example, if adherence to the smart EMAs remains low, the frequency of same can be reduced and only the most important smart EMAs can be chosen for scheduling.

Self-management of health and disease may be improved. With contextual information available, it is possible to correlate smart EMA ratings with user context (environment, activities, time of day) and provide the user with an insight in how these contextual constructs determine their mental well-being. Such insights may allow users to better control their presence in positive and negative situations, thus improving self-management of stressors.

Context aware triggering of EMAs has been the subject of research endeavours since the early 2000s. For example, in the CAES (Context-Aware Experience Sampling) project (Intille et al., 2003), a platform for iPAQ PDA devices running Microsoft PocketPC was developed. This platform, which is no longer supported, was capable of presenting questionnaires to the user with triggers generated internally or in auxiliary devices such as a camera, heart rate monitor or GPS device. The platform was later merged with the MyExperience1 mobile data collection tool, which was developed as a BSD–licensed open source project using .NET CF2 and Microsoft SQL Compact Edition. MyExperience (Froehlich et al., 2007) uses a three tier architecture in which sensor information can be used to define triggers, which in turn result in the launch of an action. The functionality can be defined using XML and a proprietary scripting language. Another application, Paco, which is short for Personal Analytics Companion, allows the online design of EMAs with limited triggers, such as use of the app and the user hanging up the phone.

To the authors' knowledge, there currently is no publicly available platform that provides a flexible and versatile solution to the triggering of EMAs based on user behaviour and context for any of the popular smart phone operating systems. Hence the ULTEMAT platform described in Section 2 was extended with this ability with the overall aim of developing a generic platform for the delivery of context-aware EMAs.

With these event-based triggers, the ULTEMAT platform allows intervention providers, researchers or other content providers to define types of smart EMAs for the user and to describe rules that govern the presentation of these smart EMAs to the user. These rules can use any of the sensors on the smart phone, but also other data sources available to the system. For mobile devices, one can typically obtain information on the user's location, their current activities, environmental factors such as the weather, noise level and light levels, interaction with other people, use of the mobile device and planned activities (if present in the user's calendar). Hence, a mood rating could be scheduled to be delivered to the user when the user is in a certain location, when deemed to be in a quiet place where interaction with the intervention is appropriate, or when mood is inferred to be low (e.g. user inferred to be inside during the day with low activity and low light levels measured). Likewise, recommendations or feedback could be provided to the user when the latter is taking a break from their daily activities, fidgeting with their mobile phone, or when user adherence with the intervention is deemed low. As an example of the latter, reminders to be physically active could be sent to users when the weather is known to be good and the user has not been physically active recently.

The ULTEMAT framework for smart EMA triggering

ULTEMAT uses a flexible, extendible architecture such that new triggers and new EMAs can be easily added to the existing code base. Fig. 8 illustrates typical use of the system for both classical, time-triggered EMAs and smart EMAs.

Fig. 8

Information flow for classical and smart EMEs in ULTEMAT framework.

With reference to Fig. 8, the creation of EMAs and communication of these EMAs to user devices is similar to the process described in Section 2. Once received on the user’s device, the EMA will be scheduled for delivery. For the examples in Fig. 8, the time based EMA for a mood rating will be presented to Jane Smith at a predetermined moment in time (15:48). In contrast, the event-based trigger for the Feedback message to John Burke will result in the EMA being presented to John during his morning break, which is signalled by inactivity between 10 and 11 am.

The software architecture used to control the above described scheduling of EMAs is depicted in Fig. 9. It consists of a software component with which EMAs can be constructed based on the information received from the server (EMAFactory). These EMAs either are of the traditional type with time-based scheduling or of the ‘smart’ type which is defined by the use of custom triggers derived from user behaviour, context or environment. Each EMA also defines, if, and how it is presented to the user. As previously mentioned, EMAs can be of the traditional single or multiple item questionnaire type, but can also consist of information gathering via more extensive user interaction such as a short game, or a measurement performed without any user interaction. Triggering of an EMA will typically be managed using a notification on the smart phone. By interacting with the notification, users can activate the actual EMA. Some EMAs such as those that gather information from phone-based sensors, do not require user intervention and may for this reason not use the notification manager. Instead, these will be started automatically when the EMA is triggered.

Fig. 9

ULTEMAT software architecture.

Starting in Fig. 9 on the left hand side, upon reception of a message from a remote server, the EMA is added to a queue of EMAs. Typically, the processing done at this stage is checking when and how the EMA should be displayed to the user, scheduling this event and adding the EMA content to the EMA database used for this purpose. Once the scheduled time is reached or the EMA is triggered by the defined event, the user is notified of the impending EMA. Typically, this means that a notification is shown to the user as the standard pattern used on Android and iPhone devices to make users aware of events. Upon the user selecting the EMA in the notification screen, the phone will trigger the action associated with the EMA type.

Where the EMA results in information that needs to be reported to the server, this is again stored in the EMA database. These values will subsequently be reported back to the server.

Fig. 9 illustrates this process for two types of EMAs, a Rating Request and a Feedback Message. In case of the rating request, once the rating schedule time is reached, a notification is shown. When the user selects this notification, a dedicated screen will be opened to Display the Request, requesting the user to rate their mood, sleep or other parameter as defined by the EMA. The resulting rating is stored in the on-phone database and sent back to the server during the next synchronisation event. Feedback messages will similarly be sent through the phone's notification screen to make the user aware of the availability of new feedback. When the user selects the notification in the notification screen, the user is automatically directed to a Message Inbox with all new and historic feedback messages.

ULTEMAT as a system component

The ULTEMAT platform was implemented as an Android component and applies its context-aware capabilities independent from the server component. As a result, ULTEMAT can be made to interact with existing server side EMA components by adhering to a shared communication protocol (which will require some adaptations in either the app or the server component). Furthermore, to make use of the context aware capabilities of the ULTEMAT platform, information on such triggers would need to be added to the EMA information sent to the mobile device. Due to the modular approach taken in developing the ULTEMAT framework, it is straightforward to include the ULTEMAT software in any Android system used for the delivery of EMAs. By default, Android separates visual resources, such as layouts, texts, graphics and multi-media and as a result, the ULTEMAT component can easily be modified to adhere to the prevalent visual characteristics or requirements. Moreover, as the user interface used for any EMA object is defined separately from the EMA object, any existing user interface can be deployed for this purpose. Hence, a currently existing EMA system that delivers EMAs to the user based on predetermined time schedules could easily be adapted to make use of the ULTEMAT framework. With its availability for Android (an iPhone version is planned), ULTEMAT is currently available for approximately 50%–70% (depending on geographical location) of smart phone users.

Discussion

In this paper, we introduced the ULTEMAT platform and we illustrated use of ULTEMAT in MoodBuster, a mobile app designed for, and used in the H2020 funded E-COMPARED project for the time-based delivery of EMAs. Various studies have shown that, whilst mobile devices are a useful medium for the gathering of EMAs, adherence to these EMAs is often low, also when audio signalling is used to notify users of the EMA requests (Broderick et al., 2003). We postulate that the use of context triggered EMAs may be able to improve adherence to the EMAs and presented the functionality built into the ULTEMAT software that enables such context aware triggers.

The ULTEMAT framework currently allows scheduling based on time, app use, physical activity, location and responses to EMAs. Due to the modular approach used in the design of ULTEMAT these triggers can be easily extended to include any other trigger that can be derived on the phone or remotely from sensors, events or user interaction. Examples of such triggers are weather related parameters, parameters derived from the user's social media, social interactions and inferences based on longer-term analysis of user behaviour. Although the use of such information can be intrusive, we believe careful choices by clinicians as to what information can truly assist the user in their therapy and the existence of strong therapeutic alliances will help users accepting any potentially perceived intrusion in privacy.

The use of ULTEMAT in the E-COMPARED project included delivery of feedback from clinicians and automated feedback to patients. Although feedback messages are a very simple example, such delivery of treatment content in a more ecologically valid way has come to be known as Ecological Momentary Interventions (EMIs). Whereas EMAs play an important role in eliciting information from the user about their experiences in their everyday lives, EMIs attempt to provide the intervention aspect of a therapy at the most suitable moment in time (Heron and Smyth, 2010).

Scheduling and content of such EMIs may be determined by the severity of symptoms as measured by EMAs. For example (Depp et al., 2015) compared an intervention for bipolar disorder using mood EMAs and the presentation of coping strategies based on symptoms and early warning signs reported (EMI), to a paper-and-pencil mood chart control with access to static information on coping strategies. They found that depressive symptoms in the group using the EMI intervention were significantly lower than in the control group at follow up shortly after the intervention, thought these differences were not retained. As a further example of use of EMIs, in (Dror Ben-Zeev, 2014) the response of schizophrenic patients to a mood related EMA was used to decide whether a more in-depth assessment should follow. Results of the latter were then used to determine the content of EMIs presented to the user subsequently.

Although in its infancy, smart phones offer an exciting means of controlling EMIs that goes far beyond scheduling based on EMAs as described in the previous paragraph. Similar to the smart EMAs presented in Section 4, we can think of smart EMIs as EMIs that make use of the phone’s context awareness to determine content and timing of the EMI. In such a scenario, EMAs and EMIs would go hand-in-hand to determine what content to offer at what time. For example, context information automatically gathered and assessed on the mobile phone may trigger an EMA. If the EMA confirms the inference arrived at, an appropriate aspect of the intervention could be offered to the user immediately, or at the earliest suitable moment in time. The latter may yet again be confirmed using an EMA. To give a concrete example: although a user may have practiced the application of coping strategies in a situation of perceived threat or mental crisis as part of a CBT intervention, the user may not be able to recognise this situation in time, nor may they have learnt these skills to the extent that they can be applied successfully without prompts. A context aware EMA would recognise this situation and present itself to the user to assess the user’s mental status in response to the perceived negative situation or event. Subsequently, the smart phone would present an EMI suggesting the most appropriate coping skill to the user at the earliest possible opportunity. Of course, such strategies could also be employed to contexts that are regarded to be positive such that the patient can be encouraged to increase the frequency of such contexts.

From a technical perspective, there is no real distinction between EMAs and EMIs. For this reason, as far as their implementation is considered, they can be treated as a single entity, which we call the Ecological Momentary Entity (EME). Interestingly, current research investigates whether EMAs induce assessment reactivity in mildly depressed (van Ballegooijen et al., 2016), which, if positive results are found, would suggest that also from a clinical perspective EMAs and EMIs share more commonalities than their ecologically valid presentation.

Conclusions

This paper introduced ULTEMAT, a new framework for the delivery of Ecological Momentary Assessments (EMAs) and illustrated its use for the time-based scheduling of EMAs in the MoodBuster application. Various studies have shown that, although EMAs can play a beneficial role within therapies, response adherence to these EMAs is often low. We postulate that the use of context-awareness in the scheduling of EMAs and determination of their content may be able to improve both adherence to the EMA requests and their information content. Such EMAs we refer to as ‘smart EMAs’ and we present ULTEMATs capabilities to implement smart EMAs. ULTEMAT is equally suitable for the delivery of smart Ecological Momentary Interventions, or smart EMIs. Moreover, as the distinction between (smart) EMAs and (smart) EMIs is troubled, certainly from a technical perspective, we refer to (smart) EMAs and EMIs combined as (smart) Ecological Momentary Entities, or EMEs. Although smart EMEs will yet need to prove their worth, they hold a great promise for future connected health interventions as they allow for the scheduling of intervention content when and where this matters most.