Predicting which type of push notification content motivates users to engage in a self-monitoring app.

Despite the unprecedented access to self-monitoring health apps, lack of optimal user engagement remains a significant challenge. Push notification prompts with contextually tailored messages offers a promising strategy to improve engagement. To increase the efficacy of push-notifications on engaging individuals with health apps, greater attention to the modifiable components of push notifications that influence responsiveness is needed. This study examines the effect of message content and frequency of push notifications, along with past app usage on responding to notifications within 24 h, and engaging with self-monitoring in JOOL Health smartphone app. Mixed models were applied on a de-identified data set of 18,000 contextually tailored push notifications sent by JOOL Health App to 1414 participants. The content in sent messages on behavioural topics were mapped into either tailored suggestions or tailored insights. Our findings suggest that push notifications with tailored suggestions were more effective overall in encouraging self-monitoring, but amongst frequent app users, push-notifications containing insights was associated with greater self-monitoring. People who were not using the app as frequently were less likely to respond to a prompt. This study suggests that push-notification content does have an impact on subsequent use of key app features, and app developers should consider what content is likely to work best for who, and under what circumstances. Secondary data-analysis of commercial apps presents a unique opportunity to elucidate and optimize health behaviors.

Introduction

Self-monitoring is a demonstrated effective behavior change strategy across an array and scope of domains (Michie et al., 2009; Hartmann-Boyce et al., 2014; French et al., 2014; Olander et al., 2013). Evidently when individuals increase monitoring frequency, their outcomes improve substantially. For example, overweight subjects who monitored diet frequently achieved significant weight loss (Carter et al., 2017). Similarly, individuals with depression increased their emotional self-awareness and subsequently reduced depressive symptoms by increasing rate of monitoring from one to six times a day (Kauer et al., 2012).

In recent years, partly driven by the convergence of demands for accessible resources and high rates of mobile phone ownership, several smartphone apps have emerged in marketplace as solutions to disseminate self-monitoring resources rapidly to populations (Carter et al., 2017; Carroll et al., 2017; Rahman et al., 2017). Smartphones are ubiquitous, relatively small and convenient for people to carry as they go about their daily lives. As a result, compared to traditional paper-based approaches, apps can prompt individuals to monitor at most opportune moments as they go about their daily lives, as well as provide contextually tailored feedback in the moment. From a public health perspective, apps have improved access to self-monitoring resources, as evident by an unprecedented number of monitoring apps now available on app stores (Hingle and Patrick, 2016; Zhao et al., 2016).

However, the lack of optimal user engagement with apps remains a significant challenge, with a significant proportion of app users disengaging from repeat use (https://www.digitaltrends.com/mobile/16-percent-of-mobile-userstry-out-a-buggy-app-more-than-twice/, n.d.). Automatically capturing data using passive sensing approaches can mitigate disengagement to some extent (Kim et al., 2016), however automated approaches are not yet advanced enough to reliably capture entire breadth of monitoring parameters. Furthermore, benefits of active participation in self-monitoring process are evident (Harkin et al., 2016). As a result devising strategies that promote users to actively engage with self-monitoring in apps is of significant importance.

Prompting, via a variety of modes, including telephone calls, emails, text messages and via app push-notifications is a promising strategy to encourage and sustain repeat use (Alkhaldi et al., 2016). Within mobile apps, push-notifications are the most commonly implemented, and also the most direct engagement strategy, appearing as a brief message at programmed times on the screen. When touched, notifications open up the programmed page within the apps. While there is evidence that the use of these techniques can lead to increased app usage, findings have been inconsistent across studies, with some studies reporting null-findings and others indicating that prompts are often ignored (Alkhaldi et al., 2016; Morrison et al., 2017; Freyne et al., 2017). This is perhaps not surprising, given the large number of push notifications that users can be exposed to in a single day (Pielot et al., 2014), and the current lack of understanding about which aspects of notifications enhance or undermine responses to prompts.

To increase the efficacy of push-notifications for encouraging desired app usage, greater attention to the modifiable components of push notifications that may influence responsiveness is needed. Based on previous research, this is likely to include notification content (Schulze and Groh, 2016; Fischer et al., 2010; Mehrotra et al., 2015). Research from internet interventions indicates that sending messages with content tailored to user's characteristics is an effective strategy to persuasively motivate users (Neff and Fry, 2009; Webb et al., 2010). This is because tailored content is perceived as more personally relevant, which is a key motivator for processing information attentively (Kreuter et al., 1999). This technique is referred to as content tailoring. With mobile monitoring apps, there are increased opportunities to employ “deep content tailoring”, which involves not only tailoring messages based on fixed characteristics (e.g., gender), but incorporating context specific information based on previously reported data and engagement. In self-monitoring apps, notifications employing deep content tailoring usually include either information on progress and insights from self-monitored data, or suggestions based on previously collected data aimed at shaping user's knowledge. Although both tailored insights and suggestions are commonly used, it is unknown which motivates users the most to respond to notifications, and furthermore to engage in monitoring activity.

Aims of this study

To date, studies investigating how prompts affect engagement have not incorporated methodologies to investigate outcomes at the level of an individual prompt (Alkhaldi et al., 2016). As a result, it is not possible to investigate how potentially optimizable factors, such as the content offered in the prompt, or contextual factors such as past usage of the app, affect users immediate engagement in response to a prompt. This study, which analyses data from a popular mobile well-being app, contributes towards addressing these gaps in knowledge.

Using a de-identified data set of 18,053 push notifications sent by JOOL Health App, through a randomization protocol, this study investigates how the odds of interacting with the app within 24 h after receiving a push notification is affected by a) content in push notification (tailored suggestions or tailored insights) b) usage of the app, and c) interaction between usage and content. It is hoped that our findings will provide useful information to app developers interested in optimizing push-notifications to enhance engagement with important app features that require some user input.

Methods

JOOL Health App push notification protocol

JOOL Health Inc. offers digital workplace wellbeing and behavioural change intervention to companies in the US. The intervention, delivered through a mobile app, encompasses self-monitoring and health messaging. Individuals monitor daily 10 different parameters – sleep, presence, activity, creativity, eating, energy, willpower along with perceived alignment with community, work and personal purposes (Fig. 1).

Fig. 1

JOOLHealth App - monitoring and feedback messages.

App users periodically receive push notifications, which serve dual purposes; to encourage users to open the app and self-monitor, and also to provide contextual feedback. The app uses a micro-randomization algorithm to alter the timing and content of push notifications (see Fig. 2). According to the algorithm, a user is eligible to receive push notifications at one of six time points on days where they are eligible to receive a notification. These six time points were chosen because office workers who are the target users of this app are less busy and thus most likely to pay attention to the notification at these times. There are two steps to the decision process. First, a randomized decision to choose a time point from the six possible time points in that day is made. The time point in a day was selected randomly, with equal probability, to ensure that sent messages were distributed uniformly across the day, over time, to draw the samples across daily changing contexts and motivation levels. Second, for a chosen time point, a decision to either send or not send a push notification is made using equal probability. If the decision results in sending push notification at a time point, then the user will not be available to receive a push notification for the remaining time points of that day. This ensures that users will receive no more than one notification per day.

Fig. 2

Micro-randomization algorithm used to alter timing and content of notification.

At any given time point, when a decision to send a notification has been made, a user's context is determined from previously monitored data, and content tailored to the context is included in the sent push notification. The content is drawn from a library of curated messages devised by JOOL Company staff with expertise in health messaging and communication on the purpose, energy, willpower, sleep, presence, activity, eating well, and creativity topics. The messages were framed to motivate, facilitate and maintain behaviour change in individuals at different stages of behaviour change (Prochaska and Velicer, 1997). When the app is programmed to send a message, a user's context at the selected time point is determined first. The context is derived from user's current and past self-monitoring data, app usage, and environmental measures such as time of day, and the day of the week. Next the tailoring algorithm identifies a subset of messages from the library that are meaningful to users' context at the decision time. A message is selected from all eligible messages with a probability distribution determined by JOOL's tailoring algorithm. JOOL staff synthesized the content of the sent message and assigned it into the following categories: (Michie et al., 2009) tailored suggestion, e.g. “Taking longer, slower breaths (…) can boost your willpower”) (Hartmann-Boyce et al., 2014) tailored insight, e.g. “Your willpower outlook's high tomorrow (…) an opportunity to build a healthy new habit”.

Study procedure

This study analyzed the content of push notifications sent to all app users between March 2017 to August 2017. Results on the decision to send notifications and effects by different times will be reported in a separate study.

JOOL staff with access to identifiable information created a de-identified anonymised data set for analysis. The data set contained participant ID, notification content type (suggestions or insights), time at which the notification was sent, and engagement markers such as time lapsed since app installation, time lapsed since most recent monitoring, number of times monitoring was done since app installed, rate of monitoring, and engagement with the app in response to notification. Rate of monitoring was defined as the number of times monitoring was done since the app was installed, divided by the time lapsed since app installation. Engagement with the app in response to notification was defined as self-monitored or not within 24 h of receiving the notification. Information on gender and range based values for age and BMI was also available for each participant id. All analysis activities were carried out on a secure R-studio server hosted on JOOL's servers with restricted IP access controls. Since the study uses existing data records, that contain only non-identifiable data about individuals, following discussions with the Executive Officer of the Social and Behavioural Research Ethics Committee at Flinders University on the nature of the study, written confirmation stating the study is deemed to be exempt from ethical review was obtained.

Analysis approach

Linear mixed effect model were used to analyze how content in push notification influences engagement in self-monitoring. These models allow us to account for dependencies shown by some variables and take the full data into account. We devised the model in R environment (with a logit link and “nloptwrap” optimizer), using the glmer function from lme4 package (Bates et al., 2017).

The dependent outcome variable was engaging with monitoring in the app within 24 h of receiving the notification, with a value either 1 (monitored) or 0 (did not monitor). Predictor variable and covariates were specified as fixed effect variables. Message content in notification which has a binary value (1 = Tailored suggestions, 0 = Tailored insights) is the predictor variable of interest. The following four variables measuring different facets of app usage at the time of notification were also specified into the model: (Michie et al., 2009) time lapsed since recent app use, which is the number of days between the date the notification is sent and the date when the app was last used; (Hartmann-Boyce et al., 2014) time lapsed since app installation, which is the number of days between the date the notification is sent and the date when the app was installed; (French et al., 2014) number of app uses, which is the number of times the app was opened between the date the notification is sent and the date when the app was installed; and (Olander et al., 2013) overall frequency of app use, which is calculated by dividing the values of number of app uses and time lapsed since app installation on the date notification is sent. In prior research these metrics have been studied as quantitative indicators of engagement (Yardley et al., 2016; Perski et al., 2017; ). Age, gender and BMI were specified as covariates as women, people with lower BMI, higher education level and age are more likely to engage with online lifestyle interventions (Brouwer et al., 2010; Schneider et al., 2012). Due to the anonymization processes of JOOL Health, age and BMI measures of app users could only be analyzed and reported as ranges.

We started with a null model with random intercepts for participant ID, day of week, and time of day, and random slope for participant, and incrementally introduced one variable at a time. We iteratively compared the model (with the introduced variable) against the reduced model without that variable. In each case, we concluded that the added fixed factor was significant, if the difference between the likelihood of these two models was significant. We perform the likelihood ratio test using the anova function and selected the model if chi-square ratio tests of the log-likelihood values were significant (p < 0.01). After identifying the candidate model with best fit, we determined which specified variables were significant (p < 0.01). Subsequently we introduced interaction terms between significant variables and assessed whether including the interaction further improved the fit. From the final model, effects for fixed variables and interactions are presented as unstandardized odd ratios with confidence intervals. Characteristics of population and notifications in the data set are descriptively presented using means and standard deviations.

Results

Descriptive

The study data set had 1265 app users, of whom 63.97% (n = 790/1235) were females. Amongst the study sample, 28.86% (n = 357/1237) of app users were under 30 years, 42.44% (n = 525/1237) in between 30 and 50 and the remaining 28.70%(n = 355/1237) in the over 50 years old category. Using a BMI cutoff >25 for unhealthy, 52.88%(n = 652/1233) participants were either overweight or obese.

A total of 18,053 push notifications with a median of 12 (average = 14.67) per individual were sent during the 3-month period through the micro-randomization algorithm process. Overall 59.64% (n = 10,767/18,053) of notifications in the data set were of ‘suggestions’ content type. Chi-squared test indicated no significant differences existing in the distribution of content type sent across day of week and time points, at which messages were sent, indicating randomization was uniform.

Equally, time points at which messages were sent were not associated with age, gender and BMI. However, the weekday at which messages were sent varied by age, (χ2 (53.48, 2), p < 0.001), gender (χ2 (53.75, 2), p < 0.001) and BMI (χ2 (22.12, 2), p = 0.001). Since prior to randomization, availability of participants at each day and time was checked, these results indicated that user's available day was not uniform across gender, age and BMI. However, these variables are controlled for in the current analysis and are unlikely to bias the results of the current analyses.

Model

The Likelihood ratio tests indicated that the model specified with the predictor variable and all covariates is the candidate model with best fit. In this model, the effect was significant for predictor variable message content and two of the four engagement metrics, viz., frequency of app use, and time period lapsed since last message. Next we updated the model with an interaction between message content and the two significant engagement metrics, which further improved the model fit (χ2 (Olander et al., 2013)=49.82, p < 0.001). Consequently, the interaction was deemed significant. The effects for predictor variable and interaction terms in this model are presented as unstandardized odd ratios with confidence intervals in Table 1.

Table 1

Table of fixed effects estimates (*p < 0.05).

Predictors (at the time of prompt)	Odds ratio	CI		p-value
Message type
Tailored insights (reference)	1
Tailored suggestions	3.56*	2.36	5.36	<0.001
Engagment metrics
Days lapsed since app was installed	1	1	1	0.964
Number of app uses since installed	1	1	1.01	0.246
Days lapsed since recent app use	0.27*	0.25	0.29	<0.001
Frequency of app use	2.64*	1.63	4.30	<0.001
Demographics
Gender
Female (reference)	1
Male	1	0.88	1.13	0.993
Age
under 30 (reference)
Middle (30–50)	1.01	0.9	1.14	0.828
50+	0.96	0.87	1.06	0.435
BMI
Normal range (reference)	1
Overweight	0.92	0.84	1.01	0.079
Interactions
Tailored insights (reference)	1
Tailored suggestion x Days lapsed since recent app use	1	0.99	1	0.547
Tailored suggestion xFrequency of app use	0.17*	0.10	0.28	<0.001

Questions

Which type of feedback in prompts is better?

Overall prompts with tailored suggestions improved likelihood of interacting with app significantly compared to tailored insights (OR = 3.56, 95% CI = 2.36–5.36).

How do app usage metrics predict response to prompts?

Amongst the variables measuring app usage, only two of the four variables, time lapsed since recent app use and overall frequency of app use had significant effects on odds of self-monitoring. Neither time lapsed since app installation and number of app uses had an effect on odds of interacting in self-monitoring in response to prompts.

As time lapsed since last app use increases, the odds of interacting with the app and self-monitoring in response to a prompt decreases (OR = 0.27, 95% CI = 0.25–0.29). Higher frequency of app use at the time of prompt significantly increases the odds of interacting in response to prompts (OR = 2.64, 95% CI = 1.64–4.30). These results indicate people who aren't using the app as frequently, or when people receive prompts long apart from when they stopped using the app, are less likely to respond to a prompt.

Interactions between feedback type and app usage metrics on response to prompts

There was a significant interaction between message type and frequency of app use, such that sending prompts with tailored suggestions to individuals with high frequency of app use at the time of prompts significantly reduces the odds of interacting with app and monitoring (OR = 0.17, 95% CI = 0.10–0.28). We confirmed through linear regression that there were no significant independent associations between frequency of app use and type of push notifications sent. This indicates that prompts with tailored insights are better than tailored suggestions when individuals become frequent app users. The interactions between time lapsed since recent app use and message types were not significant.

Discussion

Principal findings

This study aimed to enhance our understanding of how best to engage users in key mobile app features using push-notifications. Our analysis of JOOL Health's mobile app user's push notification data showed that the odds of interacting with the app to self-monitor were significantly higher when the sent prompts contained suggestions content rather than insights content. In addition, frequent use of the app increased the odds of interaction with the app in response to a prompt, and as frequency of monitoring increased, sending prompts with content containing suggestions rather than insights reduced the odds of interaction with app and engaging in monitoring (OR = 0.17, 95% CI = 0.10–0.28). Equally the odds of interacting with the app in response to a prompt decreased when the time lapsed since last app use increased, however the interactions with message types were not significant. These results provide preliminary evidence that interaction with an app after a prompt is sent is dependent upon the type of strategy contained within the content of prompt, and frequency of app use. Frequent app users are significantly more receptive to prompts with insights from their data compared to suggestions containing strategies/tips.

Comparison with previous work

There is a proliferation of mobile health apps using monitoring and feedback approaches to promote healthy behaviors (activity, healthy eating, sleep and mental wellbeing) with a large number of users (Abroms et al., 2011; Middelweerd et al., 2014; Bardus et al., 2016). Sending periodically push notification prompts to draw users attention through the app is a common strategy to encourage interaction (Morrison et al., 2017; Freyne et al., 2017). Amongst other factors likely to influence user's receptivity to prompts (e.g., timing, aesthetics, frequency, etc (Muench and Baumel, 2017)) an important and optimizable one is the type of content in the prompt. Users are receptive to prompts containing interesting and persuasive content, as such content is generally motivating (Schneider et al., 2013). However, there is a gap in knowledge on the type of content in prompts that increases the likelihood of interacting with the app, and thus enhance receptivity to the momentary intervention available within the app (Leon et al., 2014). Thus far literature on receptivity to prompts has mainly focused on timing and mode of prompts.

According to a systematic review, the categories of prompt contents investigated in this study, suggestions or feedback category types are common forms of behavior change techniques used in prompts (Alkhaldi et al., 2016). The literature from health messaging indicates that participants' receptivity to messages increases when content is tailored and relevant to a user's state, as well as less –repetitive (Kreuter et al., 1999). Overall we found that receptivity to push notifications containing suggestions was better than those containing insights. In the data set we analyzed, a majority of push notifications were sent to users in early stages (see Fig. 3). During early stages there is less historical information compared to an advanced phase of app use. Consequently push notifications of content type suggestions with pointers and tips are most likely more tailored to the state of user compared to insights from app data, and hence more effective. However, as individuals become engaged and more regular users of app over time, they are more receptive to insights over suggestion type prompts. This could be due to the fact that as user become regular and invested in monitoring functionalities, they are, having achieved mastery, more likely motivated and invested in uncovering potentials for improvements through examining and learning from their data. For instance, frequent users of weight loss apps highly customize their diet and exercises (Serrano et al., 2017). Equally, as individuals become frequent app users, the depth and granularity of data available on them continues to increase. With increased information, the depth of tailoring for insights improves, which may suggest why insights perform better over suggestions as frequency increases.

Fig. 3

Distribution of push notifications sent over time.

As per the Transtheoretical Model of Health Behavioural Change, frequency of app use can be indicative of stages of health behaviour adoption. In the early phases of adopting a new behavior, users are more likely motivated by suggestions, and equally at this stage due to limited data, quality of tailored suggestion is likely better than tailored insights. However, over time, as user's become invested with the app and use it frequently, they transition to the behaviorial maintenance phase. During this phase, both their receptivity to, and quality of, tailored insights the app can offer increases. This indicates that users in early stages of behavioural change need educational messages to help increase their intentions to change. When users are already motivated, they are likely to receive greater benefit from insights. In this study we also found that interaction decreases as time lapsed between prompts increases. App developers should consider increasing prompting frequency to non-adopters in-order to increase engagement (Schneider et al., 2013).

Strengths and limitations

This study has some limitations that should be acknowledged. First, self-selection bias might limit the generalizability of finding, since the analysis is on data obtained from a conveniently sampled population using a commercial app. Though, this does suggest findings are likely to be applicable to real-world app users interested in this topic. Second, self-monitoring within 24 h of push notification was used as the measure of engagement. While this is an important marker, it is possible to engage with the app without self-monitoring, and as such we may have potentially missed some interaction with the app. To address these limitations, replication of the findings in other samples, in apps focused on other topics, and with additional engagement measures is recommended. The study also has several strengths. Notably, the gender balance was better in this study. Other factors that may impact on user's receptivity to prompts were controlled for (e.g., timing) or held constant across conditions (aesthetics, screen placement), allowing us to tease out effects of message type and frequency with more confidence. There was also a degree of randomization between the type of interventions offered within app, and the sample size was sufficiently large to explore interaction effects.

Conclusion

Overall, the data from this study suggests that push-notification content does have an impact on subsequent use of key app features, and app developers should consider what content is likely to work best for who, and under what circumstances. Amongst JOOL app users, tailored suggestions were more effective overall for encouraging self-monitoring, however this was not the case amongst frequent app users. Amongst frequent app users, push-notifications containing insights was associated with greater self-monitoring. Future studies are needed to expand on these findings. Secondary data-analysis of commercial apps presents a unique opportunity to do so, especially when micro-randomization has been employed.