Implementing assessments via mobile during the acute posttrauma period: feasibility, acceptability and strategies to improve response rates.

Background: PTSD is posited to develop in the acute posttrauma period. Few studies have examined psychopathology symptoms within this period due to the demands on individuals in the first month after a trauma. Mobile devices can overcome these barriers. The feasibility of using mobile devices for this purpose, however, is unclear. Objective: The present study evaluated the acceptability of administering PTSD symptom assessments via a mobile application throughout the acute posttrauma period. Method: Participants (N = 90) were recruited from a Level 1 Trauma Center within M = 4.88 days of experiencing a traumatic event. A mobile application was placed on their smartphone that administered a daily self-report assessment of PTSD symptoms for 30 days. Participants were compensated US$1 for each assessment completed.
Results: The overall response rate was 61.1% or M = 18.33, SD = 9.12 assessments. Assessments were accessed M = 65.2 minutes after participants were notified to complete them and took M = 2.52 minutes to complete. Participants reported that the daily assessments were not bothersome and were moderately helpful.
Conclusion: The present study suggests that using mobile devices to monitor mental health symptoms during the acute posttrauma period is feasible and acceptable. Strategies are needed to determine how to best take advantage of these data once collected.

A significant portion of those exposed to a traumatic injury will develop posttraumatic stress disorder (PTSD) in the following year (Zatzick et al., 2007). There is a need for strategies to identify and treat those at risk shortly after such events (Kearns, Ressler, Zatzick, & Rothbaum, 2012). A key challenge in developing such strategies has been a lack of knowledge about how such disorders develop during the acute posttrauma period. There are considerable demands on trauma victims at this time such that participation in research during this period is difficult (Zatzick et al., 2001). Solutions that allow researchers and clinicians to interact with victims during this period in an unobtrusive manner are needed.

Mobile devices can conduct assessments and provide intervention during the acute posttrauma period. In 2016, 77% of American adults owned a smartphone (Pew Research Center: Internet, Science & Tech, 2017). Mobile devices allow for regular communication with trauma-exposed individuals at the times in their day that are most convenient for them. This communication occurs asynchronously to reduce participant and researcher burden. Preliminary work on the use of mobile-based assessments has suggested that responses to self-report measures on a mobile device are comparable to those obtained via pencil-and-paper methods (Price, Kuhn, Hoffman, Ruzek, & Acierno, 2015).

Despite the proposed benefits of this strategy, it is necessary to determine the extent to which it is feasible and acceptable to this population. Work conducted in other samples points to its utility of the acceptability. Prior work with a depressed sample obtained a 70% response rate for data collected via mobile devices five times per day for seven days (Putnam & McSweeney, 2008). Similarly, a study assessing substance use during adolescence reported an 88.83% response rate (Phillips, Phillips, Lalonde, & Dykema, 2014). Assessments were conducted three times per day for 14 days. A study using a trauma-exposed sample conducted six assessments per day for two weeks and obtained a response rate of 67.5% (Dewey et al., 2015). However, frequent assessments may prove overly burdensome during the acute posttrauma period.

Several pilot studies offer preliminary evidence about the acceptability of mobile assessments during the acute posttrauma period. Price and colleagues (2014) used text messages to assess PTSD symptoms over a two-week period after a traumatic injury in a sample of 24 adults. Using a single question per day and no compensation for compliance, an overall response rate of 63.1% was obtained. Another study used a similar text message strategy with 19 adolescents who were recruited from a hospital after a traumatic injury (Davidson, Bunnell, & Ruggiero, 2017). Using a single daily question for a period of 30 days, a response rate of 88% was obtained. Only one study was found that used a mobile application to administer assessments longer than a single item (Price et al., 2017). Participants (N = 23) were recruited after a traumatic injury and completed daily assessments of five questions for 30 days. The response rate was 44.93%, which was markedly lower than that of the text messaging studies. A recent study used text messages to track PTSD and pain symptoms for 14 days after an injury (Pacella, Girard, Wright, Suffoletto, & Callaway, in press). The mean response rate was 86%. Taken together, these results suggest mobile devices are a possible method to collect data from recently trauma exposed individuals, but assessment length may affect response rates.

The previously described literature has several limitations. The sample sizes across most studies was small, which limits the generalizability of these results. Two studies used single item assessments, which is unlikely to sufficiently evaluate the complex development of mental health problems. Additional work is needed on the implementation of mobile-based assessment during the acute posttrauma period.

The present study further evaluated the use of mobile-based assessments during the acute posttrauma period. The aims were to determine the response rates for a multi-item self-report assessment administered via mobile application in a large trauma-exposed sample. Additionally, the time it took individuals to access an assessment and the time needed to complete a longer assessment were examined.

Methods

Participants

Participants (N = 90) were recruited from a Level 1 Trauma Center in the Northeastern US. Inclusion criteria were access to a smartphone and having experienced an injury that met criterion A for the PTSD diagnosis according to DSM-5 (Table 1). Criterion A injuries involved actual or threatened death, physical injury or sexual violence (APA, 2013). Exclusion criteria were suicidal ideation, police custody, psychosis or moderate or severe traumatic brain injury.

Table 1.

Sample demographics.

	N	%
Male Gender	57	63.3
Completed College	36	40.0
Race: White	80	88.9
Race: African American	4	4.4
Race: Asian American	1	1.1
Race: Pacific Islander	1	1.1
Race: Native American	4	4.4
Trauma Type: Motor Vehicle Crash	45	50.0
Trauma Type: Physical Assault	1	1.1
Trauma Type: Recreational/Work Accident	19	21.1
Trauma Type: Fall	13	14.4
Trauma Type: Burn	8	8.9
Trauma Type: Other	4	4.4
Mobile Phone: iPhone	52	57.8
Mobile Phone: Android	35	38.9
Mobile Phone: Other	3	3.3
	M	SD
Age	35.10	10.41

Measures

Standardized Trauma Interview (STI; Foa & Rothbaum, 2001)

The STI is a 41-item interview assessing relevant aspects of the trauma. The STI was used to determine if the most recent traumatic event met the DSM-5 criterion A for a diagnosis of PTSD.

PTSD Checklist-5 (PCL; Blevins, Weathers, Davis, Witte, & Domino, 2015)

The PCL-5 is a 20-item self-report measure that assesses PTSD symptoms experienced over the last month according to DSM-5 criteria. Items assess symptoms across four symptom clusters of PTSD (re-experiencing, negative mood, avoidance, hyperarousal) on a 0–4 point Likert scale. Total scores range from 0–80. All administrations of the PCL were anchored to the index event for the current study.

Patient Health Questionnaire-8 (PHQ-8; Kroenke, Spitzer, & Williams, 2001)

The PHQ-8 is an 8-item self-report measure that assesses depression symptoms over the past two weeks. Ratings are made on a 0–3-point Likert scale regarding the frequency with which a symptom occurred. Scores range from 0–24; with higher scores indicating more severe depression.

Sheehan Disability Scale (SDS; Sheehan, )

The SDS is a 3-item self-report measure which assesses functional impairment. Items assess symptom impairment on three domains (work/school, social life, family life/home responsibilities) on a 0–10 point Likert scale. Higher scores indicate higher impairment.

Mobile assessment

Each mobile assessment consisted of 11 items. These included the eight items of the abbreviated PCL-5 (Price, Szafranski, Van Stolk-Cooke, & Gros, 2016) and an additional item from the PCL-5 assessing sleep (PCL-5 item 20). The tenth item assessed pain on a scale of 0 to 10. The final item was a free response item based on the collaborative care model (Zatzick et al., 2004) in which participants were asked to name what concerned them the most that day.

Satisfaction with mobile assessments

Satisfaction with the mobile assessment protocol was assessed with questions on the helpfulness of the assessments and the extent that the assessments were bothersome. Ratings were made on a 1 (not at all) to 7 (extremely) scale. Additional items asked if one assessment per day was appropriate and if the length of the assessment was appropriate, too short or too long.

Procedure

Recruitment

Potential participants were identified via the census for the trauma service at a level 1 trauma centre (Figure 1). Research assistants reviewed electronic charts and identified those who met the inclusion/exclusion criteria. Those patients were approached at bedside where they were further assessed. Eligible participants provided written consent and then completed an assessment battery. Participants downloaded a mobile application, Metricwire (Waterloo, ON), from their device’s app store. Data are presented from participants who completed ≥ 1 mobile assessment. There were no differences on baseline PTSD, depression and disability symptoms between those who did not complete mobile assessments and those who did.

Figure 1.

Diagram of participant flow through the study.

Mobile assessments

Participants received a daily notification to complete an assessment on their mobile device for 30 days. Notifications arrived randomly between 7:00–8:00 PM. Participants had 14 hours to complete an assessment. Responses were uploaded immediately. Participants were allowed to complete surveys for a total 90 days but were notified they could stop after 30. Participants were compensated US$1 per assessment for 30 days.

Follow-up assessments

Follow-up interviews were conducted via telephone one and three months after the time of recruitment. The one-month interview included a satisfaction survey about using the mobile application.

Results

The response rate for the first 30 days was M= 18.33 (SD9.12), or 61.1%. A portion of participants (n = 20, 22.2%) completed more than 30 assessments (M = 45.10, SD= 17.18). The average number of all assessments completed was M = 22.33, SD= 16.36. Missing data was minimal. Among the 21,000 Likert scale items administered, < 1% were skipped. For the 2100 free text items administered, 5.7% were skipped. Participants accessed assessments M = 65.2, SD = 99.74 minutes from the time of the notification. Assessments took M = 2.52, SD = 7.37 minutes to complete.

The relation between response rates and psychopathology at baseline and follow-up were examined (Table 2). Response rate were uncorrelated with PTSD, depression and disability symptoms at all time points, except for a significant positive correlation with disability at one-month follow-up. The correlation between response rates and one-month disability was small, r = .23, and should be interpreted with caution given the other null correlations.

Table 2.

Descriptive statistics and correlation of each measurement with response rates.

	M	SD	Correlation with Response Rate
PCL Baseline	21.29	15.44	−0.04
CL 1 Month	17.06	16.17	0.09
PCL 3 Month	15.62	16.85	−0.05
PHQ Baseline	10.73	6.54	0.08
PHQ 1 Month	7.34	6.2	0.1
PHQ 3 Month	6.42	6.53	−0.01
SDS Baseline	18.81	9.18	0.1
SDS 1 Month	15.45	9.64	0.23*
SDS 3 Month	10.43	9.44	0.03
Response Rate	23.33	16.36	−0.04
Injury Severity Score	13.24	9.75	0.46
Length of hospitalization (in days)	7.84	7.37	0.22
Education	-	-	< .001
Gender	-	-	2.591

Note: PCL = PTSD Symptom Checklist. PHQ = Patient Health Questionnaire. SDS = Sheehan Disability Scale. * = p < .05. ** = p < .01. 1 = Mean difference, significance tested with independent sample t-test.

Satisfaction data obtained at one-month follow-up suggested that the application was viewed as moderately helpful in dealing with their post-trauma experiences (M = 4.08, SD = 1.99) and minimally bothersome (M = 1.75, SD = 1.45). A majority felt that one assessment per day was the right amount (73.6%) and that the 30 days of assessments were appropriate (82.8%).

Discussion

These results support mobile devices as a feasible method to collect data during the acute posttrauma period. Assessments were perceived as moderately helpful and minimally burdensome. Furthermore, these results demonstrate it is possible to obtain both free response and Likert scale responses. Short answer items may also provide meaningful qualitative data about recovery. These results were obtained over a longer assessment period (30 days) than prior studies. Finally, the sample in the present study was the largest to date, which increases confidence in the generalizability of these findings.

The response rate provides a useful metric for future research. A subset of participants may not complete daily assessments due to technical difficulties or a lack of interest. Studies that used texting, which is less likely to have technical difficulties relative to an application, reported similar findings (Pacella et al., in press). Those who did not complete any mobile assessments did not differ from those who did, however, which suggests there may be few differences in these groups at baseline. Relatedly, researchers should expect a response rate of 68.6% (range: 45–88%) from those that do respond. Most of the assessments were completed within 2.5 hours of receiving a notification, but a significant portion took advantage of the full 14-hour window. This variability in responding is attributed to the ongoing challenges individuals face during the acute posttrauma period.

The amount of missingness due to a response rate of 61.1% requires the use of aggregation methods to use these data for clinical care (Schwartz & Stone, 1998). There is currently little guidance as to what level of aggregation ensures important information is retained when collapsing across time. In determining such methods, researchers should consider the temporal specificity needed to address a specific hypothesis and ensure that adequate data will be available. For example, data from the current study would be able to answer questions about changes in PTSD symptoms over several days but may be unable to answer questions about daily changes in PTSD symptoms.

These results also highlight the benefit of passive data collection. Passive data collection gathers information via sensors embedded in (e.g. microphone, pedometer, GPS) or connected to (e.g. fitness trackers) a mobile device with minimal or no input from the participant. Using these data streams would mitigate the problem of missing data and potentially reduce the number of self-report assessments that are needed from participants. Indeed, using these data in conjunction with self-report data could offer better insight into post-trauma recovery. A recent study demonstrated how GPS location data recorded by a smartphone was correlated with depression symptoms in a community sample (Saeb et al., 2015). Similar strategies could be useful when applied to acutely trauma exposed samples. Future work is needed on how best to implement these data collection methods.

The present study had several limitations. First, half the sample experiencing a motor vehicle crash. These findings may not generalize to other acutely trauma-exposed samples. Second, in-app notifications were used to notify participants about an assessment. Notifications via other methods, such as text messages that have a 98% open rate (Frost & Sullivan, 2010), may increase response rates. Third, the sample was demographically homogenous, and these results may not generalize to other demographic groups. Finally, mobile assessments began while participants were hospitalized. Participants may be more compliant while in the hospital due to a desire to adhere to medical care or less likely due to the severity of their injuries.

Despite these limitations, the results provide evidence for the use of mobile devices to assess mental health recovery during the acute posttrauma period, a period that is important for understanding how PTSD develops. These data, and that of recent work, provide guidance on how to efficiently use mobile devices to examine the development of PTSD shortly after a trauma.