ICAT: Development of an Internet-Based Data Collection Method for Ecological Momentary Assessment Using Personal Cell Phones.

Rapid advances in mobile data-transfer technologies offer new possibilities in the use of cell phones to conduct assessments of a person's natural environment in real time. This paper describes features of a new Internet-based, cell phone-optimized assessment technique (ICAT), which consists of a retrospective baseline assessment combined with text messages sent to the participants' personal cell phones providing a hyperlink to an Internet-stored cell phone-optimized questionnaire. Two participation conditions were used to test variations in response burden. Retention rates, completion rates, and response times in different subgroups were tested by means of χ² tests, Cox regression, and logistic regression. Among the 237 initial participants, we observed a retention rate of 90.3% from the baseline assessment to the cell-phone part, and 80.4% repeated participation in the 30 daily assessments. Each day, 40-70% of the questionnaires were returned, a fourth in less than 3 minutes. Qualitative interviews underscored the ease of use of ICAT. This technique appears to be an innovative, convenient, and cost-effective way of collecting data on situational characteristics while minimizing recall bias. Because of its flexibility, ICAT can be applied in various disciplines, whether as part of small pilot studies or large-scale, crosscultural, and multisite research projects.

Introduction

By using repeated assessments in a person’s natural environment in real time, ecological momentary assessment (EMA, also-called experience sampling or ambulatory assessment) addresses shortcomings in cross-sectional research designs such as recall bias. For example, because of the unique data it can help to collect, EMA has become increasingly popular over the last 20 years and has contributed to considerable advances in the study of phenomenology, etiology, psychological models, biological mechanisms, and treatment (Ebner-Priemer & Trull, 2009). The development of personal digital assistants (PDA, also called handheld computers or palmtops) in particular has enabled the use of more complex and sophisticated EMA protocols (Shiffman, Stone, & Hufford, 2008).

Despite various advantages of PDAs such as the possibilities to prompt (e.g., by “beeping”), to stamp data records with the date and time of collection, to program alternating questions in response to previous answers (e.g., skips), and the portability of the device (see Shiffman, 2007, for a review), a number of drawbacks has restricted their use for EMA. For example, most people do not own a PDA, which means that researchers have to provide participants with PDAs and train them to use them (Bolger, Davis, & Rafaeli, 2003). Also, PDAs tend to be labor intensive in terms of malfunction troubleshooting, battery replacement, and data uploading (Armeli, Todd, & Mohr, 2005), and they may be mislaid or may break down, leading to loss of the stored data (Bolger et al., 2003). In addition, once the study protocol has been installed in the PDA, changes to settings (e.g., altering the frequency of prompts or the questionnaire structure and content) cannot be easily made during the data-collection process.

To overcome these disadvantages, researchers recently started using participants’ personal cell phones to collect data either by means of phone calls (Courvoisier, Eid, Lischetzke, & Schreiber, 2010) or text messages (Kuntsche & Robert, 2009). In contrast to PDAs, most people own and generally carry their cell phone with them at all times, making it the ideal device for collecting EMA data (Courvoisier et al., 2010).

This paper describes the features of a new Internet-based, cell phone-optimized assessment technique (ICAT) that combines the advantages of using the Internet with those of personal cell phones. This takes advantage of the rapid advances in mobile phone technologies made in the last decade. In terms of infrastructure, the geographical coverage of cell-phone networks has improved dramatically and now enables reliable high-speed mobile data transfer from almost anywhere in developed countries at any time (Dunnewijk & Hultén, 2007). In addition, the growing popularity and convenience of smartphones means that Internet navigation on cell phones is easy and convenient, and that an increasing number of people access the Internet via their cell phone – or are at least familiar with this feature (Brambilla, 2010; Dunnewijk & Hultén, 2007; International Telecommunication Union (ITU), 2010a, 2010b; Lim, Chou, & Melewar, 2008). The so-called Internet-based, cell phone-optimized assessment technique (ICAT) we are describing in this paper consists of a retrospective baseline Internet questionnaire assessment and SMS sent to the participants’ personal cell phones providing a hyperlink to Internet-stored, cell phone-optimized questionnaires. In particular, this study investigates the following:

Retention rates (participation over time minus attrition, dropout) from the baseline Internet assessment to the first cell-phone assessment, and from there until the end of the study. In a previous study, 83.3% of the registered young adults participated with their personal cell phones in an SMS-based survey, 89.1% of whom continued participation until the end (Kuntsche & Robert, 2009). In a recent review, Morren, van Dulmen, Ouwerkerk, and Bensing (2009) reported an average retention rate (i.e., 1-attrition) of 81.0% across 30 different EMA studies among pain patients.

Completion rates (compliance), i.e., the number of entirely completed EMA questionnaires compared to the number of incomplete, blank, and unreturned questionnaires. In previous research, calling young adults on their cell phone six times a day over a 1-week period resulted in 74.9% of calls being answered with a slightly higher proportion for women (76.0%) than for men (71.5%) (Courvoisier et al., 2010). In the review mentioned above, the average compliance (defined as percentage of diary entries or completed items) was 83.4% (Morren, van Dulmen, Ouwerkerk, & Bensing, 2009).

Response times from the dispatch of the hyperlink invitation sent by SMS to the reception of the cell-phone questionnaires.

In addition, retention rates, completion rates, and response times were compared between men and women and between two different participation conditions that varied in terms of response burden.

Methods

Participant Recruitment and Registration Procedure

Two different strategies were used to recruit the 250 participants intended for this study between September and November 2009. First, an e-mail was sent to all of the approximately 11,600 students at Lausanne University inviting them to participate. In less than 24 h, registration was closed because the required number of 200 participants had been reached. Second, announcements were published in a local newspaper and on several Internet pages (facebook.com, etudiants.ch, anibis.ch), and leaflets were distributed at three higher education institutions in Lausanne and Geneva. These strategies resulted in 37 additional participants. Thus, in total 237 participants (mean age 23.4, SD = 4.2, 41.8% male) registered.

The recruitment documents provided (1) information on the aim of the study (i.e., to gather information about the personal, social and environmental determinants of evening alcohol consumption over the course of the week), (2) assurance that all answers would be treated confidentially, (3) the possibility of leaving any question unanswered, and (4) a link to the study’s homepage. As an incentive, anyone who participated for at least 20 days (two thirds of the study duration) would receive a randomly-drawn prize, e.g., a cinema ticket, a bookstore voucher, or a restaurant voucher, with a monetary value ranging from USD 25 to 40.

The study’s homepage again provided detailed information on the aims, confidentiality, the voluntary nature of disclosing personal information, and the incentives. In addition, a hyperlink allowed participants to test whether Internet access was enabled on their cell phone. Before registering, participants had to give their formal consent by ticking a checkbox indicating that they had understood and agreed with the participation conditions. Having entered their cell-phone number, participants received a unique validation code via SMS confirming their registration and granting them access to the baseline Internet questionnaire. The procedure was approved by the ethical committee of Lausanne University (Canton de Vaud Protocol No. 223/08).

Implementation of ICAT

The Internet server-based software used to conduct ICAT was developed by Valentin Vago1. Data were gathered in three ways (Figure 1 ): (1) A baseline questionnaire had to be completed online directly after registration; (2) over 30 subsequent evenings, invitations were sent by SMS asking participants to answer cell phone-optimized questionnaires via the cell-phone browsers (EMA); (3) qualitative telephone interviews were conducted approximately 2 weeks after the EMA.

Figure 1

Flowchart of the data collection procedures.

More information about the software, the technical features, and the implementation requirements can be directly obtained upon request from the developer at info@irata.ch.

On the first or second Friday evening after registration and completion of the baseline questionnaire, the 30-day EMA via cell phone started. SMS containing a hyperlink were sent randomly on the hour between 7 p.m. and 11 p.m. in order to assess the possibility of prompting participants randomly at given time points.

Clicking on this hyperlink automatically opened the cell-phone browser and downloaded a new copy of the Internet-stored questionnaire, which was specially devised for completion via cell phone: (1) only a small number of short questions were included, (2) drop-down menus for response categories were used to facilitate answering with the cell-phone keyboard, and (3) only text (i.e., no images or animation) were used so that the size of each questionnaire was less than 10 kilobytes. Based on the Browser Capabilities Project (http://browsers.garykeith.com), the system was able to identify the different cell-phone browser versions and automatically displayed the questionnaires in the most user-friendly way by taking into account the browser specifications. To minimize recall bias, questionnaires could be accessed only once and only within the 12-h period following the reception of the SMS.

Participation Conditions

To investigate variations in the response burden (Bolger et al., 2003), participants were randomly assigned to two different participation conditions (Figure 1). In the 1-SMS condition, participants received one SMS per evening randomly on the hour between 7 p.m. and 11 p.m. In total, they were asked to complete 30 cell phone-optimized questionnaires. In the 2-SMS condition, participants randomly received one to three SMS per evening (with an average of two, i.e., about 60 questionnaires to be completed in 30 days; see Figure 1).

Measures

The Internet baseline questionnaire contained questions on sex, age, education level, as well as different alcohol use measures (i.e., frequency, usual quantity, and risky drinking), drinking motives and personality dimensions. Completion of the baseline questionnaire took around 15 min.

In the cell-phone questionnaires, participants were asked to indicate, in 15-min response categories, their location (i.e., at home, at work, traveling by train, bus, car or on foot, in a pub, restaurant or nightclub, in a cultural or sporting venue, in a public park or outdoors) and the persons present (i.e., alone, with large or small group of friends, family members, colleagues) during the 60 minutes prior to the dispatch of each SMS. Using the same timeframe, participants were also asked to indicate the number of consumed alcoholic drinks separately for four beverage types (i.e., beer, wine, spirits, premix/alcopops) and their current mood assessed in seven broad categories (i.e., enthusiastic/excited, serene/confident/inspired, alert/active/strong, scared/afraid, distressed/upset, nervous/irritable/jittery, tired). Pretests revealed that completion of each cell-phone questionnaire took less than 1 minute. The SMS-sending schedule and the response time were recorded automatically for each questionnaire.

Qualitative Feedback

After the 30-day EMA, participants were invited to give personal feedback either online or via a telephone interview. The qualitative responses were scrutinized and summarized to gain a better insight into the user-friendliness of ICAT, possible resistance to disclosing personal information, the intrusiveness of receiving SMS invitations at random points in the evening, as well as the effects of the response burden in general and other potential problems.

Analytic Strategy

To test differences in proportions between groups (e.g., completion rates between the 1-SMS and the 2-SMS condition and between men and women), we used χ² tests. To test differences between the participation conditions in retention rates over the 30-day period, we used Cox regression; hazard ratios (HR) and 95% confidence intervals (CI95%) are also shown. To test differences between the conditions in the proportion of returned questionnaires per day, we used logistic regression, and odds ratios (OR) and CI95% are given.

Results

Retention and Completion Rates

In total, more than 90% (214 out of 237 participants) of those who registered and filled out the baseline questionnaire participated in the EMA via cell phone (Table 1 ). More than 80% (175 out of 237) of the EMA participants repeatedly took part in the study (i.e., they completed at least five short cell phone-optimized questionnaires). There were no significant differences in the retention rates of the 1-SMS conditions and the 2-SMS condition (randomly between one and three SMS invitations per evening) or across gender.

Table 1

Number of participants in the baseline assessment, in the cell phone-based EMA, and those who repeatedly took part in the EMA

		N at baseline	N in the EMA in total		N repeated EMA participation1
Notes. 1At least five questionnaires returned in the cell-phone-based EMA; none of the χ² values were significant at the 5%-error level. a) χ² calculated on N at baseline compared to N in the EMA in total. b) χ² calculated on N in the EMA in total compared to N repeated EMA participation. c) χ² calculated on N at baseline compared to N repeated EMA participation.
Total		237	214	(90.3%)	175	(81.8%)
Condition
	1-SMS	118	104	(88.1%)	87	(83.7%)
	2-SMS	119	110	(92.4%)	88	(80.0%)
			a) χ²(df = 1) = 1.25		b) χ²(df = 1) = 0.48
					c) χ²(df = 1) = 0.00
Sex
	Male	99	93	(93.9%)	78	(83.9%)
	Female	138	121	(87.7%)	97	(80.2%)
			a) χ²(df = 1) = 2.58		b) χ²(df = 1) = 0.489
					c) χ²(df = 1) = 2.16

Table 2 summarizes group differences in the completion rates in the cell phone-based EMA of those with repeated participation. Overall, 52.2% of all questionnaires were returned completed entirely and 6.9% of all questionnaires were returned incomplete (i.e., at least one group of items left unanswered). Of all missing items, questions about the location were most often left unanswered (33.7%), followed by questions about the persons present (30.5%). Participants in the 2-SMS condition had a lower proportion of unreturned questionnaires (i.e., not responding to an SMS invitation) and a slightly higher proportion of incomplete questionnaires than participants in the 1-SMS condition. Consequently, the proportion of entirely completed questionnaires was about two thirds in the 1-SMS condition but only less than half in the 2-SMS condition. The proportion of entirely completed questionnaires was higher on weekdays than at weekends. Over time, the proportion of unreturned questionnaires increased, yet the number of incomplete questionnaires decreased slightly.

Table 2

Proportion of submitted, returned and unreturned questionnaires among participants who replied to at least 5 questionnaires

		SMS sent	Returned questionnaires		Unreturned or blank questionnaires	Test values
			Completea	Incompleteb
Notes. ***p < .001. a) All groups of items (location, friends, drinks) completed. b) At least one group of items unanswered (e.g., no location indicated). c) Sunday to Thursday. d) Friday and Saturday. e) Days #1 and #2 (Friday and Saturday). f) Days #3 to #9. g) Days #10 to #16. h) Days #17 to #23. i) Days #24 to #30.
Total		7,930	52.2%	6.9%	40.9%
Conditions
	1-SMS	2,610	64.5%	5.8%	29.7%	χ²(df=2) = 238.3 ***
	2-SMS	5,320	46.2%	7.4%	46.4%
Sex
	Male	3,463	55.9%	6.4%	37.7%	χ²(df=2) = 33.9 ***
	Female	4,467	49.3%	7.3%	43.4%
Day
	Weekdayc	5,293	54.8%	6.9%	38.2%	χ²(df=2) = 27.9 ***
	Weekendd	2,637	49.2%	6.9%	43.9%
Weeks
	1st weekende	509	55.6%	10.0%	34.4%	χ²(df=8) = 204.5 ***
	1st weekf	1,841	61.9%	8.1%	30.0%
	2nd weekg	1,863	53.4%	7.6%	39.1%
	3rd weekh	1,857	49.3%	5.6%	45.1%
	4th weeki	1,860	43.4%	5.4%	51.2%

Concerning retention and completion rates across the entire 30-day period of the EMA, the two upper lines in Figure 2 represent the proportion of participants who were still participating in the EMA (i.e., returned at least one questionnaire at a later point). In the 1-SMS condition, the proportion of people still participating after 20 days remained at around 70% (81 out of 104 participants), whereas this figure was about 60% (73 out of 110) in the 2-SMS condition. However, although they had to complete twice as many questionnaires, the overall retention rate in the 2-SMS condition was not significantly lower than in the 1-SMS condition, as results from the Cox regression demonstrate (HR = 0.780, CI95% = 0.561–1.086).

Figure 2

Participation rate at baseline (B*) and in the course of the EMA according to participation condition (Day #1 to #30).

Large day-by-day variations were found in the proportion of returned cell phone-optimized questionnaires: About 40% to 70% of the questionnaires were returned on a daily basis (Figure 2), whereas as dropout increased over time, the overall percentage of returned questionnaires decreased as well (OR = 0.965, CI95% = 0.960–0.971). Thus, the difference between the “participating” line and the “questionnaire returned” line provides the percentage of nonreturned questionnaires on a given day among those still participating. Results of a logistic regression also revealed that overall percentage of returned questionnaires was lower in the 2-SMS condition than in the 1-SMS condition (OR = 0.846, CI95% = 0.770–0.929).

Response Time

Overall, a fourth of the returned questionnaires were sent back less than 3 min. after the SMS was sent, while half of the questionnaires were returned within less than 25 min. (Table 3 ). Men consistently returned their questionnaires more quickly than women. By contrast, the number of SMS received in the evening (1-SMS condition vs. 2-SMS condition) or on weekdays or weekend days did not have a considerable and consistent effect on the response time. As the study progressed, the response time tended to rise, albeit only slightly, i.e., participants took more time to reply to SMS in the third and fourth week than in the first 2 weeks.

Table 3

Response time among participants who replied to at least 5 questionnaires

		N returned quest.	25th percentile	Median	75th percentile	Test values
Notes. The displayed time format is hours:minutes:seconds; *p < .05; ***p < .001. χ² test = median test for K populations. χ² calculated on the number of answers above the total median time vs. number of answers above the total median time in each population. a) Sunday evening to Thursday evening. b) Friday and Saturday evening. c) Days #1 and #2 (Friday and Saturday). d) Days #3 to #9. e) Days #10 to #16. f) Days #17 to #23. g) Days #24 to #30.
Total		4,686	0:02:58	0:24:36	1:41:41
Conditions
	1-SMS	1,834	0:02:49	0:22:53	1:50:46	χ²(df=1) = 0.7
	2-SMS	2,852	0:03:09	0:25:20	1:34:43
Sex
	Male	2,156	0:02:04	0:13:48	1:11:05	χ²(df=1) = 111.7***
	Female	2,530	0:04:46	0:38:06	2:02:38
Day
	Weekdaya	2,607	0:02:43	0:24:07	1:34:23	χ²(df=1) = 0.2
	Weekendb	2,079	0:03:29	0:25:04	1:51:37
Weeks
	1st weekendc	334	0:03:49	0:20:25	1:12:16	χ²(df=4) = 20.6***
	1st weekd	1,289	0:02:46	0:18:08	1:27:15
	2nd weeke	1,135	0:02:55	0:24:57	1:45:44
	3rd weekf	1,020	0:03:03	0:31:53	1:56:01
	4th weekg	908	0:02:52	0:27:41	1:47:38

Qualitative Feedback of Participants

Most of the feedback received, either as one of the 23 online anonymous statements or in the four telephone interviews, underscored the ease of use of the system, which resulted in a low response burden: “It was a nice study, easy to do, and wasn’t time-consuming”2 (P14). “I found this system of answering questionnaires pretty convenient. [...] I didn’t find this study a major imposition. There was hardly a day when I had trouble answering. [...] I would be happy to participate in similar studies in the future” (P20). Since the question and answer categories were always identical, participants quickly became familiar with the questionnaire format: “The predefined answer categories are good because they allow you to answer quickly” (P22). “I liked the way the questionnaires were organized. [...] Taking part in the study wasn’t that difficult, so it was easy to answer the questionnaire every day” (P27).

All original statements by the participants reported here have been translated by the authors from French into English.

Although participants had to use their personal cell phone and disclose their phone number, they did not perceive the repeated reception of SMS at random times as being overly intrusive (P8, P10). Some participants even mentioned the lack of SMS after 11 p.m., which prevented them from recording their alcohol consumption later in the evening: “I found that the SMS were usually sent too early. I could seldom answer the beverage questions” (P16). “I would have liked to answer over the course of the entire evening” (P22).

Discussion

This study investigated whether EMA could be efficiently conducted by means of the ICAT using Internet-stored questionnaires accessed via participants’ personal cell phones. In particular, we investigated retention rates, completion rates, and response times in general and in subgroups, e.g., between those in the 1-SMS condition and those in the 2-SMS condition and between males and females.

Despite being asked to disclose personal information such as their cell-phone number, current moods, whereabouts and alcohol use, and to complete questionnaires at different times every evening (random prompting), we observed a high retention rate from the baseline assessment to the EMA (> 90%) and a high proportion of those having participated five or more times in the EMA part (> 80%). These retention rates are similar to those reported in previous EMA studies that used either PDAs or text messages for data collection (Kuntsche & Robert, 2009; Morren et al., 2009).

With about half of the questionnaires being returned, the completion rates were somewhat lower than in previous studies that used either PDAs or cell-phone calls (Courvoisier et al., 2010; Morren et al., 2009). However, these studies had used various means to instigate completion rates such as special instructions, training prior to the EMA (Kuntsche & Robert, 2009; Morren et al., 2009) and calling participants every 30 minutes over a 2-h period each time they failed to answer the first call (Courvoisier et al., 2010). By contrast, in this study, participants were not trained, and neither the researchers nor the software interfered with reminders or anything else in the “natural” progression of the fieldwork. Also the incentives were less than half the amount previously used in the same population (i.e., young adults in Switzerland; Courvoisier et al., 2010). Moreover, in contrast to, say, pain patients (Morren et al., 2009), the participants did not have a strong personal interest in the study.

If we take all these aspects together, the retention rates obtained here appear more than satisfactory and might in fact be further increased in future application of ICAT by means of reminders or other methods to increase compliance (see Morren et al., 2009). Interestingly and in contrast to a previous study in the same population (Courvoisier et al., 2010), men had higher retention and completion rates and responded more quickly to SMS invitations than women. We can only speculate that men might be more familiar with or more interested in the main topic (alcohol use) or technological aspects (see media literacy below) of the study. Concerning variations in response burden, it was interesting to see that the two conditions (1 questionnaire vs. 1 to 3 questionnaires per evening) did not differ much in terms of participation rates and response time. Together with qualitative feedback, this shows that answering Internet questionnaires via their personal cell phones one or more times per evening seems to be convenient for the participants. Some even wanted the SMS schedule extended to later in the evening. However, one should be cautious in interpreting the results from the qualitative part: Since only one in ten participants responded to the invitation to provide feedback, those who did might represent a positive selection of all participants.

Limitations and Future Directions

The main technical concern of collecting EMA data by means of participants’ personal cell phones remains the connection to the cell-phone network. A device that is switched off or has no reception for whatever reasons means that the participant receives the SMS later than expected. Participants were therefore instructed to complete the questionnaires within the shortest possible time. In addition, the clearly defined reporting timeframe (e.g., “between 8 p.m. and 9 p.m.”) was meant to help the participants remember their moods, cognitions, and behaviors even if they did not, or could not, respond immediately. Considering all these aspects together, the fact that half of the participants completed the EMA questionnaires in less than 25 min. following dispatch of the SMS invitation should be considered a rather short response time.

The application of ICAT also depends on the media literacy of participants, the type of cell phone, associated costs, and the network coverage in the given country. This study included young people in higher education in the French-speaking part of Switzerland who were likely to own an Internet-enabled cell phone, to use the device for various purposes, and to benefit from very good network coverage. The ICAT might therefore be less suitable for subgroups inexperienced in using cell-phone applications. Moreover, cell-phone ownership and the costs of cell-phone Internet connections may vary in different locations and across countries, which is particularly important for use in crossnational or crossregional (city vs. countryside) studies.

However, cell-phone technology, coverage, and usage are rapidly developing worldwide. For example, at the end of 2010, 5.3 billion cell-phone subscriptions were active worldwide (International Telecommunication Union (ITU), 2010b). In 2009, cell-phone subscriptions exceeded landline phone connections and even the number of inhabitants in most European countries, e.g., by a factor of 1.22 in Switzerland, 1.28 in Germany and The Netherlands, and 1.51 or higher in countries such as Lithuania, Russia, and Montenegro (International Telecommunication Union (ITU), 2010a). The forthcoming LTE (Long Term Evolution) system will give cell phones an even higher speed connection to the Internet and to other data services in almost all inhabited locations in developed countries (Dunnewijk & Hultén, 2007; International Telecommunication Union (ITU), 2010a). For example, the number of people in Switzerland who accessed the Internet via their cell phone increased by a factor of 14 from 2005 to 2008 (i.e., from 144,806 to 2,133,901 according to the most recent information; Brambilla, 2010). As a consequence, most of the current cell-phone plans include data transfer as a basic feature.

Conclusion

Combining a baseline Internet questionnaire with an Internet-based cell phone-optimized questionnaire to be answered via participants’ personal cell phones, ICAT is an innovative and convenient way of conducting an EMA and collecting information on situational characteristics with a minimal recall bias and independent of the person’s current location. This makes the method suitable for multiple EMA applications including small pilot studies as well as large-scale, crosscultural, and multisite studies. Due to its flexibility, ICAT can be applied to a large variety of research subjects in a range of disciplines. We believe that ICAT has the potential to become a standard data collection tool for any discipline that deals with self-reports and beyond.