Text messaging versus email for emergency medicine residents' knowledge retention: a pilot comparison in the United States.

We evaluated the effectiveness of text messaging versus email, as a delivery method to enhance knowledge retention of emergency medicine (EM) content in EM residents. We performed a multi-centered, prospective, randomized study consisting of postgraduate year (PGY) 1 to PGY 3 & 4 residents in three United States EM residency programs in 2014. Fifty eight residents were randomized into one delivery group: text message or email. Participants completed a 40 question pre- and post-intervention exam. Primary outcomes were the means of pre- and post-intervention exam score differences. Data were analyzed using descriptive statistics, paired t-test, and multiple linear regressions. No significant difference was found between the primary outcomes of the two groups (P=0.51). PGY 2 status had a significant negative effect (P=0.01) on predicted exam score difference. Neither delivery method enhanced resident knowledge retention. Further research on implementation of mobile technology in residency education is required.

RCT Entities:   Population Interventions Outcomes

A majority of residents are part of a techno-literate generation known as Millennials [1]. Traditional classroom lectures and readings are not preferable learning methods for this generation. Millennials are accustomed to using technology and mobile phones in their everyday lives, including their education. A Pew Research Center study found that 95% of 18 to 29 year-old American cell phone owners reported using text messaging services daily [2]. A subsequent study found that 77% of the college students reported sending and receiving emails on a typical day [3]. These technologies could be an efficient way to enhance residency education.

Integrating text messaging and email into curriculums could improve students’ learning efficiency. One study reported that sending concise exercises to students via mobile devices demonstrated favorable results in class attendance, students’ performance, and motivation [4]. Using text messaging to deliver medical knowledge also provided a positive influence in nursing academic studies [5]. Hassini found that asynchronous communication through email could lead to a richer learning experience with students [6].

With the significant use of text messaging and mobile devices by millennials, our study compared the effectiveness of text messaging versus email to improve resident knowledge retention of emergency medicine (EM) core content. An email delivery method, not a formal classroom lecture, was chosen as a comparison group due to its practicality and common use by the millennial population. We hypothesized that EM residents who received daily text messages would have better knowledge retention than those who received an email. The study was a multi-centered, prospective, randomized study conducted in three United States EM residency programs accredited by the Accreditation Council for Graduate Medical Education over three different, two-month, periods in 2014. All three institutions are university-based residency programs; one three-year program, and two four-year programs.

The study subjects were EM residents from postgraduate year (PGY) 1 to PGY 3 & 4. Inclusion criteria included residents who possessed a mobile phone and were familiar with text messaging. We excluded residents who opted out, and those who were co-investigators of the study. Participants created Study Identifier Numbers (SIN) that included their training year and gender. SINs were linked to their email addresses, phone numbers, and pre- and post-intervention exam scores. Residents were randomized within PGY into two groups; text or email. PGY 3 & 4 were grouped together due to their similar roles and responsibilities in the emergency department.

Researchers created EM-related educational material in text messaging format. The material was derived from the American Academy of Emergency Medicine written board review textbook, “Emergency Medicine: A Focused Review of the Core Curriculum,” which is used to prepare residents for the American Board of Emergency Medicine (ABEM) written qualification exam, a required exam for board certification. A group text messaging system, Celly available from: https://cel.ly/, was used to deliver scheduled text messages twice a day. One hundred messages were sent in a two-month period to the text group. The email group received a one-time email in the middle of the two-month period, which contained the exact material that was sent to the text group (Supplement 1).

All participants completed a pre- and post-intervention exam, a 40 question exam derived from the “Physician’s Evaluation and Educational Review VII and VIII.” The pre-intervention exams assessed residents’ baseline knowledge, while post-intervention exams assessed the amount of retained knowledge after exposure to the interventions. Residents still had access to multiple resources to enhance their education, including material that was used to create the education material and exams. We only tested the effectiveness of the delivery method, text or email, to increase knowledge retention of EM content (Supplements 2, 3).

Our main outcome was the difference between pre- and post-intervention exam scores (exam score difference). We used descriptive statistics to analyze the demographic data, and paired t-test to compare the mean of exam score differences between delivery groups. We collected subject characteristics, gender and PGY, to perform an exploratory secondary analysis using multiple linear regressions to evaluate the effect of subject characteristics and intervention group on knowledge retention (predicted exam score difference). The data were analyzed using Stata ver. 10.1 (Stata Co., College Station, TX, USA).

The study was approved by the institutional review board of the University of California, Irvine, the University of Medicine and Dentistry of New Jersey, and Kern Medical Center (IRB number: HS#: 2012-8946).

A total of 58 EM residents, 41 male and 17 female, participated in the study. We enrolled 16 of PGY 1, 16 of PGY 2, and 26 of PGY 3 & 4 (Table 1). Although we observed a better knowledge retention in the text group, we did not find any significant difference in knowledge retention between text and email groups (P=0.51). We found no difference in the means of exam score differences between the two groups when we compared among gender or PGY (Table 2). A multiple linear regression showed that gender had no significant effect on the predicted exam score difference as seen in Table 3 (P>0.05). PGY 1, PGY 3 & 4 also had no statistically significant effects on predicted exam score difference (Table 4). However, we found that PGY 2 status had a statistically significant negative effect on the predicted exam score difference (P=0.01). Data file of intervention exam are available from Supplement 4.

Table 1.

Emergency medicine resident demographics

Characteristic	Intervention group		Total no. (%)
	Text no.(%)	Email no. (%)
Overall	28 (48.3)	30 (51.7)	58 (100.0)
Gender
Male	17 (41.5)	24 (58.5)	41 (100.0)
Female	11 (64.7)	6 (35.3)	17 (100.0)
Postgraduate year
1	7 (43.7)	9 (56.3)	16 (100.0)
2	7 (43.7)	9 (56.3)	16 (100.0)
3 & 4	14 (53.8)	12 (46.2)	26 (100.0)

Table 2.

Comparing means of pre- and post-intervention exam score difference between intervention groups (using paired t-test analysis)

Characteristic	Intervention group		P-value
	Text mean (95% CI)	Email mean (95% CI)
Overall	-1.82 (-6.41 to 2.76)	-3.77 (-7.72 to 0.19)	0.51
Gender
Male	-3.44 (-9.84 to 2.95)	-3.15 (-7.33 to 1.04)	0.93
Female	0.68 (-6.64 to 8.01)	-6.25 (-20.69 to 8.19)	0.27
Postgraduate year
1	-0.29 (-17.34 to 16.77)	-1.94 (-6.75 to 2.86)	0.80
2	-3.50 (-13.41 to 6.31)	-10.33 (-15.16 to -5.51)	0.13
3 & 4	-1.75 (-6.88 to 3.38)	-0.21 (-8.88 to 8.47)	0.73

CI, confidence interval.

Table 3.

Multiple linear regression of exam score difference by gender and intervention group

Variable	Coefficient	Standard error	95% Confidence interval	P-value
Female	1.1	3.33	-5.58 to 7.77	0.74
Text	1.73	3.03	-4.35 to 7.81	0.57

Predicted exam score difference=-3.99+(1.10×female)+(1.73×text).

Table 4.

Multiple linear regression of exam score difference by PGY and intervention group

Variable	Coefficient	Standard error	95% Confidence interval	P-value
PGY 1	-1.94	3.04	-8.04 to 4.17	0.53
PGY 2	-8.06	3.04	-14.16 to -1.96	0.01
PGY 3 & 4	-1.92	2.68	-7.29 to 3.45	0.48
Text	1.64	2.92	-4.21 to 7.49	0.56

Predicted exam score difference=(-1.94×PGY 1)-(8.06×PGY 2)-(1.92×PGY 3 & 4)+(1.64×text).

PGY, postgraduate year.

These findings suggest that using text messaging to deliver educational content may benefit resident education. However, educators need to be aware of factors that affect text messaging implementation. Because busy schedules could prevent messages from being viewed by residents, finding optimal timing of message delivery is essential [7]. Studies showed a greater benefit when text messages were sent at optimal intervals with predetermined times and repetition, since repetition of pertinent material continually exposes residents to the knowledge that can become better retained [8,9]. In our study, the text messaging system had a maximum character restriction. We used symbols and abbreviations in text messages to minimize the character counts, while managing to deliver meaningful concepts in their entirety. For example, arrows were used to represent ‘increase’ or ‘decrease.’ This character restriction may lead to de-contextualization and misinterpretation of the material [10]. Therefore, text messaging might be of better use in delivering concise and simple educational content.

In our exploratory secondary analysis, we found that PGY 2 status predicted lower knowledge retention. This finding may be confounded by the nature of EM residency training curriculums. PGY 1 residents are able to navigate their clinical and educational experience at their own pace as they are introduced into their residency programs. PGY 2 residents are expected to see a higher volume of patients, and many critical care rotations are often included in this year. Focus on clinical workload could distract these residents from their learnings. PGY 3 & 4 residents are also responsible for a high volume of patients, but their experiences may enable them to efficiently manage their time. Upcoming ABEM qualifying examination also encourages senior residents to spend more time on test preparation. Program directors should closely monitor the learning behavior of PGY 2 and prepare to implement additional education strategies to maintain their appropriate progress.

Overall, the means of exam score differences were negative in value, meaning that residents had higher pre-intervention exam scores than post-intervention exam scores. We administered a set of pre-intervention exams shortly after an EM ABEM in-training exam, an exam used to assess an individual resident’s progress towards obtaining ABEM certification. Negative exam score differences may be accounted for by residents’ efforts in preparing for the in-training exam. Future studies should look into administering exams at a time where underlying interests would not influence the outcome of scores.

We noticed that exam score differences in the text group were less negative than those of the email group, alluding to more knowledge retention in the text group. However, this finding was not statistically significant even though we obtained an adequate sample size. We calculated that 34 subjects, 17 subjects per intervention group, were needed to achieve a power study of 80% to detect a 15% difference in exam scores between the two interventions. A larger sample size may yield significant and more generalizable results. Additionally, we did not calculate a sample size for our exploratory secondary analysis, nor stratified the randomization according to gender.

It is possible that residents shared their text messages or email with other residents, however, this would not affect the outcome of the study since the educational content of email and text messages were exact. We also did not track whether the text messages or emails were ever viewed by the residents. A crossover design could yield more significant findings by testing the capacity to achieve better knowledge retention with one intervention over the other, in the same participant. Alternatively, future studies could seek to use both interventions as complementary tools of education delivery to enhance knowledge retention.

In conclusion, use of text messaging to deliver EM education material did not enhance resident knowledge retention when compared to email. PGY 2 status predicted lower knowledge retention. Further studies about text messaging effectiveness and its implementation in residency education are still required.