Mobile learning device increased study efficiency for radiology residents but with risk of temporary novelty effect.

BACKGROUND: Digital resources in learning are increasingly available and offer new possibilities in education. Mobile learning devices (MLD) such as tablets provide easy and flexible access for users.
PURPOSE: To investigate whether the introduction of MLDs in radiology education affected time spent on studies over a longer time frame and whether learning behavior and attitudes changed.
MATERIAL AND METHODS: The radiology residents employed during 2015-2016 were invited to participate in this 12-month MLD intervention study. Results were evaluated using online questionnaires at six months (6 m) and 12 months (12 m).
RESULTS: Thirty-one residents were included, of whom half were in the early stages of residency (<2 years). After the MLD introduction, most participants (91% [6 m] and 83% [12 m]) estimated increased time spent on studies. Of these, 32% stated "a lot more" at 6 m but only 8% at 12 m (P = 0.12). The MLDs showed positive effects on the experience of radiology studies, as a majority of participants stated better quality and effectiveness in their studies (100% [6 m]-92% [12 m]), that MLD facilitated access to educational materials to a high degree (83% [6 m]-75% [12 m]), and that studies had become better and more fun (96% [6 m]-100% [12 m]).
CONCLUSION: The use of MLDs seems to facilitate learning effectively for radiologic residents. However, a larger scale study is required as a trend of decreasing figures in the longer term was seen, but our results did not show a significant reduction of time spent on radiology studies. © The Foundation Acta Radiologica 2019.

Introduction

Digital resources in learning are increasingly available and offer new possibilities in education. In mobile learning, in particular, major advantages have been described, e.g. increased access to teaching materials and low-cost solutions, leading to more situated and conceptual learning and facilitating communication between teacher and learner, as well as providing the opportunity for self-assessment while learning (1,2). Additional advantages include the possibilities of integrating moving images and films, as well as polls and interactive tests (3–6).

However, new technological possibilities also come with some disadvantages and challenges, which must be solved (7–9). These include the possibility of distraction from learning because of the multifunctional character of these technologies, as well as the risk of erasing borders between personal, professional, and educational use (1,10–12). Another major problem addressed in several articles is the need for a reliable internet connection, which still is a major problem when implementing mobile learning (1,13), especially if local regulators set barriers.

In 2014, resident physicians at the Department of Radiology in our county had no personal computer/workspace and the availability of educational resources was unsatisfactory. There was thus a proposal to assign a mobile learning device (MLD)/tablet to each resident physician. Work in clinical radiology is highly computerized and often complex, requiring three separate image displays simultaneously for the same patient. Although work takes place mostly on computers, the need to flip between clinical images and other e-resources on the same computer is a potential challenge. Furthermore, computers are often not personal but shared by several radiologists. Thus, a personalized MLD, easily accessible from wherever the residents are, with content and organization that is familiar to the owner, might be useful.

A literature review revealed that a few resident programs in radiology in the United States had introduced tablets with specially designed training resources. The University of Colorado conducted a project called the iPad Toolbox (14) in which every resident physician was given an iPad 3© (Apple, Cupertino, CA, USA) loaded with various e-services, including a basic textbook for radiology and review book for each radiology area, RAD Primer (Amirsys, Salt Lake City, UT, USA), and IMAIOS e-Anatomy (Montpellier, France), as well as other resources such as articles and lectures. The aim of their study was to evaluate if learning time increased. Three months after the introduction of the MLD, self-reported study time had increased.

Similarly, a study performed at Tufts Medical Centre, USA, including 21 radiology resident physicians, showed that self-reported study time increased and perceived experience of studying radiology improved six months after MLD introduction (15,16).

Finally, a study at Beth Israel Deaconess Medical Centre, USA, in which 38 residents were given an MLD with access to IMAIOS e-Anatomy and STATdx (17), showed that a large majority of residents used their tablets on a daily basis after a six-month intervention period.

Previous studies of MLD introduction in the radiology field have been short in duration, i.e. up to six months. A risk in these kinds of interventions is the novelty factor, with fading learning interest in the longer term.

The aim of the present study was to investigate whether introduction of MLD in radiology education affected time spent on studies over a longer time frame, and whether learning behavior and attitudes changed.

Material and Methods

Study design

Inclusion criteria were as follows: all residents employed in the Radiology Department in our County in 2015–2016 were invited to participate in this 12-month mobile learning intervention study. There were no exclusion criteria.

The outcome was evaluated using an online questionnaire designed by our study group (Suppl. Table 1). Questionnaires were completed before the MLD intervention, as well as at six months and 12 months. At the six- and 12-month timepoints, some extra questions were included, asking participants to estimate their usage of the different software solutions and their overall experience of using the MLD. The questionnaire was anonymously completed online; thus it was not possible to follow individual answers across the polls. The online link was emailed to the study participants. A reminder was sent to all participants after about one week.

Hardware

All participants were given a choice of two MLDs: an iPad© Air 2 (Apple, Cupertino, CA, USA) (size: 9.7 in., weight: 437 g) or an iPad© mini 3 (Apple, Cupertino, CA, USA) (size: 7.9 in., weight: 331 g). These options were selected for technical reasons or based on personal preferences of the study group and experience of the product at the hospital. The two different sizes were chosen to satisfy possible personal preferences of the participants. The hospital’s public wireless network was of good quality and was used as an internet source by the study participants.

Software

The decision on which content to include was made by the project manager after review of the literature. The idea was that the content could be developed gradually based on requests from users and new applications and e-books becoming available, and on the results of the final evaluation. Specific applications are summarized in Suppl. Table 2. In brief, the content consisted of three parts: (i) e-books; (ii) other radiological software; and (iii) administration and support.

E-books

We initially aimed to offer several e-books based on the concept described in the iPad Toolbox (14), i.e. basic-level and reference books. This failed for economic reasons, as the cost of e-licenses for all books for all participants would have been much higher than the costs of our present system with a printed copy of every book free to borrow in our local library at the clinic. The choice of e-books was therefore limited to the book Fundamentals of Diagnostic Radiology by Brant and Helms (published by Lippincott Williams & Wilkins). It covered the entire radiological area in one book, split into multiple volumes.

Other radiological software

Trial licenses were purchased for IMAIOS e-Anatomy, STATdx, and RADPrimer. IMAIOS e-Anatomy provides a wide range of anatomical knowledge directed toward radiology, while STATdx offers a large knowledge base with many sample images, accessed through an iPad-customized Internet page. We chose to subscribe to five concurrent users to fulfil the need for reference literature. RADPrimer offers a variety of cases with clear explanations and a radiological textbook linked to the cases. A drawback of this source is the presentation of CT scans, where only a single slice is offered instead of scrollable stacks. Radiology Assistant (radiologyassistant.nl) is a popular site among resident physicians in radiology and is available as a downloadable app for a small charge, which was paid by the hospital. Several additional apps were judged useful. For the whole list of applications, see Suppl. Table 2.

Initially, our goal was to give residents access to teaching files from our own local Picture Archiving System (PACS). Unfortunately, this was technically more complicated than the project’s time frame allowed and has not yet been implemented.

Administration and support software

As an application for taking notes, Notability (Ginger Labs, San Francisco, CA, USA) was chosen, because it is easy to learn and use, allows simultaneous audio recording, and can be automatically backed up to cloud services. To organize and read PDF files, GoodReader (Good.iWare Ltd., San Francisco, CA, USA) was chosen, as it can be synchronized with cloud services, and provides a readable storage structure and good opportunities to make notes directly in the documents. To write and read Word, Excel, and PowerPoint documents, preloaded applications on the iPad were available, but participants could also use original Microsoft solutions free of charge.

Maintenance and updates

It was desirable that updates and installation of new applications and so on could take place smoothly and remotely without the need for physical access to the MLD; thus, we initially planned to use the Apple Mobile Device Management (MDM) software. However, this was not possible owing to local restrictions at the hospital. Instead of pre-installed applications and settings via MDM, it was decided that the users would be responsible for installation. To facilitate this process, we prepared an information document with recommendations regarding applications, accounts needed, and tips for using the MLD. This was also presented in a 1-h workshop for participants, which most of the participants attended. The study set-up did not require participants to download or activate all recommended applications, but they were informed about all available applications and were strongly recommended to download all materials. Upon receiving the MLD, the residents were required to sign a contract stating that the MLD was on loan and that although the main purpose of the device was education and assistance during work, private use also was permitted as long as copyright laws and internet policies of our county were followed.

Data analysis

Data from the online questionnaires of the three timepoints were collected and analyzed in SPSS 24.0. The questionnaires (Suppl. Table 1) included questions on: background data (e.g. years in residency and previous MLD experience); time spent on radiology studies; learning preferences; source preferences; MLDs impact on education; and implementation of MLDs.

Statistical analysis

Statistics were calculated using SPSS 24.0 for Windows software (IBM Statistics, Armonk, NY, USA). For continuous variables, non-parametric Mann–Whitney U test was used and for categorical variables χ2 test was used. Estimated time spent on radiology studies was converted from categorical to continuous to enable combination of usage at work and out of work. Thus the four categorical answers were converted: “0–1h” set as 0.5 h; “1–3h” set as 2 h; “3–6h” set as 4.5 h; and “6h” set as 6 h. A p-value <0.05 was considered significant.

Results

Descriptive

All 25 residents working at the radiology department at the start of the study were invited to participate in the MLD intervention. Of these, two declined to participate after baseline data collection and were not included in the intervention part. The reasons for non-participation were planned parental leave and nearing the end of residency, respectively. In addition, six new residents joined the clinic during the study period and were invited to participate. These participants were observed for only six months, i.e. until the end of the study, and answers were analyzed as six-month data. Three participants went on parental leave during the study period. As shown in Table 1, two-thirds (68%) of the participants were male at baseline evaluation. Half of the participants were at an early stage (<2 years) of radiology education. The majority (77%) of participants stated previous experience of tablets before the beginning of the study. Forty-one percent used their tablets for educational or work purposes. At the start of the project, all participants stated that they believed that the tablet would be helpful during the residency program, but most expressed the reservation that the tablet would have to be filled with relevant educational resources for this to be the case (free-text question in the baseline survey).

Table 1.

Descriptive data.

	Baseline (n = 22)	6 months (n = 23)	12 months (n = 13)
Response rate (%)	88	79	65
Years in residency education (%)
<1	41	35	8
1–2	9	26	39
2–3	23	17	15
3–4	14	13	23
4–5	14	9	15
Gender
Male:female (%)	68:32	70:30	69:31
Tablet used before this project (%)
Often	27
Sometimes	50
Never	23
Tablet used for work/education before this project (%)
Often	18
Sometimes	23
Never	59

Study time

Out of 10 participants, 9 (91%) estimated increased time spent on radiology studies six months after introduction of the MLD, compared with 83% at 12 months (Fig. 1a). None reported less time spent on studies after MLD intervention. Most participants (91% at six months and 85% at 12 months) used the MLD every week (Fig. 1b). Almost two-thirds (61%) of participants used the MLD > 3 days per week at the six-month follow-up; this figure decreased to approximately one-third (31%) at 12 months (P = 0.083). Self-estimated time spent on radiology studies is shown in Fig. 2. Total time spent on radiology studies increased from 4.8 ± 2.5 h (median = 2 h) at baseline to 6.6 ± 2.6 h (median = 4.5 h) at six months (P = 0.027). Though not quite significant (P = 0.093), the numbers of hours studied decreased again at 12 months (5.1 ± 2.1 h [median = 5 h]). No significant difference was seen when looking separately at time spent on studies at work and out of work (Fig. 2).

Fig. 1.

The majority of participants stated increased time spent on studies (a) as well as regular use of their IPAD (b) after the implementation. However, a tendency of decreasing numbers of participants studying “a lot more” and of frequent users, i.e. >3 days/week were seen after 12 months (P = 0.083 and 0.12, respectively). The vast majority of participants indicated better quality and effectiveness of radiology training after iPad implementation (c).

Fig. 2.

Self-estimated time spent on radiology studies at baseline and after 6 and 12 months of iPad intervention. “Total hours” shows summarized time spent on radiology studies every week.

Learning preferences

Regarding learning preferences at baseline, one-third (32%) of participants preferred learning based on patient cases currently seen in their daily work, whereas 46% preferred this at the end of the study period (P = 0.396). On the contrary, learning based on the participant’s current clinical division (i.e. more general study on the current body area) decreased from 59% of participants (baseline) to 39% (12 months), though non-significant (P = 0.234). Least common was learning by reading a book from cover to cover, which was preferred by 5% of residents at baseline and 15% at 12 months (P = 0.268).

Source preferences

At the end of the study, all participants stated that they used IMAIOS e-Anatomy, RADPrimer, and STATdx at least once in a while (Fig. 3). At six months, a majority were already using these resources (IMAIOS e-Anatomy = 96%, RADPrimer = 91%, and STATdx = 86% of participants). The e-book Fundamentals of Diagnostic Radiology had considerably lower figures, i.e. 52% at six months and 46% at 12 months. Of these four resources, IMAIOS e-Anatomy was most frequently used, followed by STATdx and RADPrimer. Of the non-free applications that users could optionally download and be compensated for, Radiology Assistant had been downloaded by 46%, GoodReader by 31%, and Notability by 31% of participants at the end of the study.

Fig. 3.

Percentage of residents using IMAIOS e-Anatomy, STATdx, RADPrimer, and the e-book Fundamentals of Diagnostic Radiology after 6 and 12 months of use of their mobile learning devices.

As shown in Fig. 4, the most valuable app at the end of the study, as judged by the participants, was RADPrimer, followed by STATdx and IMAIOS e-Anatomy. In this question, residents could mark more than one particularly valuable resource. One night on ER was marked as particularly valuable by 8% of participants.

Fig. 4.

Subjectively scored most valuable e-resources after 6 and 12 months of mobile learning.

Impact on education

All participants thought that MLD facilitated access to educational materials, and the majority (83% [six months] and 75% [12 months]) indicated that MLD facilitated it to a “high degree”. Regarding the quality and effectiveness of radiology studies, all participants felt that MLD introduction had had a positive effect after six months (61% stated “much better” and 39% “a little better”; Fig. 1c). A majority (92%) still reported a positive effect on quality and effectiveness after 12 months of MLD use, whereas 8% stated unchanged quality and effectiveness, but none stated worse. The overall experience of studying radiology improved, with all residents indicating after 12 months that their studies were “better and more fun” since the introduction of the MLD; the corresponding figure at six months was 96%.

Implementation

Most participants (74%) were satisfied with the introduction and did not need any more information on how to get started or how to use the tablet. One person (4%) reported problems due to insufficient training and/or information, and the remaining 22% thought the introduction was sufficient but more information could have added value.

Discussion

In the present study, we show that the majority of radiology residents estimated increased time spent on studying radiology after the introduction of a personal MLD (Fig. 1b). However, this was most prominent during the first observation period (six months) and seemed to decline after 12 months, indicating a novelty effect. Residents were overall very positive about the introduction of the MLD: 92% indicated that their education had improved regarding quality and effectiveness; 75% stated that the MLD facilitated access to educational materials to a high degree; and all residents reported that their studies were more efficient and more fun. RADPrimer, STATdx, and IMAIOS e-Anatomy were the most frequently used and appreciated apps. However, as the main aim of the study was to evaluate the introduction of an MLD rather than different radiology apps, the content was limited to a few preselected sources. This decision was also influenced by the costs of additional content.

Time spent on education can be a challenge in the clinical setting. In our study, most residents stated an increase in time spent on education after introduction of the MLD, especially at the first observation timepoint (six months), where 32% stated “I study a lot more now” and 59% “I study a little more now.” Although numbers were slightly lower at 12 months, a majority indicated increased time spent on studies at this timepoint as well. However, comparing the estimated number of hours studied per week, the differences were not particularly marked, especially between baseline and 12 months. It is possible that time spent on radiology studies was actually unchanged after a year and that the feeling of increased time spent on studies was misinterpreted. However, estimation of absolute time spent in hours per week can be difficult, especially if it varies from week to week; this might be easier with focused studying time. The use of MLD could have led to more flexible study patterns with more integration with clinical activities. In the present study, some behavioral changes in reading preferences might support this suggestion. Thus, though non-significant, a higher number of residents preferred to read based on cases they came across in the clinic, i.e. 46% at 12 months versus 32% at baseline. This integrated study might be more difficult to define and estimate when reporting weekly hours spent, but easier to report as a feeling of more time spent on studies.

Previous studies focusing on MLD interventions in radiology residency programs have shown increased time spent on learning radiology after 3–6 months’ intervention (14,15). This is in concordance with our results, in which subjectively increased study time was reported by most residents. However, the longer duration of intervention and observation in our study allowed us to also note a decreasing trend (though not quite significant) in study time, i.e. the proportion of residents saying that they spent “a lot more time on studies” decreased from 32% at six months to 8% at 12 months (P = 0.083) and the total estimated time spent on radiology studies decreased from 6.6 h at six months to 5.1 h at 12 months (P = 0.093). This stresses the importance of challenging the novelty factor and stimulating continuous learning, preferably by using a mix of educational materials and offering interactive instruments, including formative teaching solutions such as self-assessment and teacher-controlled tests and polls, as well as easily available and pedagogical e-books, articles, and movies (1,2).

Regarding access to educational materials and learning efficiency, our results are in agreement with those obtained for the iPad Toolbox (14), showing improvements after introduction of the MLD. In the Tuft study (15), 91% of participants stated that their experience of studying had improved, whereas this figure was 100% at six months and marginally lower (92%) at 12 months in our study.

STATdx and IMAIOS e-Anatomy were the resources used regularly by most of our study participants. In the iPad Toolbox (14) it was reported that 6% thought STATdx was the most important resource, whereas in our study 39% considered it a major resource. Contrarily, electronic textbooks were seen as the most important resources in the iPad Toolbox (14) but had a more peripheral role in our study. There are several possible reasons for this difference. We provided only one e-book in our project, which might have shifted the focus from e-books to other available resources on the tablet. Our chosen book simply might not have fully covered the teaching areas to a satisfactory depth and extent. This could also have been affected by the observed shift towards case-based learning preferences, which might require the in-depth knowledge and differential diagnoses found in more specific books. A possible way to deal with the e-book shortage in our MLD would be to provide access to our university library with more general licenses. This proved difficult during the current project but should be considered for future implementations.

Previous studies (3–5) have shown the advantages of interactive lectures; unfortunately, we were unable to provide these in this project owing to technical difficulties. Furthermore, our residents wished to have a local teaching file database developed at our clinic, possibly because our own patient cases may seem to have greater significance for and connection with our clinical environment, and because the clinical information in these patient cases is often more complete and offers more complexity than general teaching cases. Furthermore, images from our own scanners with our own protocols have a certain “look,” which might differ from that of textbook cases. Use of a familiar type of radiology image might help the resident to focus on the pathology and not on technical differences among the images.

One limitation of the study was the small study group with risk of a type II error in statistical analysis. Power analysis looking at the result of the study suggests more individuals to get significance with the same proportions. As the study design included all possible participants, i.e. all residents were offered participation, this was not easily overcome. Furthermore, the actual difference in the various variables was not clear before the study started and thus an accurate power analysis was not possible. Another limitation was that the questionnaire was responded to anonymously and thus follow up of individual responses over time was not possible. We therefore chose a non-paired statistical analysis after consulting a local statistician. Retrospectively, a paired study design would have been preferable.

In conclusion, the use of MLDs in radiology residency programs may facilitate learning by making study materials more easily available and making study more effective and more fun. Time spent on study increased after the introduction of an MLD; however, a novelty factor was noted with a trend of decreasing figures in the longer term, which needs to be addressed when introducing MLDs. Further studies addressing this are warranted, as well as studies focusing on different learning resources and pedagogic tools in the radiology field.

Click here for additional data file.

Supplemental material, ARR889871 Supplemental Material1 for Mobile learning device increased study efficiency for radiology residents but with risk of temporary novelty effect by Ann-Sofi Björkman, Anna Spångeus and Mischa Woisetschläger in Acta Radiologica Open

Click here for additional data file.

Supplemental material, ARR889871 Supplemental Material2 for Mobile learning device increased study efficiency for radiology residents but with risk of temporary novelty effect by Ann-Sofi Björkman, Anna Spångeus and Mischa Woisetschläger in Acta Radiologica Open