High User Acceptance of a Retina e-Learning App in Times of Increasing Digitalization of Medical Training for Ophthalmologists.

INTRODUCTION: The aim was to identify changes in continuing education and training in ophthalmology in the context of the COVID-19 pandemic and advancing digitalization and to analyse the acceptance of e-learning tools among German ophthalmologists using a novel Retina Case App as an example.
METHODS: The participants' training behaviour before and during the COVID-19 pandemic was surveyed. Furthermore, the acceptance and usability of the Retina Case App were evaluated using the System Usability Scale (SUS). A possible influence of the app on everyday clinical practice was assessed.
RESULTS: A total of 145 ophthalmologists participated in the survey. The frequency of continuing medical education did not decrease for 62.8% of ophthalmologists during the pandemic. A significant increase in at least monthly use of online courses or lectures has been observed (90.3% vs. 28.2%, p < 0.001). No significant difference was identified in terms of frequency of use of print and digital journals or printed textbooks. The majority of participants stated that online training platforms are well suited to replace the absence of face-to-face events (73.8%). The mean SUS score was 87.7 (SD 11.9), which categorizes the app's usability as excellent. The majority agreed that the newly developed app enables faster learning (82.1%) and leads to increased motivation (71.7%). Most ophthalmologists (80.7%) felt that regular use of the app would improve confidence in the treatment of retinal diseases.
CONCLUSIONS: The COVID-19 pandemic has led to a significant change in training behaviour in ophthalmology towards e-learning and online courses, which has not been accompanied by a general decline in training activity. The exemplarily investigated application showed a high user acceptance among ophthalmologists.

Introduction

The COVID-19 pandemic has made face-to-face events temporarily impossible during the years 2020 and 2021, which has given a significant boost to digitalization in medical training and education. Like other medical disciplines, ophthalmology is facing the challenge of providing continuing ophthalmic education using digital concepts. One of the approaches that has been increasingly pursued recently is the concept of e-learning.

E-learning is described by Sangra et al. [1] as a teaching and learning approach that uses electronic media and devices to improve communication, interaction, and access to education and training. It represents an educational model that facilitates the understanding and development of learning [1]. Frequently mentioned advantages of e-learning are flexible and fast access possibilities and uncomplicated updating or dissemination of learning content. Today, e-learning can therefore be an interactive and often asynchronous form of knowledge transfer, where learners themselves decide on their learning pace and on the time and place of retrieving learning content [2, 3]. The central questions of the present study are to what degree the COVID-19 pandemic has led to a change in continuing ophthalmic education and training and to what extent a didactically prepared e-learning app (Retina Case App) with guideline-orientated content is accepted by ophthalmologists.

Methods

Recruitment

As face-to-face meetings were limited during our survey period from January to June 2021 due to the COVID-19 pandemic, ophthalmologists were contacted digitally. The sponsor of the e-learning app, supported by the University Eye Hospital Muenster, conducted a nationwide advertising campaign among German ophthalmologists in private practice and those working in clinics via email in order to raise awareness for the app. In addition to information about the app, the email contained a link to an anonymous questionnaire to be answered on Google Forms (Google LLC, Mountain View, CA, USA). The email distribution lists included all ophthalmologists of university eye clinics in Germany, the majority of municipal and private eye clinics in Germany, and with the support of the sponsor, also ophthalmologists in private practice. In total, 800 ophthalmologists were contacted. With this approach, we have tried to reach a wide spectrum of German ophthalmologists, regardless of gender, age, place of work, and field of activity.

Questionnaire

Age, gender, employment (medical practice, hospital), and field of activity (conservative, surgical) were queried. The various statements of the questionnaire were rated on a 5-point Likert scale from “completely disagree” to “completely agree.”

The questionnaire was divided into two parts. The first part focused on surveying the learning and training behaviour of the participants before and during the COVID-19 pandemic. For this purpose, the participating ophthalmologists needed to rate different statements in this regard. Additionally, the frequency of use of different training modalities was compared before and during the COVID-19 pandemic on an ordinal scale ranging from “daily” to “never.”

For the second part, participants tested the user software developed for this work and rated the app using the System Usability Scale (SUS). Furthermore, the participants were asked to assess various statements regarding the possible influence of the app on their daily clinical routine. It should be noted that the participants had not used the app before.

System Usability Scale

The SUS was developed by John Brooke in 1986 and is a valid measure for assessing the usability of an application. A questionnaire is used to assess how user-friendly a system is perceived to be. The questionnaire consists of five positively and five negatively worded statements. The response is given on a scale from 1 (strongly disagree) to 5 (strongly agree). The SUS score ranges from 0 to 100, whereas a higher score means better usability. It is important to note that the calculated score does not represent a percentage value. One can rather assume percentiles, where a score of 68 corresponds to an average value on the 50th percentile [4, 5]. The SUS is established, widely used, and freely available. There is a well-accepted adjective scale, based on the benchmarks set by Bangor et al. [6]. A score above 85 suggests an excellent, highly usable system, a score of at least 71 is “good” and a score of 51 is associated with “ok” usability. Finally, a SUS score below 51 is considered “poor” [6].

Retina Case App

The Retina Case App has its focus on the diagnosis and therapy of retinal diseases. In this context, typical and unusual cases were compiled and professionally presented in an e-learning app. They can be assigned to four major topics: age-related macular degeneration (9 cases), retinal vein occlusions (7 cases), diabetic macular oedema (5 cases), and “pitfalls” (21 cases). The last major topic “pitfalls” represents a category of various other retinal diseases such as myopic choroidal neovascularization or central serous chorioretinopathy. In addition to a patient history, each case includes corresponding retinal images with various imaging modalities (fundus photography, optical coherence tomography, fluorescein angiography, etc.). The case description is accompanied by a question with three possible answers. The user receives feedback on the selected answer as well as further explanatory texts on the discussed case according to the guidelines and statements of the ophthalmic societies (shown in Fig. 1). The app was programmed by Georg Thieme Verlag (Georg Thieme Verlag KG, Stuttgart, Germany) with financial support from Bayer (Bayer AG, Leverkusen, Germany), whereby none of the above-mentioned parties had any influence on the content of the app. The app's content was curated by a team of retina specialists from the University Eye Hospital Muenster. The app is available free of charge in the respective app stores and can be accessed in German or English language.

Fig. 1

Illustration of the Retina Case App. a Main menu with four major topics: (1) AMD, (2) RVO, (3) DME, and “pitfalls.” b In each topic (e.g., “pitfalls”), different cases can be selected. c A case description with corresponding retinal images (OCT, fluorescein angiography, fundus autofluorescence, etc.) is followed by a question with three possible answers. The images can be enlarged by clicking on each of them. d Subsequently, the answer options are discussed and the correct answer is given. In addition, the learner is offered further information from recent literature. AMD, age-related macular degeneration; RVO, retinal vein occlusions; DME, diabetic macular oedema.

Statistics

The statistical evaluation of the questionnaire was carried out with Microsoft Excel 2019 (Microsoft Corp., Redmond, WA, USA) and IBM SPSS Statistics 27 (IBM Corp., Armonk, NY, USA). Descriptive statistics with percentages were used for categorical data. The Fisher-Freeman-Halton Exact test was selected for correlation analysis in terms of agreement with the different statements and the different subgroups. We used the McNemar-Bowker test, comparing the frequency of use of different training modalities before and during the COVID-19 pandemic. The different figures were created using Microsoft Excel and Adobe Illustrator 2021 (Adobe Inc., Mountain View, CA, USA).

Results

Demographics

A total of 145 ophthalmologists participated in the survey (n = 86, 59.3% male). The SUS questionnaire was completed by a total of 115 participants. Most participants were aged between 31–40 years and 51–60 years, respectively (n = 43, 29.7% and n = 37, 25.5%). The majority of participants (n = 94, 64.8%) worked primarily as a conservative ophthalmologist and 57.2% (n = 83) worked in a medical practice (as opposed to being surgically trained or being employed at a hospital).

Impact of the COVID-19 Pandemic on Continuing Education in Ophthalmology

Despite the lack of face-to-face events during the COVID-19 pandemic, it can be noted that the frequency of continuing medical education did not decrease for 62.8% of ophthalmologists (n = 91), while online education platforms gained importance during the pandemic (n = 135, 93.8%). Most ophthalmologists considered online training platforms to be well-suited to replace face-to-face events (n = 107, 73.8%), and only 8.3% (n = 12) of the participants felt increasingly insecure in their daily practice due to the lack of face-to-face events and reduced exchange among colleagues (shown in Fig. 2).

Fig. 2

Agreement with the above statements regarding the impact of the COVID-19 pandemic on continuing education in ophthalmology (%).

The data show that before the COVID-19 pandemic, printed professional journals were the most frequently used medium for continuing education. 90.9% of the participants (n = 130) used printed professional journals for continuing education on an at least monthly basis, followed by printed textbooks (n = 111, 77.6%) and digital journals (n = 98, 68.5%). Prior to the COVID-19 pandemic, only 28.2% of ophthalmologists (n = 40) used online continuing education platforms, e.g., digital courses (CME courses) or portals to retrieve recorded lectures on an at least monthly basis.

Overall, during the pandemic, no significant difference was identified in terms of frequency of use of print and digital journals or printed textbooks. However, an increasing use of online continuing education platforms (apps, online courses, or lectures) can be witnessed during the pandemic (p < 0.001). For example, participation in online courses or lectures, at least monthly, has increased from 28.2% (n = 40) up to 90.3% (n = 130) (shown in Fig. 3, 4).

Fig. 3

Comparison of the frequency of use of passive training methods (%).

Fig. 4

Comparison of the frequency of use of interactive training methods (%).

Acceptance and Influence of the Retina Case App

The mean SUS score was 87.7 (SD 11.9), which categorizes the app's usability as excellent [6] (shown in Fig. 5). The majority of ophthalmologists agreed that the newly developed app enables faster learning (n = 119, 82.1%) and that its ease of use and accessibility leads to an increased motivation compared to conventional training methods (n = 104, 71.7%). In more than half of the cases, the app stimulated case discussions among colleagues (n = 84, 58.3%). Furthermore, most ophthalmologists (n = 117, 80.7%) felt that regular use of the app would lead to greater confidence in the diagnosis and indication of intraocular injections, whereas 67.6% of respondents (n = 98) stated that the cases presented would encourage them to modify their diagnostic and therapeutic decisions. Finally, it can be summarized that the vast majority (n = 129, 89.0%) of clinicians can imagine this learning format as a recurring training modality (shown in Fig. 6).

Fig. 5

SUS showing SUS Scores from 0 to 100 with corresponding acceptability, grade, and adjectives. The resulting SUS Score of the Retina Case App was 87.7, which categorizes the app's usability as excellent.

Fig. 6

Agreement with the above statements regarding the Retina Case App in everyday clinical practice (%).

Analysis of the Different Subgroups

We compared the respective statements between the different subgroups. Women were overall less likely to agree that changes during COVID-19 led to a decrease in continuing medical education (p = 0.048). There was also a statistically significant difference between the different age groups in relation to the statement: “App motivates more than conventional training methods.” Ophthalmologists over 60 years of age, in particular, seemed to agree less often with this fact (p = 0.032). Other than that, there were no differences of statistical significance concerning gender, age, place of work, and field of activity.

Discussion

This study evaluated changes in continuing education among ophthalmologists in the context of the COVID-19 pandemic and advancing digitalization and it analysed the acceptance of e-learning tools among German ophthalmologists using an exemplary Retina Case App. In brief, this study provides evidence that (1) the COVID-19 pandemic has led to a significant change in training behaviour in ophthalmology, which has not been accompanied by a general decline in training activity, (2) online events and e-learning offerings have essentially gained in importance, (3) the exemplary Retina Case App showed a high usability and acceptance rate.

Not surprisingly, the constraints of the COVID-19 pandemic are reflected in participants' responses regarding the use of e-learning offerings. It is particularly noteworthy that, according to the participants interviewed, the restrictions and changes in the context of the pandemic have not led to less training activity. The very significant decline in face-to-face courses has led to a substantial increase in popularity of e-learning and online courses and, interestingly, not to an increased use of digital and printed journals. In our opinion, this very fact supports the assumption that e-learning modalities designed to compensate for the loss of face-to-face events should have some degree of interactivity. According to the majority of participants, online training platforms were considered a suitable substitute for the loss of face-to-face events. Our expectation that the absence of face-to-face events and the lack of exchange among colleagues would lead to increased uncertainty among ophthalmologists could not be confirmed by our results. These observations lead us to assume that e-learning concepts, for example, also in the sense of hybrid models, will probably increasingly find their way into continuing medical education in the post-COVID era.

In this context, the free and already existing online platforms and applications for ophthalmology should also be mentioned. The English-language website “EyeWiki” has the character of an eye encyclopaedia. Similar to the German-language platform “EyeFox,” which is a news portal with video offerings and encyclopaedia functions, it lacks interactive e-learning features. The “Eye Handbook,” supported by the American Academy of Ophthalmology, can be considered an example of a successfully implemented e-learning app that is popular within the ophthalmic community. It is among other things an ophthalmic encyclopaedia, a platform for case discussion, and also a diagnostic tool [7]. In addition to these examples, there are numerous smartphone apps and online platforms in the form of diagnostic tools and e-learning concepts, especially in the English-speaking world. To our knowledge, however, many e-learning concepts often do not offer didactically prepared and case-based continuing education opportunities. In a comprehensive analysis of the current range of ophthalmic apps available by Stanzel and Meyer [8] and Hogarty et al. [9], the lack of evidence and certification of many of these applications is considered a problem with regard to systematic use in everyday clinical practice. Although these observations relate primarily to diagnostic app offerings, we believe that e-learning platforms for ophthalmology will also benefit from further research in this area as well as support from ophthalmic societies.

Above all, studies on e-learning must always be seen in the context of their year of origin. The rapid advancement of technology sometimes leads to difficult comparability. Furthermore, the assessment of e-learning approaches is very diverse and not standardized in the literature. One reason for this lies in the different objectives of e-learning studies. Often the focus is directed towards the actual learning success and only rarely the design of the e-learning method is the subject of research, although it may well affect the method's efficiency and acceptance [10]. Moreover, the evaluation of e-learning in relation to learning outcomes also shows a very heterogeneous picture. Some authors postulate an at least equal learning success with e-learning, but with less time expenditure and improved cost efficiency, compared to traditional learning methods [2, 11, 12]. Other works report no clear added value for e-learning in terms of learning success [13]. From the learner's point of view, e-learning offerings should prove beneficial in terms of time-saving, availability, interaction, and the possibility of feedback, in order to be accepted and preferred over traditional methods [14].

Only few studies compare the knowledge level of users before and after participation in an e-learning programme [12, 15]. Overall, there is currently insufficient data to assess whether e-learning in medicine has a measurable, positive effect on clinical decision-making or even on patient outcomes. Here, studies are needed that consider patient health as a clinical endpoint of an evaluation of e-learning [16].

For the evaluation of the applied app, we chose the SUS. It is a widespread, standardized questionnaire for evaluating usability, which by now has been used to assess various software applications. The method has proven to be valid, reliable, and sensitive [17]. Our results show that users consider an interactive software as a profitable learning method with a high degree of usability (SUS score >85). We believe that this high degree of usability is an essential requirement for successful e-learning concepts. In this regard, the majority agreed that the app's ease of use and accessibility led to an increased motivation compared to conventional training methods and over 80% reported faster learning success.

In our study, most ophthalmologists agreed that regular use of the app improves confidence in the treatment of retinal diseases and encourages to modify clinical decisions. However, an objective assessment of individual learning progress by using the e-learning app was not performed in our study. Future studies should include a standardized assessment of learning progress and also of the time participants spent with the learning application. Furthermore, the credibility of industry-sponsored e-learning tools, such as the one presented, can be enhanced through accreditation by ophthalmic societies, which could help increase the use of this flexible and versatile educational modality among the ophthalmic community.

In addition to these aspects, there are other limitations to consider in the context of our study: The primarily digital recruitment of the participating ophthalmologists might have supported the popularity of digital teaching concepts, which is reflected in the ophthalmologists' responses to the questionnaire. Likewise, it seems possible to us that differences in technology affinity and possibly also age could have had an influence on the acceptance and perceived usefulness of the app. A possible recall bias for the retrospectively assessed learning behaviour before the COVID-19 pandemic represents another limitation. Despite these limitations, however, we believe that our results reveal an overall trend with regard to the conclusions drawn and suggest that the COVID-19 pandemic has led to an increasing shift of ophthalmic education into the digital space. A survey conducted by Chatziralli et al. [18] in April 2020 among ophthalmologists with teaching responsibilities underpins these observations.

In summary, our study shows that the COVID-19 pandemic has led to a significant change in training behaviour in ophthalmology towards e-learning and online courses, which has not been accompanied by a general decline in training activity. Furthermore, the exemplarily investigated e-learning application shows high user acceptance. The restrictions in the context of the pandemic that made it difficult to hold face-to-face events clearly highlighted the advantages of e-learning, such as accessibility and interactivity. Although they will certainly not replace face-to-face events after the end of the pandemic, e-learning tools can certainly represent an interesting addition to the ophthalmic training spectrum in the future.

Statement of Ethics

Ethics approval was not required (Ethical committee of the Medical Association Westfalen-Lippe and the Westphalian Wilhelms-University of Muenster).

Conflict of Interest Statement

Florian Alten: Speaker's fee Bayer; Nicole Eter: Speaker's fee and research funding Heidelberg Engineering, Novartis, Bayer, Sanofi Aventis, Allergan, Bausch & Lomb; Christoph R. Clemens: Speaker's fee Heidelberg Engineering, Novartis, Bayer; Alexander Kamouna and Eugen Grabowski declare no conflicts of interest. The app was designed by Georg Thieme Verlag with financial support from Bayer. Neither Georg Thieme Verlag nor Bayer had any influence on the design and implementation of the study presented. The content of the publication reflects solely the opinions of the authors, not those of the named project partners. The app is available free of charge in the appropriate app store. Written informed consent was obtained from all participants. The study was conducted in accordance with the World Medical Association Declaration of Helsinki.

Funding Sources

No funding was received to assist with the preparation of the manuscript.

Author Contributions

Christoph R. Clemens and Florian Alten took the lead in creating the content of the investigated “Retina Case App” and conceived the idea for the presented study. Alexander Kamouna and Christoph R. Clemens developed the questionnaire and took the lead in writing the manuscript. Alexander Kamouna carried out the statistical analysis of the results. Alexander Kamouna, Florian Alten, Eugen Grabowski, Nicole Eter, and Christoph R. Clemens provided critical feedback and assisted in the editing of the manuscript.

Data Availability Statement

The data that support the findings of this study are available on request from the corresponding author.