A digital online platform for education and certification of diabetic retinopathy health care professionals in the Region of Southern Denmark.

PURPOSE: The incidence of diabetes continues to increase across the world. As the number of patients rises, so does the need for educated health care professionals. Diabetic retinopathy (DR) remains one of the primary complications in diabetes, and screening has proved to be a cost-effective measure to avoid DR-related blindness. Denmark has an established screening programme, but no formal training of the people responsible for analysing retinal images.
METHODS: We here present an online learning platform that offers a diabetic eye screening course for health care professionals undertaking screening responsibility in the Region of Southern Denmark. The course is divided into lectures, each focussed on identifying different levels of DR or detecting related lesions. The course is free to use on-demand, contains instructional videos, interactive tests and exercises, and it is concluded with a certification test. The tools on the platform can in addition be used to generate data for research purposes, such as comparing users or experts in detection of lesions or annotating data for the development of machine learning models.
RESULTS: More than 150 participants have so far completed the course, and the platform is being adopted for education in other regions of Denmark.

Introduction

Diabetes is the most common metabolic disease in the world. The incidence of diabetes has increased in recent years, and about 422 million people are affected worldwide. This number is expected to surpass 600 million by 2045 (Saeedi et al., 2019). Easy and affordable access to qualified health care professionals and facilities is an important factor in diagnosing and treating diabetes and diabetes‐related complications in e.g. heart, kidneys and eyes. With the majority of diabetes cases found in low‐ to middle‐income countries (World Health Organization, 2019), these aids may not be available to all people with the disease. As the diabetes incidence continues to rise, the need for additional educated health care professionals also increases.

Diabetic retinopathy (DR) is the most frequent complication of diabetes and a leading cause of blindness in the working age population in the western world (Pappuru et al., 2018). DR accounted for more than 860 000 cases of preventable blindness globally in people aged 50 or above (GBD 2019 Blindness and Vision Impairment Collaborators, & Vision Loss Expert Group of the Global Burden of Disease Study, 2021). Periodic eye examination is essential for the detection and planning of treatment, as timely treatment has been shown to reduce incidence of DR‐related vision loss by 90% (Early Treatment Diabetic Retinopathy Study Research Group ,1991), and screening for DR is proven to be a cost‐effective measure (Stefánsson et al. ,2000).

In 2016, it was estimated that the number of people with diabetes in Denmark was 230 000 and that this number would increase by 27 000 each year. From March 2014 to February 2015, 77 968 people had known DR status in Denmark (Andersen et al., 2016). Denmark has a fully implemented screening programme where diabetes patients attend screenings at regular intervals at either private ophthalmology clinics or public hospitals. If presence of sight threatening DR is suspected, patients are referred for treatment at public hospitals.

Since 2007, Denmark has been divided into five regions where each region is partly responsible for its own public health sector. The Danish population is unevenly distributed across these five regions, with the most populous region being the Capital Region of Denmark with more than 1.9 million inhabitants and the least populous being the region of Northern Denmark with approximately 600 000 inhabitants. Due to unequal distributions of eye care providers and regional prevalence of DR, regional differences exist in the screening programme. Even though the established screening programme uses a common set of guidelines for DR disease staging, regional traditions may still play a factor in the care and amount of monitoring and follow‐up consultations patients receive. It is important that clinicians are equally qualified and adhere to the same guidelines to ensure that people across the nation have access to the same quality of care. Education plays an important role in this regard, ensuring that the relevant clinical health care professionals are subject to the same educational material and that certifications are based on the same set of quiteria and tests.

Despite the introduction of the nationwide screening programme, no formal education of DR health care professionals has so far been established. All practicing ophthalmologists have been considered qualified to perform DR screening, but it has been left up to the individual hospital departments and private clinics to ensure that the staff have been equipped with the right set of tools and instructions in order for them to adhere to the common set of guidelines established by the Danish Ophthalmological Society (Grauslund et al., 2018) of using the International Clinical Retinopathy Disease Severity Scale (ICDR, Wilkinson et al., 2003) in screening and grading of DR.

Steno Diabetes Center Odense (SDCO) was established in 2018 with the goal of strengthening and supporting the effort on diabetes care in the Region of Southern Denmark, as well as collaboration between stakeholders such as regional hospitals, local municipalities and private clinics. This includes consolidation of the effort to prevent irreversible vision loss brought on by DR in people with diabetes. This was cemented through the establishment of a centrally localized grading centre where DR health care professionals grade retinal fundus photographs gathered from SDCO, as well as six outpatient clinics in the region. From March 2019 through September 2020, photographs from more than 10 000 patients were analysed at the grading centre. SDCO is part of a national collaborative network of Steno centers located in each of the five Danish regions.

The previous training environment at SDCO existed in a physical classroom, where an instructor provided the learning material for the participants. The learning material consisted of printed images of eyes, where any markings or gradings given by participants had to be evaluated by the instructor manually. This process limits how quickly participants receive feedback and may also be limited by how much time the instructor is able to set aside. In this work, we present a virtual ocular learning platform (VIOLA) for formal training and certification of health care professionals tasked with the day‐to‐day analysis of diabetic eye images and grading of DR in diabetes patients. VIOLA brings the training online, where students and teachers can participate in training activities anywhere and anytime.

In Denmark, DR screening is traditionally performed by practicing or hospital‐based ophthalmologists, but there is also substantial support from support staff, including nurses, optometrists and medical students. As the latter group is often handling imaging in direct involvement with patients, it has been a priority for us to design courses specified for attendees at specific levels of knowledge, as this will make it easier for them to engage with patients and answer some DR screening–related questions. Since 2018, the educational programme has been offered for free to all ophthalmologists and screening personnel in Denmark.

Methods

We have created VIOLA, a digital, online platform that offers a diabetic eye screening course divided into lectures, each focussed on identifying different levels of DR or detecting specific DR lesions. Lectures in the course are created by an instructor and can be divided into modules that provide the learning material for the participants.

Each lecture consists of instructional videos (Fig. 1), exercises and tests. The duration of lecture videos ranges from 3 minutes and 31 seconds to 12 min and 11 seconds with an average duration of around 7 min. Tests are either multiple choice questions, where participants are asked to relate to different statements on DR and DR classification, or grading tests. In grading tests (Fig. 2), participants are presented with an image, to which they are asked to assign the correct grade. After completion, the participant submits their answers and is given immediate feedback on the results in comparison with expert answers.

Fig. 1

Example screenshot from an instructional video on the use of the ICDR scale. The video is the third module in lecture 1. In this module, participants are presented with example lesions (“blødninger”: haemorrhages and “hårde exudater”: hard exudates) and how they relate to the individual ICDR levels. [Colour figure can be viewed at wileyonlinelibrary.com]

Fig. 2

Example screenshot from a grading test on identification of different DR levels (DR level 0 through 3). This is the sixth module in lecture five on level 3 DR. In the Image in the top of the figure, the participant is asked what level of DR the image represents, and they then choose an answer from the menu to the right. [Colour figure can be viewed at wileyonlinelibrary.com]

Exercises can be so‐called “marking” or “drawing” exercises. In marking exercises, the participants are asked to click on locations in an image where they suspect the presence of specific types of DR lesions (Fig. 3). Prior to clicking, the participant selects the type of lesion from a drop‐down menu yielding a colour‐coded selection tool and clicks on the desired location. Clicks can be removed, if they have been made by mistake. After completion, the participant submits their answer and is given immediate feedback. Both quantitative and visual feedback is provided, showing the number of correct or incorrect markings as well as an image superimposed with all markings given by experts, and indications of where the participant agrees or disagrees with the experts. In drawing exercises (Fig. 4.), users are asked to draw outlines of specific lesions. Participants select a colour‐coded drawing tool from a drop‐down menu and can then draw the outlines of specific lesion types. Both a quantitative drawing score based on the Otsuka‐Ochiai coefficient (Romesburg, 1984) and qualitative feedback is given after completion. To minimize the risk of participants incorrectly being awarded a false‐positive marking, all images used in the drawing and marking exercises have been independently annotated by two different experts, and the common quantity between experts is used as reference standard.

Fig. 3

Example screenshot from a marking exercise for different lesions; haemorrhages: magenta, hard exudates: red and cotton wool spots: yellow. This is the fourth module in lecture five on level 3 DR. [Colour figure can be viewed at wileyonlinelibrary.com]

Fig. 4

Example screenshot from drawing exercise for different lesions; haemorrhages: magenta, hard exudates: red and cotton wool spots: yellow. This is the fifth module in lecture five on level 3 DR. The problem text reads in English; “Draw five haemorrhages, five cotton wool spots and five Intraretinal microvascular abnormality (IRMA) changes”. [Colour figure can be viewed at wileyonlinelibrary.com]

Prior to conclusion, participants are examined on the knowledge obtained throughout the course by completing a final certification test. If this test is completed with a satisfying result, the participant will be certified to perform diabetic eye screenings within the bounds of SDCO. The certification test is similar to the grading tests, where participants are presented with an image, to which they are asked to assign the correct level of DR. The reference standard is given as the consensus evaluation by two experts.

All instructional videos, tests and exercises can be interrupted and resumed at any time, allowing participants to complete the course in their own time.

The content on VIOLA is customizable, meaning that at any time instructors are able to expand or remove content as well as rearrange lectures, exercises and tests. This does not require any access to the underlying platform architecture or any programming experience. This in turn means that the platform can be used to create other courses, e.g. a course on identifying glaucoma. The underlying architecture is also flexible in a way that allows it to be customized for courses within completely different domains.

Virtual ocular learning platform provides instructors with the tools to generate the material for marking and drawing exercises as well as grading tests, i.e. the expert references such as the image DR level, markings and drawings of lesions used in tests and exercises are created using tools on the platform. Figure 4 illustrates the drawing tool that is used by both users and experts to draw the various lesions. For grading tests, drawing and marking exercises, the platform is equipped with a number of tools available to the participants when examining images. These tools are: zoom, convert to grey‐scale image, adjust red colour component or brightness.

Currently, VIOLA offers two courses on diabetic eye screening: a basic screening course and an advanced screening course, where the difference between the two is in the final certification test as well as the drawing exercises, which are not included in the basic version. The focus of the basic course is directed more towards participants learning to differentiate between no DR (ICDR level 0) and any DR (ICDR level 1 to 4). In the certification test of the advanced course, participants are awarded 0.5 point for correctly identifying level 4 DR, and an extra 0.5 point for identifying the presence of active proliferative DR (PDR). In the basic course, 1 point is awarded regardless of whether they identify active PDR. For DR levels 1, 2 and 3, participants are awarded a point if the assigned grade is within 1 level of the expert reference, while for level 0 and 4, they must assign the exact level given by the expert. After the certification test has been completed, participants can obtain a certificate by providing their name, after which a printable certificate is generated. The outline of the advanced screening course is given in Table 1. Table 2 provides a detailed description of example tests and exercises.

Table 1

Course structure of the advanced diabetic eye screening course on the VIOLA platform.

Lecture	Modules
1. Introduction	1 Diabetic eye screening (Video) 2 Diabetic eye screening (MCT) 3 ICDR scale (Video) 4 ICDR scale (MCT)
2. Level 0 DR	1 Level 0 DR (Video) 2 Level 0 DR (MCT) 3 Identification of DR (Any level, G)
3. Level 1 DR	1 Level 1 DR (Video) 2 Level 1 DR (MCT) 3 Microaneurysms (M) 4 Identification of level 1 (G)
4. Level 2 DR	1 Level 2 DR (Video) 2 Level 2 DR (MCT) 3 Haemorrhages, hard exudates, cotton wool spots (M) 4 Haemorrhages, hard exudates, cotton wool spots (D)
5. Level 3 DR	1 Level 3 DR (Video) 2 Level 3 DR (MCT) 3 Intraretinal microvascular abnormality (IRMA, M) 4 Haemorrhages, cotton wool spots, IRMA (M) 5 Haemorrhages, cotton wool spots, IRMA (D) 6 Identification of level 3 (G)
6. Level 4 DR	1 Level 4 DR (Video) 2 Level 4 DR (MCT) 3 Retinal neovascularization (NV, M) 4 Panretinal photocoagulation scars (PCT, M) 5 NV and PCT (D) 6 Identification of level 4 (G)
7. OCT	1 OCT (Video) 2 OCT (MCT)
8. Certification	1 Grading of DR level 0 through active and inactive level 4 (G)

Left column gives the name of the individual lectures (seven plus the certification test). Right column gives the name of the submodule in each lecture. Each module is classified as either video, multiple choice test (MCT), marking (M) or drawing (D) exercise or DR grading test (G). VIOLA = Virtual ocular learning platform.

Table 2

Description of example test and exercises on VIOLA.

Lecture‐Module	Description
3–2	Three questions on retinal microaneurysms. Select single correct answer from multiple choices. (MCT)
4–5	Twelve images. For each image, indicate either DR level 0 (four images), level 1 (four images) or level 2 (four images). (G)
5–4	Five images. For each image, find five retinal haemorrhages, five cotton wool spots and three intraretinal microvascular abnormalities. (M)
6–5	Three images. For each image, outline five retinal neovascularizations and five panretinal photocoagulation scars. (D)
8–1	Certification test. Forty images. For each image, indicate the correct DR level among level 0 or level 1 (ten images) and level 2, level 3 or level 4 (thirty images). If level 4 DR, indicate whether active proliferative DR is present. (G)

The left column indicates the lecture and module where the test or exercise is placed in the course. The numbers refer back to Table 1, where the lecture number is given in the left column and the numbers of the lecture submodule is indicated in the right column, i.e. 5–4 refers to lecture 5 module 4, which is a marking exercise. Tests are either multiple choice (MCT) or grading (G). Exercises are either marking (M) or drawing (D) exercises. VIOLA = Virtual ocular learning platform.

The only requirement for using VIOLA is a computer or laptop with internet access. Currently, the platform supports the following internet browsers: Google Chrome, Microsoft Edge and Mozilla Firefox. As the images presented in tests and exercises are high resolution, it is recommended that participants use computers with a decent‐sized screen or external monitor. The drawing and marking exercises require some processing, but no more than can be expected from a relatively modern computer. The underlying data structure supports an English version of the programme, but so far, only the Danish version has been implemented. In theory, the content on VIOLA can be offered in any language, it just requires that the teaching material is translated.

In the teaching and learning structure, VIOLA carries a resemblance to popular online learning platforms such as Codecademy (Sims & Bubinski, 2011) that teaches programming skills in a number of different computer programming languages. Similar to VIOLA, the course material on Codeacademy is arranged into lectures with associated exercises, and users are given immediate feedback on performance and course progression. PracticeAnatomy (Adaptive Learning research group, Faculty of Informatics of Masaryk University, 2016) serves a similar purpose to VIOLA of teaching material related to medical science. Among the types of exercises found on PracticeAnatomy are marking tests, where participants are asked to find specific parts of the human anatomy, or provide the name of highlighted anatomical structures. PracticeAnatomy likewise provides users with immediate feedback on performance as well as course progression. The material on PracticeAnatomy is fixed and cannot be modified or customized without access to the underlying platform architecture.

Khan Academy (Khan, 2008) is another online teaching environment, which allows students to study a wealth of subjects within different areas. Similar to VIOLA, Khan Academy also allows instructors to generate teaching and practice material without needing access to the underlying platform or computer programming skills. Unlike VIOLA, Khan Academy does not offer any functions for generating material for marking and drawing exercises.

Apart from the teaching and learning outcomes of the VIOLA, the platform also serves the purpose of generating data for research. The drawing and marking tools were recently used in a study comparing agreements between experts for detection of DR abnormalities in fundus images (Grauslund et al., 2021). In 300 retinal fundus images representing the full ICDR scale, all lesions were marked independently by two experts. The total number of DR‐associated lesions marked by the experts was 39 723 and 32 069 respectively.

Results

From the video length and median exercise and test times, the time to complete the advanced screening course in one sitting can be estimated to be around seven hours. Figure S1, S2 present box plots for time spent on the different tests and exercises as well as the certification test by participants who have completed the advanced course on VIOLA. Generally, participants spent the least amount of time on multiple choice tests, with the maximum time being less than 5 min. For grading tests the median time spent ranged from 9 min and 59 seconds to 31 min and 21 seconds depending on the module. Based on the median, the average time spent grading a single image ranged from 1 min and 2 seconds for grading of level 1 DR (grading test 3–4) to 1 min and 34 seconds for grading all levels (grading test 6–6). The median time used to complete individual marking exercises ranged from 6 min and 22 seconds to 51 min and 40 seconds, and for drawing exercises, the median time spent ranged from 20 min and 20 seconds to 22 min and 30 seconds. The final certification test has a median time to complete of 1 hr, 8 min and 55 seconds for users undertaking the advanced screening course, with an average of 1 min and 43 seconds used per image.

Figure 5 shows participant markings for different retinal lesions (cotton wool spots, intraretinal microvascular abnormalities (IRMA) and neovascularizations). The visualizations in the top row of the figure illustrate correct clicks for each type of lesion, and the bottom row shows incorrect clicks. Clicks are included from alle exercises and images. Comparing the clicks for correct and incorrect markings can be used by instructors to determine the areas where participants are likely to confuse regular image features with retinal irregularities. For cotton wool spots, the example in Fig. 5 illustrates that participants in some cases make mistakes in the periphery of the retinal mosaic. For IRMAs and neovascularizations, the figure shows that a lot of incorrect clicks occur around the macula. For IRMAs, there are also a significant amount of incorrect clicks in the peripheral mosaic. The same data can be generated for the other types of lesions, and this information can be used to expand the course and elaborate the modules relating to these lesions.

Fig. 5

Click marking visualizations of correct clicks (top) and incorrect clicks (bottom) in right retinas by course participants for cotton wool spots (yellow), IRMA (cyan) and neovascularizations (blue) superimposed on an example retinal mosaic. Only clicks that were not removed by the users are included. Densely coloured areas indicate more clicks. [Colour figure can be viewed at wileyonlinelibrary.com]

Figure 6 illustrates the location in images where participants most frequently initialize examination of images for the presence of specific lesions. The Figure shows the first two clicks made by each participant for specific lesions. In the example shown, they are haemorrhages and microaneurysms. Clicks are included regardless of being correct or incorrect, but only if they were not removed by the user. The information can be used by the instructor to determine whether it is necessary to emphasize areas where lesions are at risk of being overlooked if graders neglect these.

Fig. 6

Click marking visualizations of the location of the first two clicks (both correct and incorrect) not removed by participants in left and right retinas by participants for haemorrhages (magenta) and microaneurysms (green) superimposed on an example retinal mosaic. Densely coloured areas indicate locations where most participants initially suspected the presence of lesions. [Colour figure can be viewed at wileyonlinelibrary.com]

Discussion

Virtual ocular learning platform (VIOLA) offers an online training and certification course for health care professionals who undertake diabetic retinopathy screening work in the Region of Southern Denmark. The platform permits users to participate anywhere and anytime it suits them. Users are given immediate feedback on the results of tests and exercises completed in the course, giving them an opportunity to reflect on their own performance and redo them if needed. Users are given progress and performance statistics, so they can keep track of how they are progressing through the course. Putting the course online and offering it on‐demand with instructional videos and interactive tests and exercises makes it flexible for the user. It also ensures that health care professionals are presented with the appropriate learning material and given the correct instructions for performing screenings according to the guidelines put forth by the Danish Ophthalmological Society. Furthermore, course times are not dependent on when a qualified instructor is free to conduct the lecturing, meaning that personnel can be certified as the need arises.

As illustrated by the plots in Fig. S1 and S2, there is a large variation in the time used by individual course takers. There is some uncertainty in the numbers, since times are computed from when participants initially started the test or exercise, and when the results were submitted. As such, all the time where participants did not actively interact with the platform is also included. While it makes it difficult to determine the total time, and time used in individual tests and exercises, it shows that users have seemingly taken advantage of the on‐demand opportunity the platform offers.

Considering the recent events around the global COVID‐19 pandemic, which has resulted in lockdowns of educational institutions and remote work for many people around the world, VIOLA fits into a new online education paradigm where online and remote teaching is becoming a more recognized and accepted approach to education. Obviously, due to the nature of the online and on‐demand format, it limits the participants ability to ask and receive answers on any questions they might have relating to the material. As VIOLA is in continuous development, we rely on feedback from the users to improve and adapt the platform to ensure the best possible experience.

In our study comparing the agreement between experts in the detection of diabetic retinopathy‐associated lesions, we found good or excellent agreement (>0.77) in most types of lesions with respect to intraclass correlation coefficients (ICC); microaneurysms (0.81), haemorrhages (0.83), hard exudates (0.91), cotton wool spots (0.91), panretinal photocoagulation scars (0.99) and IRMA (0.77). Retinal new vessels (0.11) and central laser treatment scars (0.07) demonstrated quite low ICC values. Overall, for marking exercises, the common quantity between two graders is a valid way to evaluate most lesions.

The expert annotated data can also be used in the developments of artificial intelligence algorithms for clinical decision support and automated grading in DR. The dataset of pixel level annotations of retinal lesions and abnormalities in DR has recently been used to develop a deep learning artificial intelligence algorithm to perform automatic segmentation of retinal features.

With the increasing interest in data‐driven artificial intelligence algorithms, a number of online data tools with varying levels of complexity have become available for annotation of a wide variety of data types, e.g. Make Sense and COCO annotator (Brooks, 2018, Skalski, 2021). The annotation tools in VIOLA are easy to use and specifically targeted the medical domain by focussing on two fundamental tasks in medical image analysis, namely image classification and segmentation. As the user base of the platform will hopefully grow through certification and recertification of course participants, we plan to generate even more data in the future. For example by adding data without expert annotations and permitting users with good performance statistics to grade, mark or draw on these, we can build larger datasets that can be used in the development of new and stronger models in the future.

Figure S1. Box plots of times spent on MCTs (top left), marking exercises (top right), grading tests (bottom left) and drawing exercises (bottom right) in the VIOLA courses. Plots show the median values (yellow line), interquartile ranges and outlier times.

Figure S2. Box plot of times to complete the VIOLA certification tests.

Click here for additional data file.