Creating a "Good Clinical Practices Inspection" Authentic Online Learning Environment through Educational Design Research.

Educational design research (EDR) was applied to address the challenge of providing more public health professionals with opportunities to develop their knowledge and skills related to Good Clinical Practices (GCP) inspections. The conduct of clinical research in accordance with the principles of GCP is necessary to ensure that clinical research participants are not exposed to undue risk, and that data generated from the research are valid and accurate. The safety and efficacy of pharmaceuticals, medical devices, and other healthcare products depend upon the application of GCP. The World Health Organization has been providing a global face-to-face GCP Inspection course in Indonesia for a decade, but budget constraints only allow this course to be offered once a year to 15 participants. This paper describes how the move from a face-to-face course to an online authentic learning environment was accomplished through EDR and concludes with a discussion of design principles derived from the research. The results of a beta test of the new online course with 12 participants are also described in the paper. © Association for Educational Communications & Technology 2020.

Although clinical research is the costliest aspect of the entire development process for new drugs, it is a necessary investment because clinical research provides the best evidence to establish the safety and effectiveness of health and medical products and practices (Collier 2009; Eichler and Sweeney 2018). Much of what is known today about the safety and efficacy of specific medical products and treatments has come from randomized controlled trials (RCTs), the “gold standard” for studies of this kind (Concato et al. 2000). RCTs form the foundation for “evidence-based medicine,” but such research can be relied upon only if these studies are conducted according to principles and standards collectively referred to as “Good Clinical Practice” (GCP) (Shalala 2000; World Health Organization 2005; Council for International Organizations of Medical Sciences 2016). The conduct of clinical research in accordance with the principles of GCP helps to ensure that clinical research participants are not exposed to undue risk, and that data generated from the research are valid and accurate (Institute of Medicine 1999; Bhatt 2011; Garattini et al. 2016). By providing a basis both for the scientific and ethical integrity of research involving human subjects and for generating valid observations and sound documentation of the findings, GCP not only serves the interests of the parties actively involved in the research process, but also protects the safety and wellbeing of people who will be prescribed the tested drugs in the future.

Traditionally, the system of quality assurance has relied on audits and inspection of the clinical trial sites by relevant regulatory authorities. Developing the knowledge, skills, and attitudes of inspectors so they can perform their jobs competently—safely, effectively, and efficiently—is important in any endeavor. When it comes to clinical trials involving human subjects, ensuring that international ethical principles are observed, that the rights, integrity and confidentiality of trial subjects are respected and protected, and that clinical trial data are collected in a reliable and valid manner, the competency of professional staff responsible with oversight of good clinical practices is exceptionally critical because failures can have devastating effects (Clark et al. 2019; Eichler and Sweeney 2018).

Advances in the field of medicine are dependent on the quality of research that is conducted, especially with respect to clinical trials. The “Public Engagement and Clinical Trials” report released by the U.S. Institute of Medicine in 2012 points out,

“Clinical trials are the linking step that enables basic research findings to emerge at the patient’s bed-side and the physicians’ examining rooms. The questions clinical trials seek to answer change over time, depending on advances in basic research and the population health problems they are intended to address. Potential new treatments must be tested in humans in order to find out whether they succeed and whether they cause harm.” (p. 1).

On the regulatory front, experience shows that deviations from approved trial protocols are common during clinical trials, some with significant detrimental impact on patient safety (European Medicines Agency 2012). The haunting past of devastating human research should never be forgotten (Weyzig and Schnipper 2008; Bansal et al. 2015). As this paper is being written, health professionals around the world are racing to create and test new vaccines for SARS-CoV-2 in the wake of the COVID-19 pandemic. Although this urgency is justified, GCP must be followed (Lanese 2020).

GCP Inspection Course: From Face-to-Face to Online

An important set of competencies that professionals in the world of pharmaceutical research and regulatory agencies must possess are related to GCP inspections (WHO, 1995). For the past ten years, the World Health Organization (WHO) has been helping regulatory agencies to expand and enhance GCP inspectors’ skills through face-to-face GCP inspection courses provided at its Global Learning Opportunities (GLO) learning centre at the Indonesian regulatory agency (Badan POM). The week-long course begins with an interactive three days in the classroom followed on the fourth day by a field activity whereby participants are taken to a clinical trial site for a mock inspection. The course participants then report back their findings from the mock inspection on day 5. The course has been typically offered once a year, as part of a clinical trial package along with clinical trial authorization and clinical data evaluation for registration courses.

In order to reach more participants and reduce costs, WHO worked with Badan POM to convert the face-to-face course into an authentic online learning opportunity using an educational design research (EDR) approach (McKenney and Reeves 2012). The beta version of the online course was first offered between September 2018 and March 2019 to 12 participants who had successfully completed the previous face-to-face course (EPELA 2019). Although the face-to-face and online courses have great deal in common, there are important differences in objectives, learning activities, tools, and technologies deployed in the two courses. This paper describes how the move from a face-to-face course to an online authentic learning environment was accomplished and concludes with a discussion of design principles derived from the EDR process.

Research Approach

A genre of educational research known variously as educational design research (EDR) (McKenney and Reeves 2012, 2019), design-based research (DBR) (Barab and Squire 2004), and by other names was applied in this project. EDR ideally begins with the identification of a serious problem relevant to practitioners. The main problem addressed by this EDR initiative was that because of budgetary and logistical issues, the physical GCP inspection course could only be offered once annually and could accommodate no more than 15 learners per year. There was a clear need to provide GCP Inspection learning opportunities more often than once a year and to accommodate more learners.

As defined by McKenney and Reeves (2012, 2019), one major outcome of EDR is a robust innovation, in this case, an authentic GCP inspection eLearning course. The second major outcome of EDR is new theoretical knowledge, in this case, a set of refined design principles that others may be able to use in future efforts to convert traditional courses into authentic online learning environments.

This EDR project moved through three major phases: 1) Analysis and Exploration, 2) Design and Construction, and 3) Evaluation and Reflection. The Analysis and Exploration phase of this EDR initiative began in September 2017 with the identification of initial design principles based upon the authentic tasks eLearning model developed by Herrington, Reeves and Oliver (2010). These design principles were derived from previous course conversion projects carried out by the WHO (cf. Kartoğlu, Vesper, & Reeves, 2017) and an updated review of the literature. In addition to design principles, additional outcomes from this first phase included a clear set of learning objectives for the online GCP inspection course and a set of technical specifications for interactive affordances deemed desirable for the online learning environment.

The Design and Construction phase included applying the authentic learning design principles to the prototyping of an online GCP inspection course, from rough drafts on paper to interactive beta versions of the course using a new online platform called EPELA (Extentio et Progressio, Authentic e-Learning) (http://www.epela.net/). For example, one of the primary design principles delineated by Herrington et al. (2010) is that authentic tasks should be seamlessly integrated with assessment. This is accomplished in the second half of the course when participants collaborate to conduct a GCP inspection review for a real-world client. The primary outcome of this second phase of the EDR initiative was a prototype e-learning GCP inspection course.

The Evaluation and Reflection phase encompassed multiple iterations of expert review, risk assessment, field testing, and other formative evaluation strategies using mixed-methods research strategies (Johnson et al. 2007) to test and refine the prototype e-learning course. As described below, a beta test of the refined online course with 12 participants was an important part of this third phase.

Analysis of Two Different Delivery Modes

The face-to-face GCP Inspection course takes a total of 27 hrs, conducted consecutively during a working week, divided into seven sessions of between 80 and 280 min each, using a variety of learning methods including illustrated lectures, demonstrations, brainstorming, small group work, readings, role plays, simulations and games. The on-site mock GCP inspection occurs on the fourth day of the week (360 min in total), and fifth and last day of the course provides the course participants with the opportunity to report the results of their mock inspection from the day before.

By contrast, participants in the online version of the GCP inspection course, learn within the context of an interactive simulation of a clinical trial spread over 24 weeks, virtually walking through all aspects of a trial together with a small team of online co-learners. The authentic tasks encompassed in the online simulation are focused on application of GCP principles, maintaining a quality management system, engaging in risk-based thinking, preparing for an inspection, carrying out an inspection, and reporting the results of the inspection. These online activities are mainly arranged as collaborative learning activities for two or three participants working together. Figure 1 presents a sample page from the online GCP inspection course.

Fig. 1

A sample page from the online GCP inspection course (Module 3 – Risk based thinking)

Both courses have pre-course activities intended to engage participants and to encourage them to get connected with each other in a supportive manner. In the face-to-face course, various hands-on games and activities are conducted to promote learner engagement (Kartoğlu 2018) whereas in the online course, a ‘Two truths and one lie’ game is played online seven days prior to starting the online course. Online participants also use Flipgrid (Green and Green 2018) to present themselves and express their expectations from the course in video format not to exceed 90 secs (https://flipgrid.com/3b1f95). In each version of the course, an anonymous pre-course questionnaire is distributed to assess the knowledge of the group, to provide an overview of the contents, and to whet participants’ appetite to learn. In the face-to-face course, the questionnaire responses are totaled and displayed to the participants on a flipchart whereas in the online course, a group performance matrix showing the results both in absolute numbers and percentages is displayed online after all participants have completed the questionnaire.

Although the physical course is quite interactive, the content in the face-to-face course is presented in a somewhat prescribed manner and follows more of a compartmentalized fashion of a clinical trial anatomy. Following a classical introduction to GCP, other topics such as organization/management, documentation, informed consent, facility, and report type are presented to the participants. By contrast, the online course presents the content components more broadly and delves into details through engagement in authentic tasks. Even in the first online module that introduces GCP principles, online participants are challenged with four case studies (excerpted from magazines and newspapers) to review and identify risks that may impact noncompliance with GCP principles. Following this introduction, quality management system, risk-based thinking, preparing for an inspection, and inspection modules are presented, each infused with authentic activities as well as didactic resources such a brief video lectures and readings.

Although the face-to-face GCP inspection course and the online course have a great deal in common, there are significant differences. The face-to-face course focuses on the theory first and then walks participants through the principles with the help of various games, group work, and discussions. The online course differs greatly in focus by enhancing and expanding the competencies that will make participants “better” inspectors. In doing so, the online course introduces a “risk-based” approach to inspections. The analysis of the objectives of the two versions of the GCP inspection course presented in Table 1 reveals important differences. For example, the face-to-face course has 68% lower-level cognitive skills objectives and 32% higher-level cognitive skills objectives whereas the online course has 44% lower-level cognitive skills objectives and 56% higher-level cognitive skills objectives.

Table 1

Analysis of objectives in two versions of GCP inspection course based on cognitive process dimensions defined by Anderson and Krathwohl (2001)

Level	Cognitive skill	Number of objectives
		Face-to-face course	Online course
Higher	Creating	2	6
	Evaluating	7	9
	Analyzing	2	2
Lower	Applying	1	6
	Understanding	13	6
	Remembering	3	1

The online course lasts 24 weeks with an expectation that participants will engage with the course activities 6–10 hrs per week. For the first nine weeks, the learners sharpen their inspection skills by tackling simulated realistic problems, working individually in a few cases, but more often collaborating in small groups. The final 12.5 weeks of the online course involve a complex and highly authentic real-world task. During this time, participants in groups of three, review actual documents, plan an inspection visit by defining roles and responsibilities of the inspection team, conduct an opening meeting, interview critical research staff, conduct a full inspection, evaluate and report on findings, and present findings to the real client.

An ongoing clinical trial conducted by MCRI (Murdoch Children’s Research Institute, Parkville, Victoria, Australia), BioFarma (Bandung, Indonesia), and RSUP Dr. Soeradji Tirtonegoro Hospital (Yogyakarta, Indonesia) is used as the authentic case. This case is a phase IIb, double-blind, randomized, placebo-controlled parallel group study of the efficacy, safety and immunogenicity of an oral candidate rotavirus vaccine. For data integrity and training purposes, all records are modified to protect the identities of participants.

The most important advantage of the online course over the face-to-face course is introducing an authentic clinical trial client to participants with full access to all documents and staff. Although the face-to-face course participants are taken to a real clinical trials site, in terms of timing, the mock up inspection that takes place onsite is quite time limited (six hours), whereas in the online course, participants have ample time to review and digest every single document available from the clinical trial site during the 88 days allocated for completing the inspection authentic task. Figure 2 illustrates the flow and details of the online course. Table 2 summarizes the activities, tools and technologies used in face-to-face and online GCP inspection courses.

Fig. 2

The online course flow and details

Table 2

Summary of the activities, tools and technologies used in face-to-face and online GCP inspection courses

Description	Face-to-face course	Online course
Pre-course survey for participants	None	A 12-question survey is sent to all participants to evaluate their previous e-learning experiences.
Emails	Emails are used for communicating with participants on acceptance, sharing documents and links prior to their attendance to the course.	Emails are the main channel of communication before, during and after the online course. It is used for guidance throughout the course, reminding about the deadlines, sharing mentor’s observations in summary format for each task. Mentors also communicate with participants on an individual basis on their diary entries.
Icebreaker	A hands-on icebreaker activity is introduced at the very start of the course for everyone to get to know each other better.	An online icebreaker activity is introduced at the very start of the course for everyone to get to know each other better.
Introduction	Introduction session covers expectations and objectives. Participants work in groups to express their expectations (anonymously) from the course. Facilitators compare participants’ expectations with the course objectives to highlight expectations that will not be covered during the course and reach a verbal agreement on the objectives of the course.	Participants present themselves and express their expectations from the course in Flipgrid videos as a pre-course activity. A Skype audioconference is attended by all mentors and participants to explain the workflow, and expectations from participants.
Pre-course questionnaire	A questionnaire with 33 true/false statements is given to all participants to provide an overview of the contents before starting the course and stimulating their thinking about the GCP subject areas. Responses are collected anonymously to mark them in a matrix to understand the group’s performance to help facilitators to make final touches to the program.	An online questionnaire with 30 true/false statements is answered by all participants. Once all participants answer, the performance matrix is automatically displayed on screen.
Document library	Selected documents are shared with participants prior to the course to prepare them for the work ahead during the course.	All participants have access to document libraries and are directed to specific documents for each task.
Video library	‘The Constant Gardener’ movie is watched by the whole group on Wednesday evening (Day 3) and a discussion is held on the clinical trial aspect of it.	Short (not to exceed 10 min) expert videos are produced specifically for the course to support the theory. In addition, some case studies are presented in video format. As for the last assignment (virtual inspection of a clinical trial), a facility tour video helps participants to familiarize themselves with the facility and processes.
360 degrees spherical photography	None	360 degrees spherical photography is provided for critical locations in the clinical trial site for the last assignment. These photographs put the participant in the middle, and they can turn around, zoom in and out to see all details in the location.
Illustrated lecture	Materials are presented in illustrated lectures where a facilitator uses audiovisual aids for a facilitator to teach and help participants to learn, such as flip charts, overheads and other materials for the session.	There are no lectures as such in the online course.
Learning diary	None	Participants are required to create a learning diary, specifically at the end of each block (module) and during the final authentic task. They are asked to consider what they have learned, what specifically they are going to take back to work, and comments. In these diaries, participants also reflect on what has worked, and what has not worked during the tasks and how would they do it if they face a similar situation. All diaries are read by mentors and feedback is given via email on an individual and private basis.
Discussion forum	There are plenary discussions in face-to-face course. These discussions are based on the planned activities.	An online discussion forum is used mainly by participants raising questions and commenting on the posts by others. Mentors intervene either to summarize the discussion or when an expert view is required.
Mid-course questionnaire	A multiple-choice questionnaire containing 34 questions is given to participants on Day 4 afternoon and they are required to write down their names. The main objective of this questionnaire is for mentors to meet each and every participant before they leave the course and discuss the answers provided by participants privately. This allows a detailed review of subjects covered during the course. The whole idea is for all to come to same understanding regarding certain course subjects.	None
Final assignment	None	Following nine weeks of work, participants in teams are requested to work as GCP inspectors for the given clinical trial. They plan and conduct the inspection and present their findings followed by a discussion.
Participants’ performance evaluation and certification	All participants who attend all sessions of the course receive a ‘participation’ certificate. Their performance is evaluated through facilitators’ observations during group work, presentations, and discussions. Facilitators meet on a daily basis to evaluate the day’s activities as well as discussing participants’ performance and decide on an action plan if necessary to motivate and involve participants more on coming days.	Authentic assessment is used wherein the assessment is integrated within the task, rather than in the form of a separate test. Authentic assessment is designed to focus on completion of a real-world task and does not hold any individual’s work up against the work of others. In other words, authentic assessment identifies strengths and weaknesses with respect to nature of the task, but does not compare or rank learners. Attendance in this authentic e-learning program does not qualify participants to receive a certificate. To receive a “successful completion” certificate, participants must complete all tasks, comment on work of other participants and groups (peer-review), participate in discussions raised in the Discussion Forum by mentors and other participants, start at least one discussion in Discussion Forum, and keep a diary for each module.
Letter to myself	At the end of the course, participants are requested to write a letter to themselves (in their local language if they wish) as a reminder for the things they are planning to implement within three months of their return back to work. These letters are sealed and collected back and mailed to participants in three months by the course management.	None
Course evaluation	Course evaluation is done through a written form containing eight questions, mostly open-ended and with two questions with ranking. The results are compiled and shared with the whole group via email. Following written evaluation, all participants and facilitators are asked to reflect on the week’s work and in turn they all speak on their feelings. These sessions are recorded and an edited short version is released publicly through Vimeo/YouTube (Refer to http://bit.ly/2Ga14x1 for 2017 GCP inspection course reflections – running time 09:33 min)	Course evaluation is done through a Skype audio conference with participation of all mentors and participants.

The learning diary incorporated in the online version is a very supportive tool that promotes learning, but it is not incorporated into the face-to-face course. At the end of each online module and following the final project, participants are asked to pause and reflect on what they have learned in this particular section, which things they are going to take back and use in their work settings, and what would they do differently if they face similar problems next time. This allows them to return to the experience by recollecting the salient features of the experience and attending to feelings by sharing both positive and negative feelings about the experience. The learning diary also allows them to re-evaluate the experience by integrating their new knowledge into their conceptual framework. Reflecting on learning experience through the learning diary is critical throughout learning process (Kolb and Fry 1975; Schön 1987). Participants’ diaries are periodically read by mentors only and feedback is provided on a one-on-one basis.

Refinement of the Authentic Online Learning Design Principles

Several years before the EDR initiative described in this paper began, the World Health Organization employed a similar approach in the process of converting an award-winning experiential learning course that took place on a bus traveling down the “cold chain” for time- and temperature-sensitive pharmaceutical products in Turkey to an online interactive experiential learning environment (Kartoğlu, Vesper, & Reeves, 2016). As defined by McKenney and Reeves (2012), educational design research generally has two major outcomes: a robust innovation (that is the online course itself), and a set of design principles that can be applied in future projects. In moving the unique experiential learning environment of the bus course into an online environment, Kartoğlu et al. (2016) reported three central design principles, and these same principles were followed and refined in creating the GCP inspection authentic eLearning environment.

Design Principle 1: Rather than Perfectly Duplicate, Replicate where Possible and Innovate where Necessary

Creating an online learning environment based on a face-to-face course demands that designers and mentors understand both the strengths and weaknesses of each delivery method. In such conversions, some things simply cannot be duplicated, cloned, or mirrored since being in a classroom and sitting in front of a computer connected to Internet are very different experiences. Nonetheless, some activities were possible to be integrated into both course modalities. For example, in the ‘Introduction to GCP Principles,’ a card game requires participants to match a total of 14 GCP principles with 14 corresponding applications of these GCP principles. In the face-to-face course, the game is played with cards attached to the wall and participants matched 14 cards in small groups. Then, each group is visited by the other groups and facilitators, and the results were discussed. In the online version, we added more challenge in that anytime the participant made a mistake in matching, he/she lost everything that have been correctly done before, thus forcing them to start again. This matching activity in the face-to-face course takes 30 min, however, in the online course, the time is not prespecified although learners must complete it within 2 days. Figure 3 illustrates the online version of the GCP principles and applications matching activity.

Fig. 3

Online version of the matching cards game

The beta-course of the online course initially enrolled 12 participants who had previously participated in 2017 face-to-face GCP inspection course. Within the first month of the beta-course, six participants became unresponsive and dropped out of the course. This attrition level was not unexpected because the online course required 6–10 hrs of work per week, and earlier beta tests of other WHO online courses had experienced similar attrition levels. It is a large time commitment for working professionals, all of whom had completed the face-to-face version of the course beforehand. Although the beta-test continued with the remaining six volunteers, three of them were much more active than the other three. One of the active participants said:

“I could see that the e-course was designed to be as interactive as the conventional course and this is amazing. I can imagine how hard work did it take to design an electronic course to be highly interactive like this! The activities are fun and the case studies are very challenging, I like that!”

Another participant mentioned mentors commenting on participants’ work summarizing real world tasks in a paper:

“I like that the mentors comment on our work on each single part in each task, this is so helpful.”

Participants also mentioned how online resources contributed to their learning:

“Last year in Jakarta we kept repeating the phrase ‘risk-based thinking’ but honestly I didn’t really get what it meant and how it applied in clinical trials. This time, in this e-learning course, the two presentations (it was genius), the tasks and the references given helped me to understand the concept.”

“I appreciate that the course is not only aiming to give us knowledge but also critical skills.”

“I liked the idea of the task in the form of a video that we watch and then comment on it. I feel like I am part of the video, it is so real.”

When designing the online version of the face-to-face GCP inspection course, the design team strove to take advantage of what the technology allowed the learners to do. For example, the learning diary online participants kept fostered their engagement in reflection and meta-cognition over time. In the face-to-face course, such level of reflection probably occurred only after the learners have returned back to their jobs. Another advantage of the online course is affording ample study time for each task. The face-to-face course is quite intense, and it is challenging for slower pace participants to keep up with time-limited tasks. Although the online course has defined periods for each task to be completed, the arrangements are so flexible that participants could arrange their own timing. For example, for a task that needs to be completed in six days, this is typically explained as follows:

“You have to complete this assignment in six days. Although six days are allocated to complete this assignment, the actual amount of time required is estimated at around 3 hours or less.”

The reflection phase of this EDR project indicated that the first Design principle (Rather than perfectly duplicate, replicate where possible and innovate where necessary.) did not require modification. But a corollary design principle did emerge: When innovating, seek to maximize the authenticity of the learner’s experience. We strove to do this by engaging participants in completing an GCP inspection for an actual real world client. The slogan of the e-learning design team at the WHO is “Go authentic,” signifying the attempt to always seek to instill as much authenticity as possible into each and every learning task.

Design Principle 2: The Collaboration that Is Essential to Instantiating Authentic Tasks-Based Learning Strategies Online Is a New Experience for Most Learners and Must Be Carefully Nurtured

The online course has a total of 13 tasks with completion times ranging from a day (solving a crossword puzzle – individual task) to 88 days (MCRI/BioFarma clinical trial inspection – group task). Naturally, individual tasks required less time compared to group tasks. For example, case studies with similar weight had to be completed within 2–3 days for individual tasks, but participants were given 6 days for group work. Group work was always a challenge for participants being in different time zones and having different cultural backgrounds. Some groups were more successful in coming together than others. For example, one group came up with an additional means of communication among the group members by using WhatsApp. It was critical to watch how participants were collaborating to scaffold their interactions when needed. This was mainly done by the course manager.

Reflection on the second design principle (The collaboration that is essential to instantiating authentic tasks-based learning strategies online is a new experience for most learners and must be carefully nurtured.) also did not indicate a need to change the principle, but another corollary principle emerged: Even when course participants have worked together before, constant monitoring of their collaboration is needed to ensure that authentic teamwork is supported. This monitoring requirement means that the course manager must be prepared to work even more hours than the participants to stay informed about what is happening across the different small groups of learners.

Design Principle 3: The Fidelity of the Simulated Experiential Learning Environment Does Not Have to Be Exceptionally High as Long as it Enables Learners to Suspend Disbelief and Feel that What they Are Experiencing Is Real

Since the replication of actual physical environments is not possible in an online environment, it requires some degree of suspension of disbelief by the participants (Herrington et al. 2003). If well-designed and carefully produced, technology can contribute to participants’ suspension of disbelief and increase their engagement with the course. In this sense, the online facility video that takes participants through all steps of the clinical trial and the 360-degree spherical photography of major work areas in the hospital help participants to feel as if they are really in the facility. One participant reported:

“[On 360-degree spherical photographs] It is so real, I turn wherever I want and I can even see it closer [zoom in function]”.

Of course, technology can only help to a certain extent. The course managers and mentors provide ‘the human touch’ that values each and every participant as fellow professionals while also serving as the more knowledgeable other. The effectiveness of this online course is heavily dependent upon participants feeling connected to the mentors. One participant wrote:

“I would like to thank Christine [mentor] for all her comments and her care to respond to everyone.”

Reflecting on this third design principle (The fidelity of the simulated experiential learning environment does not have to be exceptionally high as long as it enables learners to suspend disbelief and feel that what they are experiencing is real.) suggested an important corollary design principle: Even when participants successfully suspend disbelief and experience the learning environment as authentic, highly skilled mentors are necessary to sustain learner engagement. Although the new e-learning course makes the GCP learning opportunities more accessible to learners from around the world, it greatly increases the demands on the course manager and mentors.

Additional Findings from the Beta Test of the Online Course

The beta course was conducted to identify any specific challenges that might hinder the provision of an efficient and successful eLearning GCP inspection course. All participants who stayed with the course found that the course structure and communication were not difficult, however, remaining challenges are being addressed to provide professional development both for mentors and course participants:

Time management: The participants preferred to take the course when at home. They faced difficulty engaging with the course and completing online assignments when at work or while travelling on professional missions. Time periods were suggested for each task according to its complexity and whether the task involves individual or group work. The course manager sent reminders about deadlines regularly.

Internet access: The biggest concern was the poor access to the Internet. Given that the participants in the online course primarily come from developing countries, this will remain a challenge for the foreseeable future.

Time gap between countries: The time difference among the participants varied from 1 to 8 hrs depending on their location. This time gap complicates organizing groups. In order to change the grouping from module to module, time differences were taken into account, however, there were cases where bigger gaps were inevitable.

Language barrier: For most developing countries, English is not a first language. Therefore, it takes time to understand tasks. The online course uses quite simple language to ease the comprehension. Some sections were rewritten based on the comments received from beta-course participants for the coming courses. The course also provides ample time for discussion. Mentors contact participants if there are delays in the submission of their work.

Different levels of knowledge and experiences: Differences in expertise and experience can hinder active participation if some participants feel they are not up to the task due to less experience and or believe that they are unable to complete assignments. Collaborative learning was used to support “the more knowledgeable other” notion (Vygotsky 1978). Mentors were also active in communicating with participants and scaffolding their interactions whenever needed.

Protection to subjects’ (in clinical research) privacy and confidentiality: We had to modify all records used in the major authentic task in order to protect the subjects, without eliminating the authenticity of the real clinical research. It was a formidable task in terms of number and volume of documents to be modified as well as introducing different level of mishaps in various documents.

Anxiety associated with technologies such as recording video, online discussion, online submission, teleconference, video conference: Ideally, the mentors can help low tech participants engage in the effective use of all technologies and encourage participants to practice tech skills on their own. However, our experience shows mentors are often not as confident in their tech skills as we expected, and thus more experienced mentors must also help other mentors who may be still learning to use some of the platforms and tools.

Cost Effectiveness

A face-to-face GCP inspection course for 15 participants and 4 mentors costs approximately 60,000 USD including the core expenses, travel and per diem of all participants (depending where they come from). With current budget restraints, only one course is offered in a calendar year. Development of the eLearning version of the course required an approximately 110,000 USD investment and involved 18 months of time from project conceptualization to completion. Since the course is 24 weeks long, it can only be offered twice a year. Each course preferably enrolls not more than 20 participants in order to have close interaction and appropriate scaffolding since the course is not a self-paced one. Annual running costs are expected to be USD 15,000. The availability of the online course affords participants who might not be able to travel to Indonesia opportunities to benefit from this important learning experience.

Future Research

The authors are continuing to conduct EDR to meet the twofold goals of improving professional development opportunities for people involved in GCP inspections, vaccine management (Kartoğlu et al. 2017), and other areas such as medical education (Chen and Reeves 2019) while at the same time refining the design principles underlying the development of authentic e-learning.

Limitations

This paper describes only one cycle in an ongoing iterative process of testing and refining the GCP authentic online learning environment described in this paper. Most of the data comes from self-reports from course participants and observations from mentors.

Final Reflections

Could the online version of the GCP inspection course have been developed using a traditional instructional design approach such as ADDIE (Branch 2009)? This does not seem likely (Oh and Reeves 2010). ADDIE has been justifiably criticized for being too heavily front loaded with analysis and for not weaving formative evaluation into the design process as early and as frequently as is done in EDR (Allen and Sites 2012). The experience of this project and other efforts to develop truly innovative authentic e-learning courses carried out by the World Health Organization over the past decade demonstrates the value of employing EDR rather than traditional instructional design techniques alone. EDR evolves through three main phases, each of which may be repeated multiple times: analysis/exploration, design/construction, and evaluation/reflection.

In the first phase of EDR, researchers, designers, and practitioners collaborate closely with one another to learn about the root causes of the problem being tackled. They achieve a depth of understanding of the needs the course must address far greater than achieved through typical needs assessment activities.

During the design/construction phase, a creative, multidisciplinary team reviews the theoretical knowledge relevant to the problem; brainstorms innovative solutions; considers various options; and creates prototypes to try in real educational settings. The extent of prototyping involved in EDR encourages designing the ideal first, and later dealing with the real.

Once a viable prototype solution is developed, iterative formative evaluations take place to test and revise both the design and the ideas/assumptions on which it is built. Multiple methods are applied including observations, interviews, questionnaires, usability studies, quasi-experiments, and full scale beta tests.

In addition, throughout the EDR process, theoretical knowledge is utilized and refined. For the World Health Organization, EDR has yielded several award-winning e-learning courses that are distinctly different from most other online courses found in the public health sector. We encourage readers of this paper to try EDR and to adopt our motto, “Go authentic.”