Consensus Meeting Report "Technology Enhanced Assessment" in Covid-19 Time, MENA Regional Experiences and Reflections.

PURPOSE: Majority of the assessments, appraisals and placements have been disturbed, with some being cancelled, postponed, or modified in design. New approaches for assessment should be well-thought-out. This work attempts at capturing the collective wisdom of educators in the Middle East and North Africa region (MENA), providing an understanding of the online assessment conceptual framework in the era of COVID-19 that tells the story rather than determining cause and effect, and identifying the biggest gaps that derail the digital transformation.
METHODS: A qualitative inductive study using the grounded theory approach was implemented following a synchronous virtual online meeting, a summary of the reflections as well as experiences of medical education experts was prepared. Data for this qualitative study were collected from the meeting. The meeting was video-recorded and transcribed by the researchers. Thematic analysis was performed by three separate researcher coders. The authors then discussed together until they reached a consensus.
RESULTS: Three main thematic areas were identified: 1) feasibility, 2) exam fairness/equity and 3) acceptable graduate attributes, society/community acceptance.
CONCLUSION: The COVID-19 era necessitated revisiting of our assessment strategies to cope with new changes within the available context. Rapid adaptation is required.

Introduction

Global pandemic COVID-19 has resulted in innovative public health measures including world measures: school closures, lockdown of educational and public institutions, self-isolation for sick individuals, and most recently “lockdown” this also includes assessments and placements.1 Traditional exams have been dropped for thousands of students; universities have moved at pace to transform the assessment processes, while some were cancelled, postponed, or changed in format.2 There was an agreement that holding modified assessments this year is better for students in the long term than overriding assessments to a later date or cancelling them completely. This will give students the best possible chance to make progress better than delaying the assessment with all the burden and worries to a later unspecified date.3 There are essentially two reasons why assessment of student learning is important; assessment is needed for improvement, and for recognizing what students know, and what they do not know for effective learning.4

The benefits of using Technology Enhanced Assessment (TEA) are manifold such as improving authenticity as well as alignment with learning outcomes, helping to clarify marking criteria, and spreading the assessment load for staff and students. However, even more compelling, education technology can also be used to aid formative assessment; helping to boost engagement, identify knowledge gaps, track student progress over time, and support further/deeper learning.5 There is clearly an urgent need to re-institute on-site teaching and kick-start curricula, yet the pandemic continues to pose a grave threat to public health (and will do so for many months to come as the world emerges from its extended national lockdown in a phased approach, educational institutions across the world are faced with a difficult and complex task.6 Revisiting old curricula and modalities is advised to work backwards on design and process within the context. Revisiting old curricula and modalities is advised to work backwards on design and process within the context. This includes new approaches for assessment to be instituted in the existing framework of assessment 7 Standard assessment formats in medical school include written exams (like Multiple Choice Questions (MCQs), Single answer Questions (SBA) and EMQs (Extended Matching Questions) and practical skills via Objective Structured Clinical Examination (OSCE).7 There are minimum requirements that needs to be met: 1) Relevance: making sure there is a proper choice of tools used; 2) Effectiveness: in making sure needs are met of different geographical origins; 3) Trustworthiness: making sure there is a degree of confidence in choices made.

Purpose

This work attempts at capturing the collective wisdom of educators in the MENA region and provides an understanding of the online assessment conceptual framework in the era of COVID-19 pandemic that tells the story rather than determining cause and effect and identifies the biggest gaps that derail the digital transformation.

Setting

The Ain Shams University Middle East North Africa-FAIMER Regional Institute (ASU-MENA-FRI) is a regional institute for the Middle East and North Africa, for academic leaders. The institute offers a multi-session program in a hybrid design offering both in-site and distant training, hosted by the Ain Shams University Faculty of Medicine in Cairo, Egypt. Mentor-Learner web (ML web) which is the method of teaching that emphasizes the reflection on one’s learning aiming to improve efficiency of learning and professional development. Learning through Participation in the M-L web is also one of the key requirements for certification of completion of the FAIMER Fellowship.

Materials and Methods

Study Design

Qualitative grounded theory design from the conducted meeting on “the use of technology in virtual assessment” was used in this study.

Participants

Eleven FAIMER fellows and four FAIMER faculty with previous medical education background oversaw conducting and moderating the meeting. Medical educators engaged in medical training and assessment were invited to participate, specifically FAIMER fellows who were enrolled in an online technology enhanced learning module for one month, experts in online knowledge assessment, expert in online clinical assessment, regional experts and faculties from different medical institutions.

Sampling Technique

One hundred medical educator participants attended the meeting from 8 countries attended the session Purposive sampling technique was performed to provide a range of views on online assessment.

Time Frame

Preparation for the event took 2 months and the Announcement took place for 2 weeks to allow registration to the event. This was followed by the creation of a WhatsApp group for the participants three days before the event to give enough time for technical and administrative instructions.

Data Collection

Following the creation of a WhatsApp group, a panel discussion entitled “The use of Technology in virtual assessment was established. This panel was part of ASU-MENA-FRI-Technology Enhanced Learning (TEL) mentoring and learning (ML) web activities. The discussion meeting was held on the 2nd of May 2020 via a Microsoft Office application. The meeting report methodology is anchored according to the Consolidated Criteria for Reporting Qualitative Research (COREQ) checklist.8 Meeting arrangements included announcement, registration, enrollment, and implementation. Announcement was disseminated to the experts through the ASU MENA FRI Official Communication Channels. Interested participants registered. Once registration was received, participants were added to a WhatsApp group for preparation of the meeting, guidance, communication, inquiries for the expert panelists and discussion. The meeting was conducted on Microsoft Office teams with introduction of the objective of the study, then the 1st talked about virtual knowledge assessment for 10 minutes describing the solutions and possibilities done so far. That was followed by discussion and comments for 20 minutes from participants. The 2nd talk was tackling virtual clinical assessment for 10 minutes describing the solutions and possibilities done so far followed by inquiries from participants ending with a wrap up and key points from the participant perspectives. The discussion was in English and lasted for 1 hour. Even though there was a moderator, invitations for comments were pursued on open ended format, and participants were free to follow their priorities in the discussion. Clarifications and summaries were made at regular intervals and at the end of the discussion by the moderator, who ensured that equal participation. During the webinar, all participants’ contributions, questions, and reflections in the chat box were transcribed and analyzed with other data from an online TEL module. Themes and categories were interpretative and constructed by the researchers from data collection.

Analysis of the Findings

The recorded meeting discussions were then transcribed verbatim, word by word. We include in brackets () if there is silence, laugh or coughs. We stated clearly if it is the interviewer or the respondent talking. Comments or question by the Interviewer was labeled by typing I: and any comments or responses from the respondent was labeled with R. then the data was uploaded to a Google document. Authors, who were part of the main event, read through the transcribed meeting to ensure its accuracy. Data coding was done by three independent qualified coders using writing in margins. The initial number of the created codes was 9 codes, and it is mainly dependent on the other research question. They examined the data by open codes methods, ie, line by line to construct the meaning. The initial codes were reduced three timed and refined to codes that are mutually exclusive and include all raw data that was identified as usable. The type of the codes used are In Vivo (inductive or grounded) codes that derived from the data by using code names drawn from participant quotes or interpretation of the data. The key points were identified by repetition – of the same word/phrase, synonyms, technical terminology words/phrases and linguistic connectors – eg because, instead, similar as the coding progresses they were clustered in similar categories and units that share similar meanings or concepts of related data in several cycles, which created meaning for those clusters with labels. This is followed by classification and categorizing repeatedly to allow for deeper immersion. The preliminary classification categorizes raw data into groupings-themes. The themes expressed meanings and condensed into shorter formulations which reduce the text but maintain the context. As themes emerge, analysis becomes more organized and structured, and the method of analysis chosen for this study was an inductive approach based on the grounded theory. A thematic index of all the codes was made and written on the margins of the transcription. The authors then discussed together until they reached a consensus. This is followed by interpretation of meaning beyond organizing to deeper and creative’ interpretations of the texts.

Ethical Considerations

Ethical approval was obtained from the ASU MENA FRI institute and Participants’ informed consent including publication of anonymized responses was obtained.

Results

One hundred medical educator participants attended the meeting from 7 countries (Table 1).

Table 1

Participant’s Characteristics

Participant’s characteristics
Sex:
Male	24
Female	76
Academic profession:
Assistant lecturer	5
Assistant professor	34
Lecturer	36
Professor	25
Participating countries:
Egypt	38
Bahrain	3
Iraq	23
KSA	30
Kurdistan	2
Libya	2
Sudan	2

Abbreviations: ASU MENA FRI, Ain Shams University - Middle East North Africa – FAIMER Regional Institute; COREQ, Consolidated Criteria for Reporting Qualitative Research; EMQs, Extended Matching Questions; MCQ, Multiple Choice Questions; MENA, Middle East North Africa; ML web, Mentor-Learner web; OSCE, Objective Structured Clinical Examination; OSPE, Objective structured practical examination; OSVE, Objective structured video exams; SBA, Single answer Questions; TEA, Technology Enhanced Assessment; TEL, Technology Enhanced Learning; VOSCE, virtual Objective Structured Clinical Examination.

Three themes were identified (Figure 1):

Figure 1

Main thematic findings for conducting online assessment.

Feasibility: Do Universities Have the Infrastructure to Do Online Assessment?

Universities are diverse in terms of their available resources; in turn the feasibility of online assessment can differ drastically between countries even between the universities within the same country.

1 – Number of trained assessors: Assessment methods; All these assessment methods can be achieved through any virtual meeting application which has an option of different breakout rooms like Zoom or Microsoft Teams, but it requires an adequate number of well-trained assessors. The availability of a sufficient number of assessors is not applicable in many settings, as it depends on the resources of the university and availability of staff members. Furthermore, assessors should be trained to apply the assessment method and the technology that will be used to evaluate the examined clinical skill. Methods such as online structured VIVA require two assessors per station and require assessors training to increase objectivity and decrease subjectivity, while Objective structured video exams (OSVE) and virtual OSCE require less number.

Online assessment to evaluate clinical skills is considered a revolution in education that enriches the process as it changes the human interaction and gives a way for further success.

Of course, We have to use technology in teaching and assessment, but it needs training.

2 – Connectivity issues: A lot of participants were concerned with whether their universities/servers have what it takes to conduct online assessments. Internet connectivity in some universities is not well established, in addition to some university servers.

It was said that the exam will be affected by the capacity of the server if there is a large cohort … In my opinion, the server here has no role, and I won’t need it … It is solely related to Microsoft.

Most experts from different institutions were academicians, but some were working in the clinical fields and stated that the clinical assessment in some settings was postponed based on Governmental decisions. So virtual clinical assessment was not implemented by all institutions. In addition, Currently, in College of Medicine Dar Al Uloom KSA, the clinical skills teaching, and assessment were completely postponed to the next semester, a situation that was compulsory because of the rapid course of preventive measures taken by the Saudi government to prevent COVID-19 spread, including early suspension of educational sector followed by the partial then the total curfew.

Exam Fairness/Equity

Nothing is perfect, everything has its drawbacks; using technology is no different. Several concerns were aroused, and multiple factors affected the online exams’ fairness, some of which were tackled by the participants.

Exam security and Online proctoring: This subject was a spiky one, because of two reasons: firstly: the ethical obligation and dilemma of accepting that the students will be monitored, and secondly: no proctoring program guarantees a 100% no cheat rate at the exam. One of the participants raised this point: Is it accepted by students and their families to monitor them? Also, some participants expressed a lot of difficulties in using proctoring programs being expensive and requiring a lot of planning and manpower to achieve the most perfect results. Some participants and panelists made some suggestions for decreasing the chance for cheating by performing an open book exam as well as preparing innovative exam logistics to prevent cheating in virtual Objective Structured Clinical Examination (VOSCE).

Even when giving an assignment, how can we assure that this student himself is doing the assignment?

Applications, which allow video meetings, can be used for proctoring during these exams as well, but with proper preparation of the proctoring process.

Psychomotor skills are the most difficult to be evaluated virtually.

70–80% of learning outcomes related to clinical skills could be tested online, except for the psychomotor skills or physical examination skills.

Acceptable Graduate Attributes, Society/Community Acceptance: What It Takes to Become a Physician, Fear of Graduating an Incompetent Physician

Online assessment to evaluate clinical skills is considered a revolution in education that enriches the process as it changes human interaction and gives a way for further success. A variety of tools can be used online to assess students’ clinical skills. As mentioned in the previous theme, clinical skills that need to be evaluated for medical students include history taking, physical examination, data interpretation, management and clinical acumen, communication skills and professionalism. Because of the current situation of COVID, the panel also agreed that there was a need to modify the assessment methods for the innovation and depart from the traditional instruction, using a variety of online assessment tools. Although there were a lot of virtual assessment tools, the experts stated that most (70–80%) of learning outcomes related to clinical skills considering the available virtual clinical skills assessment methods could be tested online, except for (20–30%) the psychomotor skills or physical examination skills. This necessitates blending regular face to face assessment with the virtual assessment to fully evaluate the psychomotor skills and enhance the concept of conducting further regular face to face evaluation once possible. As a matter of fact, the meeting highlighted a basic idea, which is to align the teaching and learning with the assessment process, valuing leadership decisions especially, that most settings used the ordinary clinical skills education method. Moreover, it made us realize how important it is to video record the clinical skills in the future and utilize the recordings in the learning process and in virtual examination as well. Our participants stated that multiple online assessment modalities have been implemented in medical and dental schools. Although VOSCE was considered a reliable solution for assessment, it was not applicable for exit exams. In most medical schools, there are many students; thus, the application of online clinical assessment in exit exams will not ensure that all the required skills are assessed. Due to the current pandemic, there is a concern regarding the availability of the role players, also it will be difficult for the students to attend the clinics to record the consultation for the VOSCE.

The experts concluded a summary of the different tools that can evaluate clinical skills through virtual learning as alternatives to written exams.

Written assessment can assess some psychomotor skills via MCQ, data interpretation and “Script concordance test” which is used to assess clinical reasoning in ambiguous or uncertain situations. It investigates the multiple judgments in real-life situations that are made in the clinical reasoning process.

“Virtual Objective structured practical examination (OSPE) or integrated OSPE”: OSPE stations (clinical cases, videos with positive clinical findings, ECGs, radiographs, or lab results) are uploaded to the LMS and the questions are in the form of MCQs, short answers, matching or ordering questions. In some universities, the virtual integrated OSPE is used to assess the critical thinking and clinical reasoning of senior students, to reach a comprehensive diagnosis and formulate a sequential treatment plan of cases, including different clinical disciplines. The time consumed per each station depends on its content, the difficulty is considerably low, and its validity and reliability are moderate.

OSVE: in which the students watch a video of a clinical examination of one patient with positive findings for the student to diagnose or with a procedural mistake for the student to criticize then respond to a series of questions. It is considered as the best tool, and of moderate reliability. But it is considerably difficult, with a high level of technology challenge facing it, and low validity.

Objective structured video clinical examination (OSVE): OSVE This assessment method depends on a video session of a clinical examination of one patient with positive finding for the student to diagnose or with a procedural mistake for the student to criticize, moreover, it can assess the history taking, data interpretation, communication skills and professionalism, this variability in testing the different aspects of psychomotor skills is a major advantage for this type of assessment, plus it is of moderate reliability.

Discussion

Educators in the MENA region agree that universities are diverse in terms of their available resources which resulted in different forms of online assessment between countries and even between the universities within the same country. Some also mentioned that virtual clinical assessment was not implemented by all institutions. This result is consistent with the fact that the COVID-19 pandemic struck the world suddenly and it leads to the global medical education situation, which necessitated working online and raising medical educators’ understanding of online teaching and assessment.9,10 Most MENA countries have adopted distance learning including private universities building on earlier investments in electronic platforms and content, but the majority of countries and public universities are struggling with abrupt necessity of providing large-scale online teaching and learning. Given this variability of the current situation, not much reliable information is available on how countries are managing interim or final exams. Some countries have cancelled or postponed exams and/or replaced in-class exams with online exams or alternative, project-based assignments. For example, Egypt cancelled midterm university exams. With few exceptions, most countries, including Egypt, Morocco, and Saudi Arabia, are subscribing to the principal to subscribing to the principal to “contain the problems of this current academic year in this academic year and not let them impact the following academic year.”11,12

In addition, participants mentioned that online assessment could be achieved through any virtual meeting application, which has an option of different breakout rooms like zoom or Microsoft Teams, but it requires adequate training of the assessors. Ali (2003) stated that traditional assessment techniques could be modified to meet the assessment requirements of online instruction.13

Regarding the number of assessors, participants stated that online VIVA requires two assessors per station and requires assessors training to increase objectivity and decrease subjectivity, while OSVE and virtual OSCE require less number. This is because during the VOSCE, examiners rotated around rooms without any difficulty as the student interface remained on the same remote workstation.14

MENA educators are concerned with exam security and fairness due to the ethical obligation and acceptance of online monitoring and secondly the non-availability of a proctoring program that guarantees a 100% monitoring. This concern is almost universal as shown by Academic Integrity in Online Assessment of Irish university union in which they mentioned that the lack of online proctoring or invigilation leaves room for various modes of contract cheating.15

One of the major concerns of the experts is graduate attributes and society acceptance of physicians, fear of graduating an incompetent physician lacking some clinical skills that need to be evaluated for medical students including history taking, physical examination, data interpretation, management and clinical acumen, communication skills and professionalism. This could be overcome by using a variety of techniques to assess student learning in online courses as suggested by Jorge Gaytan.16 The same suggestion was recommended by MENA expert in the tool’s summary of the different tools that can evaluate clinical skills such as Virtual OSPE or integrated OSPE, objective structured video exams as well as MCQs. In MCQs by using a clinical scenario in the question stem where series of responses are needed, or coordination of actions are required. As clinical judgment items require prioritizing or weighing management decisions.

Conclusion

The COVID-19 era demanded revisiting our assessment strategies to cope with new changes. Rapid adaptation is required. Recommendations for adopting online assessment are mandatory.

Based on the rich discussion, the take-home messages were compiled into these tips.

Modification of methods of instruction by educators.

Aligning assessment with teaching and learning methods; carefully selecting assessment tools to be aligned with the learning objectives including essay exams, projects, problem-solving, OSCE, OSPE and others.

Recording Clinical skills video in the learning process and in virtual context as an examination tool.

Appropriate selection of the clinical assessment and proctoring applications.

Appropriate selection of Software for the online written assessment, eg, Class marker, blackboard, or Moodle.

Suitable choice of applications for Formative assessment tools, eg, Kahoot has to be appropriately chosen.

Training Assessors on the online assessment method and the used software.

Preparation of case scenarios of simulated cases and videos.

Preparation of Virtual OSCE logistics.

Developing rubric for online assessment.