Alternative Strategy Adopted in Practical Assessment for Pathology Postgraduate Certification Examination Amidst COVID-19 Nationwide Lockdown: An Indian Experience with a Novel Hybrid Method of Assessment.

BACKGROUND: A major concern and challenge faced by the educational institution during coronavirus disease (COVID-19) pandemic and consequent lockdown is the timely conduct of the summative assessment. Unlike cognitive assessment, real-time practical assessment through online mode, without compromising principles of assessment is difficult. This study was performed to analyze an alternative hybrid approach adopted for the postgraduate practical summative assessment in pathology.
MATERIALS AND METHODS: Evaluation of the process of summative assessment was done where internal and external examiners synchronously assessed examinees real-time on-site (face to face) and off-site (remotely through live videoconferencing) respectively. A pre-validated questionnaire on a novel approach for assessment and feedback (with close-ended on 5-point Likert scale and open-ended questions) were administered to nine participants comprising three final year postgraduates in pathology, four examiners (two internal and two externals) and two technical experts from the Department of Pathology. Overall scores obtained by all examinees were recorded. A pre-validated questionnaire and feedback were administered to the participants to assess the novel approach to assessment.
RESULTS: The mean performance score obtained by examinees was 63.5% in the summative assessment. Feedback analysis revealed that 85% of participants strongly agreed and the rest (15%) agreed that essential minor deviations introduced in this novel mode of assessment did not compromise basic principles and goals of assessment. Quality of assessment through this mode of assessment was graded as good to excellent. All participants responded that this mode of examination may be applied in the future under similar circumstances.
CONCLUSION: This study concludes that an alternative novel hybrid method of real-time synchronous assessment with on-site and off-site examiners may be a feasible and successful model for conducting summative assessment and can be practiced in time of need.

Introduction

The coronavirus disease (COVID-19) pandemic has brought global life to a standstill with many places being closed and limited essential normal activities being allowed.1 One relevant question posed to universities was the timely conduct of summative assessment as a part of course completion and certification.2 Many institutes and centres postponed it.2 Postponement of the course completion examination may not be the right solution. The need of the hour is a positive approach devising an option where the principles of assessment are not compromised and, simultaneously, evaluation can be completed as per the scheduled time while fully abiding by all precautionary measures to prevent the spread of COVID-19 infection.3

In calamities and times of adversity, to ensure uninterrupted educational course progress educationists need to devise appropriate strategies and measures for summative assessment. A real-time online assessment has good prospects in such situations. The online mode of examination is mostly practised for evaluating the cognitive domain and preferred for formative assessment.4,5 However, there is a paucity of literature on psychomotor evaluation through online mode. A recent experience shared by Wong of the University of British Columbia states that educational adaptability is the way forward in the situation posed by COVID-19.6

The inability of the examiner to travel during lockdown/post lockdown discouragement of travel as per advisory, rules out the possibility of the traditional method of the in-person presence of external examiner for evaluation, while a completely new method of only online mode of practical examination may not be advisable or fair as per assessment principles.7,8 An amalgamation of virtual and real-time approach may be the best option available to conduct course summative examination during COVID-19 pandemic with travel restrictions for external examiners.7 An asynchronous, assessment of video recorded Objective Structured Clinical Examination (OSCE) conducted by Sturman et al, showed good results reflecting fair acceptance by examinee and examiners.9 However, synchronous (live/real-time) online mode in the practical examination would be a novel approach for summative assessment. Postgraduate courses in India are 3-year courses with an end-of-course summative certification examination. This entails both practical and theory examination. The practical examination involves assessment of practical skills by examiners invited from other institutes and internal examiners are also appointed. The present study was designed to analyze an alternative mixed approach adopted through synchronous live/real-time, on-site, and off-site evaluation by internal and external examiners respectively, in summative assessment of postgraduate course certification in pathology during COVID-19 pandemic lockdown.

Materials and Methods

Ethical clearance was obtained from the Institutional Ethics Committee of the All India Institute of Medical Sciences Rishikesh before the study (AIIMS/IEC/20/358 Dated 5/06/2020).

This was an observational qualitative study conducted at a Tertiary Care Medical Institute (located in Uttarakhand, India) on postgraduate summative assessment in pathology. The examination was carried out in the Pathology Practical laboratory with the examinee and internal examiners placed onsite and external examiners off-site (remotely available) through video linking. Convenience sampling was the only method which could be followed. The study population included nine participants comprising three final year postgraduates in pathology, four examiners (two internal and two externals) and two technical experts from the Department of Pathology. External examiners were stationed at their respective institutes at a different location in India (Lucknow and Chandigarh) during this novel assessment. The study was conducted from May 1 to June 16, 2020.

A pre-validated questionnaire for a novel approach to assessment and feedback (with close-ended on a 5-point Likert scale and open-ended questions) were administered to examiners, examinees and technical experts and the responses were recorded and analyzed.

Pre-Examination Process Followed at Institute

Final-year postgraduate students appearing for the pathology practical examination were informed 1 month before their summative assessment about synchronous face-to-face (in person) and real-time videoconferencing by internal and external examiners respectively. Information on mode of examination was also shared with external and internal examiners to make necessary preparation for the assessment.

The Process Followed During Summative Practical Assessment

The Skype platform () for online videoconferencing was used during the practical assessment. Skype is freely available, with a secure connection, and has an option of admitting multiple participants as well as sharing the screen. A test run was done 3 days before to check for connectivity and video/audio quality. The meeting link was shared with all external examiners enabling them to connect, observe, and interact during the ongoing practical examination.

All examinees were well versed with OSPE format, which was practiced for formative assessment in their 3-year postgraduate course. Strict timing was maintained for all the practical exercises during the assessment. All OSPEs were prepared by external examiners and shared with internal examiners half an hour before each practical exercise. Examinees were instructed to perform the test and write their response for the OSPE which were immediately scanned and shared through email with both external examiners for evaluation followed by immediate live interaction through video-conferencing. External examiners also administered a few OSPE through screen sharing with slides in PowerPoint Presentation for clinical history and images while each examinee viewed these in individual laptops and their response was subsequently assessed.

Overall principles followed for assessment were largely based on the Accreditation Council for Graduate Medical Education (ACGME) to cover patient care, medical knowledge, practice-based learning, systems-based practice, and academic skills.10 The evaluation was according to the assessment blueprint shared with the examiner inclusive of a few necessary alterations introduced for this mode of examination (Table 1); these were suggested by external examiners.

Table 1

List of Practical Exercises Covered in Postgraduates Practical Assessment in Pathology

	Section	Exercise/Task	Method
Day 1	Clinical Pathology	Long case	OSPE & Interaction
		Urinalysis (Skill assessment)	OSPE
		Clinical chemistry: Standard curve preparation and interpretation	OSPE
	Autopsy	e-autopsy	OSPE & Interaction
	Hematology	Hematology glass slides	OSPE & Interaction
	Cytopathology	Cytopathology glass slides	OSPE & Interaction
	Blood banking	Blood grouping by Gel Method (Skill assessment)	OSPE
Day 2	Histopathology	Histopathology glass slides	OSPE & Interaction
	Histopathology & cytopathology techniques	Staining (Skill assessment)	OSPE
	Ancillary tests	Spotters	OSPE
	Pedagogy	Examinee to take a class on topic provided for medical undergraduates	Subjective

In clinical pathology (Table 1) a long case was administered by external examiners through PowerPoint showcasing case history, Laboratory values, and relevant images through OSPE and interaction to explore the examinee’s ability of case workup. After this, on-site skill assessment was done by internal examiners for biochemical tests, and standard curve preparation interpretation and examiners interacted with the examinee. Urinalysis by strip test and microscopy examination and interpretation of result was assessed through a checklist by internal examiners.

Autopsy assessment (Table 1) was done through e-autopsy which was administered via PowerPoint display and screen sharing facility through videoconferencing which displayed case history and gross images while the examinee noted and recorded each finding. Interaction with external examiners was arranged individually with each examinee immediately after completion of the exercise. A blood banking skill test was on blood grouping through the gel method and was assessed on-site through a checklist by internal examiners.

An exhaustive list of cases for which glass slides were available was shared with external examiners (along with ancillary test glass slides, reports, datasheets) 1 week prior to practical examination. OSPE stations for histopathology, hematopathology and cytopathology glass slides (Table 1) were designed by external examiners and shared with internal examiners on the day of examination. In addition, slides of immunohistochemistry/special stains/cell block preparation, high-performance liquid chromatography result sheets, flowcytometry datasheet and hematology analyzer result sheets were provided as applicable to these cases. The grossing of specimens was discussed through WhatsApp video during which external examiners interacted with examinees individually. WhatsApp Messenger is a free software application that allows users to make voice and video calls.

Assessment on ancillary test (Table 1) was done through screen sharing in videoconferencing with PowerPoint displaying a case history and its e–images of glass slides and gross images/other relevant laboratory findings to cover electron microscopy, immunofluorescence, fluorescent in situ hybridization and polymerase chain reaction assay gel electrophoresis findings and the examinees wrote their response on paper sheets which were promptly scanned and shared through email with external examiners for assessment. For the pedagogy exercise, topics were given to the examinees on the first day of examination and assessment was done on a subsequent day by all the examiners through videoconferencing. Examinees were assessed on the skills of delivering a lecture, including appropriate sequencing and clarity in lecture content, ability to interact with the audience and voice modulation. Additionally, the quality of PowerPoint slides and the proper use of audiovisual aids were assessed.

Results

All three examinees were successful in this novel mode of assessment and scores ranged from 58% to 68.3% with a mean of 63.5%.

Assessment of examination on various parameters and feedback obtained from examiners and examinees was evaluated which included questions regarding the structure of the examination, quality of exercises and OSPE stations, and overall process of examination followed. All agreed that deviations introduced were necessary and strongly agreed that despite the deviations basic principles and goals of the assessment were not altered (Figure 1).

Figure 1

Response on structural deviation in assessment required in this novel mode of examination.

The overall quality of practical exercise and OSPE stations related to each practical exercise were rated from excellent to good by most. Audio-video quality in videoconferencing during the examination was rated as fair (Figure 2). Most (75%) of respondents expressed that a major part of the interaction was done by external examiners (75%) as compared to internal examiners (25%).

Figure 2

Evaluation of quality for each practical exercise and OSPE stations.

As per the feedback of examinees, examiners and technical experts, most agreed that the overall conduct of the examination was satisfactory and it was conducted fairly. Each task was well administered and all participants agreed that this mode of examination may be used in the future in the event of external examiners being unable to reach the examination center (Figure 3).

Figure 3

Feedback for practical examination by examinees, examiners and technical experts.

Examinees, examiners and technical assistants mentioned various facilitating and hindering factors for this new method of practical examination. They also provided suggestions to improve this modality of practical examination (Table 2).

Table 2

Facilitating and Hindering Factors Encountered During Examination and Suggestions for Improvement (as Opined by Participants)

Facilitating Factors	Hindering Factors	Suggestions
Strong IT support and Internet Cooperation, active participation and expert guidance from external examiners Integrity and full cooperation of internal examiners (External examiner) Could see examinees at will through out the exam (Examiners) Limited number of examinee (Examiners) Adequate time management (Examinee & technicalassistant) Patience of Examiners and examinee (Examiner) Uninterrupted power supply Google meet good platform for videoconferencing Zoom platform better for PowerPoint display (Examiner) Travel time saved (Examiner) Flexibility for other regular work at their location along with assessment (External examiner)	Interruption in network services and Audio visual connectivity First time use of online platform In the beginning had a hitch because of new experience in videoconferenceing but later became comfortable (Examiner) Eye strain and heaviness by end of day due to looking at desktop screen for long periods. Dependence on Internal examiners Non availability of electronic OSPE stations with e-slides Scanned slide use was considered as an option however examiners opined that a first time scanned slide for assessment will not be fair for examinees. Technical problems as no prior experience of practical assessment by videoconferencing by examinee	Camera & big screen with better internet and audio/video quality required Digital slides with real time interaction can be better Mock examination 1–2 day prior to main examination to acquaint examiner and examinee could be helpful. OSPE sheets can be given to fill in answers Electronic answers for OSPE which can be viewed online (as a solution for scanning answer sheets) Multiple desktop and scanners to be available for scanning written answer sheets to save time on scanning and sending answer sheets to external through email. A dedicated place with audio video set up and sound proof room can facilitate interaction.

Abbreviation: IT, information technology.

Discussion

The education system has been equally challenged compared to other sectors during the current COVID-19 pandemic. A new outlook and needs-based approach are recommended in teaching-learning methods and assessment to overcome limitations induced by the current pandemic.3,7,11,12 In the last two decades, the medical literature has been flooded with papers promoting technology in the education system. There has been selective usage of technology as per need and feasibility in the education system, as it has its own pros and cons. Various medical schools in the world and even the Medical Council of India, has been encouraging colleges to use digital platforms and to use technology as a replacement in various disciplines such as anatomy, pharmacology, and other types of laboratories.1,2

There is sufficient literature on the use of technology for teaching-learning activities, cognitive assessment but limited literature on technology use for practical assessment in examination excluding simulation.9,13–17 Assessment of practical skills is indeed a concern during such a situation. Educators as a part of medical school response to the COVID-19 situation need to work towards educational adaptations and mitigating strategies that should be principle-based.6 In the present circumstances, educationists need to facilitate the residency program and ensure timely conduct of final certification assessment.

A directive specified by a university from the United Kingdom (UK) on the option of using online/videoconferencing for practical exams correctly emphasizes that the basic purpose of assessment should not be compromised. An alternative mode of assessment must be able to ensure that learning outcomes are closely comparable to the original assessment approach.18 Even in the research discipline, videoconferencing has been adopted as a mode for the viva-voce, as noted in communications uploaded on the website of The University of Sheffield, research services, which was revised on March 31, 2020 due to the situation posed by COVID-19.19 In such a challenging time, non-face-to-face interaction is being recommended for the well-being of learners.7,20 Portfolios and scores of internal assessment should be taken into account.3 Extra points may be allocated on student’s participation towards combating COVID-19.7

A review of the literature reveals an encouraging experience by Sturman et al, on the asynchronous mode of assessment made through recorded video on the performance of participants in OSCE stations.9 However, an in-person assessment adds value in the assessment.9 Modification of the assessment framework facilitated the assessment of final year medical students through OSCE during the Covid-19 pandemic.6 Khan et al demonstrated medical students performed better in the pathology practical examination through the assistance of video projection as compared with the traditional method and recommended it to be used for formative and summative assessment for pathology practical assessment based on scores obtained and student and faculty feedback.21

Therefore, a synergy of traditional methods and online videoconferencing for assessment is a promising step for the evaluation of postgraduates in practical examinations. There is a lack of published literature relevant to this novel hybrid model of assessment (synchronous on-site and off-site live real-time) with in-person and remote assessment for assessing practical/psychomotor skills. Sharing our experience on the use of this modality for practical assessment may prove helpful, to others exploring ways of dealing in similar situations. In our study, overall scores obtained by the examinees revealed that despite a new model of assessment they performed reasonably well. The highlight of this novel mode of assessment was that the basic principles of assessment remained intact despite some structural deviation essential to conduct this type of examination. Deviation should not alter the overall goal of assessment or the blueprint of assessment.9

Considering the decline in clinical autopsies worldwide, an alternative and promising approach is e-autopsy.22–24 Application of technology has facilitated e-autopsy for teaching and assessment as an effective alternative to the traditional method.24,25 Evaluation through e-autopsy as an equivalent exercise to traditional post-mortem was acceptable to all respondents of the present study.

The aim of assessment in clinical pathology exercises is focused on evaluating the students’ approach to case and investigations required.26 The examiner explores the examinee’s ability to select tests to be applied, its reliability, principle, normal/abnormal values and approach to further investigations for prognosis or therapy in reference to a clinical case. This was successfully dealt with through e-assessment using videoconferencing with history and examination findings shared by the examiners.

Remotely used slide scanners and online real-time evaluation of examinees can be an option for histopathology assessment though cost may be a limiting factor.27 OSPE has high interrater reliability and has been found to be in the best interest of the assessment process.28 Experience at OSPE stations in this mode of assessment was like face-to-face assessment. The participants response reflected that quality was not compromised in any of the practical exercises in this novel mode of examination. There was varying opinion about stress during the examination and this may be because stress is subjective and multifactorial.29

Facilitating factors in this mode of assessment include a non-threatening positive atmosphere and adherence to the schedule. Availability of a video link enabling external examiners to see the examinees is also a major facilitating feature in this mode of examination. The integrity of internal examiners is an essential facilitating factor in this hybrid mode of examination.

It would be unfair for the examinee to expose them suddenly to a new method/mode in their final postgraduate certification examination.30 It was encouraging to get an opinion from most of the participants that the performance of examinees in this new model was comparable to the traditional method of assessment. This novel hybrid method of examination simulated a traditional examination except for external examiners being placed in a remote location.

A limitation of this mode of assessment was that only a limited number of postgraduates can be assessed within the stipulated time. No previous experience with a new modality of assessment can be a hindrance as mentioned by examiners.31 Strain to the eyes more than traditional exam method and heaviness in the head at the end of each day may be due to looking at the screen continuously for long hours are other disadvantages.

Our successful experience for practical assessment through a hybrid method of onsite and offsite real-time evaluation brings out the following important features, which should be applied for smooth conduct of examination through this mode:

Clear communication with all stakeholders prior to adopting a new mode of assessment.

Clear non-ambiguous instructions provided to the examinee to allay any confusion.

A mock examination sensitizes examinee and internal examiners. It would enable internal examiners to identify any shortcomings.

One of the prerequisites is to have an uninterrupted internet connection with good bandwidth and power supply. In addition, a videoconferencing platform as per the requirements should be selected while any new unfamiliar software or videoconferencing platform should be avoided to prevent any unforeseen problems.

A guidance paper released by the World Health Organization's regional office of Europe has recommended development and implementation strategies for hastening training and early certification for health care disciplines as a step towards resource development for COVID-19 response.32 The spread of COVID-19 has raised concerns on the scarcity of health care professionals and this calls for facilitating an expedited process for timely completion of postgraduate courses and strengthening of the workforce.32 Worldwide, there has been a transition to online learning with the advent of virtual platforms as an acceptable form of examination tool which is here to stay.33–35

Amidst the COVID-19 pandemic this may be adopted as a new normal mode of practical assessment for course examination until the situation stabilizes. Similarly, in other natural calamities such as earthquake, landslide, flood and many more man-made damage/danger such as security threats, such novel mode of practical examination may be the only possible option. It may be applied in the event of the sudden inability of an external examiner to travel with no replacement available at short notice. Under such conditions by using this hybrid approach, the very same external examiner can still participate through videoconferencing.

There is a limitation of this study which is response bias as only three out of nine participants were examinees and since the study was conducted during the examination we were more likely to have positive comments, however we ensured that anonymity of response was maintained. The study was also conducted after exams were finished and exam results declared hoping to cater to the same bias.

Conclusion

A novel hybrid method of synchronous real-time assessment with on-site and off-site examiners may be a feasible and successful model for conducting the summative assessment. This novel mode can be adopted when the situation arises for a comprehensive practical skill assessment of students. However, in normal circumstances, it cannot replace the traditional method of in-person (face-to-face) assessment.