Faculty's perspective on skill assessment in undergraduate medical education: Qualitative online forum study.

BACKGROUND: India is at the nascent stage of competency-based medical education. Faculties trained in medical education are the main driving force for change. The present study explores the perception of faculties about the current practices and problems in medical/dental/nursing undergraduate assessment, barriers to adoption of best practices, and solutions for addressing them.
METHODOLOGY: A qualitative study was designed and data collected through an asynchronous online discussion forum. A group of 31 health professionals (FAIMER fellows selected on the basis of active participation in department of medical education of respective colleges) participated in the forum. An open-ended topic guide with prompts was designed. The forum was initiated by release of discussion topics (threads) at the start of the month and remained in forum throughout the month. Researchers moderated and recorded day-to-day events. All online forum data were coded line by line and analyzed using conventional content analysis.
RESULTS: Four categories generated were: (1) Low utility of current skill assessment system due to low validity and reliability; (2) Barrier in adopting newer assessment tool due to the absence of felt need of faculties and students, mistaken beliefs, and limited resources; (3) Poor implementation of newer assessment tools such as formatives and objective structured clinical examination with no blueprinting; and (4) Solutions proposed were regular formative assessment, criterion-based examination, quality-assured faculty development programs, and administrative support.
CONCLUSIONS: Barriers in adopting newer assessment tools are related to the faculty's perception and resource constraint. This can be addressed by quality-assured faculty development programs and effective implementation of competency-based education. Copyright:

Introduction

Two decades earlier, assessment in medical education was only in the form of written and oral examination limited to the cognitive domain. Medical schools later faced a variety of challenges from society, patients, doctors, and students. Some of them were exponential growth in knowledge, associated technologic (”disruptive”) innovations, and societal changes as given by Densen.[1] This led to the development of new curricula, introduction of new methods of learning and assessment, and a realization of importance of faculty development. “Changing examination system without changing curriculum has a much more profound effect on the nature of learning than changing curriculum without altering examination system” as said by Miller.[2] This depicts the importance of assessment. Assessment and evaluation are crucial steps in the educational process.[2] Assessment has a powerful steering effect on learning and curriculum.[3] Many universities around the world have recognized importance of assessment and changed their curriculum to competency based that assures, achievement of professional competence at the end of course.[45] In competency-based education system, assessment drives learning by active involvement of trainee in his/her learning and assessment, creation of an authentic environment for learning and assessment, emphasis on direct observation, and use of frequent formative feedback. In India, the existing examination system is predominantly cognitive-based.[6] Faculties being busy in administration, patient care, and research, spare very limited time for updating teaching-learning practices. As a result, students are assessed with the same traditional methods and not able to develop a compassionate doctor–patient relationship. Newer tools such as objective structured clinical examination (OSCE)/objective structured physical examination are used infrequently and ineffectively. Assessment is currently a one-time exercise conducted at the end of the year as a surrogate marker of competence. Assessment which should be consumer-driven is actually provider driven by the medical teacher.[7] Faculties trained in medical education or member of the department of medical education are main driving force for change in the education system. The present study was undertaken to explore the view of these faculties related to the current practices and problems in undergraduate assessment, barriers to adoption of best practices in assessment, and propose solutions for addressing them.

Methodology

Setting and Participants

The study setting was faculty development 2-year fellowship program at CMCL-FAIMER Regional Institute. The program invites the application annually from health-care fraternity in the Southeast Asian region. After screening process and telephonic interview, 16 fellows from multidisciplinary health-care field were selected based on their involvement in medical education. These fellows together of year 1 and year 2 comprised 31 health professionals from different states of India and abroad having medical, dental, and nursing background. In addition to the onsite session of 1–2 weeks’ fellows, actively participated in monthly discussion topic – online forum (Listserv) throughout the year. It was a private group where fellows registered with id and password. Fellows participated as a part of mandatory requirements for fellowship completion. As per FAIMER Data Use, Sharing, Authorship and Ethic policies (updated Dec 2018), electronic consent was obtained from entire Listserv after informing them about the purpose of data collection, plan of analysis, and intent of publication in future. Complete enumeration sampling technique with 31 registered fellows was employed in the current study.

Design

This study was designed as a qualitative online forum[8] among registered fellows (participants). The online forum was selected because participants belong to different states/countries, and we aspired for a comprehensive understanding of existing assessments system in various health-care institutes.

Online forum topic

The study was an outcome of ML web e-learning monthly discussion forum on topic “Skills assessment” moderated by researchers. Three topics were discussed with prompts as given Table 1.

Table 1

Topic and prompts for the online forum

Topic	Prompt
Perception regarding prevailing assessment system	What is the current assessment system in your collegeTo what extent is assessment system in your medical/dental/nursing college able to meet international standards for creating competent health-care professionalsWhat are the lacunae in the current assessment system
Attempt to adopt newer assessment tools/techniques	Had you tried to adopt new assessment tools for skills?Were you successful?What barriers did you face?
Solution to address the problem	How did you address those barriers?Can you suggest any other solutions to make newer assessment skills adaptable, feasible, and sustainable?

Data collection procedure

At the start of moderation month, the asynchronous online forum was initiated by releasing discussion thread on the topic given in Table 1 in Listserv group and remained for the whole month. Participants could post messages about the topic at their convenience in any form as experiences, conversations, response to other messages with use of pauses, emoji, etc., As all participants were comfortable with English, only English language was used. The number and length of messages were not limited, yet at least one message per topic in a period of first 7 days followed by responding to two fellows during discussion was must as a part of course completion requisite. Answering the topic question and responding to other participants took place at various times, so the exact time of data saturation was difficult to determine. Criteria for focus group data saturation were set at least two responses for each category and subcategories generated, and no new categories further added in consecutive days.[9] SS oversaw the communication, steered discussion with prompts, and maintained netiquettes. MK did record-keeping of discussion on daily basis, memos, and notes. Any technical issue if encountered was solved by site maintenance team of FAIMER. MK posted summary of the topic at the end of 15 days and participants were encouraged to give feedback and add further to discussion.

Data analysis

Online forum data were analyzed simultaneously with data gathering by conventional content analysis using inductive approach.[10] Discussion was printed out as transcripts, were thoroughly read and reread for line by line coding to generate initial codes. These were later grouped and regrouped in subcategories, reduced further to generic categories. Finally, categories were established. This was an ongoing process with data gathering done by two independent researchers (MK and HS) trained in qualitative studies. Any discrepancies, in categorization, were resolved by discussion.

Rigor of study

To ensure reliability and validity of data, Guba and Lincoln's criteria, including credibility, dependability, transferability, and confirmability, was used.[11]

Approaches to improve credibility in this study were existing long-term engagement with participants, member check, and investigator triangulation. Long-term engagement was assured by nature of the FAIMER Fellowship Program. For member checking, discussion points were summarized by the author and posted on the Listserv for cross-checking by participants. Investigator triangulation was achieved by analyzing forum data separately by two independent investigators.

For dependability of data, all stages and processes of the study were recorded daily in detail to make the study process traceable for reviewers.

To ensure the confirmability of data, an audit trail was done by TS (acknowledged).

With regard to the transferability of findings, participant's comments were presented without alterations. Demographic characteristics of participants were reported for further examination by readers.

Results

Thirty-one fellows participated in the online discussion with most of them having > 5 years of teaching experience. Fifty-one percent belong to pre- and paraclinical, 10% to clinical department, while 4% and 1% were dental and nursing faculty, respectively. There was a wide variety of participants with respect to positions, duration of service, and department [Table 2]. This thread recorded 94 messages.

Table 2

Characteristic of the study participants (n=31)

Characteristics	n (%)
Post
Assistant Professor	6 (19.3)
Associate Professor	12 (38.7)
Professor	13 (41.9)
Duration of service (years)
<5	3 (9.6)
5-10	11 (35.4)
>10	17 (54.8)
Discipline
Pre- and paraclinical	16 (51.6)
Clinical	10 (32.2)
Dental	4 (12.9)
Nursing	1 (3.2)

All online forum data were coded, and finally, four categories emerged as shown in Table 3.

Table 3

Categories generated from responses of faculty

Categories	Generic categories	Subcategories
Low utility of the current assessment system	Validity of test/tool	Poor reliability
		Difficulty in assessing IPC
		Driven by availability of patient
		Assess outcome of skill and not skill itself
		Limitation in addressing higher level of Miller pyramid
		OSCE diminishes importance of learning in real environment
	Relaxed evaluation criteria	Passing criteria combined practical and theory
		Flexibility for student to drop few subjects/topics
		Quota chasing/fulfilling number of procedures without focus on how it is done
Barriers	No felt need	Not sensitized
		No knowledge of benchmarking
		No knowledge of skill assessment techniques
	Belief/perception	Past experiences
		Comfort zone
		Perception that formatives increase anxiety to student
		Newer assessment will increase workload
	External factors	Large number of students
		Resistance from senior faculty and administrative
		Policies not oriented to teaching
		No effective training
	Resources	Lack of simulation lab (infrastructure)
		Lack of trained workforce
		Time constraint
Poor implementation strategies	Formatives for internal assessment	No program evaluation done with limited/no feedback given to student
		WPBA not done
		Postponing skill development to internship
	Preparation of assessment	No blueprints of examination
		Limited focus on must know, show how
	Nonuniform curriculum	Nonuniform curriculum throughout universities and colleges
Solutions	Formatives assessment	Should be frequent with feedback mechanism
		Focus on proficiency
	Competency	Criterion reference testing
		Innovations in policy suit to local need
		WPBA internship
		Alignment of assessment to curriculum objective
		Accreditation, program evaluation
	Faculty development program	Longitudinal faculty development
	Resources	Appropriate use of technology
		Assured availability of resources by sharing among institutes

OSCE=Objective structured clinical examination, WPBA=Workplace-based assessment, IPC=Inter Personal Communication

Low utility of the current assessment system

Participants felt that the current assessment system is not appropriate to create competent physicians, as they lack validity. Some of the problems pointed were poor reliability, unable to assess skills (higher level of Miller pyramid), while only the outcome of skill is assessed. Assessment tools used like ”Higher-order MCQ are criticized for their inability to distinguish whether a high score in exam is due to true knowledge or random guessing” (Associate Professor, Physiology). Long cases assessment of a student, based on their presentation of a single long case is likely to be subjective and dependent on personal and intellectual preference of examiner. It is mostly a rare case and patient–student interaction often goes unobserved. Students are assessed by final diagnosis of the case rather than process of reaching to diagnosis.

As quoted ”While examining the blood pressure of subject, student is judged on basis of correct reading only.” “End posting exams student is only assessed on final diagnosis as examiner does not have so much time to be their during history taking” (Assistant Professor, Community Medicine).

Some newer assessment skill is rebuked to diminish the importance of learning in a real environment like OSCE, while some are patient-driven. In some universities, no separate head is considered for passing in practical and theory.

”Students need not pass OSCEs in each discipline or even specified number of OSCEs in each discipline, as it is an integrated assessment. Students may omit certain disciplines which they find difficult to master and focus on other disciplines and still manage to pass final examination” (Associate Professor, Pediatrics).

”There are student in the university who do not read or attend anesthesia class but still manage to pass surgery exams” (Associate Professor, Anesthesia). Interns are given completion based on a number of procedures conducted and not on how it was done. Participants were of opinion that ”Our syllabus is exam-oriented. On that run, we try to finish it somehow within the time limit and students try to get through exam somehow. So students are just getting examination driven knowledge.” (Professor, Orthopedic) Assessment and teaching/learning activities are not oriented toward educational objective.

Barrier

Participants have tried to bring change in the assessment system at their level in respective colleges. However, they had to face many barriers related to difficulty in motivating other faculty members. Need for change in assessment tool is not perceived as felt need of faculties and students, as they are not aware or sensitized for newer assessment tools. In addition, faculties have no knowledge of how to do benchmarking of assessment.

”Lack of awareness about need as well as required know-how about benchmarking of assessment. Without this benchmarking, it is difficult to develop strategies and design tools appropriate for its assessment.” (Associate Professor, Medicine).

Bad experience in the past while bringing change in assessment tools with limited support from seniors and colleagues, participants avoid to make further efforts. They are comfortable with current assessment tools, and some feel that formative assessment increases stress among students.

Some external factors that prevent participants from adopting newer assessment techniques are high student–teacher ratio, resistance from senior, and lack of administrative support. There are limited resources in terms of workforce, infrastructure (simulation lab), and workload. Existing policies at workplaces are not in favor of creating the best learning environment.

”Excellence in teaching is not rewarded with promotions. There should be rewards for excellence in teaching on a level similar to research (sorrow emoji).”(Professor, Pediatrics).

Poor implementation strategies

Formative assessment with feedback is not practiced regularly. Workplace-based assessment (WPBA) is not done, and interns are assessed by logbook on required amount of procedures to be done. Students perform the procedures only during the internship period. Blueprinting of paper is not given due importance. Due to the absence of standardized protocol, “Must know” area is not emphasized proportionately. In addition, summative assessment is not implemented uniformly across all medical colleges.

”Ophthalmology paper is for 40 marks in AP and Telangana states (states of India) where same ophthalmology paper is for 100 marks which makes a lot of difference in time given by candidate, preparation and student outcome”(Professor, Ophthalmology).

Solutions

Solutions suggested were frequent formatives with feedback focusing on proficiency rather than knowledge. ”Need to move from comfort zone of normative procedures to criterion-referenced” (Associate Professor, Anatomy), i.e., the performance of students is assessed against a standard criterion and not just in comparison to others. This will assure competency. Sub-competencies and objectives should be defined as per local need in supervision of Medical Education Department. Accreditation needs to be stringent, “It is the responsibility of accreditation agencies to certify that educational program in medical schools can deliver what they promise. Longitudinal Faculty development program should be made mandatory.”(Professor, Pediatrics).

Participants also suggested solutions, pertaining to teaching-learning practices in resource-constrained settings as follows:

”Availability of cadavers, histology and pathology specimens for teaching and learning without resorting to expensive virtual tools.” (Professor, Anatomy).

”Sharing of resources among all institutions: e. g. online library resources shared at the national level” (Associate Professor, Nursing).

”Efficient Use of developed technological resources and talent in the IT industry for Technology-enhanced learning” (Assistant Professor, Ophthalmology).

There is a need of an optimum combination of learning with technology that is accessible and sustainable.

Discussion

The main finding in the study was adherence to traditional curriculum, poor assessment methods, lack of faculty development program, and resource constraint. Student assessment is done in traditional way and curriculum is not reformed for the last 21 years. However, step has been taken in the last 3 months to reform medical education in India.[12] Multidisciplinary faculty of various cadre involved in department of medical education of respective colleges is the strength of this study. Due to geographically distant locations, the discussion was based on asynchronous online platform, and therefore, data saturation could not be confirmed. We tried to overcome it by setting up arbitrary criteria for data saturation by brainstorming among researchers. In the current study, it was found that there is a need to review the current assessment system as it tests only “know” and limited “know-how” area of Miller pyramid. The current method of skill assessment is in the form of long case that is criticized to have low utility value (low reliability, low validity, and low impact) as also identified by Olson et al.,[13] Norcini,[14] and Ponnamperuma et al.[15] Long-case examination is based on one unobserved case that affects its content validity.[16] It has been observed in other studies too that long cases poorly assess history-taking ability, communication skills, and physical examination skills.[1718] Literature review by Memon et al.[19] found lack of transparency and fairness in oral assessment. OSCE though is a reliable tool to assess skill is criticized for depicting an incomplete picture of clinical competence as in reality clinical environment is complex in nature.[20] In line with views of participants related to poor evaluation, a study done by Deswal and Singhal[21] also found problems in medical education in India, such as outdated curriculum, nonuniform assessment system, and poor internship supervision.

Participants had tried to bring change in assessment tools through fellowship projects but had to face many barriers mostly related to lack of felt need from other faculties/students and misconceptions regarding newer assessment techniques. This may be due to the poor quality of faculty development program. Medical Council of I ndia has mandated the basic medical education course, but with workload to train 40,000 medical teachers by 4000 trained teachers is hampering the quality of workshop.[2122] There is a lack of academic leadership with no support from seniors as observed in the current study. Academic leaders could help in engaging purposes with people's moral, building capacity to generate forces for change, understanding change process, developing learning culture and culture of evaluation, and fostering professional development at all possible levels.[2223] Participants in the study felt that existing teacher–student ratio is low which is in line with Deswal and Singhal who claimed 30%–40% shortage in medical teachers.[21] Deswal and Singhal also opined about shortage of clinical material and no incentive given to college for establishing good clinical/simulation labs.[21]

Some colleges had initiated newer assessment skills but not effectively implemented. Formatives aid in deep learning while immediate feedback, direct student learning, and highly appreciated by student.[24] This feedback is lacking during the internal assessment. Most of the medical colleges have a fixed standard (50% cutoff) for examination. This compromises validity of proven valid tools like OSCE.[25] Therefore, standard setting is required to improve the quality of assessment. Blueprinting of examination is rarely practice in India.[2627] This leads to inadequate coverage of curriculum and educational domain over a stipulated period of time.

The solution suggested were regular formatives with feedback mechanism with the introduction of logbook, OSCE, and other skill assessment tools.[28] Standard setting should be done for assessing the performance quality of students instead of standard cutoff.[29] Multiple assessment methods are necessary to capture all or most aspects of competencies required for medical graduates.[3] Therefore, rather than evaluating by individual methods, student must be evaluated assessment program as a whole.[30] Interns should be assessed by WPBA and criterion-based licentiate examination at the end. As accreditation plays an important role in maintaining quality of medical education, all medical schools should be accredited by the MCI and National Accreditation and Assessment Council. In resource-limited settings, plausible solutions for using effective assessment tool can be managed by sharing of resources between medical colleges, use of technology students to create low-cost “Made in India” models.

Participants were of the view that competencies should be driving force for medical training and curriculum planning. MCI had taken promising steps toward the implementation of CBME with recent release of undergraduate curriculum to be implemented from batch 2019.[12] The findings of this study were based on view of 31 participants, which are very less compared to large number of medical colleges in India. Therefore, to get better insight into implementation of skill assessment and improving the student outcome, we further recommend longitudinal collaborative action research projects.

Conclusions

The current skill assessment system lacks ability to develop the array of abilities of a fresh graduate, to perform expected roles in providing health care to the community. There are internal barriers (mistaken beliefs, lack of felt need, low motivation, and insufficient knowledge) and external barriers (constraint of resources, unfavorable policies) to successfully adopt newer assessment tools in India. These barriers can be addressed by a quality-assured faculty development program and strategies to combat resource constraints.

Financial support and sponsorship

This study was supported by the FAIMER, CMCL, Ludhiana, India.

Conflicts of interest

There are no conflicts of interest.