Formative assessment in practical for Indian postgraduates in health professions education: A strategic initiative towards competency-based education.

BACKGROUND: Observational assessment (OA) or Workplace Based Assessment (WPBA) is being increasingly used to assess the trainees by direct observation to shape their learning. The current deficiencies in our conventional assessment system are due to lack of conceptualization of assessment as a process for continuous improvement and learning, leading to non-utilization of many available tools of assessment.
OBJECTIVE: The present study aimed to introduce formative assessment (FA) for postgraduates at all the constituent colleges (Medical, Dental, Ayurveda, Nursing) under the ambit of Datta Meghe Institute of Medical Sciences (Deemed University) Sawangi (M) Wardha, (Maharashtra), India.
METHODOLOGY: Observational Assessment tools used for FA were Mini CEX (Mini Clinical Evaluation Exercise) and DOPS (Direct Observation of Procedural Skills)in addition to Objectively Structured Clinical/Practical Examination (OSCE/OSPE). Six encounters of OAs were conducted at the end of each semester. The OA tool used was either the MiniCEX or/and DOPS, depending on the type of subject (Preclinical, Para-clinical, Clinical).
RESULTS: A significant improvement in scores from one semester to other as the postgraduate learner progresses, was observed. The overall response from the postgraduate students and faculty for FA was satisfactory.
CONCLUSION: Formative assessment in practical's, was feasible, acceptable and effective program to improve the overall learning and competency of postgraduates. Copyright:

Background

“Assessments drives students’ learning” is the accepted quote. But what is rather strange is how assessment directs, drives, and influence learning. This understanding holds the key for an educator, teacher, or policymaker to direct the entire process of learning through appropriate assessment tools.[1]

One of the models of competency, the Miller's Pyramid, provides a basic framework for bringing into focus the four levels of competence: ‘Knows’, ‘Knows how’, ‘Shows how’, and ‘Does’ (Miller, 1990).[2] Multiple choice questions or short answered questions or long answered questions can test the first two levels while the latter two components can be tested by objective structured practical/clinical examination (OSPE/OSCE) and other performance-based assessment tools.

In India, the Postgraduate Medical Education Regulations 2000 of the Medical Council of India (MCI) states that postgraduate training be competency-based and also suggests the use of ‘Logbook’ for monitoring the learning process. However, these regulations do not provide any processor road map of in-training assessments.[3] There are few gray areas in the document of Graduate Medical Regulations of MCI 1997 like ‘integrated teaching’, which has got somewhat disappointing results due to lack of provision of operational details or/and monitoring details.[4] While there is a provision of internal assessment for periodic assessments during the undergraduate (UG) program in Indian Medical Regulations, there are no such pre-requisites for postgraduate (PG) courses in India.

PG training is directed not merely at the attainment of knowledge, attitude, and skills but also observable responsiveness and appropriate functioning in real-life situations. The purpose of PG education is to create specialists who would provide high- quality healthcare and advance the cause of science through research and training. At the same time, they would also serve as physicians of the first contact for the community, practicing affordable, accessible primary healthcare to the rural, needy, marginalized, and underprivileged populace.

It follows that even the most ideal of conventional assessments conducted in examination settings will fall short of measuring these outcomes. There is indeed a need to observe and assess the trainees in real situations so that necessary corrections can be provided to the trainees.[5] Observational assessment (OA) or Workplace-based assessment (WPBA) is being increasingly used to assess the trainees by direct observation and to shape their learning.[67] The current deficiencies in the assessment system of India are due to a lack of conceptualization of assessment as a process for continuous improvement and learning, leading to non-utilization of many available tools. Hence, considering the Indian scenario, using the advantages of OA over the traditional assessment methods and its educational utility; the study aimed to introduce formative assessment (FA) in practicals for PGs at all the constituent colleges (Medical, Dental, Ayurveda, Nursing) under the ambit of Datta Meghe Institute of Medical Sciences (DU), Wardha (MS), India. The objectives of the study were to train and sensitize the faculty/supervisors and PG trainees for FA in PG practical as “Assessment for Learning” and to evaluate the program for its feasibility and effectiveness.

Materials and Methods

The study was conducted from June 2015 to October 2016, in all the Departments (having Postgraduate courses) of Medical, Dental, Ayurveda, and Nursing Colleges, of DMIMS (DU), Wardha (MS), India. The study participants were the PG students of the first and second academic year and faculty/guides/supervisors of the respective departments. The approval from Institutional Ethics Committee was obtained (Ref No.: DMIMS (DU)/IEC/2015-16/155).

It was an interventional prospective analytical study. OA tools used for FA were mini clinical evaluation exercise (MiniCEX) and direct observation of procedural skills (DOPS) in addition to OSCE/OSPE. Six encounters of OAs were conducted at the end of each semester. The OA tool used was either the MiniCEX or/and DOPS, depending on the type of subject (preclinical, paraclinical, clinical). There was the flexibility of the sites for using these tools as outpatient department (OPD) setting, in-patient department (IPD) settings, community/fieldwork, operation theater etc., as per the requirement of the subject/department.

Sensitization of the faculty was done for the concept of ‘FA in practical for postgraduate’. The faculty was trained for using the tools for OA, i.e., MiniCEX and DOPS as well as in providing developmental feedback on direct observation. The training was given by conducting a one-day workshop for the faculty. Similarly, the PG students were also sensitized by the respective Heads of the Department regarding the importance and implementation strategy of FA in practical.

All the departments identified various competencies and skills to be attended by the junior residents and prepared checklists for OA as well as various stations for OSPE/OSCE. Procedural skills were identified as per the level of residency of the PG students. All the skills or competencies identified by subject experts to be used for MiniCEX or DOPS and OSCE/OSPE stations were prevalidated by the respective Heads of the Department followed by rigorous validation by faculties of the Department of Assessment and Evaluation, School for Health Professions Education and Research.

The scores assigned for each FA (per semester) in practical for PGs were as follows:

3 OAs at the end of first semester = 30 marks

3 OAs at the end of second semester = 30 marks

OSPE/OSCE examination (8 stations) =40 marks

(Total of 100 marks for each academic year).

Later, as per the operational strategy for FA in practical, validated tools for identified skills were implemented for FA in practical at the end of each semester. The result was prepared by the respective Department and submitted for final analysis. Feedback from PG teachers/supervisors and PG students were solicited for the feasibility, practicality, and applicability of FA in practical for PGs in the form of the validated structured feedback proforma. All the enrolled PGs (n = 331) and all registered PG teachers/supervisors (n = 153) participated in the study.

Results

Analysis of marks/scores obtained in Formative assessment

Table I depicts the mean scores of the FA (OA and OSPE/OSCE) of the PGs of various institutes under the ambit of the university. The mean scores of Junior Resident 1 (JR1) and Junior Resident 2 (JR2) are calculated for the FA. Overall,

Table 1

Institution wise distribution of marks of Formative assessment (Observational assessment and OSPE/OSCE) of the postgraduates

Institute	Postgraduate student (JR- Junior resident)	Semester -I 1st OA (30 marks) Mean score	Semester-II		Total 100 marks Mean score
			2nd OA (30 marks) Mean score	OSPE/OSCE (40 marks) Mean score
Medical	JR-I (n=99)	16.03	18.71	24.55	59.30
	JR-II (n=85)	17.26	20.74	26.17	64.18
Dental	JR-I (n=37)	16.40	18.71	26.27	60.26
	JRII (n=36)	16.44	17.72	23.94	58.11
Nursing	JRI (n=10) #	17.98	20.33	25.63	63.94
		1st OA (40 marks) Mean score	2nd OA (40 marks) Mean score	OSPE/OSCE (40 marks) Mean score	Total (120 marks) Mean score
Ayurveda	JRI (n=26)	Not conducted$	26.69	23.51	50.21/80 Marks
	JR II (n=38)	24.91	25.92	25	75.83/120 Marks

$JR1 of Ayurveda were not having first OA due to late admissions. (JR - Junior Residents). #For Nursing, as the total duration of nursing post-graduation is of two years in comparison to three years for other disciplines, the FA was being taken only for JR1 (Batch 2015-16). Different modalities like Thesis review presentations and log book are proposed to be applied for JR2 of Nursing

The mean scores of FA ranged in between 58.11 and 64.18 out of 100 marks

The highest score of 64.18 marks out of 100 is of the JR2 of medical college

For the institute of Ayurveda, the mean FA score for JR1 was 50.21/80 marks and for JR2 it was 75.83/120 Marks

In all the institutes, for both JR1 and JR2, mean scores of 1st OA were improved in the 2nd OA test

The mean scores of OSPE/OSCE were also above 50% for both JR1 and JR2 in all the institutes.

Line diagrams [Figure 1] depicting the trend of the mean scores of OA for JR1 of Medical college showed improvement in scores.

Figure 1

Trend of the scores of observational assessment for JR1 of Medical college

Out of all the departments of medical faculty, having PG students (JR1) of the Batch 2015-16, in 4 departments (ENT, Medicine, Pediatrics, Radiology) the mean FA score came below 50%. Similarly, for JR2 of the batch 2014-15, in one department (Community Medicine) the mean FA score was found to be below 50% [Figures 2 and 3].

Figure 2

Department-wise distribution of mean scores of formative assessment (Observational assessment & OSPE/OSCE) of the postgraduates (JR1) of Medical college

Figure 3

Department wise distribution of mean scores of Formative assessment (Observational assessment & OSPE/OSCE) of the postgraduates (JR2) of Medical college

On similar lines, out of all the departments of dental faculty having PG students (JR1) of the Batch 2015-16, in only one department (Oral Medicine & Radiology) the mean FA score came below 50%. For JR2, all the departments of dental faculty had the mean FA score above 50% [Figure 4].

Figure 4

Score of Formative Assessment for JR1 of Dentistry

Total FA score of all departments of Ayurveda faculty for JR1 is calculated considering one OA test and OSPE/OSCE marks (as the first OA was not conducted due to late admissions. See Table No. 1). The total marks are out of 80 for JR1 of Ayurveda. For JR1, all the departments of Ayurveda faculty had the mean formative assessment score above 50%. All the departments of Ayurveda College, having PG students (JR2) of the Batch 2014-15, the mean FA score was above 50% (>60 marks).

For the faculty of Nursing, in all the departments, the JR1 scored above 50% for the FA.

Feedback analysis for Formative assessment

The response rate of feedback from all PG students was 92% whereas from PG teachers/supervisors of all the constituent colleges was 79%. For almost all the parameters of feedback, the PG students showed satisfaction towards the various aspects of FA. The rating average ranged between 3.86 and 4.19 [Figure 5]. Overall rating by the PGs and faculty for the process of FA showed that cumulatively 88.27% PGs and faculty rated the FA as Good (72.22%) to Best (16.05%).

Figure 5

‘Rating average’ for feedback from postgraduate students and faculty regarding FA (measured on 5 point Likert scale)

Key suggestions to improve Formative assessment in the department given by students and teachers

FA provides a good opportunity for PGs to self-assess their practical knowledge

Should be undertaken every fortnightly or monthly

Helps to improve the PG's practical and communication skills with patients

Should be included in summative assessment too

OSCE should be done in both IPD and OPD settings

Combination of OSCE, MiniCEX/DOPS should be applied to every patient's procedure (minor surgical) for the betterment of results

More procedures to be covered through DOPS/MiniCEX

It should not only be taken for exam purpose but also taken once in a week to improve patient examination skills

Should be implemented as a routine day to day system (such as during rounds or immediate after wand rounds)

Supervised learning may be useful

For improvement in FA, only procedural skills should not be chosen for DOPS but it should include problem-solving skills in the form of case scenario and differential diagnosis, clinical application of the procedure, etc

Repeated conduct should be monitored for improvement in outcome.

Hindrances faced by postgraduate students and faculty during the implementation of Formative assessment in the department

Majority of the PG students opined that the FA was beneficial and there were no significant hindrances during its implementation.

Some of the hindrances informed by others were as follows: -

Time constraint, Space constraint, in some departments

If it is conducted in routine clinical working hours, there is a rush by both PGs and faculty to get back to the routine clinical duties. Hence, FA should be taken in a separate time slot

Need to have many more faculty/assessors available in the department on the day of FA, in major clinical departments with a significant number of PGs

It is tedious to prepare a checklist for each and every procedure. If there is no trained staff for how to prepare the checklist or how to conduct DOPS, then responsibility lies with only trained staff in the department

More time is required for doing all things meticulously. Because of the task of completing clinical and/or departmental work, faculty may not give much time for everyone during FA.

Discussion

Health educators have differentiated assessment according to its purpose; assessment for learning (FA) and assessment of learning (Summative assessment). When the information from an assessment is used solely to make a judgment about the level of competence or achievement, it is the summative assessment. FA refers to assessment that is particularly intended to provide feedback on performance to improve and augment learning. It is not the instrument that is formative, it is the use of the information gathered, by whatever means, to adjust teaching and learning that merits the “formative” label.

Post Graduate Medical Education Regulation 2000 document[8] of the Medical Council of India (the regulatory body of Medical Education in India) mentions that the PG curriculum should be ‘competency-based’. It should ideally develop competent, confident, concerned, compassionate, and globally relevant healthcare professionals who must provide comprehensive healthcare. However, there is a need for specific guidelines regarding assessing those competencies. At present, the three- year long PG training is assessed at the endpoint only (summative assessment). This means that the assessment is done for the product and not for the process. There are very few colleges/Universities conducting FA in theory for PG in India. But very little literature is available for FA especially for assessing competency through practical, for all health professions.

DMIMS (DU), Wardha is a Deemed University (in Rural setting) having Medical, Dental, Ayurved, and Nursing Colleges, which are regulated by their respective Apical Councils. These councils do not encompass any form of formative assessment or continuous internal assessment in their PG training program. Knowing the importance of FA for learning, we conducted a study for PG students during the academic year 2015-16 for practical (in the domain of skills and attitude) and tested for its feasibility and effectivity.

In this study, it was observed that the mean score of OA conducted for the second time was improved as compared to the first OA test scores. Though these scores were not statistically significant the increasing trend was observed for both JR1 and JR2. OSPE/OSCE was one of the modalities of the FA. OSCE was used as a tool for both formative and summative assessment of Japanese PG in clinical training at Hiroshima,[9] wherein it was found that trainees viewed OSCEs positively and appreciated their effectiveness from a pedagogical perspective, and OSCE positively affected the trainees’ approach to learning. Similarly, Miller JK,[10] in his study, on ‘Competency-based training: objective structured clinical exercises (OSCE) in marriage and family therapy’, an adaptation of the OSCE procedures was done for competency-based training of Marriage & Family Therapy students. Instead of using the procedures as a summative examination as is typical in medical education, the study proposed how to use them as formative exercises in the development of student competence.

The study by Dijksterhuis M G et al.[11] aimed to explore perceptions of PG and teachers about factors that determine active engagement in FA. It was stated that engaging in FA with a genuine impact on learning is complex and quite a challenge to both students and teachers. Increased acceptability along with the effective implementation of FA structure, individual perspectives on feedback, a supportive learning environment, and credibility of feedback are all important in this process. Lases LSS et al. (2019)[12] too opined that a healthy learning environment is associated with improved resident well-being. Every one of these should be taken into account when the utility of FA in PG medical training is evaluated. On the other hand, as per Watling and Ginsburg (2019),[13] the emergence of learning from a cauldron of assessment and feedback can seem like alchemy. Whereas feedback stresses development and learning, assessment stresses judgment and decision making. Mixing these distinct ingredients creates a risky recipe for learning.

In this study, all the PG residents and teachers/assessors were asked to give feedback regarding the feasibility and utility of this program in a structured questionnaire. Feedback analysis depicted the acceptance of FA by majority of students and faculties.

Conclusions

This study of ‘Formative Assessment in Practical for postgraduates in Health profession at DMIMS (DU) Wardha (MS), India’, revealed that it was a feasible, acceptable, and effective program (with minor modifications at respective departments) to improve the overall learning of students. From the results, it can be concluded that the overall response from the PG students and faculty for FA was good. The results showed a significant improvement in scores from one semester to other as the PG learner progressed.

Limitations

As it was a pilot study conducted only for 1 year, the prospective progress of the PG student could not be computed.

Recommendations

Looking at the need for FA for Learning in Health Education, the following recommendations are made.

To include OSCE/OSPE in summative assessment of PG training

Increasing the number of OAs to have more reliability and thus increasing validity

Incorporation of FA (practical) in PG training program as ELIGIBILITY CRITERIA (minimum 35% or 50%) for progression

Inclusion of the concept of Continuous Internal Assessment in Post Graduate Assessment, as in the case of undergraduates

It should be included in the curriculum as a part of the scheme of examination

Bank of OSCE station for PG should be prepared at the departmental level.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.