Acceptability and feasibility of mini-clinical evaluation exercise as a formative assessment tool for workplace-based assessment for surgical postgraduate students.

BACKGROUND: Despite an increasing emphasis on workplace-based assessment (WPBA) during medical training, the existing assessment system largely relies on summative assessment while formative assessment is less valued. Various tools have been described for WPBA, mini-clinical evaluation exercise (mini-CEX) being one of them. Mini-CEX is well accepted in Western countries, however, reports of its use in India are scarce. We conducted this study to assess acceptability and feasibility of mini-CEX as a formative assessment tool for WPBA of surgical postgraduate students in an Indian setting.
METHODS: Faculty members and 2nd year surgical residents were sensitized toward mini-CEX and requisite numbers of exercises were conducted. The difficulties during conduction of these exercises were identified, recorded, and appropriate measures were taken to address them. At the conclusion, the opinion of residents and faculty members regarding their experience with mini-CEX was taken using a questionnaire. The results were analyzed using simple statistical tools.
RESULTS: Nine faculty members out of 11 approached participated in the study (81.8%). All 16 2nd year postgraduate surgical residents participated (100%). Sixty mini-CEX were conducted over 7 months. Each resident underwent 3-5 encounters. The mean time taken by the assessor for observation was 12.3 min (8-30 min) while the mean feedback time was 4.2 min (3-10 min). The faculty reported good overall satisfaction with mini-CEX and found it acceptable as a formative assessment tool. Three faculty members (33.3%) reported mini-CEX as more time-consuming while 2 (22.2%) found it difficult to carry the exercises often. All residents accepted mini-CEX and most of them reported good to high satisfaction with the exercises conducted.
CONCLUSIONS: Mini-CEX is well accepted by residents and faculty as a formative assessment tool. It is feasible to utilize mini-CEX for WPBA of postgraduate students of surgery.

Introduction

Workplace-based assessment (WPBA) has emerged as an attractive concept in the field of medical education. It has many advantages as the assessment is conducted under direct supervision in a real environment, thus resembling the day-to-day clinical encounters more closely than traditional ways of assessment. It focuses more on actual performance rather than mere knowledge. Although it can be used both for formative and summative assessment, the strength lies in its use for formative assessment to promote learning. Since its introduction many tools have been developed and added to armamentarium of WPBA, namely, mini-clinical evaluation exercise (mini-CEX), direct observation of procedural skills, case-based discussions, clinical work samples, and multi-source feedback.

Mini-CEX is a useful tool that can be used for formative assessment and hence for WPBA. First introduced by the American Board of Internal Medicine in 1995 for assessment of postgraduate students, mini-CEX has been designed to provide feedback following direct observation of a clinical encounter.[1] The assessor scores the performance using a generic form that is applicable to most clinical scenarios which is followed by a feedback session. The duration of encounters varies from 10 to 15 min including the feedback. Thus, mini-CEX appears promising as it is conducted in a real environment, in a short time with feedback incorporated. The traditional way of assessment does not allow opportunity to provide feedback and thereby avoids the use of this important influence on learning. Mini-CEX is being used extensively for formative assessment in West[2345] but has been used sparingly in India.[67891011]

Although appealing, WPBA has not enjoyed a uniform level of acceptability and success,[8] and there are concerns regarding the feasibility of carrying out multiple assessments amidst busy clinical schedules by multiple assessors to ensure reliability and simultaneously justifying the formative nature of the exercises being conducted.[1213]

We attempted to assess the feasibility of carrying out mini-CEX as a formative assessment tool in Indian setting and also tried to find out whether it is acceptable to our students and faculty in surgery.

Methods

The study was carried out at a Government Medical College and Tertiary Care Teaching Hospital after approval from the ethical clearance committee and consent from the participating faculty and residents. Purposeful sampling including all 16 2nd year postgraduate surgical residents (total population sampling) was done. Focus group technique was employed to evaluate the shortcomings of the existing system of internal assessment (IA) and to sensitize the residents to mini-CEX. Two focus group discussions (FGDs) were conducted for residents. In the first FGD, the views of the residents regarding the existing system of IA were taken including the responses to their reaction to the marks displayed after the assessment. The advantages and disadvantages of the existing system were discussed in detail in a friendly atmosphere. They were encouraged to suggest alternatives to offset the disadvantages of the existing system of IA. Mini-CEX was then introduced as a formative assessment tool that could overcome some of the disadvantages inherent to the traditional way of IA. At the end of the first FGD, the students were provided with printed material on mini-CEX to help it understand better.

In the second FGD, residents were exposed to mini-CEX in detail and were told how the exercises would be conducted and what is expected out of them. The mini-CEX evaluation form was discussed in detail. Simultaneously, one to one discussion was carried out with the faculty members to sensitize them toward mini-CEX with emphasis on how to provide feedback. They were also provided with the printed material on mini-CEX, and the mini-CEX evaluation form was discussed in detail with them.

Informal feedback was taken from the students and faculty members at regular intervals to assess difficulties in running the project, and appropriate measures were taken to address them. After achieving the requisite number of exercises, feedback from residents and faculty members was taken using a predesigned and validated questionnaire on a five-point Likert scale [Figures 1 and 2]. The questionnaire was used to generate responses to assess the acceptability of mini-CEX as a formative assessment tool for WPBA.

Figure 1

Feedback questionnaire form for faculty on their experience with mini-clinical evaluation exercise

Figure 2

Feedback questionnaire form for residents on their experience with mini-clinical evaluation exercise

Results

The results of the study are as follows.

Analysis of first focus group discussion

The responses of postgraduate students on display of performance following (conventional) IA varied from “works as a stimulant to study hard” to the feeling of “unpleasantness, depression and being worthless.” Many of them admitted being “surprised with the result” as they thought that the “case presentation” went fine and they wanted to know “what went wrong” to which they had no opportunity.

The following shortcomings of the existing conventional IA were identified during the conversation.

The assessment is grossly inadequate as it is conducted once in 6 months as a single case presentation

There is chance of getting an easy case and performing well by the candidate and vice versa

There is chance/luck of getting a strict or lenient examiner that may affect the score

There is no provision of feedback to help the candidate know about his performance.

Feasibility to conduct mini-clinical evaluation exercise

A total of 60 mini-CEX were conducted by 9 faculty members for 16 2nd year postgraduate students of the department over the period of 7 months. Each resident underwent at least three and a maximum of five encounters during this period. This amounted to nearly seven exercises conducted by each faculty member over 7 months, i.e., one mini-CEX per month per faculty. This was therefore not difficult for the faculty members and does not seem to increase their workload significantly. The initial plan of distribution of students among faculty members for mini-CEX was found not working well as the students or faculties were not available on same day and time because of their professional commitments. The distribution was then rearranged to ensure that the student remained available to the assigned faculty member.

No extra resources were required to carry out the exercises except printed evaluation sheets. Despite difficulties, mainly limited to the motivation of faculty to continue taking mini-CEX and some unforeseen circumstances such as attrition of few faculty members and therefore their unavailability for the project, it was observed that it is feasible to conduct mini-CEX without stretching the resources.

Complexity of mini-clinical evaluation exercise conducted

Forty-four (73.3%) clinical encounters were considered of moderate complexity by faculty while the number of low- and high-complexity clinical encounters was 12 (20%) and 4 (6.67%), respectively.

Competencies assessed

The competencies assessed during mini-CEX included interviewing skills 15 (25%), physical examination skills 15 (25%), counseling skills 4 (6.7%), clinical judgment 2 (3.3%), and an overall impression 24 (40%). None of the residents was assessed on professionalism alone [Table 1].

Table 1

Responses of faculty on mini-clinical evaluation exercise on a five-point Likert scale

Questionnaire item	1	2	3	4	5
	Strongly disagree	Disagree	Can’t say	Agree	Strongly agree
I directly observed the student’s performance	0	0	0	0	9
It took me 10-15 min to complete the exercise	1	2	0	6	0
I provided the feedback in a constructive way	0	0	0	6	3
The exercise is easy to carry out	0	0	0	7	2
I feel that mini-CEX can sample more areas for assessing	0	0	0	7	2
student’s competence than the traditional internal assessment
I found it difficult to examine the students more often	0	6	1	2	0
Mini-CEX requires more time and commitment than the usual method of internal assessment	0	5	2	2	0
Evaluation of a candidate by mini-CEX is better than traditional way of internal assessment	0	0	2	5	2
Mini-CEX can supplement the traditional way of internal assessment	0	0	1	5	3

Mini-CEX: Mini-clinical evaluation exercise

Time taken for observation and feedback

The mean time taken by the assessor for observation was 12.3 min (8–30 min) while the mean feedback time was 4.2 min (3–10 min).

Acceptability of mini-clinical evaluation exercise by faculty

All faculty members found mini-CEX acceptable as a formative assessment tool. Most of them expressed that it can be used to supplement the traditional way of assessment. The feedback from the faculty regarding their experience with mini-CEX is presented in Table 1.

As is reflected in the table, some of the faculty members reported that mini-CEX was more time-consuming than initially thought. Although all of them agreed that the exercises were easy to carry out and sample wider areas as compared to the traditional IA, some found it difficult to conduct the exercises often. The following are some of the favorable comments in the faculty feedback questionnaire regarding their experience with Mini-CEX:

It is better way to assess students than traditional way

It is interesting to interact with different students frequently.

However, one faculty member found mini-CEX similar to the case presentation by residents during routine ward and emergency rounds.

Acceptability of mini-clinical evaluation exercise by residents

All residents well-accepted mini-CEX for IA and most of them reported good to high satisfaction with the exercises conducted. A few of them liked the relaxed environment during the exercises, but a few of them found that being supervised by faculty while they were interviewing/examining the patient made them nervous. All were convinced that mini-CEX could be incorporated for IA. Some of them came up with innovative ideas to further enhance the quality of IA using mini-CEX. The feedback from the students is summarized in Table 2.

Table 2

Responses of residents on mini-clinical evaluation exercise on a five-point Likert scale

Questionnaire item	1	2	3	4	5
	Strongly disagree	Disagree	Can’t say	Agree	Strongly agree
I was assessed by 3 or more assessors	0	0	0	3	13
Adequate time (10-15 min) was given by the assessors for the exercise	0	2	0	8	6
Being observed by the assessor affected my performance adversely	4	7	2	2	1
Feedback on my performance was provided	0	0	0	12	4
The feedback was constructive (my strengths were mentioned, and weaknesses were pointed out in a nonthreatening way with comments on how to improve)	0	0	0	9	7
I was troubled by multiple times the assessment was conducted	10	6	0	0	0
Having examined by many assessors is a good idea	0	0	0	11	5
Mini-CEX can be incorporated for internal assessment	0	0	0	12	4

Mini-CEX: Mini-clinical evaluation exercise

In the words of residents:

Mini-CEX is a great opportunity to get assessed by different examiners and know the shortcomings from time to time so that we can improve gradually

This type of assessment is better than our exam where we are assessed for our learning of 3 years in just 1 day on few cases

It was more relaxed than routine exams, which was inherently different for our set up

The exercises appeared very casual

The exercises should have been more frequent and at the end of 6 months should have been graded by a point-based system as is done in IITs.

Performance of residents

There was consistent improvement in the performance of the residents with the conduction of the exercises, however, we did not observe any significant differences in their performances when different assessors assessed them.

Discussion

We observed that conducting mini-CEX is feasible in the existing setting without stretching the resources, this has also been observed by other authors.[6910] We did not had any difficulty in ensuring the workforce, clinical material, and stationary required to carry out the exercises. The biggest hurdle was the motivation of the faculty members for participating in the study. Two of the senior faculty members expressed their inability to participate citing their unavailability due to other professional commitments. This was despite the assurance that mini-CEX was not time-consuming and can seamlessly be incorporated in their existing schedule. Rest of the faculty accepted to participate although there was some difficulty initially to keep their motivational levels up. Once the faculty became comfortable with mini-CEX, the task became progressively easier.

The exercises conducted during this study were not scheduled with respect to time, considering the first exposure of mini-CEX and also to provide faculty with the liberty to complete the exercises at their own comfort. This was found to be a limiting factor in conducting the desired number of exercises in time. We learned that it is better to schedule the exercises at the outset for their smooth conduction. Mini-CEX was well acceptable to the faculty and students alike despite few apprehensions. The primary concern of some faculty members that mini-CEX requires more time was apparent as in few exercises the time taken was nearly 30 min. This reflects inexperience of the faculty members with mini-CEX as they tried to assess multiple skills at the same time and considered it comparable to a directly supervised traditional clinical case presentation. This can be overcome by providing the faculty with a small training on mini-CEX. Although we deliberated them about mini-CEX at the outset, more training in conducting the exercises would have offset this apprehension. In a review on perceptions on WPBA, improper training of assessors has been cited as one of the major shortcomings in acceptance of WPBA.[14] Other authors have also expressed the importance of proper training of assessors.[6151617] Another concern of faculty regarding difficulty in conducting the exercises often also reflects the low motivation on their part. The time taken for conduction of each exercise was small, so we believe that this apprehension is more a mental block than a real problem. Massie and Ali have also cited this as one of the shortcomings that affect the acceptance of WPBA adversely.[14] In a system, once some form of WPBA is firmly incorporated, and faculty assignments are well laid down, there will be less inertia in conducting the exercises.

The students accepted the tool very well. This was reflected by their high level of satisfaction with the exercises performed. Some of our residents reported anxiety as the teacher was watching them during their performance. This is not surprising as in the present system of assessment; there is usually dissociation between the examiner and the student while the student is actually “performing” on the patient. This has also been observed by other authors[618] except Goel and Singh.[10] With more exposure on ‘being observed’ and the recognition that it enhances learning, we have sincere hope that this will not disturb the student as also conveyed by Malhotra et al.[7]

Mini-CEX provides a unique opportunity to discuss the weaknesses of the candidate identified at the conclusion of the exercise. This empowers them for consistent improvement in the weak areas so identified. We observed consistent improvement in the performance of our residents over the period when mini-CEX was conducted.

This is the first study on the use of mini-CEX for formative assessment of postgraduate students of surgery form India. The present study is also unique as we could overcome some of the shortcomings reported by other Indian authors on mini-CEX, like the varying status of the assessors,[6] unable to complete the requisite number of exercises,[611] exercises conducted in limited settings,[611] less number of exercises,[9] each candidate examined by only one examiner,[9] and evaluation of a single competency.[11] The findings from this small study may not be generalized and more similar studies need to be conducted to support the evidence generated from this initial work.

Conclusions

Mini-CEX is acceptable to the faculty members and residents as a tool for formative assessment for WPBA. It is feasible to carry out mini-CEX without stretching the existing infrastructure and humankind. Mini-CEX has a potential to supplement or replace the existing system of IA.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.