Mini-Clinical Evaluation Exercise as a Tool for Formative Assessment of Postgraduates in Psychiatry.

CONTEXT: Introduction of competency-based medical education in India has necessitated a diversion from traditional methods of teaching and assessment. AIM: The aim of this study was to assess the feasibility and acceptability of mini-clinical evaluation exercise (mini-CEX) as a tool for formative assessment of postgraduates in psychiatry at our institute. SETTINGS AND
DESIGN: It was a prospective, quasi-experimental study carried out in the Department of Psychiatry, PGIMS, Rohtak (India).
MATERIALS AND METHODS: After obtaining written informed consent, trainees (18) were sensitized and faculty members (5) were trained to use mini-CEX and regarding the feedback process with the help of workshops. STATISTICAL ANALYSIS USED: Data collected were analyzed using descriptive statistics. Mean differences between the sessions were assessed using a paired sample t-test. P < 0.05 was considered as level of significance.
RESULTS: Statistically significant difference was found between session 1 and session 6 for counseling skills but not for three sessions of diagnostic skills. Mini-CEX was accepted as a better learning method by 61% of the postgraduates, 36% felt that it improved their clinical skills, 64% reported a boost in their confidence, and 89% looked forward to these encounters. Majority of the postgraduates and faculty members suggested that the mini-CEX should be regularly used in academic settings and should be introduced early in the training.
CONCLUSIONS: Mini-CEX is a feasible formative assessment tool for postgraduates in psychiatry and an effective method of learning by the postgraduates, inducing a significant improvement in the counseling skills of students. Teaching faculty also accepted mini-CEX as an effective method of formative assessment. Copyright:

Introduction

Assessing the postgraduates for various clinical skills at workplace is important though it needs a lot of resources and time. With a shift from traditional model to competency-based medical education, the role of formative assessment is paramount. This becomes all the more important in a specialty like psychiatry where interviewing and counseling skills are critical for successful clinical practice. However, medical students report that they are rarely being observed during patient encounters and one main reason for this is lack of faculty time.[1] Direct observation is mandatory for the reliable and valid assessment of interviewing and counseling skills.[2] It also requires multiple assessments over time.[3] In-training assessments done at the end of a term introduce a “halo effect.”[4] Mini-clinical evaluation exercise (mini-CEX) developed by the American Board of Internal Medicine helps to overcome these issues.[5] The mini-CEX involves direct observation of trainees in a focused clinical encounter followed by immediate feedback. Mini-CEX has not been used that often as a feedback tool in psychiatry, probably because of lack of awareness and limited resources. To best of our efforts, we could identify only three studies in this regard.[678] Hence, we planned to take up this study.

Materials and Methods

After obtaining the ethical approval from the Institutional Ethics Committee and written informed consent from the participants, the study was conducted from April 2019 to September 2019 in the Department of Psychiatry, Pt. BD Sharma PGIMS, Rohtak. The department has an annual intake of ten postgraduate students for M. D Psychiatry course of 3 years.

For the purpose of this study, all the 2nd- and 3rd-year trainees (19) and faculty members (6) were invited to participate in this study. Consenting postgraduates (18) were sensitized and teaching faculty members (4) were trained to use mini-CEX and regarding the feedback process with the help of workshops conducted by the investigator (SS). Feedback from the participants was used to address any anticipated difficulties. A mock drill was conducted to pilot test the introduction of mini-CEX with 3 first year postgraduates. These residents were not involved in the main study nor was this data used in the analysis. A tentative roster of mini-CEX sessions was prepared, ensuring that the trainee rotates through all the 5 assessors (including investigator) and that no two consecutive sessions are with the same assessor. The minimum time interval between two consecutive mini-CEX encounters was 2 weeks. One of the postgraduates faced only 3 mini-CEX encounters of diagnostic skills and left the institute. After the completion of the study, feedback from faculty members and postgraduate students was gathered using the feedback forms specially designed for the purpose and validated by the senior faculty members who did not participate in the study and medical education unit members. Postgraduates were asked to complete another scale also, Learning Self-Efficacy Scale for clinical skills.[9]

Permission to use these scale has been granted by the authors.

Data collected from mini-CEX encounters and feedback from faculty members and students were analyzed using SPSS. Descriptive statistics were generated for each of the domains on the mini-CEX. Mean differences between the sessions were assessed using a paired sample t-test.

Results

Every postgraduate (except one student as mentioned earlier) faced 9 mini-CEX encounters: 3 encounters for diagnostic skills and 6 encounters for counseling skills. A total of 156 mini-CEX encounters were carried out by 5 faculty members for 17 (18) postgraduates. Level of case complexity was based on the clinical expertise and judgment of the assessing clinician [Table 1].

Table 1

Basic details of mini-clinical evaluation exercise encounters

Number of residents	Diagnosis (n=18)	Counseling (n=17)*
Gender
Male	9	9
Female	9	8
Residency year
II	8	8
III	10	9
Number of assessments	156
Patient setting, n (%)
IPD	44 (28.20)
OPD	112 (71.79)
Patient, n (%)
New	84 (53.85)
Follow-up patients	72 (46.15)
Case complexity, n (%)
Low	50 (32.05)
Moderate	99 (63.46)
High	7 (4.48)
Patient’s diagnoses, n (%)
Bipolar disorder	28 (17.94)
Depression	59 (37.82)
Schizophrenia	45 (28.84)
OCD and alcohol dependence	7 each (8.96)
Others	10 (6.44)

*One of the postgraduates had left after 3 encounters. OCD: Obsessive-compulsive disorder; OPD: Outpatient department; IPD: Inpatient department

Table 2 shows the statistical comparison between various sessions of mini-CEX on two different competencies, i.e., diagnostic skills (interviewing and mental state examination) and counseling skills. A comparison of sessions 1 and 3 of mini-CEX for diagnostic skills shows that there was no statistically significant difference between 2 sessions except in the domain of evaluator satisfaction.

Table 2

Comparison of mean scores for diagnostic skills assessment sessions and for counseling skills assessment sessions

Diagnostic skills (n=18)	Session 1 (mean)	Session 3 (mean)	SD	t	df	P
Interview	3.6111	3.8333	1.395	−0.676	17	0.508
Mental status examination	3.33	3.83	1.294	−1.638	17	0.120
Humanistic qualities/professionalism	4.05	4.05	1.714	0.000	17	1.00
Organization/efficiency	3.61	3.72	1.182	−0.399	17	0.695
Overall clinical competence	3.38	3.77	1.092	−1.511	17	0.149
Evaluator satisfaction	7.22	5.88	1.414	4.00	17	0.001
Resident satisfaction	6.77	7.05	1.673	0.704	17	0.491
Counseling skills (n=17)*	Session 1 (mean)	Session 6 (mean)	SD	t	df	P
Humanistic qualities/professionalism	4.47	6.64	1.380	−6.503	16	0.000
Counseling skills	3.52	6.29	1.032	−1.040	16	0.000
Organization/efficiency	3.47	6.11	0.86	−12.66	16	0.000
Overall clinical competence	3.52	6.29	0.970	−11.75	16	0.000
Evaluator satisfaction	6.17	7.29	1.166	−3.951	16	0.001
Resident satisfaction	6.82	7.47	1.221	−2.184	16	0.044

SD: Standard deviation

However, a comparison of session 1 with session 6 of counseling skills shows a statistically significant difference among all the domains.

Figures 1 and 2 depict the residents and faculty feedback on mini-CEX. In response to open-ended questions, 61% of the postgraduates mentioned that the mini-CEX was a better method of teaching as compared to other methods, 36% felt that it improved their clinical skills, 64% reported a boost in their confidence, and 89% looked forward to these encounters. Majority of the postgraduates did not find any drawback in the exercise though around a third (28%) of them reported experiencing anxiety and stress in the initial parts of assessment. A significant proportion (88%) of residents suggested that the mini-CEX should be regularly used in academic settings. On a similar note, most of the consultants who were already trained in various assessment exercises found that mini-CEX was better than the traditional assessment methods such as outpatient department and bedside case discussion or case conference as it provided immediate feedback (80%), remains focused on the task assigned for the encounter (60%), and has a structured pattern of evaluation (60%). However, subjective bias and time constraints for assessment were considered to be significant drawbacks by 40% of the faculty members. Despite these shortcomings, most of the faculty members (70%) recommended mini-CEX as a good formative assessment tool and that it should be introduced early in the training. Table 3 shows the positive perception of residents in all three domains on the Learning Self-Efficacy Scale. [Table 3].

Figure 1

Faculty feedback on mini-clinical evaluation exercise . *Questions that were negative and scores were reversed

Figure 2

Student feedback on mini-clinical evaluation exercise (n = 17). *Questions that were negative and scores were reversed

Table 3

Learning Self-Efficacy Scale for clinical skills

	Strongly disagree (%)	Disagree (%)	Neutral (%)	Agree (%)	Strongly agree (%)
Cognitive
I can recall how to perform psychiatric evaluation	0	0	5.5	61.1	33.3
I understand the content of psychiatric evaluation and can demonstrate it to others	0	0	5.5	22.2	72.2
I can verbally explain the purpose and principle of conducting psychiatric evaluation	0	0	0	72.2	27.8
I can verbally explain the sequence and interrelationship between each step	0	0	16.7	61.1	22.2
Affective
I think I spend more time on clinical evaluation than on others	5.5	11.1	38.8	33.3	11.1
I think I gain more doing so than in other activities	0	0	0	61.1	38.9
I tend to pay more attention to information related to psychiatric evaluation	0	11.1	0	72.2	16.7
I tend to actively look for information related to psychiatric evaluation.	0	0	16.7	50	33.3
Psychomotor
I can precisely imitate the instructor’s steps and actions of psychiatric evaluation	0	0	11.1	72.2	16.7
I can smoothly complete the steps of psychiatric evaluation	0	0	16.7	66.6	16.7
I try to monitor my clinical skills for improvements.	0	0	0	66.6	44.4
I try to monitor my clinical skills and make proper adjustments as needed	0	0	0	83.3	16.7

Discussion

The direct observation-cum-evaluation approach is an effective way of learning for residents, particularly in the specialty of psychiatry, where diagnostic as well as communication skills are of paramount importance. Direct observation and timely feedback lead to a significant improvement of the clinical skills of the trainee residents, and findings of the index study support this assertion.

Because of the time constraints, we could focus only on two competencies, i.e., diagnosis and counseling skills. It was noticed that the difference between the mean score for each of the items on mini-CEX for session 1 and session 6 of counseling skill assessment was statistically significant. Findings of the index study corroborate that of Gupta et al.[10] on pediatric residents and Saeed et al.[11] on undergraduates. Khalil et al.[12] in their study with pediatric residents drew similar inference and concluded that it was feasible to use mini-CEX as a formative assessment tool with high acceptability among both faculty and residents.

However, the difference of mean scores between session 1 and session 3 of diagnostic skill assessment was found to be statistically significant only in the area of evaluator satisfaction. The absence of significant difference in mean score for most of the items on diagnostic skill evaluation can be attributed to insufficient number of sessions for this competency. It can be possible that a few more sessions could have led to better diagnostic skills and hence significant difference in most of the items on mini-CEX. Alves de Lima et al.[13] used mini-CEX for 17? cardiology residents and suggested that although the tool was valid and reliable, it was not feasible. It was evident in their study that it was not easy to achieve the number of encounters required and a greater number of sessions are required to mark a difference as has been suggested by the developers of the tool.[5]

The feedback from the participants reflects that both the students and faculty accepted the mini-CEX very well. It is further corroborated by the feedback? of students wherein they looked forward to mini-CEX encounters as well as desired for more of these. Although some of the residents (28%) reported initial anxiety and stress, they became comfortable as the encounter progressed. These students wanted to continue with further encounters, and most of the students agreed that this exercise helped them to learn better and improved their clinical skills. A significant proportion (88.8%) of residents suggested that the mini-CEX should be regularly used in academic settings.

Most of the faculty members who were already trained in various assessment exercises found that mini-CEX was better as it provided immediate feedback (80%), has focused (60%) and structured pattern of evaluation (60%), and should be introduced early in the training. However, subjective bias and time constraints for assessment were perceived to be significant drawbacks by both faculty and students, which was in line with the results of previous studies.[101314] However, the completion of all the planned encounters proves its feasibility. Regular use, constant reinforcement, and positive attitude can overcome these constraints.

This is further corroborated by the perceptions of the residents on the Learning Self-efficacy Scale, which strongly suggested improvement in clinical skills in all the domains – cognitive, affective, and psychomotor. It underlines the perception of improvement in self-skills with mini-CEX among residents, which justifies its utility as suggested by other studies also.[1015] It can be, therefore, implied that mini-CEX as a tool has good acceptability and feasibility, as suggested by the feedback responses, and this substantiates the findings of earlier studies.[678101617]

Conclusions

The study concludes that it is feasible to use mini-CEX for formative assessment of postgraduates in psychiatry. Furthermore, it was considered to be an effective method of learning by the postgraduates and was found to induce a significant improvement in the counseling skills of students. Teaching faculty also accepted mini-CEX as an effective method of formative assessment.

The strength of the index study lies in the fact that it is for the first time in this part of India, and the faculty and residents were exposed to workplace-based method of formative assessment using a wide diversity of psychiatric diagnoses. However, due to time constraint, the number of mini-CEX encounters for each competency was restricted and assessment could be made only in two competencies.

We continue to use mini-CEX for formative assessment as a routine practice in the department and plan to introduce mini-CEX as a method of formative assessment as early as possible in the training.

Ethical clearance

Study was conducted after approval from the Institutional Ethics Committee.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.