Educational impact of the mini-Clinical Evaluation Exercise in resident standardization training: a comparative study between resident and professional degree postgraduate trainees.

OBJECTIVE: We aimed to explore differences in the educational impact of the mini-Clinical Evaluation Exercise (mini-CEX) on resident (RE) and professional degree postgraduate (PDPG) trainees, as well as influencing factors, to provide suggestions for hospital managers, trainers, and trainees.
METHODS: We performed a retrospective analysis of all scores among first-year resident standardization training trainees registered during 2017 to 2019 at Xinqiao Hospital of Army Medical University, to identify differences in mini-CEX outcomes between REs and PDPGs.
RESULTS: We collected data of 154 registered trainees for retrospective analysis, including 57 PDPG trainees and 97 RE trainees. The mean (standard deviation) overall performance score of PDPGs was 84.18 (4.25), which was higher than that of REs (81.48 (3.35)). In terms of domain analysis, PDPG trainees performed significantly better than REs in history taking, physical examination, clinical diagnosis/treatment regimen, and the knowledge examination; communication skills/humanistic care were comparable between the groups.
CONCLUSIONS: PDPGs performed better than REs in overall competency, history taking, physical examination, clinical diagnosis/treatment regimen, and the knowledge examination. A better knowledge base, supervisor-dominated one-to-one teaching mode, higher self-esteem and learning goals, and more sophisticated responses to feedback were potential contributors to a superior educational impact of the mini-CEX.

Introduction

Competency-based medical education (CBME) has been used in the past few decades by trainers, trainees, and regulatory bodies for clinical competency training of medical students. CBME is defined as “an outcomes-based approach to the design, implementation, assessment, and evaluation of medical education programs.”[1,2] Unlike traditional education that can be characterized as “fixed time, variable outcomes,” CBME focuses on desired outcomes. The assessment of clinical skills is a key component of CBME. However, traditional assessment, which is usually conducted using objective structured clinical examinations that include multiple-choice questions, focuses on knowledge in the basic or clinical sciences rather than competencies in actual clinical encounters, thereby making assessment of clinical competence challenging.[2,3]

Workplace-based assessment (WPBA), which takes place as part of daily medical work rather than in an artificial setting, has been embraced as a response to the challenge of competency assessment using traditional approaches. WPBA has been identified as a component of formative assessment for competencies and outcomes of resident training in many institutions worldwide.[4-6] Multiple WPBA methods have been developed, including the mini-Peer Assessment Tool (mini-PAT), mini-Clinical Evaluation Exercise (mini-CEX), direct observation of procedural skills (DOPS), and case-based discussion, among which the mini-CEX is the most widely recommended and used.[7] The mini-CEX was developed by the American Board of Internal Medicine in the 1990s, based on the traditional CEX or long case assessment.[8] In the mini-CEX, an evaluator observes a trainee’s performance during a normal clinical encounter lasting 15 to 20 minutes. The evaluator then rates the trainee on medical interviewing skills, physical examination skills, professionalism, clinical judgment, counseling skills, organization, efficiency, and overall competence using a 9-point scale, followed by immediate feedback at the end of the consultation lasting 5 to 15 minutes.[9] The reliability and validity of the mini-CEX in evaluating trainees’ clinical competence have been confirmed by many researchers and institutions.[10-12] Moreover, because of the associated feedback component, the mini-CEX is also regarded as a powerful teaching and learning tool, to improve clinical achievement among trainees.[13,14]

In response to CBME and to ensure high competence levels among clinicians for high-quality health care, the Chinese government launched the resident standardization training (RST) program in 2015. Owing to its superb reliability, effectiveness, convenience, and multifunctionality, the mini-CEX is recommended as a formative assessment tool in the Chinese RST program.[15,16] After specialized modification based on the original format, the mini-CEX has been used for the evaluation of interns, residents, and postgraduates in clinical medicine and medically associated professions during the 3-year RST program. Our hospital is a certified RST institution, with more than 300 trainees enrolled in RST each year, and the mini-CEX is conducted monthly during rotations.

Previous studies have shown that the educational impact of the mini-CEX can be substantially influenced by several factors.[17,18] One factor is the purpose for which the mini-CEX is used. Compared with summative assessment or a combination of formative and summative assessment, using the mini-CEX as formative assessment has been demonstrated to be more beneficial to learning.[19] Because feedback exerts one of the most important influences on achievement, the manner in which supervisors provide feedback and trainees’ perception and interpretation of the feedback received are potentially important influencing factors. Other factors associated with the context, users, implementation, and outcome of the mini-CEX have been reported to influence the educational impact of this method.

According to official documents and previous studies, resident (RE) and professional degree postgraduate (PDPG) trainees are the two main groups compulsorily enrolled in the Chinese RST program. Although both RE and PDPG trainees have finished the 5-year medical undergraduate course, a national entrance examination must be passed to become a PDPG trainee, thereby distinguishing the two categories of trainee. However, there are limited comprehensive analyses of differences in the implementation and educational impact of the mini-CEX between RE and PDPG trainees as well as little exploration of potential factors influencing this impact. In addition, all trainees enrolled in the Chinese RST program undergo mini-CEX evaluation for the same procedure, which may result in suboptimal educational outcomes among trainees.[20] With the aim to provide suggestions for hospital managers, trainers, and trainees to optimize the educational impact of the mini-CEX on different groups of trainees, we retrospectively analyzed the differences in educational impact of the mini-CEX between RE and PDPG trainees enrolled in RST at our hospital, and we investigated the potential influencing factors.

Methods

In this study, we first performed a retrospective analysis of all scores among trainees for whom we collected data in this study, to determine the differences in mini-CEX outcomes between RE and PDPG trainees. We then identified potential influencing factors that may lead to differences in the mini-CEX outcomes between RE and PDPG trainees.

Assessment using the mini-CEX

To ensure reliability and validity, as well as optimize the educational impact of the mini-CEX, one or two senior doctors with extensive clinical and medical teaching experience are selected as evaluators. A workshop comprising a pretest, special lecture, video, and group discussion is held for all selected evaluators to introduce the background, principle, concept, purpose, function, and procedure of the mini-CEX, so as to ensure that all evaluators have a good understanding of the mini-CEX. At the end of the workshop, simulation practice of mini-CEX assessment is conducted.

All registered RST trainees are provided with information about the mini-CEX during lectures, with emphasis on using it as a learning tool via immediate feedback from the trainers, based on trainees’ clinical performance. Identical written materials and videotapes are provided to all trainees. During rotations, the same mini-CEX is performed one to three times per discipline, as an assessment and teaching tool for every trainee registered in the RST program. We have adopted a modified Chinese-language mini-CEX form (Appendix I), which is based on the original developed by the American Board of Internal Medicine. The modified mini-CEX includes 4 domains and 18 subdomains covering 6 dimensions of clinical competence in medical interviewing skills, physical examination skills, counseling skills, clinical judgment, humanistic qualities/professionalism, and organization and efficiency. Each subdomain is rated on a 5-point scale, with scores of 1 and 2 indicating unsatisfactory performance, 3 and 4 satisfactory performance, and 5 excellent performance. Unsatisfactory performance (scores 1–2) is defined as “poor performance” (1 point) and “partially incorrect in principle, requiring intervention” (2 points). Satisfactory performance (scores 3–4) is defined as “partially incorrect, not in principle” (3 points) and “correct” (4 points). Excellent performance (score of 5) is defined as “proficiency in all dimensions.”

At the end of each academic year, all trainees are expected to undergo a knowledge examination, mini-CEX assessment, comprehensive defense, and direct observation of procedural skills, to evaluate educational outcomes of the previous 1 year in the RST program. In implementation of the final mini-CEX each academic year, each trainee is rated by three experienced examiners who are not known to each other, to reduce measurement error. Final scores are determined by averaging the scores recorded by the three examiners.

In the present study, we retrospectively collected data from first-year trainees registered in the RST program at Xinqiao Hospital of Army Medical University during 2017 to 2019. RE trainees who obtained a medical master’s degree or medical doctoral degree were excluded from this study. Ethical approval for the present study was obtained from Medical Ethics Committee of Second Affiliated Hospital of Army Medical University, PLA. We obtained verbal informed consent from all participants to use their scores in this study. In addition, we removed all the personal information from all data used in the retrospective analysis.

Statistical analysis

Continuous variables are expressed as mean (standard deviation; SD) and categorical variables are summarized as proportions. Statistical analysis was conducted using SPSS version 13.0 software (SPSS Inc., Chicago, IL, USA). Statistical comparisons were made using the Student t-test or Pearson correlation analysis. A p-value <0.05 indicated a significant difference.

Results

Data of 154 first-year trainees registered in the RST program at Xinqiao Hospital of Army Medical University during 2017 to 2019 were collected for retrospective analysis, including 57 PDPG trainees and 97 RE trainees. All RE trainees had completed an undergraduate medical education course whereas PDPG trainees were enrolled in a postgraduate program. Among the total, there were 41 men (71.93%) in the PDPG group and 42 men (43.30%) in the RE group. The mean age (SD) in the PDPG and RE group was 24.84 (1.56) (range 23–29) years and 24.23 (0.95) (range 23–27) years, respectively.

Table 1 provides descriptive statistics for trainees’ scores on the mini-CEX and knowledge examination. The mean score (SD) of the PDPG group for overall performance was 84.18 (4.25), which was higher than that of RE trainees (81.48 (3.35), p < 0.05) (Figure 1). Domain analysis indicated that PDPG trainees performed significantly better than RE trainees in history taking (p < 0.05), physical examination (p < 0.05), and clinical diagnosis/treatment regimen (p < 0.05); there was no significant difference in communication skills/humanistic care between the two groups. PDPG trainees also showed significantly higher scores in the knowledge examination than RE trainees (p < 0.05) (Figure 1). The mean (SD) observation and feedback time of the PDPG group was 19.25 (1.56) minutes and 7.65 (0.33) minutes, respectively, which was not significantly different from that of the RE group, 20.16 (3.83) minutes and 8.08 (1.72) minutes, respectively.

Table 1.

Descriptive statistics for trainees’ scores on the mini-CEX and knowledge examination.

	mini-CEX					Knowledge examination
Group	History taking	Physical examination	Clinical diagnosis/treatment regimen	Communication skills/humanistic care	Overall
PDPG	4.34 (0.74)	4.31 (0.63)	4.47 (0.60)	4.30 (0.40)	84.18 (4.25)	97.05 (10.67)
RE	4.16 (0.34)	4.05 (0.82)	4.08 (0.82)	4.43 (0.82)	81.48 (3.35)	84.88 (10.51)

Note: Values are presented as mean (standard deviation).

Abbreviations: CEX, Clinical Evaluation Exercise; PDPG, professional degree postgraduate; RE, resident.

Figure 1.

Scores on the Mini-Clinical Evaluation Exercise (CEX) (a) and knowledge examination (b) for professional degree postgraduate (PDPG) and resident (RE) trainees.

Discussion

China has the largest population in the world, as well as an aging demography. To meet the increasing demand for qualified general practitioners, the National Health and Family Planning Commission of China proposed the RST program in 2015, to train medical graduates to become qualified practitioners.[15,21] Qualified practitioners should have good skills in medical interviewing, physical examination, professionalism, clinical judgment, counseling, organization, efficiency, and overall competence. However, longstanding Chinese medical education methods are heavily focused on medical technology, with little content addressing humanities, ethics, communication skills, or public health; thus, at present, the goals of the RST program cannot be met. In addition, as part of the education process, evaluation plays a very important role. However, the traditional evaluation approaches highlight diagnostic accuracy, treatment effectiveness, and clinical technology rather than humanistic care and medical ethics.[22] Thus, new teaching and evaluation methods are warranted to improve the quality of care.

The mini-CEX is a multifunctional tool for assessing trainees’ clinical skills as well as for supplementing teaching via immediate feedback from a knowledgeable rater, to help students identify their strengths and weaknesses in continued learning.[13] In the mini-CEX, a trainee’s clinical competence during a normal clinical encounter is comprehensively rated in six dimensions, including medical interviewing skills, physical examination skills, counseling skills, clinical judgment, humanistic qualities/professionalism, and organization and efficiency. Owing to its effectiveness, convenience, and timesaving properties, the mini-CEX is regarded as one of the most powerful WPBA methods and is used in a large number of medical schools for residency assessment.[23] The Chinese government also recommends the mini-CEX as a formative assessment tool in the RST program.

RE and PDPG trainees comprise two major groups compulsorily enrolled in the Chinese RST program, who receive the same clinical training and mini-CEX evaluation.[20] Interestingly, we determined that there were significant differences in the educational impact of the mini-CEX between these two groups in our study. Specifically, PDPG trainees had better performance than RE trainees in most domains of the mini-CEX. PDPG trainees’ mean score (SD) for overall performance was 84.18 (4.29), which was significantly higher than that of RE trainees with 81.48 (3.37). PDPG trainees also performed significantly better than RE trainees in history taking, physical examination, and clinical diagnosis/treatment regimens, although similar scores were observed in terms of communication skills/humanistic care in the two groups. Several factors may account for the observed differences. It has been reported that the educational impact of the mini-CEX may be influenced by various factors. Lörwald et al.[17] conducted a systematic literature review and qualitative synthesis of these factors. Their qualitative study revealed four themes and nine subthemes associated with the educational impact of the mini-CEX, including context, users, implementation, and outcome. In another meta-analysis conducted by Lörwald et al.,[18] the authors systematically studied the potential influences of mini-CEX implementation on educational impact. Their analysis revealed that mini-CEX quality and participant responsiveness were positively associated with the educational outcomes of the mini-CEX. The knowledge base of trainees has also been regarded as an important factor influencing their clinical performance. Studies in behavioral economics and medical education have suggested that clinical performance is guided by medical theoretical knowledge.[24,25] In this study, differences in scores between the PDPG and RE groups may be partially owing to knowledge base discrepancies. The national entrance examination is a written test focusing on medical knowledge assessment that is taken by students who wish to pursue postgraduate studies. Thus, PDPG trainees usually have a better medical knowledge base, allowing them to obtain higher scores in the mini-CEX. In addition, PDPG trainees obtained higher scores on the end-of-year-summative theoretical knowledge test (97.05 [10.67) than RE trainees, further proving that the former group had a better knowledge base. In addition, each PDPG trainee has a supervisor, who is usually an associate professor or professor with extensive clinical experience. This one-to-one teaching scheme provides plenty of opportunities for PDPG trainees to improve their clinical knowledge and skills with the guidance of an experienced senior doctor, which can also lead them to perform better on the mini-CEX.

Studies have identified feedback as one of the most important factors influencing educational impact.[26] Receiving appropriate critical feedback promotes the identification of strengths and weaknesses within trainees’ clinical competencies, which is crucial for effective teaching and learning. Feedback in clinical education may be defined as “specific information about the comparison between a trainee’s observed performance and a standard, given with the intent to improve the trainee’s performance.”[27] Not only the provision of feedback but also its content and how it is provided are important. The effects of feedback are sometimes equivocal and confusing because feedback can both increase and decrease motivation and performance. More than 33 variables have been revealed to affect the process and outcome of feedback.[28] In particular, four variables influence the outcome of feedback received by trainees: trainees’ initial skill level, self-esteem, goal-setting behavior, and feedback content. In this study, although feedback was provided by the same raters and delivered to trainees using the same method, the trainees had different levels self-esteem and goal-setting behaviors. In China, the huge population has led to the country’s medical education system becoming the largest worldwide, which has also brought about many challenges for the medical educational system. In the past decade, the Chinese government has struggled to balance the medical needs of the country’s huge population with the shortage of medical staff by adopting various durations of medical degree. When students enroll in a medical college directly from high school, the curriculum spans a duration of 3 (for a diploma), 5 or 6 (for a bachelor’s degree), 7 (for medical master’s degree) or 8 years (for medical doctoral degree). In another stepwise education system, students can progress from a bachelor’s to a master’s degree in 3 years, and then to an MD or PhD in another 3 years.[29] Students who obtain the medical bachelor’s degree are eligible to take the licensure examination and work as licensed doctors after passing the examination. By completing postgraduate medical courses, students can gain rich clinical experience and maturity and can go on to become world-class academic researchers and clinical practitioners. Thus, compared with the RE group, PDPG trainees usually have higher self-esteem and learning goals, leading them to respond more positively to feedback. In addition, medical literature review is an essential component of the postgraduate curriculum whereas this is optional for undergraduate students. Thus, PDPG trainees tend to be more versatile in their responses to the same feedback as they likely pay greater attention to feedback outside the assessment system at different time points, which results in a greater educational impact.

Conclusions

In this study, we retrospectively analyzed scores on the mini-CEX and knowledge examination of 154 first-year trainees registered in the RST program, including 57 PDPG trainees and 97 RE trainees. We found that PDPG trainees performed significantly better than RE trainees in history taking, physical examination, clinical diagnosis/treatment regimen, and the knowledge examination; the two groups performed similarly with respect to communication skills/humanistic care. We discussed potential reasons for the differences in educational outcomes of the mini-CEX between the PDPG and RE groups; we proposed that a better knowledge base, supervisor-dominated one-to-one teaching mode, higher self-esteem and learning goals, and more positive responses to feedback may lead to better educational scores in the mini-CEX on the part of PDPG trainees.