The Role of Integrative Educational Intervention Package (Monthly ITE, Mentoring, Mocked OSCE) in Improving Successfulness for Anesthesiology Residents in the National Board Exam.

BACKGROUND: National Board of Anesthesiology (NBA) pass rate is an important and critical step in clinical residency programs.
OBJECTIVES: This study was designed to assess the relationship between an integrative educational intervention (IEI) and the relative annual pass rate (RAPR). RAPR is defined as ratio of NBA pass rate of Shahid Beheshti University of Medical Sciences (SBMU) to the NBA pass rate of all the anesthesiology residency programs across Iran.
METHODS: In a descriptive-analytic retrospective study from 2012 to 2019, RAPR was calculated. IEI was implanted in the latter 4years period of this time interval includes: (1) individualized mentorship for residents by faculty members; (2) monthly in-training examination (ITE) in written; and (3) periodical mocked OSCE exam. Spearman's correlation coefficient was used to assess correlation between integrative educational intervention and RAPR results. P value less than 0.05 was considered statistically significant.
RESULTS: There was a statistically significant relationship between "integrative educational intervention program" and the RAPR results: Spearman's correlation coefficient = 0.655 (P value = 0.039).
CONCLUSIONS: The IEI package of Anesthesiology Department, SBMU showed a significant relationship with improvements in successfulness for anesthesiology residents in the National Board Exam (RAPR trend). More prolonged studies could prevail further aspects of these interventions.

1. Background

Trainee assessment is one of the main steps in medical education and training process; both for undergraduate and postgraduate trainees (1-3). Passing the National Board of Anesthesiology certifying exam is among the criteria for assessment of residency programs (4-6). In addition, it is a major challenge for each resident, as a main step in being certified for clinical residents in medical specialties (7). Though it is a main success to pass the exam, the time and effort needed to be prepared for the exam is a major stress-provoking factor for residents. Add to this stress, the possibility that there is a chance for failing the exam, which affects especially those who have failed. This stress is among the main factors for resident burnout (8, 9). Anesthesiology is among the residency programs with a significant degree of resident burnout in some studies, being in top of the resident burn out list (9-12). Institutional support could improve well-being in residents (11). In addition, assessments of trainees are among the fields necessitating research that is more sophisticated; mandating more collaborative research to improve assessment methods (13).

There have been a number of methods to increase the success rate and to improve the capacity of the residents in passing the exam. However, none has proved to be perfect yet and research in this field is still active. In this study, we assess the trend of pass rates in National Board Exam after implementing an integrative educational intervention (IEI) consisted of monthly written exam, periodical mocked objective structured clinical exams (OSCE) and individualized mentorship program in Anesthesiology residents, Shahid Beheshti University of Medical Sciences (SBMU), Tehran, Iran (14).

2. Objectives

The trend was assessed over the 8-year time interval to see the relationship between period for interventions and the pass rates in National Board Exam.

3. Methods

This is a descriptive-analytic retrospective study performed through an 8-year period (2012 - 2019). In this study, the trend of pass rate was assessed. This time interval from 2012 to 2019 included one 4-year interval before the implementation of the IEI and a 4-year interval in which the following package of IEI program was performed throughout the whole time interval:

1) Individualized mentorship for residents by faculty members; described in detail in another study (14);

2) Monthly in-training examination (ITE) in written multiple choice questions (MCQ) format;

3) Periodical mocked OSCE exam.

Anesthesiology Department Examination Committee (ADEC), SBMU, performed these interventions from A to Z with participation of all faculty members, based on a standard and peer reviewed departmental protocol which is described before (14). To make the annual pass rate comparison more standardized, the annual pass rate of Anesthesiology Department, SBMU was compared with the nationwide annual pass rate of the National Board exam using the following ratio:

Relative annual pass rate (RAPR) = pass rate of SBMU anesthesiology residents/pass rate of the National Board of Anesthesiology.

Spearman’s correlation coefficient was used to assess the possibility of correlation between the integrative educational intervention and the RAPR results. One-tailed correlation coefficient was considered statistically significant at the 0.05 level. All statistical analyses were done using SPSS (version 11.5; SPSS Inc, Chicago, IL, USA). P values < 0.05 were considered statistically significant.

4. Results

RAPR results are demonstrated in Table 1. These results showed that RAPR increased annually during the years that the IEI (i.e. mentorship, monthly written exams and periodical mock OSCE) was done. The assessment of relationship between IEI and the RAPR results demonstrated a significant relationship: Spearman’s correlation coefficient = 0.655 (P value = 0.039).

Table 1.

Annual Results of RAPR[a]

Year	RAPR
2012	1.00
2013	1.01
2014	1.10
2015	1.15
2016	1.06
2017	1.20
2018	1.36
2019	1.35

aRelative annual pass rate (RAPR) = pass rate of SBMU anesthesiology residents/pass rate of the National Board of Anesthesiology.

5. Discussion

This descriptive-analytic retrospective study demonstrated that there was a statistically significant relationship between the IEI (i.e. package of the three aforementioned interventions) and the annual RAPR index. In addition, there was an increasing trend in the annual RAPR. These results are indicative of global improvements in residency training based on a programmed educational intervention associated with feedback and continuity over a couple of years.

Medical knowledge has been quoted as one of the main six core competencies of the Accreditation Council for Graduate Medical Education (ACGME) in training clinical residents; including anesthesiology residents (2, 15-17). More importantly, there is an integral relationship between Miller's Pyramid framework (18, 19) and ACGME core competencies. For example, Williams et al. (20), demonstrated this relationship between Miller’s Pyramid framework on one side and the ACGME core competencies, on the other side; both sides of this relationship are cornerstones of medical education. Many others have similar results confirming the latter study by Williams et al. (21-25). These findings stress the importance of knowledge as one of the main competencies for each clinical resident mandating efforts to improve outcome of educational interventions, which is in concordance with the results of our study.

Among the wide variety of approaches, MCQ’s have an old history and are frequently designed with varying degrees of validity and reliability (26-28). Albeit these drawbacks, MCQ exams are still among the main armamentarium in assessment of clinical residency programs; focusing but not limited to the knowledge domain (29, 30). In another study, Lingenfelter et al. (31) demonstrated that ITE could provide “formative assessments” regarding “medical knowledge” of the residents. These studies support our findings regarding the relationship between performing an integrated and continuous program based in part on MCQ exams and improvement in RAPR trend.

Clinical faculty members are the mainstay of trainee assessment. However, their role is much more than just taking exams. In other words, performing formative exams supported by regular mentoring function improves their impact significantly (14, 32-35). There is significant evidence demonstrating the role of mentorship in training of anesthesiology residents, not only improving their knowledge and clinical performance of the trainee but also affecting professionalism and patient care characteristics; all being among the main ACGME core competencies (33, 34, 36-38). These studies are in favor of our findings regarding the relationship between regular mentoring and improvement in RAPR index.

Mock oral examination especially if being standardized could detect “areas of relative strength and weakness” which are different between residency programs (39, 40). In another study, it was demonstrated that multi-institutional mock oral examination can improve performance in American Board of Surgery Certifying Examination and identify the residents at risk of board exam failure (41).

Resident burnout is a main consideration in each residency program. Among the many factors that could be involved in burnout of residents, emotional factors are in the top of the list (42, 43). There is possibility that the integrative educational intervention assessed in this study could have some impacts regarding resident burnout because of improving the success rate and the possible emotional effects of mentorship. However, this needs more studies to be accomplished to see if such relationship really exists.

5.1. Conclusions

Our study demonstrated that the integrative educational intervention (IEI) package of Anesthesiology Department, SBMU had a significant relationship with improvements in RAPR trend; which means this package could be beneficial in improving the results of successfulness for anesthesiology residents in passing the National Board Exam. More prolonged studies could prevail further aspects of these interventions.