Multifaceted remediation program: experience of a residency program to rescue residents who failed the American Board of Anesthesiology basic examination.

PROBLEM: Passing the American Board of Anesthesiology (ABA) basic examination is required to progress through anesthesiology training in USA. Failing the test may be related to medical knowledge gaps, presence of negative psychosocial factors, and/or individual approaches to learning. This article describes the experience of development and implementation of a multifaceted remediation program (MRP) in residents who failed the ABA basic test. INTERVENTION: This is a retrospective analysis of four cases of residents who failed the ABA basic test between 2016 and 2017. The MRP is described. Pedagogical diagnosis, objectives, teaching strategies and assessment, and their constructive alignment are presented. Information regarding test performance is also presented. CONTEXT: This study involves accredited anesthesiology residency program in USA. Outcomes: Four subjects (11% of program residents) failed the ABA basic test. Superficial approach to learning was observed in 100% of cases. The total possible number of participants was 4. The actual number of participants was 4, and the response rate was 100%. Four residents fell under 10th percentile on the first attempt, and 100% passed the test on the second attempt. There was 38% improvement in the number of failed keywords between the two attempts. LESSONS LEARNED: Implementation of the MRP developed at our institution is successful to remediate anesthesiology residents who fail the ABA basic examination. We learned that the deep analysis of learning approaches, psychosocial factors, and medical knowledge gaps can be used to develop a remediation program based on the constructive alignment between objectives, curriculum, and assessment.

Background

The anesthesiology residency curriculum in USA is based on the development of competencies that include medical knowledge, in accordance with the Accreditation Council for Graduate Medical Education (ACGME) recommendations.1 The American Board of Anesthesiology (ABA) implemented a system of evaluation to assess medical knowledge in anesthesia consisting of sequential basic and advanced examinations. The basic test concentrates on the scientific basis of anesthesia practice. Failing the basic test may hinder progression of the resident and, in some cases, may evidence significant knowledge difficulties in the integration of scientific basis to relevant clinical situations. Additionally, indirect factors such as social, psychological, and learning approaches may play a role in test failure.2

Remediation of failing learners is an important element of educational programs. The process of remediation is complex and imposes strain on teachers, students, and academic programs alike.3 Remediation is an integral part of medical education as it affects the individual in training as well as the future professional as a part of the medical workforce.4 It is generally accepted that early remediation is more effective to get the trainees back on track to the point of achieving uniformity with their peers.5 An effective remediation program should include strategies to accompany students on a journey to fine-tune their approaches to academic success and facilitate student-centered learning in the context of active student participation that acknowledges the psychosocial environment.6–9 The remediation process should consist of a logical sequence of steps including mandating, measuring, and reporting competence rather than unstructured efforts led by medical educators.10 A remediation program should be planned taking into consideration the particular deficiencies of the resident, and it is recommended that an assigned mentor tailors the program to the resident’s specific needs.11,12 Remediation implies a holistic approach to stress evaluation and counseling as well as a judicious evaluation of the individual case.13 Finally, deep approaches to learning are associated with better performance in medical education and are strongly influenced by workplace environment.14,15

Literature addressing the effect of remediation programs focused on medical knowledge competencies is scarce. We developed and implemented a structured, individualized remediation program for residents who failed the basic part of the ABA examination. This article describes our experience with the successful implementation of a remediation strategy plan in anesthesiology residents at our institution.

Intervention design and methods

The Institutional Review Board (IRB) of Augusta University approved the educational project of which this study forms part. The IRB number is 839326, with the approval date of February 16, 2016. Residents included in the study agreed and signed informed consent to participate in this research. After IRB approval and receiving consent from the individual residents, we retrospectively studied a group of postgraduate students of an anesthesiology residency program, who failed the ABA basic examination on the first attempt between 2016 and 2017. Demographic characteristics, test scores obtained in the United States Medical Licensing Examination (USMLE) Steps 1 and 2, In-Training Examination (ITE), and ABA basic examination, were recorded. The residency program direction designated a remediation committee to design a multifaceted remediation program (MRP). Comfort level with respect to readiness for the ABA basic examination was measured with a scale ranging between 1 and 10. The comfort level instrument was administered halfway through the program and 1 week before the ABA test.

Description of the intervention

We intervened a group of anesthesiology residents at an academic residency program, who failed the ABA basic examination on the first attempt between 2016 and 2017. The intervention plan used a multifaceted approach adjusted to individual needs based on the interaction of social, emotional, and cognitive aspects in relation to the application of medical knowledge. Within the program, the purpose was to create a teaching/learning environment tailored to the individual resident’s learning approaches and specific needs. We focused on the achievement of maximum academic potential to pass the ABA basic examination on the second attempt and to improve the overall academic performance during training for the group of residents included in the analysis.

Intervention program

Following the program’s basic principles, study residents were identified. Demographic characteristics and analysis of the list of weak areas based on the keywords attached to the test score, provided by the ABA-generated feedback, were identified, and a mentor was assigned to the case. The mentor tried to understand different resident’s dimensions influencing quantitative performance in the ABA basic examination. The process started with an interview in which the resident’s perception of social and learning environment, emotional baseline status and the effect of the test failure on self-esteem, and possible causes of test failure were addressed. Data gathered during this initial encounter provided a comprehensive view of the dimensions that would serve as main topics in the MRP. The MRP encompasses psychological and social supports, identification of individual learning approaches and medical knowledge as well as strategies to close the gaps, development of test-taking strategies, and allocation of study time blocks. Details about the remediation program are provided in the Supplementary materials.

During an initial interview conducted by the mentor, the resident was asked about psychological aspects including academic motivation, efficient use of time, perceived social support, identification with the residency program, and the institution. The social dimension focused on the effect of stress on academic consequences. This dimension was analyzed in terms of interference between study and social domains and social support from family, friends, and fellow residents. The evaluation of such social factors was subjective. Healthy habits including exercise, nutrition, and sleep were explored in the interview. Data gathering after the interview allowed the mentor to identify the presence or absence of stressors in the psychological factors described earlier. If at least one stressor was present, a psychology counselor was invited to take part in the MRP for the resident. The role of the counselor included provision of advice to navigate through the educational plan offered by MRP, recommendations for time management and use of stress-reducing techniques, and identification of learning approaches. The counselor met the resident once a week and made recommendations to the mentor when deemed appropriate.

Residents’ approaches to learning were evaluated with the revised two-factor Study Process Questionnaire (R-SPQ-2F).16 The R-SPQ-2F instrument consists of 20 items evaluating two scales, each one composed of 10 items, evaluating either the superficial approach or the deep approach to learning. Additionally, the scales were subdivided into two subscales of five items each to reveal the strategies and motivations underlying the learning approaches. Responses to each item were categorized according to a 5-point Likert scale. The individual’s approach to learning was considered superficial when the resident relied mainly on the memorization of contents and saw passing the test as the only goal, whereas the resident was deemed a deep learner when he/she tried to find meaning within the texts and saw knowledge acquisition as a higher goal than passing the test per se.

The mentor identified weak keywords as per the ABA examination feedback. The mentor generated a list of book chapters and review articles covering the broad topics addressed by keywords. Study blocks to review these topics were assigned each week. A goal of individual reading of two chapters per week was established. During the weekly meeting with the resident, the chapter was discussed in a format allowing the resident to present the topic in an informal way, without audiovisual aids. The mentor emphasized the specific aspects that might be potential targets in a structured test such as ABA basic examination.

Regarding test-taking strategies, we explored the connection between clinical reasoning and test-taking behaviors.17 Based on this analysis, we planned a strategy using the following two question: Truelearn™ platform and Anesthesiology Continuing Education (ACE) questions. During the first session, mentor and resident sat together to answer a block of 50 questions. During this session, a technique was taught consisting of the following three steps: 1) reading the stem question neglecting the multiple choices; the goal was twofold, such as trying to answer the question directly and brainstorming to generate a list of aspects related to the topics presented in the question; 2) going over every single possible answer to determine whether it was true or false based on the association list found in the first step; and 3) using information from prior questions to facilitate finding a solution for new questions. Table 1 summarizes the key steps of the remediation program.

Table 1

Stages of MRP

Stages	Tools
Conformation of intervention team	Open mind, flexible, open to dialogue
Psychological and social evaluations	Academic motivation, efficient time use, perceived social support, identification with residency program/institution, healthy habits
Identification of learning approaches	Observation and application of structured questionnaires
Medical knowledge gaps and strategies	Keywords, question banks, study blocks, one-on-one discussion, simulated examinations

Abbreviation: MRP, multifaceted remediation program.

Results

During the 2-year period, four resident cases were analyzed, representing 11% of the total number of residents enrolled at an academic anesthesiology residency program, who took the test during 2016 and 2017. The total possible number of participants was 4. The actual number of participants was 4, and the response rate was 100%. All residents were in their second postgraduate year. Demographic characteristics of the subjects and USMLE test scores are shown in Table 2. The application of the revised questionnaire of two factors validated by Biggs et al showed that 100% of residents had a superficial approach to learning.16

Table 2

Demographic characteristics and baseline USMLE test scores

Resident	Age (years)	Gender	PGY	IMG	Years after medical school	USMLE Step 1 score	USMLE Step 2 score
1	30	Male	3	No	4	202	215
2	36	Male	3	Yes	10	184	202
3	35	Male	3	Yes	9	216	223
4	35	Female	3	No	2	198	210

Abbreviations: IMG, international medical graduate; PGY, postgraduate training year; USMLE, United States Medical Licensing Examination.

The ABA basic scores are shown in Table 3. The four residents (100%) passed the test after the implementation of the remediation program. The percentile for the PASSED test is not presented in the official ABA report; however, a list of keywords corresponding to questions responded to incorrectly was reported for each individual resident. The average number of keywords in the FAILED and PASSED attempts was 78 and 49, respectively, with an improvement of 38% between the two tests. The maximum improvement in terms of keywords occurred in the Basic Sciences area (50%) and the lowest improvement occurred in the Clinical Sciences (Anesthesia, Procedures, Methods, and Techniques) area (24%). The percentage of keywords presents in both attempts was 45%. Figure 1 shows the progression of scores in the simulated timed multiple-choice tests forming a part of MRP. All residents scored >65% in the last simulated 50-question test administered prior to the second attempt to take the ABA examination. The Remediation Committee considered that with those scores, the residents were ready to take the official test. Resident comfort level progression is shown in Figure 2. In-training test scores are shown in Table 3 (Supplementary materials).

Table 3

ABA basic examination and ITE scores

Resident	ABA basic examination – attempt 1 (status/percentile)	ABA basic examination – attempt 2	ITE-scaled score (%) (CA-1)	ITE-scaled score (%) (CA-2)
1	Failed/1	Passed	1	30
2	Failed/9	Passed	25	38
3	Failed/8	Passed	27	Not taken yet
4	Failed/1	Passed	24	Not taken yet

Abbreviations: ABA, American Board of Anesthesiology; CA-1, clinical anesthesia year 1; CA-2, clinical anesthesia year 2; ITE, In-Training Examination.

Figure 1

Percentage of right answers (residents 1, 2, 3, and 4) in simulated timed multiple-choice tests.

Note: Tests 1, 2, and 3 included 25 questions and tests 4 and 5 had 50 questions.

Figure 2

Resident comfort level scale (1–10) measured halfway through remediation program and at the end.

Note: 1, fully uncomfortable; 10, completely comfortable.

Discussion

Here, we present the successful development and implementation of a remediation program focused on medical knowledge oriented to pass the ABA basic examination for anesthesiology residents who failed during their first attempt. Rosenblatt et al18 showed that there is great variation among anesthesiology residency programs to remediate residents. Similar experiences are reported in other residency programs, both medical and surgical, and failure of implemented strategies is common.11,19,20 To our knowledge, no educational strategy has been tested to remediate medical knowledge in anesthesia residents.

We shifted the focus of remediation from the outcome to the pedagogical process. Our MRP was established based on the principle of student-centered teaching. The initial approach to the residents included a multidimensional assessment evaluating both academic and extra-academic factors including social and emotional stress, self-esteem, and self-perception about the causes of failure. Student’s responsibility and independence are essential determinants of lifelong learning, motivation, and self-evaluation.21 The initial mentor-resident meeting was directed at putting the student in the center of the problem analysis, giving him/her the opportunity to be part of the solution by the stimulation of insight and analysis of the environment. Although the MRP was standardized in its application to the study residents, customization was possible as individual mentors managed each case separately, based on the information obtained from the identification of social/psychological stressors, learning approaches, and medical knowledge gaps.

There is strong correlation between the approach to learning that a student adopts and academic performance.22 Results from academic literature converge on the idea that the teacher must be proactive to facilitate critical thinking and deep learning in students at different levels of education.22 We used a generic framework of student approach to learning based on constructivism and systems theory to identify motive and congruent strategy as determinants of learning approach.23,24 The combination of right motive and strategy yields the approach to learning. In our study, we performed an initial analysis of individual residents by the application of the R-SPQ-2F (validated) to diagnose the approach to learning. All the residents displayed a superficial approach. The MRP designed strategies to enhance critical thinking and avoid the use of memory as the sole resource to approach questions. The program stimulated associations within the stem question based on prior knowledge to create logical connections with the multiple answers.

Dillon et al showed that ABA basic examination scores bear a Pearson correlation coefficient of 0.59 and 0.56 with USMLE Steps 1 and 2 results, respectively.25,26 A sound explanation to this type of correlation lies in the fact that a strong foundation in basic science and its application to clinical settings is evaluated by both the USMLE and ABA examinations. The residents in our study had USMLE scores below national averages, especially for Step 1. Although the combination of low USMLE scores and individual characteristics of the residents may have anticipated an unsatisfactory result in the ABA test, the same might hold true for the second attempt. In our opinion, the effectiveness of the implemented MRP is proven by the 100% passing score rate obtained in the second attempt. Additionally, an even more dramatic improvement was observed in the residents who scored below the 10th percentile during the first attempt. Resident progression in simulated tests revealed improvement in both medical knowledge base and test-taking approaches. Although USMLE scores have been shown to be an effective predictor of academic performance during anesthesiology residency and some authors recommend including this fact into recruitment decisions,27 our study focuses rather on the remediation of residents with difficulties. Helping these residents to achieve their full potential is a more challenging yet rewarding task and adds special value to our findings. In contrast, if after recruitment of new residents, one or more individuals are considered at the high risk of failure based on low scores in USMLE tests, it may be advisable to “prophylactically” enroll them in the MRP.

MRP was founded on principles of constructive alignment.28 We combined constructivist psychology and curriculum theory to build our educational strategy. MRP established learning objectives based on a multidimensional pedagogical diagnosis. Constructivist psychology relies on the use of appropriate learning activities to build knowledge, whereas the curriculum theory states that optimal learning depends on the alignment of learning outcomes with teaching and assessment.29,30 Once we identified the factors that drove poor performance, the basic considerations and objectives of the remedial program were defined. Thereafter, a series of student-centered teaching strategies, aimed at encouraging engagement of the resident in learning activities, were implemented. Since progress evaluation was necessary to assess the performance of both the resident and the program, we aligned assessment with learning objectives, teaching strategies, and outcomes. In addition to provide a quantitative measure of progression of the resident throughout the remediation program, the timed tests were used to give feedback to the residents about right and wrong answers. Significant gains for educators and students derived from the adequate deployment of this type of assessments.30

Successful implementation of MRP depended on departmental support. Although resident work might generate significant revenue to anesthesiology departments, the academic mission must take precedence over billing.31 Our department was able to accommodate midlevel providers to cover for the residents in the remediation program without affecting the normal flow of the operating room activities. Additionally, residents included in the program continued working in the operating room during times not related to remediation. From the mentor’s perspective, the anesthesiology department was able to accommodate nonclinical time to dedicate to MRP. On average, the one-on-one encounters took approximately 4 hours/week. From the resident’s perspective, duty hours were monitored with the mentor every week. In addition, time to take tests and to participate in encounters with mentors was accommodated in 1 day/week. Additional independent work was performed by the residents out of work. The mentor monitored resident case logs on a weekly basis to ensure that exposure to cases and procedures was not compromised during MRP.

Finally, an important aspect to highlight about the success of MRP is the quality of communication between residents and mentors. This bidirectional communication entailed breaking any language barriers. Although, for two of our residents, English was not their mother tongue, their proficiency level was appropriate to work as a physician in USA. Here, we refer more to the communication barriers in regard to expectations of residents and mentors. We identified the lack of shared definitions of shared concepts. This aspect was discovered and resolved when the stem questions were approached. We consider that tackling the question with a common definition of key concepts helped the residents to answer exactly what was been asked in the stem statement. This strategy implied working on reading comprehension.32

Limitations

Our study has limitations. The sample of four residents is too small to generalize our conclusions. The psychological and social dimensions evaluated to plan the MRP were subjectively assessed, which may introduce bias as guarded responses from residents who might feel uncomfortable speaking openly with faculty could have been elicited. Test-taking anxiety was not specifically addressed or measured in our study; however, the multitude of multiple choice questions approached by the residents during MRP may have contributed to reduce this type of anxiety. We acknowledge that having no control group to prove that the intervention rather than a repeat test-taking experience was responsible for the improvement may compromise the validity of our results. Further research using controlled experimental approaches is necessary to evaluate individual effects of our strategies. The national passing rate of ABA basic examination is higher than 90%, making it difficult to recruit a large sample to test a remediation strategy. However, the fact that we were able to gather experience with the use of our remediation program in a single institution, gave us the opportunity to explore a combination of articulated strategies that in the end showed great success. The MRP has its foundations on the individual characteristics and diversity of learning approaches, as well as psychosocial environments of individuals with academic difficulties. Although MRP is not a one-size-fits-all educational tool, it can be adapted to individual characteristics by means of a judicious educational diagnosis that directs the alignment of teaching, assessment, and pedagogical outcomes. Future research is warranted in aspects of remediation including the effect of learning environment, and the behavioral patterns of teachers/mentors to optimize the effect of programs intended to improve learning outcomes.33