Formative Feedback with In-Class Question Bank Utilization Improves Resident Satisfaction with General Surgery Didactics.

BACKGROUND: Formative feedback provides low-stakes opportunities for educational improvement. To enrich our basic science didactics, formative feedback measures were incorporated into our didactics using mobile devices.
MATERIALS AND METHODS: Lecture changes included institutional paid access to a commercial question bank, a 5-item in-class pre-didactic quiz curated from the question bank and taken on the resident's mobile device, and group discussion of quiz topics. An anonymous survey was sent to participating residents.
RESULTS: Overall response rate was 71% among residents. All reported that the new lecture format was a valuable addition to the basic science curriculum (100% Agree/Strongly Agree), and formative assessments provided valuable feedback about the progress of their learning (Strongly Agree = 42%, Agree =58%). All residents reported that in-class use of their mobile device for quizzes was convenient, with majority (84%) preferring it over paper printouts. Residents were more motivated to study before lecture (Strongly Agree = 42%, Agree =42%), with majority also reporting the new format helped identify weaknesses in their knowledgebase (Strongly Agree = 58%, Agree =33%). While majority of residents agreed that quizzes motivated them to study more after lecture, a large portion disagreed (42%). Majority of senior residents reported that the process of composing quizzes prior to lecture enriched their own learning (57%) and helped them find gaps in their knowledge (71%).
CONCLUSION: Incorporating a commercial question bank within didactics gives general surgery residents formative feedback and encourages learning outside the classroom, leading to improved satisfaction with basic science didactics.

Introduction

For medical school graduates in the United States, general surgery residency is an additional 5–7 years of post-graduate education. Along with the operative requirements for final certification, residents are required to learn the fundamental basic science principles used in the management of clinical problems.1 Furthermore, continued residency accreditation requires all surgical training programs to provide residents with didactic instruction of the basic sciences and formative feedback throughout their training.2

Within surgical education, the term “basic science” is used to describe the background principles of anatomy, physiology, biochemistry, pharmacology, pathology, and microbiology that apply in the clinical setting. The Accreditation Council for Graduate Medical Education (ACGME) requires general surgery residencies to provide didactic lecture instruction to facilitate resident education. For basic science instruction in surgery, three frequently recommended texts include Sabiston Textbook of Surgery, Schwartz’s Principles of Surgery, and Greenfield’s Surgery: Scientific Principles & Practice. In their latest editions, sum total page count for these three books is 6416: over 2000 pages per book. Exceedingly rare that all three books are read in their entirety, these texts are frequently used interchangeably, illustrating the extensive volume of material to be covered during surgical training. In the face of this overwhelming and ever increasing amount of content, the methods of didactic instruction used to teach residents basic science are critical. Standard lecture methods, lacking audience interaction or group discussion, are associated with lower rates of knowledge retention among students and residents.3–5 When attending a standard didactic lecture, audience attention significantly declines after the first 10 minutes, with attendees only retaining approximately 20% of presented material.6,7 Formative feedback techniques have been incorporated into higher education with favorable results.8–11 Furthermore, in this modern era of increasing mobile device integration in education, young doctors may grow to question the efficacy of traditional didactics or sole use of a textbook; increasing audience interaction during didactics, mobile device use in medical education is viewed favorably by residents, with some research even showing better academic performance.12,13

In graduate medical education, formative feedback focuses on providing residents with low-stakes opportunities to measure their knowledge, such as asking questions about physiology during rounds or anatomy in the operating room. Contrarily, summative feedback is typically given at the end of the year, or after a clinical rotation. The biggest difference in these two forms of feedback is the finality with which they are conveyed. Formative feedback allows educators to better augment the focus of their teachings during the time residents are learning the material, in contrast to summative feedback, which is given before residents move on to the next stage of training. Research into perceptions of feedback during post-graduate education reveals that there are often disconnects between residents and faculty.14,15 Establishing consistent, adaptable methods for providing formative feedback is essential to address these misperceptions between resident and faculty, allowing trainees to become independent, self-directed learners.16

Incorporation of formative feedback with use of a commercial question bank, utilizing resident mobile devices during didactics, may be a contemporary solution for residents stifled by the sheer volume of basic science material to cover. To answer this question and gauge the reception of this approach in a general surgery residency program, we incorporated formative assessments via a question bank into our basic science didactics, with aims to provide residents with formative feedback and enrich the value of our curriculum.

Materials and Methods

This is a prospective cohort study evaluating surgery resident satisfaction with implementation of mobile devices and access to a question bank as formative feedback measures during basic science didactics. All residents post-graduate year (PGY) 1–4 participated in the didactics and in-class assessments. Chief residents (PGY-5) attended the didactics but were not required to complete the newly implemented in-class quizzes; due to their advanced insight and background surgical knowledge, chief residents acted alongside faculty surgeons as junior-faculty advisors during the discussion segment of the didactics. Satisfaction with didactic changes was assessed through the distribution of a voluntary survey to PGY 1–4 residents within the training program.

Prior Didactic Format

Basic science didactics were conducted on a weekly basis during protected educational time prior to start of surgical cases. Lecture duration was 60 minutes, with material curated from an assigned chapter in a surgical textbook, and taught by a first or second year resident under the supervision of chief residents and assigned faculty. After 50 minutes of didactic instruction, a 10-item un-graded quiz of non-standardized multiple-choice questions, composed by the presenting resident, was administered openly in a sequential fashion, typically one trainee per question. Correct answers and explanations were briefly given by the presenting resident before progressing to the next question. Other than assigned textbook chapters, there was no additional preparation assigned prior to the start of each didactic. Furthermore, no system was in place holding residents accountable for completing assigned readings.

Didactic Changes

Like the prior didactics, the newly formatted basic science lectures were presented by a first or second year resident, took place on a weekly basis during 60 minutes of protected time, and were supervised by chief residents and faculty. In addition to textbooks, residents were provided with a paid 1-year subscription to Truelearn®, a bank of standardized multiple-choice questions covering many of the basic science learning objectives emphasized by the American Board of Surgery. Residents were also required to use Socrative®, a free and readily available mobile device application used to administer formative in-class assessments. Residents were informed of the basic science lecture topics spanning the entire year, and encouraged to complete assigned readings while taking corresponding multiple-choice questions in Truelearn®.

The prior 50 minute didactic was divided and restructured. Didactic instruction was reduced to 40 minutes, preceded by a 10 minute, 5-item multiple choice quiz of standardized multiple-choice questions curated by a senior resident from Truelearn®. This quiz was administered through each resident’s mobile device using Socrative®, with residents accessing the quiz via a room code displayed on screen at the start of lecture. When completing the quiz on their mobile device, residents were shown their percentage of correct answers. All resident results were exported onto an encrypted server, with grades supervised by the program director. The didactic session following the quiz covered the basic science topics outlined in the quiz along with additional content outlined in the question bank and assigned textbook. Following the didactic, the lecture transitioned to group discussion of the same 5 questions asked during the quiz, during which open exchange between medical students, residents, and faculty was actively encouraged. Figure 1 presents a diagram comparing the sequence of the prior didactic format with the new format.

Figure 1

Didactic changes. A diagram comparing the format of the former, traditional didactic lectures to the new approach. The new lecture included institution paid access to a question bank, a graded 5-item quiz administered through the resident’s mobile device, a shortened didactic lecture, and group discussion session of the quiz contents between residents and faculty.

Resident Survey

This study was conducted in accordance with the ethical principles of the Declaration of Helsinki. Study recruitment was performed in the classroom setting prior to and after educational didactics after obtaining informed consent among participating residents. Since this research involved evaluating the effectiveness of educational instructional strategies, was conducted in a commonly accepted educational setting, and involved voluntary survey participation, this study met our institutional review board’s guidelines for exemption. Six months after implementing this new didactic format, a 16-item voluntary, anonymous survey was distributed electronically to participating residents. After obtaining informed consent, surveys were completed, with results tabulated to characterize the reception of this new didactic format. Responses to survey questions were recorded as “strongly agree,” “agree,” “disagree,” or “strongly disagree.” Senior residents (PGY 2–4) tasked with composing quizzes were sent additional questions to evaluate the impact of curating quiz questions on their learning. Chief residents (PGY-5) were excluded from the survey since they were not required to complete quizzes, acting in a supervisory role alongside faculty during didactics. Reception and satisfaction with formative feedback incorporation was gauged using overall survey feedback from all resident respondents.

Results

A total of 17 participating residents were surveyed. Twelve residents completed the survey, resulting in a response rate of 71%. Survey responses varied based on post-graduate year (PGY), with half to majority of residents from each class responding (Figure 2).

Figure 2

Overall survey response. Overall survey response broken down by resident class (PGY1-4). Chief residents (PGY-5) were excluded from the survey as they assumed supervisory roles alongside faculty during didactics.

Regarding restructuring of the didactic with formative feedback assessments (Figure 3), all residents agreed/strongly agreed that the new lecture format was a valuable addition to the basic science curriculum (Strongly Agree = 50%; Agree = 50%). All residents also felt that full access to the bank of possible quiz questions enriched the overall quality of the lecture (Strongly Agree = 58%; Agree = 42%). A majority of residents felt that quizzes motivated them to study more before lecture (Strongly Agree = 42%; Agree = 42%; Disagree =16%), helped them identify high-yield topics (Strongly Agree = 50%; Agree = 42%; Disagree =8%) and aided them in recognizing areas of weakness in their knowledge (Strongly Agree = 58%; Agree = 33%; Disagree =9%). Furthermore, all residents reported that openly discussing the quiz questions at the end of lecture was valuable for their learning (Strongly Agree = 50%; Agree = 50%), with a majority also reporting that having faculty present during the open group discussion added more value to the lecture as a whole (Strongly Agree = 42%; Agree = 50%; Disagree =8%). While a majority of residents agreed that quizzes motivated them to study more after lecture, a large portion disagreed (Strongly Agree = 25%; Agree = 33%; Disagree =42%). Lastly, all residents agreed that quizzes provided valuable feedback about the progress of their learning (Strongly Agree = 42%; Agree = 58%).

Figure 3

Response to new lecture approach. Resident survey responses regarding the new lecture format and use of in-class quizzes.

On survey of senior residents regarding quiz composition (Figure 4), all felt that the process of creating quizzes was easy/convenient (Strongly Agree = 29%; Agree = 71%). A majority agreed that creating quizzes helped them identify areas of weakness in their knowledge (Strongly Agree = 42%; Agree = 29%; Disagree = 29%). While a majority also felt that creating quizzes enriched their learning on the relevant lecture topic, a significant number also disagreed (Strongly Agree = 29%; Agree = 29%; Disagree = 42%). Overall, a majority of senior residents reported that creating quizzes was beneficial for them (Strongly Agree = 29%; Agree = 42%; Disagree = 29%).

Figure 4

Senior resident responses. Senior resident (PGY2-4) survey responses regarding their composition of in-class quizzes.

For the implementation of mobile device use during the new didactics (Figure 5), all residents reported that taking in-class quizzes on their mobile device was convenient (Strongly Agree = 58%; Agree = 42%). A majority also agreed that they would rather continue taking quizzes on their mobile device instead of switching to paper (Strongly Agree = 67%; Agree = 16%; Disagree =16%).

Figure 5

Mobile device use. Resident survey responses regarding mobile device utilization during lecture.

Discussion

Formative feedback incorporation within didactics is frequently met with positive reception by students, who report high satisfaction and more motivation to learn. Those questioning the implementation of in-lecture assessments express concern that insufficient preparation prior to lecture hinders potential learning benefits.17 While in-lecture opportunities for formative feedback has been researched extensively in medical student education, little research has been conducted using a cohort of general surgery residents, who arguably have less time for class preparation due to longer work hours and higher magnitudes of clinical responsibility.18–20 Studies investigating the efficacy of in-lecture question bank utilization in surgical resident education do not currently exist; while similar approaches within flipped classroom literature do exist, no current studies use a cohort comprised of only general surgery residents.14,21,22 Surgical residents within these studies were found to have higher lecture engagement, increased participation, and superior performance on educational assessments. Furthermore, when utilized outside the classroom, commercial question banks improve resident knowledge of basic science, as well as performance on formal assessments.23,24 The results of our study, comprised of general surgery residents utilizing a question bank within the classroom, echo many similar conclusions found in these prior studies. All residents in this study felt that the opportunity to receive formative feedback regarding their basic science knowledge added value to the curriculum and produced a more satisfying basic science didactic.

The integration of mobile devices into surgical education is a developing phenomenon with promising results.25–29 One systematic review looking at mobile device use in surgical education notes that in light current resident work hour restrictions, trainees may become increasing reliant on mobile devices as a means of education through e-learning.30 The residents within our study found integration of their mobile devices into the didactic convenient, preferring it to the idea of receiving printed materials.

Some limitations to this study are worth noting. The small size of our cohort is our prime limitation, which may predispose this study to potential bias and limit the generalizability of our results. Furthermore, this survey was not distributed prior to the implementation of didactic changes, eliminating the possibility of utilizing a control group within the current study. Despite these shortcomings, this is a pilot study, and the majority of residents within the program responded generating a sizable survey response. Review of current literature revealed no statistically validated surveys sufficient to gauge the primary endpoints of this study, namely the reception and satisfaction with formative feedback and mobile device integration into a surgical basic science curriculum; these endpoints were drawn from overall resident feedback.

Conclusion

Formative resident assessment by means of question bank incorporation into general surgery basic science curricula is a relatively new teaching approach. The preliminary findings of this study, exhibiting an improved curriculum that residents feel is more valuable, echo many of the beneficial results established in other fields of education. Incorporation of formative feedback opportunities into basic science didactics effectively highlights knowledge gaps for improvement and encourages self-directed learning outside the classroom.