Student assessment of teaching effectiveness of "bundle of changes"-A paired, controlled trial.

BACKGROUND: Inching toward optimum patient safety by training personnel is the prime aim of the ongoing medical education. AIMS: To assess whether lectures targeted to improve quality care in ICU could improve ICU practitioners' knowledge levels and to evaluate the effectiveness of teaching. SETTINGS AND
DESIGN: In this paired controlled trial, 50 ICU practitioners, i.e., anesthesia and medicine residents and nursing staff of our hospital attended a series of four lectures.
MATERIALS AND METHODS: Participants enrolled in the study attended lectures on bundles of changes in ICU, namely, introduction, ventilator bundle, central line bundle, and catheter-related blood stream infections and severe sepsis bundle. They were given a questionnaire of 15 multiple choice questions prior to and after the lectures. We evaluated their immediate knowledge acquisition and retention recall. Subsequently, they evaluated the effectiveness of the teaching programme by a questionnaire of 10 multiple choice questions. STATISTICAL ANALYSIS USED: Data for statistical analysis were tabulated and analyzed using SPSS-Pc 11.5 version software.
RESULTS: Fifty study participants completed all three questionnaires. There was an increase in the overall mean score in the post-lecture test (4.58 + 1.51 SD) (P < 0.001). Overall mean score increased significantly from 8.30 + 1.34 SD in THE pre-lecture test - to 12.02 + 1.61 SD in the postlecture re-test (3.72 + 1.39 SD) (P < 0.001). In the evaluation of teaching effectiveness 88% respondents agreed to most of the questions, signifying the effectiveness of the lectures. However, there were 10% who disagreed to the questions and only 2% strongly disagreed to all the questions.
CONCLUSIONS: Teaching programmes such as the "bundle of changes" are effective in improving immediate knowledge acquisition and retention recall of the participants if designed keeping the target audience in mind.

Introduction

The aim of teaching effectiveness is to assess and evaluate the performance of teachers in order to enhance their professional growth. Jahangiri et al. stated that “Al-though goals can vary across institutions and may be specific or global, evaluating a teacher's effectiveness is proof of the academic institution's commitment to continual improvement.”[1] Evaluation of teaching efficiency can be formative and summative. Formative assessment helps to facilitate learning and helps teachers and students give feedback on each other's performance. Summative assessments are used to determine accomplishments and give answerability, as the outcomes are related to the competence of teaching.[2] Summative appraisals focus on the sequel of teaching.[23]

To determine the effectiveness of any teaching programme, it is essential to get a feedback from the participants and analyze it to further improve the teaching curriculum. To improve the quality of care provided in ICU, it is essential to teach and train all caregivers of ICU patients at regular intervals. We hypothesized that knowledge of doctors and nurses, in ICU care of patients, increases after a designed educational programme. Four lectures were delivered. Pre- and postlecture test questionnaires, related to the course taught, were answered by the study participants.

Materials and Methods

This prospective controlled paired study was conducted with anesthesia and medicine residents and nurses of ICU serving as subjects from April 2009 to November 2009. Exclusion criteria included those involved in the teaching programme and those working in any discipline, in any capacity, for more than 10 years. Those who agreed to participate in the study signed a written consent form. The only demographic information collected was level of training of the participants. The intervention consisted of a designed educational programme, i.e., series of four lectures on “bundle of changes” to be followed or implemented in our ICU. At the end of each lecture, there was a verbal discussion and the key points were emphasized again.

The designed educational program or “bundle of changes” was as per the protocol followed by an organization based in Massachusetts, the Institute for Healthcare Improvement (IHI), which targeted prevention and reduction of three Hospital Acquired Infections (HAIs) — i.e. ventilator-associated pneumonia (VAP), central line-related bloodstream infection (CLRBI), and surgical site infection (SSI). The participants were given a written schedule of the dates of these lectures. The participants were not given any feedback on test performance.

The first lecture was on introduction to the concept of “bundle of changes” during which stress was laid on targeting zero for HAIs. The second lecture was on ventilator bundle, which are a series of interventions related to ventilator care that, when implemented together, will achieve significantly better outcomes. The key elements of the ventilator bundle comprised of -elevation of the head of the bed, daily “sedation vacations” and assessment of readiness to extubate, peptic ulcer disease prophylaxis, and deep venous thrombosis prophylaxis.[4] The third lecture was on central line bundle and catheter-related blood stream infections. The risk factors of central line catheter insertion were taught and the preventive strategies were explained to them.[5] The fourth lecture was on severe sepsis bundle. It consisted of the sepsis resuscitation bundle: tasks that should begin immediately, i.e., within 6 hours for patients with severe sepsis or septic shock and sepsis management bundle : tasks that should begin immediately, but must be done within 24 hours.[6]

A pre- and postlecture test questionnaires comprising of 15 multiple choice questions each, were given to the participants before and after taking the lectures. Another questionnaire was prepared to evaluate the effectiveness of the teaching programme. This consisted of 10 multiple choice questions with four options for all the questions, i.e., strongly agree, agree, disagree, and strongly disagree. Responses agreeing to the questions signified the effectiveness of the teaching programme. Another postlecture retest was conducted after 6 months of initial assessment, whose questions were same as the prelecture test questionnaire comprising 15 multiple choice questions.

Results

Data for statistical analysis were tabulated and analyzed using SPSS--Pc 11.5 version software. Data were expressed in proportions. Item-wise difference between pre, post-lecture tests and the re-test conducted after 6 months was compared using McNemar's chi-square test “P” value less than 0.05 was considered statistically significant.

This study was conducted in our hospital from April 2009 to November 2009. In the pretest questionnaire, the mean marks obtained by the participants were 40%, whereas in the posttest questionnaire the performance was much better, i.e., 85% which showed a significant improvement in immediate knowledge acquisition which was the primary outcome. The overall mean score increased significantly in post-lecture test (4.58 + 1.51 SD) (P < 0.001). Overall mean score increased significantly from 8.30 + 1.34 SD in the prelecture test to 12.02 + 1.61 SD in the postlecture retest (3.72 + 1.39 SD) (P < 0.001).

As shown in Table 1, the maximum improvement in the postlecture test was in defining a bundle. Sixty eight percent more respondents were able to answer this question. The next two questions in which there was a significant increase in the awareness were naming components of central line bundle (52%). There was an increase in the documentation of various risk factors, to the extent of a significant 42%, for vascular assist device infections and catheter-related blood stream infections.

Table 1

Comparison of correct responses to prelecture test, post--lecture test, and postlecture re-test questions among study participants

Forty two percent more study participants were able to differentiate between sepsis resuscitation bundle and sepsis management bundle. There was significant improvement in knowledge acquisition, i.e., 36% more participants were able to recall facts in the post-lecture test, such as defining VAP and naming the four criteria for systemic inflammatory response syndromes and 30% more were able to recall the extubation protocol checklist. Similarly for describing disadvantages of over-sedation, the increase was 26% and in enumerating the key elements of the ventilator bundle the improvement was 22%. This implies that such teachings do cause an increase in the recall of scientific facts and a significant improvement in immediate knowledge acquisition. Twenty two percent of the study participants believed that teaching of such bundles of changes would cause a tremendous improvement in patient management.

The summary of the teaching evaluation form is depicted as Table 2. To summarize, 88% respondents agreed to most of the questions, 10% disagreed to the questions, and only a mere 2% strongly disagreed to all the questions.

Table 2

Opinion of the respondents regarding effectiveness of the training program (n = 50)

Follow-up of participants

Fifty five doctors attended the initial lecture, but five were excluded because they had participated in developing the questionnaire. The 50 remaining study participants returned a baseline prelecture test and subsequently all of them completed a post-lecture test. All these 50 respondents filled up the teaching evaluation form. Of these 50 participants, 38 were resident doctors from different branches of medicine and 12 were nursing staff of ICU. All the 50 respondents completed the post-lecture re-test conducted after six months. Figure 1 shows the details of study participants and the pre-, post-lecture tests, and re-test conducted.

Figure 1

Enrolment of respondents

Discussion

There was an overall improvement in the performance of all the participants (45%) in the postlecture test questionnaire. The percentage of study participants who could answer each question correctly improved, implying that such studies do cause an increase in the recall of scientific facts and a significant improvement in immediate knowledge acquisition. Since there was an improvement in the scores in the re-test conducted 6 months later, it implies that there was a significant retention recall of facts. All the students, i.e., 100% believed that the course content of the lectures was appropriate and that their knowledge of the subject was updated by these lectures and that the presentations were organized. It is interesting to discern that 96% of them found the lecturer to be enthusiastic about teaching the course. Relative to other courses they had done, 92% respondents agreed that the lectures were not complicated and that the ambience (atmosphere) during the lectures was good. Almost 76% of the respondents felt that the teaching was adequate and the lectures need not have been more interactive. However, surprisingly only 78% respondents found the course challenging. All respondents agreed that they would be able to implement the changes as suggested in the lectures.

Feedback after a teaching programme by the participants is always recommended to evaluate success of the teaching programme.[1] Search of literature did not reveal any such attempt in the subject of Anesthesiology and critical care. Evaluation of an educational programme can essentially lead to desired changes in the objectives, course contents or teaching methods.

Giving excellent care and progressively ameliorating the medical aftermath are the main objectives.[7] Anesthesiologists endeavor to achieve.[8] There are reasons to believe that structured programmes in patient management in ICU and improving medical care provided by ICU caregivers including Anesthesiologists, Physicians, Surgeons, and nursing staff are necessary. Improved knowledge and assessment skills can potentially reduce morbidity and lower costs by decreasing stay in ICU. Interventions such as clinical examinations, daily rounds, and use of care “bundles” also improve the outcome of ICU patients.

The IHI has targeted prevention and reduction of HAIs, VAP, CLRBIs, and SSIs. Resar et al stated that “bundle is a structured way of improving the processes of care and patient outcomes: a small, straightforward set of evidence-based practices—generally three to five—that, when performed collectively and reliably, have been proven to improve patient outcomes.[4]

To construct an instrument for evaluation, we need to include technical aspects such as ambience of the lecture, i.e., Is the lecture room comfortable? Is the presentation visible to all? Questions about lecturers’ delivery such as: Is the presentation interesting? Is the lecturer confident and able to deliver appropriately? These can be answered directly and objectively by students. A strategy of using a change in the framing of statements was used so that sometimes the “desired” response was “strongly agree”: And in others “strongly disagree.” Questions 1-6 dealt with the course content and the lecturer characteristics, while questions 7-10 sought general information about the course.

We view our survey as a preliminary insight into areas that may benefit from further exploration and probably implementing it on a bigger study group. The limitation of our study was that the number of questions of the pre and post-lecture tests was small in number and possibly did not quantify the knowledge acquisition. It is unrealistic to expect that “care givers” of ICU patients can successfully manage their clinical roles without basic exposure to some form of training similar to the bundle of changes carried out by us.

To conclude, the participants were satisfied with the course content and the lecture characteristics. Teaching programmes are effective and can lead to an improvement in the quality of care provided to patients and help reduce morbidity and mortality and reduce hospital costs. Teaching needs to be more interactive with practical demonstrations of the topic being taught.