Applying a reflexive framework to evaluate a communication skills curriculum.

After creating and delivering an educational curriculum, medical educators must ultimately evaluate the effectiveness of the implemented curriculum. Seasoned educators can benefit from using an established framework to help them structure a thorough, complete curricular evaluation; however, novice educators may have difficulty in transforming the concept of evaluation into a concrete process. The RUFDATA (Reasons and purpose, Uses, Focus, Data and evidence, Audience, Timing, and Agency) framework is one such paradigm. It is a well-recognized tool consisting of a reflexive framework that can guide medical educators to evaluate their own medical education curriculum. Just as important, it enables medical educators to reflect on the reasons behind the evaluation. This insight, in turn, can foster a spirit of evaluation, thus helping to ingrain it into the local educational culture. By using the evaluation of our communication skills curriculum as an example, this article describes how educators can apply the RUFDATA framework to evaluate their own curriculum.

Introduction

Once medical educators have developed and implemented a curriculum for their learners, they must evaluate the effectiveness of that curriculum to ensure if it is fulfilling its intended purpose.1,2 Here, we define curriculum evaluation as gathering information or data about an educational program to judge its merit.3

However, a proper curricular evaluation can be difficult to conduct for faculty who lack experience or tools to methodically plan the evaluation.4–6 A reflexive framework known as RUFDATA (Reasons and purpose, Uses, Focus, Data and evidence, Audience, Timing, and Agency) is a tool that can transform the concept of evaluation into a concrete process.7

This article shows how medical educators can use the RUFDATA framework to plan and conduct an evaluation of their own curriculum. After providing a brief account of our communication skills curriculum, this article gives an overview of the RUFDATA framework and describes how to employ each component of the framework to conduct curricular evaluation, using the evaluation of our communication skills curriculum as an example.

Background

Teaching communication skills is an essential component of the curriculum in residency education.8,9 For our 2-year subspecialty residency program in respirology, we first performed an analysis with past cohorts of residents about their needs. Then, we developed and implemented a communication skills curriculum after reviewing the methods to teach communication skills in the literature. Our curriculum thus included several small group seminars that incorporated active learning,10–13 reflective essay assignments,14 and simulation exercises15–17 and covered topics such as writing proper consultation letters, communicating with patients and their families, collaborating with allied health professionals, and giving effective presentations.

RUFDATA framework

The RUFDATA framework by Saunders7 asks 7 questions that educators should ponder when planning their curricular evaluation. Reflexive questioning18 (ie, by asking open-ended questions to help the educator reflect on many possible options) helps start the evaluation process by addressing key aspects of the evaluation’s design. In other words, it helps educators decide what they want to get out of the evaluation and thus to plan and execute it accordingly.

Reasons

First, educators should ask themselves the reasons for evaluating the curriculum as the answer(s) to this question will be the answers to the rest of the questions in the RUFDATA framework. Broadly speaking, a curriculum is evaluated to demonstrate accountability to various stakeholders, to develop further knowledge about the curricular topic, and to improve the curriculum itself.19

We specifically wanted to evaluate our communication skills curriculum to ensure that our residency program was meeting accreditation standards in this specific area, the first small step in evaluating all aspects of our residency curriculum. We also wanted to ensure that our residents were learning the requisite skills to work with allied health professionals and were communicating effectively with patients and their families. In addition, we sought to determine whether there was any effect on the residents’ communication skills during their teaching presentations and whether there were any unplanned learning experiences occurring, such as “curriculum drift”20 and a “hidden curriculum”,21–25 which failed to match the expectations of our planned curriculum.

Uses

After reflecting on the reasons behind the evaluation, educators should ask how the information gathered will be used. In other words, what action can and/or will be taken once the data are collected and the results of the evaluation are known? How the information will be used, in turn, informs the focus of the evaluation and the data one collects.

We planned to use the information for ongoing quality improvement of our program ie, to conduct formative evaluation.26 The information gained from our evaluation would enable us to efficiently target areas for improvement in our teaching and assessment of communication skills for the residents. We also wanted to incorporate the evaluation results into our organizational database27 to allow future residency program directors to see what worked (and did not work) for us. If insufficient resources such as money, time, and personnel were found to be the cause of any deficiencies in our curriculum, we planned to use our evidence to lobby for investment in these resources. Finally, we sought to demonstrate our compliance in this domain for accreditation.

Focus

As it is difficult to thoroughly evaluate an entire teaching curriculum at once, educators should focus their evaluation on key aspects of the curriculum. The focus, in turn, influences the data that will be collected.

We selected to focus on the adequacy of the residents’ consultation letters, their written and verbal communication with allied health professionals, their verbal communication with patients and families, and their presentation skills during visits.

There are many other areas where a communication skills curriculum can be important, such as teaching proper patient care handover,28–30 discharge summary dictation,31–33 and crisis resource management.34–36 However, we chose to evaluate these other aspects of our curriculum in the future as we did not yet have the tools to assess the residents’ performance in these areas.

Data and evidence

After reflecting on the reasons, uses, and focus of the evaluation, educators must select the tools that they will use to collect the data and evidence necessary for the evaluation. This usually involves measuring how well the learners have achieved the learning objectives37 or measuring both the unintended and intended accomplishments of the learners.38 Some even focus on the learners’ pursuit of accomplishment, rather than the actual accomplishment itself.39 Typically, the data collection tools assess the achievements of the learners.40 These assessments include observation of behavior in the clinical workplace,41–44 observation of behavior during an objective structured clinical examination (OSCE) or simulation exercise,45–51 and chart review or audit.52–57 In addition, feedback can also be gathered from the learners and/or new graduates, through interviews or questionnaires, to determine if the teaching program has met their perceived needs.58–60

Our data consisted of random audits of the residents’ consultation letters, multisource assessment of the residents’ communication skills by allied health professionals (specifically, the nurses and ward unit clerk), and faculty’s assessment of the residents’ communication skills during presentations. We chose these tools because we were already using them before implementing our new curriculum and had baseline results with which we could compare. We considered using feedback from patients and their families but had not yet implemented this form of assessment during our evaluation. Instead, we used the faculty’s direct observations of the residents’ communication with patients and families, realizing the potential limitations of this approach.61 Additionally, we conducted exit interviews with the residents to look for any hidden curriculum or unintended consequences from our formal curriculum.

Audience

Educators should decide who the audience of the evaluation will be. In other words, who will see the results of the evaluation? The answer, in turn, depends on the level of decision-making informed by the evaluation.62 Medical educators overseeing the teaching program can use the evaluation to determine the need for improvement. They may wish to share this information with their peers who can then enhance their own teaching programs. Accreditation organizations may use the information as proof of compliance for this facet of the whole academic program. And the evaluation results, if positive, can bolster the teaching faculty’s morale and validate their teaching efforts.

We chose to share our evaluation results with our residency program committee tasked with implementing and revising the curriculum. In addition, we decided to keep the results as part of our records for accreditation and also share them with our teaching faculty.

Timing

The timing of the evaluation is another decision that educators must contemplate. For example, we conducted our evaluation 6 months before the start of the upcoming academic year. This enabled us to ensure that we had collected and analyzed all the data and that we would have sufficient time to contemplate and implement the necessary changes at the beginning of the academic year involving the next cohort of learners.

Agency

Here, educators must consider who will conduct the evaluation. Both internal evaluators (ie, people already familiar with the program or who may have helped develop it) and external evaluators have their own advantages and disadvantages. For example, an internal evaluator acquainted with the intricacies of the program and local quirks of the educational environment may probe deeper to look for weaknesses or may be more available to conduct the evaluation. On the other hand, an external evaluator might possess greater objectivity or garner more legitimacy.63 Potential disadvantages of an external evaluator include a greater cost (eg, for transportation and lodging) and difficulty in coordinating the schedule for evaluation.

After careful consideration, the program director of our residency training program, one of the developers of the curriculum, conducted the evaluation.

Evaluation results

We analyzed the data during the 12 months before and after the implementation of our communication skills curriculum.

For the evaluation of the residents’ consultation letters, we used a 9-item, 5-point scoring scale (for a maximum score of 45) developed by Keely et al54 to analyze features such as content, style, organization, and educational value. We randomly selected 3 consultation letters written by each resident. Prior to implementation of our curriculum, the mean score was 32.8 with a SD of 4.4, whereas after implementation, the mean score improved to 40 with a SD of 2.9.

For the multisource assessment of the residents’ communication skills by allied health care professionals, we asked the nurses and pulmonary ward unit clerk to rate 4 aspects of the residents’ verbal and written communication as either unsatisfactory or satisfactory. Virtually, all residents achieved a satisfactory rating before and after implementation of our curriculum. On reflection, our rating scale had only 2 options – that is, satisfactory and unsatisfactory. Thus, we were likely unable to detect nuanced differences in performance. There did not appear to be any significant, unintended deterioration in the residents’ behavior with the new curriculum (within the limits of our evaluation).

Using direct observation, faculty rated the residents’ communication with patients and their families using a 5-point scale from 1 (poor) to 5 (excellent). Among 24 encounters observed before implementing our curriculum, 6 encounters received a score of 3 (adequate), 13 received a score of 4 (good), and 5 received a score of 5, with no encounters receiving a score of 1 or 2 (borderline). Among 20 encounters observed after implementing our curriculum, there seemed to be overall improvement with 2 encounters receiving a score of 3, 8 receiving a score of 4, and 10 receiving a score of 5.

Faculty also rated the residents’ communication skills during presentations using a 3-point scale (1, poor; 2, adequate; and 3, excellent). Out of 12 presentations given before implementing our curriculum, 1 received a score of poor, 7 received a score of adequate, and 4 received a score of excellent. Overall, scores improved after implementation where, among 10 presentations, 4 received a score of adequate and 6 received a score of excellent, with none receiving a score of poor.

During individual exit interviews, residents uniformly felt that the curriculum improved their written and verbal communication skills and felt that the faculty modeled the curriculum. We could find no instances where the formal curriculum conflicted with the learning environment.

Overall, we felt that we were meeting the accreditation standards, our residents were acquiring the communication skills that were the focus of our evaluation, and that there were no unintended consequences of our curriculum.

Conclusion

The RUFDATA framework is a simple tool that we used to evaluate our teaching program. It can be widely applied to evaluate other areas in education. For example, it has been used to evaluate other focused medical topics (such as a training module to teach intrapartum care64) and even entire undergraduate university courses.65 In the future, we plan to evaluate other aspects of our communication curriculum such as crisis resource management, discharge summary dictations, and resident-to-resident patient care handover. We will also refine our tool that assesses the residents’ communication with allied health professionals and explores options to involve patients and their families in the assessment.66,67

Although there are other frameworks that can be used to evaluate curricula, such Stufflebeam’s CIPP68 and Kirkpatrick’s Four-Level model,69 the RUFDATA framework’s reflexive nature helps foster a culture of curricular evaluation by enhancing reflection on the reasons for the evaluation. Medical educators can easily apply this framework to evaluate their own curriculum.