A Process for Assessment and Quality Improvement of the Clerkship Curriculum.

Reliance on the apprenticeship model of education in the clerkship years of medical education persists despite concerns with variability in educational delivery and outcomes. Although many institutions are addressing this variability, there needs to be a clear and objective method to assess what is working. Evaluating these educational experiences is an essential component to ensure that students graduate prepared to enter residency. In 2014, A.T. Still University's School of Osteopathic Medicine in Arizona (ATSU-SOMA) introduced a curricular change to address clerkship variability by implementing an online curricular component for the core clerkship courses in the third and fourth years of medical student education. Subsequently, a new structured and objective process to evaluate these courses was designed to improve student learning outcomes in the clerkship years. A Curriculum Year Three and Four Work Group was created to develop the new process for curricular evaluation of the clerkship courses. In the pilot phase of its implementation, described herein, the process fostered stakeholder participation and buy-in, enhanced communication of expectations, increased accountability in clerkship course design, and effectively employed objective evaluation tools in determining what curricular changes were needed. The Curriculum Year Three and Four Work Group continues to revise the tools and methods to enhance the efficiency of the evaluation process and to analyze whether recommended course revisions have improved student outcomes.

Background

Reliance on the apprenticeship model of education in the clerkship years of medical education persists despite concerns with variability in educational delivery and outcomes. Although calls for change are occurring, they are slow.[1] The primary goal of the learning environment in these 2 years is working with students to practice situated cognition, in which students learn to apply what they previously learned in the classroom to actual patient care under the guidance of experienced physicians.[2] A downside of this apprenticeship model is that student learning during this time tends to be inconsistent. Some factors may be the variety of physicians with whom students spend time and the variability in training, practice, and teaching patterns.[2-4] Another factor is that students’ didactic exposure in the third and fourth years is often dependent on the educational sessions and patient cases available at their rotation site. Furthermore, accurate assessment of student clinical competence requires a multifaceted approach that extends beyond the use of activity logs[5] and preceptor evaluations,[6] especially because these types of evaluations fail to differentiate between learner abilities.[7] The Flexner Report in 1910 and the subsequent Carnegie Foundation report in 2010 both emphasize the importance of standardization and integration of formal and clinical learning.[8]

Context

To address these concerns and to generally improve student outcomes on national board examinations, in 2014, A.T. Still University’s School of Osteopathic Medicine in Arizona (ATSU-SOMA) decided to enhance its educational offerings by adding weekly, required coursework via an online platform in the third and fourth years. The 11 core clerkships for which online content was designed included Family Medicine, Internal Medicine, Pediatrics, Obstetrics and Gynecology, General Surgery, Psychiatry, Neurology, Cardiology, Critical Care, Emergency Medicine, and Osteopathic Principles and Practice. The Clerkship Course Directors (CCDs) for these clerkship courses created the learning content and delivered it through the school’s learning management system (LMS), Blackboard, so all students could easily access it from locations across the country and internationally. This was the primary responsibility of the CCDs. Unlike other institutions,[9] at ATSU-SOMA, preceptor oversight and development does not fall under the purview of the CCDs and instead is the responsibility of the Regional Directors of Medical Education, who work directly with preceptors in furthering their development as clinical educators. To help close the loop of communication, the CCDs created short written guides for preceptors to better support the online course curricula and increase consistency across sites.

Problem

After the first year CCDs taught these new clerkship courses, ATSU-SOMA realized that there was a marked variation between courses and that the Curriculum Committee needed a systematic method to evaluate and document course effectiveness. The Curriculum Committee typically interviewed the CCD and reviewed data points of interest for the course, such as examination scores. However, there was no way for the Curriculum Committee to systematically and objectively rate the data and discussion points to define minimum expectations and determine what improvements the CCDs might need to make. Thus, we saw the need to create a more robust and documented evaluation system to decrease ambiguity for both CCDs and the Curriculum Committee.

This article describes the innovative, structured, and objective process that ATSU-SOMA created in 2016 to evaluate the core clerkship courses and details how this work led to enhanced communication, encouraged stakeholder engagement, increased accountability of course directors, and effectively employed objective evaluation tools to determine what curricular changes were needed.

Creating the Evaluation Process

Forming Curriculum Committee work groups

Curriculum committees at most educational institutions generally maintain oversight of the curriculum and usually include administrators and faculty who have critical roles in the planning, creation, and delivery of curricular content.[10] After noting how other institutions effectively reorganized their curriculum committees,[11] ATSU-SOMA formed working groups of the Curriculum Committee, based on the experiences of these other institutions, to subdivide the work of curriculum review to allow for a deeper evaluation of different academic years and assessment of specific tasks by those with pertinent experience. The Curriculum Committee created a Curriculum Year One and Two Work Group and a Curriculum Year Three and Four Work Group to evaluate the curriculum in those respective years.

Evaluation of the clerkship courses was a collaborative process between the CCD and the Curriculum Year Three and Four Work Group (hereafter referred to as the Work Group), and the Curriculum Committee reviewed and approved any final recommended changes. The Curriculum Committee purposefully selected Work Group members who had varied roles, perspectives, and levels of experience to ensure that the clerkship evaluation process addressed the diverse needs and concerns of CCDs, administrators, and students. The formation of these work groups allowed faculty and other key stakeholders who did not hold formal seats on the Curriculum Committee to have a say in the process of curriculum evaluation.

The chair of the Work Group was a clinical faculty representative of the Curriculum Committee who also happened to be a CCD. Members of the Work Group included 3 clinical faculty who were also CCDs of clerkship courses, 1 basic science faculty who had experience as a course director for first-year and second-year courses, 1 osteopathic medicine faculty who could evaluate the integration of osteopathic principles in each of the courses, the director of the Office of Evaluation and Effectiveness (assessment) who had analytic experience in developing methods that ensured usable outcomes, the staff member who facilitated delivery of course materials, and the assistant and associate deans of Clinical Education who had oversight of the third- and fourth-year curriculum. The Work Group also included the associate dean of Curricular Integration as an ex officio member who needed to stay informed of the Work Group’s progress. Like other institutions that have created teams or focus groups of students to obtain input on the curriculum,[12] ATSU-SOMA recognizes the value of student input in curricular reform and has used student focus groups for feedback of courses from the first and second years. The chair of the Work Group considered adding a student to the team but found that students on the Curriculum Committee were rarely available to attend or contribute because of academic obligations, and focus groups are more challenging to form during the third and fourth years when education is so widely distributed at ATSU-SOMA sites across the country. Thus, the Work Group used components of the course feedback survey from students as the primary and most efficient source of student input (Table 1). At this early stage of development, the ultimate stakeholders, the patients, were not involved in the initial course evaluation process, but we hope to include patient feedback in the future.

Table 1.

Student evaluation of the rotation (course evaluation).

For this course, please rate each of the following items: (rate each 1 = strongly disagree, 2 = disagree, 3 = neutral, 4 = agree, 5 = strongly agree)Questions
The course had clear objectives and goals stated in the syllabus or online in the LMS.
The reading, written, and other assignments complemented the course learning objectives.
I was able to complete my coursework and logs in a timely manner.
The knowledge/concepts in the course were applicable to clinical situations I encountered during the rotation.
My knowledge base has improved significantly as a result of this experience.
Comments: (free text)

LMS, learning management system.

The overall results of student feedback determined the rating assigned for the first item in the Clerkship Course Evaluation rubric.

Creating a timeline and communication plan

At the beginning, it took 6 months to develop the course evaluation process. Creating an annual project timeline for reviewing and revising the clerkship courses was also essential and ensured that the Work Group would meet the project’s goals and deadlines each year (Figure 1).

Figure 1.

Annual timeline for clerkship course evaluations.

CCDs, Clerkship Course Directors.

The group used backward planning to ensure it would meet target dates. During the period for course reviews from July to December, the group met for 2 hours every other week to evaluate 11 clerkship courses. The role of the CCD for each course typically accounted for 0.1 to 0.2 full-time equivalent depending on the duration and complexity of the course they were developing. Because this was not their sole teaching responsibility, CCDs typically needed several weeks to months to make course changes in this early phase of curricular development.

To minimize surprises and disruptions to usual workload and schedules and to increase the willingness to make changes, the Work Group determined that all persons involved or impacted by the evaluation process needed to know what to expect. Therefore, to ensure clear and accurate communication of the course evaluation process, the chair created a document that described the committee’s process for curricular evaluation of the clerkship courses and shared it with all CCDs, curriculum committee members, and administrators. This document provided clarity by describing the step-by-step process to evaluate clerkship courses and indicated who the responsible parties were for each step in the process.

Throughout the clerkship course review process, the Work Group documented and shared the notes from each weekly meeting with group members. A rapid editorial and dissemination procedure allowed for accurate recall of the discussions, and any questions or concerns could be addressed before the process was complete. With such a wide representation of stakeholders on the committee, web conferencing was essential for equal participation of those located on or off campus and broadened our pool of potential input.

Identifying necessary data

The Work Group and CCDs worked collaboratively to define what data were needed for the evaluation process. At ATSU-SOMA, because it takes a full academic year for an entire class of students to complete a given clerkship course, a full set of performance data and student feedback results for an individual course was not available until July of each academic year. To compensate for this delay of information, the Work Group provided mid-year course data packets to CCDs in January to allow them to assess student progress mid-year (Figure 1). The course data packets included the following reports for each clerkship course:

Clerkship course final grade distribution histogram;

Histogram of clerkship national subject examination performance;

Histogram for weekly coursework scores delineated by week;

Summative student feedback with comments from the Student Evaluation of the Rotation form;

Histogram of Level 2 national board examination scores with discipline subscores;

Anonymous student feedback comments.

Depending on the goals and interests of any given institution, different data sets may need to be included in the review packet to capture each school’s unique programs.

Developing the course evaluation process and evaluation tools

Establishing and codifying the evaluation processes ensured clear communication of expectations for all involved. The clerkship course evaluation process included reviewing essential information about each course and CCD commentary about his or her course. We decided that a course rating evaluation system based on predetermined objective criteria was necessary.

When creating the evaluation tools, we determined that they needed to be collaborative to ensure that expectations were clear between the CCDs being evaluated and the evaluators. The purpose of the evaluation tools was to gain insight into the course, answer key questions about course design and outcomes, and provide objective means of evaluation. Using Google Forms, the Work Group created and distributed a questionnaire for CCDs to complete (Table 2).

Table 2.

Questionnaire for Clerkship Course Directors.

Category	Questions
Demographics	Course name, course director, class year instructed, date of report.
Attestation	Verification course director received and reviewed all course data including national subject examination scores, coursework assignment scores, CRE scores, and student feedback scores and comments.
Course design	Describe how the course content is organized from start to finish week by week. Is there a framework approach you used (eg, GI course mouth-to-anus approach)?
	What did you want students to get out of the clerkship curriculum you provided? (Your own course goals as a Clerkship Course Director)
	List any national standards used as a reference when developing the course content.
	How do you determine where the knowledge and learning gaps are for students in this clerkship?
	How are you addressing the knowledge gaps you have identified?
	How were osteopathic principles incorporated into the course and how did the students engage with this material?
	Were there any other unique aspects of this course that are not done in other courses throughout the student’s educational experience?
	How do you view Level 2 board preparation in the context of this course? Was it something you are intentionally addressing and, if so, how? If not, what are your thoughts about the best way for students to prepare for the Level 2 examinations during the third year of their education?
Communication and feedback	How do you explain the purpose and importance of the curriculum and its coursework to the students (other than the syllabus)?
	What were the 3 most common positive themes expressed by students regarding the clerkship curriculum? (Information may be gleaned from logs, student evaluations, or course director interactions with students and RDMEs)
	What were the 3 most common negative themes expressed by students regarding the clerkship curriculum? (Information may be gleaned from logs, student evaluations, or course director interactions with students and RDMEs)
Assessment	Other than the national subject examination at the end of the rotation, how else is student learning of the curricular content in the course assessed (describe your weekly assignments or any other mode of assessment)?
	What were some of the strengths and weaknesses of the course assessments?
	Based on the results of the course data packet (end of rotation examinations, CRE, coursework assessments, and SER results), which of the items you wanted students to learn from the course (ie, your personal course goals from above) were achieved and what evidence supports this conclusion?
	Based on the results of the course data packet (end of rotation examinations, CRE, coursework assessments, and SER results), which of the SOMA curricular goals listed in the syllabus were achieved and what evidence supports this conclusion?
Conclusions	What was the best thing about the course that you plan to continue and why? What needs to be improved and why?
	What changes are you planning on making for the next year?
	List Curriculum Committee recommendations from the last year and how they were or were not addressed.
	Any final remarks?

CRE, clinical rotation evaluation; GI, gastrointestinal; RDME, Regional Director of Medical Education; SER, student evaluation of the rotation; SOMA, School of Osteopathic Medicine in Arizona.

The goal of the questionnaire was to gain insight into the course development process of the CCDs. The questionnaire included several items about course design, such as CCD goals for the course, methods of communication and feedback to students, methods of student assessment, interpretation of the course outcomes, and planned changes for the future. While creating the questionnaire, group members were especially interested in the CCDs’ impressions of the questions and worked to revise those that created ambiguity, such as the differentiation of course goals from programmatic competencies.

The questionnaire represents a portion of what ATSU-SOMA wanted to evaluate in this given year; however, the questionnaire could easily be modified to meet the unique goals and interests of a given institution.

Next, the Work Group created a course evaluation rubric to evaluate courses consistently and objectively (Table 3). Like the questionnaire, the rubric includes a portion of what ATSU-SOMA wanted to evaluate for this particular year and could be modified as needed.

Table 3.

Clerkship Course Evaluation Rubric.

Measure	Needs improvement	Meets expectations	Exceeds expectations
Student evaluation of rotation, ratings, and feedback on curriculum	<50% of class rated related SER items with positive responses.	50%-70% of class rated related SER items with positive responses.	>70% of class rated related SER items with positive responses.
Used national standards in course design to meet course objectives at the appropriate educational level	No identification of resources investigated.	Identifies potential resources investigated and documents why not applicable.ORDocuments national standards used and how.	Identifies multiple national standard resources and documents how they used them.
Conceptual design of CCD’s course goals	Course goals vague and not achievable by clerkship curriculum.	Course goals clear and achievable by the clerkship curriculum.	Course goals clear and achievable by the clerkship curriculum and forward thinking about the future of medicine.
Course content on LMS	LMS missing key content materials: syllabus, weekly assignments, learning objectives.ORLMS page departs from standard template, making materials difficult to find.	Syllabus, weekly assignments, and learning objectives present on LMS and easy to find.	Syllabus, weekly assignments, and learning objectives present on LMS and easy to find.Contains added resources to enhance student learning in a manner beyond what other clerkships provide.
Syllabus	Syllabus is poorly organized and does not follow template.ORSyllabus contains extra information that could otherwise be posted on LMS (ie, specific learning objectives).	Syllabus follows standard template and is well organized. Does not contain superfluous information that should otherwise be posted on LMS (ie, specific learning objectives).	Syllabus follows standard template, is well organized, and includes a concise list of diagnoses and procedures recommended for students to learn.
Learning objectives	Not provided or hard to find.ORProvided, but yield limited educational benefit.	Provided and offer adequate educational benefit.	Provided and will enhance student learning in a manner beyond what other clerkships provide.
Integrates osteopathic learning objectives or assignments	No or very few osteopathic objectives included or without reference source.	Some osteopathic objectives included for each week with reference source.	Osteopathic objectives integrated with nearly all other medical content including reference sources.
Appropriate volume of curricular material for weekly workload	Volume either too voluminous (>12 hours per week) or too small (<4 hours per week).	Volume achievable in 4-12 hours per week.	Achievable volume and minimizes potential perception of busy work.
Course assessments tagged with SOMA curricular goals	Few weekly assignments or other course requirements are tagged with SOMA curricular goal indicators.	All weekly assignments or other course requirements are tagged with SOMA curricular goal indicators.	All weekly assignments and individual quiz questions are tagged with SOMA curricular goal indicators.
Assessment applicability to specialty and usefulness for students	Assessments not applicable to specialty.ORUnlikely to enhance performance on national examinations or preparation for residency.	Assessments applicable to specialty and likely to enhance performance on national examinations.	Assessments applicable to specialty and likely to enhance student performance on national examinations.Assessments demonstrate application to Entrustable Professional Activities for residency.
Weekly graded assignments	Graded assignments not provided weekly to encourage learner engagement.ORGraded assignments of limited educational benefit.ORGrading instructions or rubric provided to students lack clarity.	Graded assignments provided weekly and are of reasonable educational benefit. Instructions and grading rubrics provided and sufficiently clear.	Graded assignments provided weekly and of high educational benefit. Instructions and grading rubrics provided and sufficiently clear and result in differentiation of learner abilities. Example assignments or other supporting materials provided to assist student in completion.
National subject examination score	Class performed lower than national average.	Class performed at or <1 SD above national average.	Class performed >1 SD above national average.
CCD recommended changes for the next year	Unrealistic changes recommended or unsupported by course data or documented feedback.	Basic recommended changes supported by course data and documented feedback.	Well-delineated recommended changes or enhancements supported by course data and documented feedback.
Progress on Curriculum Committee prior year recommended changes	No changes made.	Changes made and impact on student learning was negative, neutral, or unclear.	Changes made and impact on student learning was positive.

CCDs, Clerkship Course Directors; LMS, learning management system; SER, student evaluation of the rotation; SOMA, School of Osteopathic Medicine in Arizona.

Identifying benchmarks in rubric format had not been previously performed in curricular reviews, resulting in inconsistent evaluations between courses. During the first year of the evaluation process, therefore, the intention of the rubric was to ensure a basic level of consistency between courses rather than to evaluate the academic content of the courses. The rating level of “exceeds expectations” was included in the rubric to encourage clerkship directors to set goals above the minimum expectations and stimulate innovation. Although evaluation categories might vary from year to year depending on the specific goals and needs of the course reviews, we felt it was important to gather baseline information about course goals and objectives, structural design, and student feedback during this pilot phase of using this new evaluation process.

Complete evaluation of courses required the Work Group to review questionnaire responses of the CCDs, course performance data, and course documents provided to the students through the LMS. After reviewing this information, the Work Group conducted live meetings with each CCD to discuss the information and ask questions about the clerkship course curriculum. Scheduled meetings adhered to the following process:

CCD gave a general overview of the intention for the curriculum and the process used in planning the course;

CCD provided suggestions for recommended changes for the next year;

Work Group members asked questions of the CCD as needed;

Work Group members shared and discussed their rubric ratings;

Work Group members made suggestions for changes to the clerkship course curriculum, which may have included or excluded recommended changes from the CCD.

Course evaluations took place from July to December (Figure 1). During the first year of evaluations using this new process, meetings lasted 1 hour per course. In the next year, the time was reduced as the Work Group and CCDs were familiar with the process and asked fewer questions. In some cases, no meeting was needed with specific CCDs because there were no apparent deficiencies and the CCD provided sufficient information in the questionnaire, course data packet, and other course materials.

At the end of each course review meeting, the Work Group and CCD discussed the proposed recommended course changes. After all courses were reviewed, Work Group members voted on all the proposed course changes. Waiting to decide on final recommended changes to each course allowed for analysis of patterns between courses and for recommendations to focus on the best practices. These recommendations were subsequently shared with the Curriculum Committee, which reviewed and modified them as necessary. The Curriculum Committee’s final recommendations were then shared with each of the CCDs in January to allow time to make course changes before the start of the next academic year in July (Figure 1).

Documenting the process and findings

Google Forms and Google Sheets were essential for keeping track of the Work Group responses and for thoroughly documenting procedures and recommendations for the Curriculum Committee, especially when questions arose. Google Forms were used to collect the ratings of each of the rubric items for individual Work Group members. This method ensured that decision-making contributions were made by all members of the group rather than using group discussion of each item in the rubric where input from those who were less vocal or less opinionated may not have been accounted for. During the first evaluation year, all Work Group members rated all of the rubric categories, and an average score was calculated for each category. In the second year, to reduce the time commitment, each group member worked on an average of 2 items in the rubric that were evaluated across courses. The chair selected qualified evaluators for each item and allowed Work Group members to provide additional commentary on other rubric items.

Google Sheets were used to track progress and improvement from one year to the next and to share information with multiple individuals using a live document. This method of information sharing was especially beneficial in the case of group member turnover and for keeping the Curriculum Committee and administrators informed of progress. Each course had its own tab containing the evaluation rubric in the Google Sheet and included a column for CCD comments and a column for the Work Group members’ comments. The Work Group created a separate Google Sheet to keep a summary of discussions from each meeting and record attendance. The group’s decision-making process was democratic and decisions made were kept in a separate tab in the Google Sheet with dates and times.

Impact of the Clerkship Course Evaluation Process and Limitations

During the pilot phase of this new process, the holistic view the Work Group gained from this innovative clerkship course evaluation process (Figure 2) indicated that much work remains to be done to improve the clerkship courses. In particular, the responses to the questionnaires for CCDs revealed that several CCDs needed additional guidance in course design and pedagogy. Thus, the Work Group created a document that detailed clerkship course design and content expectations for CCDs as a basic guide to structuring clerkship courses. A key feature of this document was to guide CCDs in creating assignments with SMART (specific, measurable, achievable, relevant, and time bound) design structure. Feedback during group CCD meetings indicated that CCDs appreciated the clarity of expectations and guidelines to follow for their courses. As a result of this process, the Work Group noted that improvements were made in course design and student assessment methods, and CCDs reported receiving fewer student complaints the next year.

Figure 2.

Steps in creating the evaluation process.

CCDs, Clerkship Course Directors.

The evaluation process also revealed that additional work on curriculum mapping for the clerkship courses needs to occur at ATSU-SOMA. Currently, our mapping involves using the 7 osteopathic core competency domains and includes 33 competencies we have defined as SOMA curricular goals. When we looked at which SOMA curricular goals were addressed by all parts of the clerkship curricula, we identified 1 goal of the 33 goals that was not addressed; the missing goal was in the area of systems-based practice. Furthermore, future steps will likely involve investigating how our curricula address the Entrustable Professional Activities of undergraduate medical education. In addition, we are assessing how clerkship courses address ATSU’s Core Professional Attributes, which are the unifying competencies expected of all graduates of our university regardless of degree program.[13]

Another positive outcome of using this evaluation process was that CCDs considered more deeply what they were trying to accomplish in their clerkship curriculum. The process made them more accountable for their courses and the potential impact on student educational outcomes. This accountability also enhanced faculty collaboration because CCDs became more engaged in discussions with their peers about course design and online student assessment methods.

One of the most important benefits of using a working group for our clerkship course evaluation process included the buy-in from an array of stakeholders, which ensured the success of the process. Furthermore, the collaboration between the Work Group and CCDs led to other improvements in the overall structure of the clerkship curriculum, including necessary adjustments to the global clerkship grading scales, revision of preceptor evaluation forms, and enhancement of the student rotation feedback surveys.

We did encounter challenges during the process. For example, a few Curriculum Committee members, who were not in the Work Group, may not have read the Google Sheets and meeting notes and had questions about the clerkships when we presented our findings to the Curriculum Committee. These discussions resulted in their offering solutions without having the Work Group’s earlier input. In addition, the Work Group was not as empowered in evaluation and decision-making as it had initially presumed, and the Curriculum Committee’s additional recommendations for change came as a surprise to the CCDs, resulting in CCD dissatisfaction. In the future, the Work Group will need the explicit support of the Curriculum Committee to effectively conduct the tasks asked of it.

To ensure that repeated cycles of curricular evaluation adequately determine the effectiveness of the process, commitment from the group members and the CCDs is essential. At ATSU-SOMA, we were pleased to have 100% participation of the CCDs; each CCD had a vested interest in seeing the curriculum and, ultimately, student outcomes improve. However, the nature of academia, with its frequently shifting roles, responsibilities, and changes in leadership, places newly implemented changes at risk of dissolution.

Overall, ATSU-SOMA’s new curriculum evaluation process enhanced communication, stakeholder engagement, and accountability of course directors; and it effectively employed objective evaluation tools to determine what curricular changes were needed. Ideally, the Curriculum Year One and Two Work Group will successfully adopt and implement a similar evaluation process for their classroom-based curriculum in the next academic year. What is clear is that for sustained, robust curricular change to occur, the CCDs, department heads, and deans need to support and direct a rigorous course review process, such as that described in this article.

Subsequent evaluation and research at ATSU-SOMA is ongoing and will involve assessing the impact this process has had on student learning outcomes by monitoring score improvements on national board examinations, subject examinations, and preceptor evaluations of student performance. Other enhancements being considered are the use of student focus groups to gain deeper insight from students about how the clerkship course curricula can be improved. Ultimately, solicitation of patient feedback and input would also be included in this process. Furthermore, the Work Group will revise the course evaluation rubric each year to address evolving areas of needed evaluation. We hope that having described our process will enable other institutions to develop and revise methods of evaluating their course curriculum.