Shifting the Curve: Fostering Academic Success in a Diverse Student Body.

PROBLEM: Diversity in the health care workforce is key to achieving health equity. Although U.S. medical schools have worked to increase the matriculation and academic success of underrepresented minority (URM) students (African Americans, Latinos, others), they have had only limited success. Lower standardized test scores, including on the Medical College Admission Test (MCAT), have been a barrier to matriculation for many URM applicants. Lower subsequent standardized exam scores, including on the United States Medical Licensing Exam Step 1, also have been an impediment to students' progress, with mean scores for URM students lagging behind those for others. APPROACH: Faculty at the Morehouse School of Medicine developed and implemented interventions to enhance the academic success of their URM students (about 75% are African American, and 5% are from other URM groups). To assess the outcomes of this work, the authors analyzed the MCAT scores and subsequent Step 1 scores of students in the graduating classes of 2009-2014. They also reviewed course evaluations, Graduation Questionnaires, and student and faculty interviews and focus groups. OUTCOMES: Students' Step 1 scores exceeded those expected based on their MCAT scores. This success was due to three key elements: (1) milieu and mentoring, (2) structure and content of the curriculum, and (3) monitoring. NEXT STEPS: A series of mixed-method studies are planned to better discern the core elements of faculty-student relationships that are key to students' success. Lower test scores are not a fixed attribute; with the elements described, success is attainable for all students.

Problem

Attaining diversity in the health care workforce has been recognized as a key component of achieving health equity, but the goals of action plans to realize this diversity have not yet been reached.[1] While a variety of barriers to attaining diversity exist, poor performance on standardized tests has been a significant obstacle to the matriculation and academic progress of many aspiring physicians. On one hand, Medical College Admission Test (MCAT) scores are predictive of academic progress and performance on other standardized exams, including the United States Medical Licensing Exam (USMLE) Step 1; on the other hand, persistent differences in MCAT performance by race have been noted.[2,3] The tendency has been to regard standardized test scores as reliable, somewhat fixed, and unbiased indicators of academic potential, which has led to ongoing problems in the medical school admissions process.

Morehouse School of Medicine (MSM), a historically black medical school established in 1975, is committed to diversifying the health care workforce and training physicians to serve the primary care needs of the medically underserved. In working toward these goals, we have developed a process that has led to a high level of academic success for our diverse student body. In this Innovation Report, we describe MSM’s success in “shifting the curve” of students’ academic achievement and analyze the key elements that have supported this outcome.

Approach

MSM uses a holistic approach to recruit and matriculate a diverse student body of which on average about 75% are African American and about 5% are from other groups that are underrepresented in medicine (URM) (mostly Latino). This holistic approach includes seeking evidence of applicants’ long-term commitment to serving underserved populations, resilience, upward trend in grades, ability to overcome challenges, and similar character traits, through a careful review of their activities, personal statements, and an interview. Students’ entering credentials (e.g., grade point average [GPA], MCAT score) are comparable to those reported nationally by race/ethnicity.[4] However, we have developed and implemented a curriculum and support system that have resulted in standardized exam scores, like USMLE Step 1 scores, that are consistently well above those expected based on the MCAT scores of our matriculants.

As part of the ongoing monitoring of individual and program outcomes and to gain an understanding of what contributes to our students’ Step 1 scores being well above those expected, we compared the standardized test score profiles of matriculating students against their Step 1 scores. We also sought to determine what factors might have contributed to this shift. Although data are available for the graduating classes of 2007–2016, we limited the analysis presented here to the classes of 2009–2014.

For this analysis, we first looked at how our students’ MCAT scores compared with the national average. We normalized MSM students’ MCAT scores, by their expected year of graduation, to the national mean MCAT score for that class. We analyzed data as class-year cohorts because of the shifts in national mean scores over the study period (increases in mean MCAT scores from 29 to 31 and in mean USMLE Step 1 scores from 217 to 229). Using MCAT score curves provided by the Association of American Medical Colleges (AAMC), matched to the relevant year, we extrapolated MCAT reference data and then normalized those data to the national mean by dividing all scores by the national mean score for that year. We graphed this information for the matriculating classes of 2005–2011 (graduating classes of 2009–2014).

We next determined first-attempt Step 1 scores for the relevant classes, with reference values extrapolated from score curves provided by the USMLE and National Board of Medical Examiners. We graphed this information (normalized to the national mean score) for the graduating classes of 2009–2014 and compared the shapes of these curves against the shapes of the curves of MCAT scores for each class.

We then calculated a Pearson correlation coefficient comparing the MCAT scores and Step 1 scores for each class.

In addition, we used the formula created by Veloski and colleagues[5] to predict students’ Step 1 scores from their MCAT scores and GPAs. On average, students’ Step 1 scores were 22.6 points (more than a standard deviation) above the scores calculated from their MCAT scores and GPAs (range −7.0 to 50.7; data not shown).

Finally, to assess the key elements that lead to these outcomes, we reviewed responses to student surveys, including yearly course evaluations and the annual AAMC Graduation Questionnaire (GQ). The course evaluations were completed at the conclusion of every course and the GQ prior to graduation each year from 2008 to 2014. We also conducted faculty and student interviews and focus groups from 2012 to 2014 (approximately 6–8 faculty members and 7–10 students participated each year). Students and faculty were randomly identified and asked to participate in these short, small-group sessions. The collected data were evaluated using concept mapping techniques, and related themes were identified. We describe these themes below.

Outcomes

For the graduating classes of 2009–2014, the majority of MSM students had MCAT scores below the national mean (see Figure 1). However, the range of their USMLE Step 1 scores overlapped with the national range, shifting a full standard deviation compared with the predicted range based on their MCAT scores (see Figure 1). Correlations of individual students’ MCAT scores with their Step 1 scores over this period averaged 0.17 (range 0.10–0.30). A scatterplot of a single year of these data (class of 2010) is shown in Figure 2.

Figure 1

Comparison of Morehouse School of Medicine (MSM) students’ Medical College Admission Test (MCAT) score curves and United States Medical Licensing Exam (USMLE) Step 1 score curves for the graduating classes of 2009, 2014, and 2009–2014 combined. Scores are normalized to the national mean by year.

Figure 2

Scatterplot of Morehouse School of Medicine students’ Medical College Admission Test (MCAT) scores and United States Medical Licensing Exam (USMLE) Step 1 scores for the graduating class of 2010 (correlation coefficient = 0.12). Scores are normalized to the national mean for 2010.

For the graduating classes of 2009–2014, students self-identified on admission as African American as follows: 62.0% in 2009, 55.0% in 2010, 76.5% in 2011, 76.4% in 2012, 91.1% in 2013, and 78.3% in 2014. Other URM groups included Latino, Native American, and Pacific Islander. From 2009 to 2014, URM students represented from 76.4% (2012) to 93.0% (2013) of each graduating class. Using the broad categories of African American, white, and Asian (East and South Asian), we found no clear differences in the outcomes reported above by race (the sample sizes for some groups were small).

At MSM, the total attrition rate is approximately 2%, and the timely progression rate, defined as the percentage of students who complete the four-year curriculum in four years, ranges from 86% to 100%. Approximately 90% of students take Step 1 after two years in the preclinical curriculum. Thus, the higher-than-expected Step 1 scores cannot be explained by a disproportionately long preclinical phase or by a high early attrition rate.

Instead, in analyzing themes from the student and faculty surveys, focus groups, and interviews, we believe that this success is due to three key domains: (1) milieu and mentoring, (2) structure and content of the curriculum, and (3) monitoring—all of which can be replicated at other medical schools.

Milieu and mentoring

Strong teacher–student relationships are common at MSM, with both faculty and students using annual surveys to comment on the “family atmosphere” and “supportive faculty” as key elements of the school’s milieu. A core group of experienced, engaged, and dedicated faculty who are passionate about teaching and are known to build strong relationships with students foster this culture. Key attributes of this group include:

Content as well as pedagogical expertise with extensive experience in teaching and deep and broad knowledge of the content and style of Step 1 items,

Significant engagement in the curriculum (most with 50–100 or more hours of direct teaching contact with students yearly), and

Dedication to students and easy availability (in time and location) for questions (course-related or otherwise) or personal/small-group tutorials (know students by name).

Although the key academic resource that supports students is the faculty, peer tutoring is also available (about 30% of students use this option). Consistent peer and faculty support is offered through learning communities that start in the first year as part of the curriculum and continue through all four years of medical school.

Structure and content of the curriculum

MSM has a two-year preclinical curriculum that is a mixture of integrated and traditional discipline-based courses. The curriculum is structured so that the students are tested every three to four weeks, with intentional repetition to enhance knowledge retention. Exam skills, study skills, and time management are explicitly addressed in sessions that are part of this curriculum. Students who do not perform well on exams early in their courses complete required, structured faculty-guided sessions with in-course enrichment addressing key concepts in active and engaged learning strategies.

Monitoring

MSM has an established practice of outcomes monitoring, with teams of faculty reviewing students’ performance on examinations on a monthly basis and providing feedback and support for students who do not perform well. In addition, if these exams reveal classwide issues with concept mastery, faculty adjust the course schedule to reemphasize key concepts. This constant collaborative monitoring also enhances faculty-led continuous quality improvement of curriculum delivery. Students’ performance is also reviewed each month by the interdisciplinary Student Academic Progress and Promotions Committee, which provides specific guidance to individual students and their advisors.

With these key elements—milieu and mentoring, structure and content of the curriculum, and monitoring—we have enabled our students to reach their potential and consistently score higher on Step 1 than is expected based on their MCAT scores. These elements and results could be replicated at other institutions, leading to substantial progress in addressing the diversity of the health care workforce, in decreasing health disparities, and in achieving health equity.

Next Steps

To better understand the impact of faculty–student interactions on the outcomes we described above, we are undertaking a series of mixed-method studies to better discern the core elements of this relationship, including additional reviews of student surveys and questionnaires, focus groups, and other assessments.

Our outcomes should be viewed in light of several limitations. First, we reported the experience of one medical school in the context of changing exams (the new MCAT in 2015) and standards (a rise in national mean scores and minimum pass levels). While MCAT scores have not been predictive of USMLE Step 1 scores at MSM, scores on internal examinations have been very predictive of Step 1 scores. Because of our tight-loop feedback for students with lower course test scores, we could have a bias toward the correction of poor performance. At the individual level, student performance on Step 1 varies widely, with some students scoring 10 points higher and others scoring 50 points higher than predicted. We have not been able to identify matriculation characteristics that predict the degree of this individual difference. Finally, the effectiveness of our learning milieu appears to be linked to the strong caring student–faculty and student–student relationships. Although these relationships can be quantified, they are more difficult to reliably replicate in another environment. Noddings[6] and others have noted the central role of caring in an effective learning environment.

In spite of these limitations, this work has profound implications. Many have documented that African Americans score lower on many standardized tests, with mean scores on exams such as the SAT and MCAT being about a standard deviation below those of the general testing cohort.[7] Indeed, other studies have suggested that MCAT scores predict, or overpredict, Step 1 scores for this group.[8,9] Thus, lower standardized exam scores are sometimes seen as a fixed attribute that is partially, but not completely, explained by socioeconomic status.[5] Our work demonstrates that lower test scores are not a fixed attribute. Extraordinary learning materials, expensive technology, and specialized simulators are not required for African Americans to succeed on these exams. Instead, what is needed is faculty with exemplary knowledge, skills, and dedication; close and trusting relationships between students and faculty; and a supportive and nurturing environment that consistently expects academic success.