Will Any Road Get You There? Examining Warranted and Unwarranted Variation in Medical Education.

Undergraduate and graduate medical education have long embraced uniqueness and variability in curricular and assessment approaches. Some of this variability is justified (warranted or necessary variation), but a substantial portion represents unwarranted variation. A primary tenet of outcomes-based medical education is ensuring that all learners acquire essential competencies to be publicly accountable to meet societal needs. Unwarranted variation in curricular and assessment practices contributes to suboptimal and variable educational outcomes and, by extension, risks graduates delivering suboptimal health care quality. Medical education can use lessons from the decades of study on unwarranted variation in health care as part of efforts to continuously improve the quality of training programs. To accomplish this, medical educators will first need to recognize the difference between warranted and unwarranted variation in both clinical care and educational practices. Addressing unwarranted variation will require cooperation and collaboration between multiple levels of the health care and educational systems using a quality improvement mindset. These efforts at improvement should acknowledge that some aspects of variability are not scientifically informed and do not support desired outcomes or societal needs. This perspective examines the correlates of unwarranted variation of clinical care in medical education and the need to address the interdependency of unwarranted variation occurring between clinical and educational practices. The authors explore the challenges of variation across multiple levels: community, institution, program, and individual faculty members. The article concludes with recommendations to improve medical education by embracing the principles of continuous quality improvement to reduce the harmful effect of unwarranted variation.

Medical education, both undergraduate and graduate, has long embraced uniqueness and variability in curricular and assessment approaches. The rationale is multifactorial: medical schools, residencies, and fellowships possess different strengths and weaknesses, promote differences in their missions, and are located in different geographic regions and communities with different needs. Changing institutional financial incentives and pressures are also important emerging factors that contribute to variation. However, the primary goal of medical education is to ensure all learners acquire the essential competencies (i.e., educational outcomes) necessary to meet societal needs (i.e., health and health care outcomes). Outcomes should drive educational processes and design. Some variability in clinical and educational processes is justified, particularly the adaptive variability (i.e., warranted or necessary variation) that enables meeting patient and community needs. Variability that fails to meet patient and community needs is considered unwarranted variation.

Unwarranted variation in curricular and assessment practices ultimately contributes to suboptimal educational outcomes, widens the fragmentation across the medical education continuum, and by extension contributes to suboptimal health care outcomes delivered by graduates. [1-3] Unwarranted variation occurs at multiple levels within the health care and educational systems. Learners are nested within training programs (medical schools, residencies, and fellowships), which are nested within institutions that are nested within communities. These nested relationships are interdependent, and this interdependence impacts clinical and educational outcomes with potentially profound effects.

In this article, we examine the effects of unwarranted variation in health care, its correlates present in undergraduate medical education (UME) and graduate medical education (GME), and the need to address the interdependency of unwarranted variation occurring between health care and educational practices. We also explore the challenges of variation across the multiple levels and conclude with recommendations to improve medical education by embracing the principles of continuous quality improvement (CQI) and reducing the harmful effect of unwarranted variation.

Unwarranted Variation in Patient Care: Definitions

Unwarranted variation has been a persistent and pernicious quality and safety problem in health care delivery. Landmark studies by Wennberg and colleagues, starting in the 1970s, found that much of the variation they documented in care delivered to patients across multiple conditions could not be explained by patient or community demographics or patients’ underlying health status and comorbidities. [4-6] Scientists have long noted the interdependent relationship between structure and process in health care delivery on outcomes. [7,8] Unwarranted variations, especially in processes of care, do not contribute to better outcomes for patients. Unfortunately, unwarranted variation continues to be a problem today. [6] Wennberg, in his book Tracking Medicine, highlights 3 specific categories of care subject to unwarranted variation: effective or necessary care, preference-sensitive care, and supply-sensitive care. [4]

Effective care, or science-informed care, is defined by the presence of high-quality scientific evidence that guides patient evaluation and treatment, meaning that the majority of patients should receive (e.g., vaccines) or not receive (e.g., X-rays for low back pain) this type of care. [9] Suboptimal quality occurs when there is underuse or overuse of care. [6,10,11] Preference-sensitive care is defined as care and interventions where more than one evidence-based option is available, or insufficient evidence exists for a preferred option, and outcomes for each option likely differ. For example, there are multiple prostate cancer screening and treatment options, and quality-of-life outcomes can vary substantially among them. [12] A major quality problem with preference-sensitive care is that the preference of the physician, rather than the patient, often dictates choice. [4,11,12] Supply-sensitive care relates to the frequency and amount of care a patient receives, such as the frequency of office visits for a patient with diabetes. [4] Preference- and supply-sensitive care are especially prone to misuse and overuse. [6] Detrimental inequities in all 3 categories of care due to bias and structural racism remain persistent. [8,13,14]

Within training environments, unwarranted variation in health care can be further amplified by substantial variability in curricular and assessment approaches across the medical education continuum. The nested relationships experienced by health professions students are critically important, and their interdependence contributes to the educational outcomes of a training program (see Figure 1). Some of the variation in both clinical and educational approaches is justified based on local needs of patients and learners and available resources. Warranted variation ideally should contribute to optimal clinical and educational outcomes. However, training programs need to devote more attention to the impacts of unwarranted variation embedded within the nested, interdependent relationships as part of their CQI efforts for both clinical and educational processes to help achieve desired outcomes.

Figure 1

In the latter stages of undergraduate medical education and in graduate medical education, learners work and learn in clinical environments, most notably microsystems. The community has a major impact on who the learners will care for. Institutional and educational programmatic culture and resources will substantially shape learners’ development. Learners will form bonds of variable degrees with patients (e.g., thickness of the bidirectional arrow). Faculty involvement can be highly variable with both learners and patients (strength of connection represented by the dotted arrows).

Educational training programs are subject to the same categories of variation outlined by Wennberg and colleagues [4] (see Table 1). Variation operates at all levels shown in Figure 1, and within each level, educational factors and processes can serve to mitigate or amplify problems of unwarranted variation in both clinical and educational outcomes. The desired integrative relationship of educational and clinical outcomes within the institutional and training program levels must emphasize the centrality of the patient (see Supplemental Digital Appendix 1, at http://links.lww.com/ACADMED/B248). Using this outcomes lens, we examine variation at each level, starting with the community, then moving to the institution, the program, and the faculty educators within the program.

Table 1

Correlates of Unwarranted Variation in Medical Education

Community-level variation

Ongoing analyses by groups such as the Dartmouth Atlas Project and Commonwealth Fund continue to demonstrate wide variability in health care costs, disparities, and quality across the United States. [6,13] A logical question is whether such variation might be perpetuated during medical education training. Early research suggests the answer is probably yes. For example, Chen and colleagues found that family medicine and internal medicine residents who trained in higher-cost regions were more likely to be higher-cost providers regardless of where they practiced after graduation. [15]

Furthermore, the location and nature of a community often determines what patients and conditions will be encountered by learners within a program. Substantial variation in experience due to community factors affects the nature of the workforce that is supplied to meet patient care needs, particularly if graduates ultimately practice in a community different from where they trained. The impact of community-level variation on educational outcomes has not been fully elucidated and warrants further study. Additionally, more research is needed on whether and how learners become competent in a new community where they might care for different patient populations compared with where they trained, a component of practice-based learning and improvement (PBLI).

Institutional-level variation

Substantial variation continues to exist between training institutions that cannot be fully explained by risk adjustment or population demographics. [6,16,17] For example, Wennberg and colleagues found substantial variation in care between training institutions with similar patient demographics in 2 separate cities. [16] This institutional-level variation also appears to affect educational outcomes. In oft-cited studies, Asch and colleagues found a strong association between the rate of major obstetrical complications at a teaching hospital and the rate of major obstetrical complications among the hospitals’ graduates in practice. [18,19] This relationship was noted to persist for over 15 years. Subsequent studies examining institutional-level performance in surgical complications and costs of care found similar patterns. [20,21]

The Accreditation Council for Graduate Medical Education (ACGME) Clinical Learning Environment Review (CLER) program has also found substantial variation in how institutions engage trainees in quality and safety processes. It can be very difficult to concomitantly improve health care and education without including learners in the process. In the most recent CLER report, less than a quarter of institutions reported including trainees in their institutional-level quality improvement and patient safety (QIPS) initiatives. [22] This highlights the potentially problematic lack of integration between institutional care delivery performance and QIPS education, with significant implications for both learners and patients.

Institutional-level variation can be magnified when learners rotate at multiple hospitals and other clinical settings. Each training site has its own culture and resources that contribute to clinical and educational variability. Quality management around clinical and educational standards and practices across multiple institutions is often limited, potentially creating an amalgam of unwarranted variation within programs. This situation may contribute to suboptimal care for patients and impede professional development if medical students, residents, and fellows are not properly prepared to work in a new environment.

This shifting of learners between institutions is often justified as a mechanism to provide learners with broader experience, a reasonable educational rationale. However, standard protocols for managing the transition between institutions are lacking. [23] Lack of attention to the impact on patients and assessment of whether such curricular practices improve educational outcomes may introduce unwanted and unwarranted sources of variation. In addition, harmful and unnecessary variation (misuse, overuse, underuse) may result from misaligned financial incentives within the institution that can filter down to learners.

Program-level variation

Accreditation and certification create high-level standards around curricular content and experience, and while programs are nested within institutions, and often more than one institution, they still have meaningful control and oversight of their curricula and assessment. Yet, the intersection of regulatory requirements with institutional needs can result in unintentionally rigid educational structures. [24,25] For example, many medical schools and residency training programs rely heavily on a rotational training model to meet regulatory (e.g., broad coverage of curricular content) and clinical service needs. However, what happens educationally (i.e., processes) during these rotations can vary substantially from school to school and residency to residency. Learner development can vary substantially depending on when and where specific rotations are scheduled, and rotations are often scheduled independent of a learner’s developmental readiness.

Additionally, the variable approach to managing transitions between rotations has serious, negative ramifications for learners and patients. [26-29] For example, Denson and colleagues found an association of increased patient mortality in internal medicine programs when patients were discharged during intern and resident service transitions. [27,28] Bernabeo and colleagues uncovered numerous stories of the negative impact of rotation transitions on patients, learners, and other health care professionals. [25] Interns and residents described developing their own “system” for these transitions in the absence of programmatic or clinical care standards. Faculty were often unaware of the impact of these rotational transitions, with some faculty arguing such transitions were simply part of trainees’ development. These studies highlight that even a single source of programmatic variability, such as how transitions occur and are managed, can have profound effects—an argument strongly in support of tighter standards around rotational transitions.

Compounding this problem is the general reticence of programs to feedforward learner information to faculty from one rotation or clinical experience to another. [30] Yet a growing number of studies are documenting what has been known anecdotally for a very long time—learners experience different learning curves across different competencies. [31-34] McGaghie has long argued that medical education should use a mastery-based approach, requiring a willingness of the medical education enterprise to let go of current assumptions that dwell time, rotation-based curriculum designs, fragmented attending coverage and supervision, and single-point-in-time high-stakes assessments are sufficient to produce optimal outcomes. [33] While there are legitimate issues for medical schools regarding Family Educational Rights and Privacy Act regulations, they do not preclude feedforward activities within the medical school if proper policies and procedures are in place. [35] For GME, feedforward should become a standard practice to support professional development, recognizing the feedforward process must be done effectively to avoid unintended consequences.

Mastery-based learning also depends on robust, systematic assessment. [33] Unfortunately, programs across the continuum vary substantially in both their approach to assessment and the quality of their assessment activities. Twenty years after the launch of the Outcome Project, most GME training programs still struggle to assess the competencies of PBLI and systems-based practice (SBP). However, an ongoing commitment by institutions and training programs to advance PBLI and SBP is still urgently needed. [36] The quality and patient safety measures and robust multisource feedback tools remain substantially underused in GME programs. [22,37,38] Clinical competency committees (CCCs), while an important addition to the assessment system, are still learning how to effectively synthesize and collate assessment data and fill assessment gaps. CCCs remain heavily dependent on faculty assessments that are one of the largest sources of unwarranted variation. [39]

Faculty-level variation

Differences in faculty clinical and educational competence are underappreciated sources of unwarranted variation. For some researchers, faculty variation in teaching and assessment is viewed positively. [40,41] For example, idiosyncrasy in assessment can be seen as a matter of 2 faculty members each emphasizing different but legitimate aspects of performance. If that idiosyncrasy aligns with effective, science-informed practice, the difference between faculty can be useful, warranted variation that can contribute to richer feedback and a fuller assessment of learner competence (provided the learner recognizes it as such). [40,41] If idiosyncrasy results from outdated or ineffective clinical practices, or the inappropriate execution of preference- or supply-sensitive care, the variation is unwarranted, thereby undermining the quality of the assessment of learner competence. There are significant long-term implications related to faculty as an unwarranted source of variation in assessment and supervision of clinical care.

Faculty clinical competence.

Programs operate from the assumption their faculty possess all the necessary competencies to serve as clinician, teacher, assessor, and coach. Furthermore, supervising faculty competence in QIPS, clinical skills, diagnostic accuracy, direct observation, assessment, and evaluation are commonly, though tacitly, considered “competent until proven incompetent” by the public and by regulators. The same assumptions are made about institutions and programs. While the vast majority of clinician–educators are unquestionably dedicated to their role, the assumption that all faculty possess all needed competencies is deeply flawed. [42-44] The majority of current clinician–educators trained in an era before there was full and effective training in the 6 ACGME general competencies. Furthermore, institutions have not sufficiently invested in faculty development in the newer competency domains such as QIPS, interprofessional teamwork, and systems thinking. Even for the 2 competencies with which most faculty feel comfortable, medical knowledge and patient care, studies demonstrate that faculty possess variable levels of competence, and this variability has real-world implications for learners. [42,44] Learners often recognize that faculty have disparate approaches to patient care that can lead to discordant feedback as well as variable approaches to supervision. This situation can create substantial difficulties for learners to determine what feedback is most ideal to incorporate into their own practice.

Faculty assessment.

Unwarranted variation commonly surfaces in faculty assessment of learners. It is well established that low interrater agreement using rating scales for work-based assessment can be problematic for both learner and patient. [44] Studies have shown faculty often use self as their primary frame of reference and attend to different aspects of a clinical encounter when directly observing a learner with a patient, often missing clear errors in care. [39,42,44] While assessments across faculty can be combined over time to provide an aggregate picture of competence, a given patient often receives care at a single point in time such as an ambulatory visit. Therefore, a correct assessment must be made in each encounter for the patient to receive quality care, even if the assessment is only based on a learners’ oral presentation to faculty.

Figure 2 highlights 2 faculty members assessing the performance of a trainee differently for the same encounter, with one faculty member missing an element of unacceptable care and another faculty “observing” performance in the best practice and acceptable zones that was not actually done by the trainee. Combining these observations becomes a mishmash of accurate and inaccurate observations and assessment that may lead to suboptimal feedback and supervision decisions. This variability also undermines the credibility of feedback and leaves to chance whether the learner incorporates the feedback that was most accurate. Finally, CCCs have no straightforward mechanism of making critical distinctions between the 2 assessments. For patients, this variation between the 2 faculty members, especially the unwarranted variation in the care they received, simply equates to suboptimal and potentially unsafe care. [44]

Figure 2

highlights the zones of care from a learner-patient encounter perspective, showing the implications of faculty idiosyncrasy and variable competence of the learner and faculty skill in assessment. “Quality of care” can be viewed as a set of performance zones: best evidence care, acceptable but not optimal care, questionable care, and unacceptable care.

Let’s use informed decision making (IDM) as an example. Multiple studies have provided guidance on effective communication practices for IDM, such as how to set the stage for the conversation, how to explore patient preferences, use of teach back, and so forth.[42,61] Best evidence care would include these evidence-based behaviors. Acceptable care might include the behavior of asking if the patient has questions but neglecting a teach back. Unacceptable care would be interrupting the patient so they cannot ask a question about the medical decision.

The trainee’s performance in each of these zones is represented by the tiled area. In other words, the trainee performed some, but not all, of the evidenced-based elements of IDM. As we can see, the trainee also provided an element of unacceptable care in the encounter, such as dismissing a patient question or concern. Faculty 1 detected some, but not all of the best evidence care and acceptable care provided by the trainee but completely missed the component of unacceptable care. From a sampling perspective, Faculty 1 could contribute some meaningful assessment information about the trainee. From the patient perspective, however, this is a supervision “system failure” by failing to recognize and address the unacceptable care. Faculty 2 detected some, but different, behaviors of best evidence care and the unacceptable care, a better outcome for the patient compared to Faculty 1 but resulting in a very different assessment for the learner.

Faculty also vary in their approach to supervision of learners, using different combinations of factors and approaches that can be exacerbated by variable approaches to assessment. [45-47] This variability in supervisory styles creates confusion for learners and potentially puts patients at risk. [48,49] Schumacher and colleagues found essentially little to no correlation between supervisors’ entrustment ratings of learners and the quality of care delivered to children in a pediatric emergency department. [50] These studies and others suggest a potentially harmful interaction between faculty (in)abilities in clinical competencies and assessment skills, and subsequent supervision decisions, especially when rating scales are employed as the primary means of making assessment judgments.

The Variation Cascade

While the interactions between the hierarchical levels of variation can have deleterious implications for patients, variation can also occur at multiple points during a patient care episode within teaching settings (see Figure 3). At each stage of the encounter, multiple learner and faculty factors are present that can enhance the probability of unwarranted variation, or frank clinical errors, to the detriment of the patient. Furthermore, faculty can choose whether to directly observe the learner at each stage. For example, in family medicine and internal medicine, faculty typically “see” patients indirectly by listening to a resident’s case presentation. This requires the sign-out process to be systematic and accurate. Unfortunately, case presentations may not accurately represent what transpired in the resident–patient encounter. [51,52] In our faculty development work, we have heard countless stories of faculty experiencing profound discordance between what they heard from a case presentation and what they discovered when they saw the patient directly. [53] Singh and colleagues also found that diagnostic error in an ambulatory visit was a common reason for unexpected patient return visits within 14 days, highlighting the importance of appropriate supervision of accurate medical interview and physical examination skills for patients presenting with new signs and symptoms. [54] This is not meant to criticize faculty, but rather to highlight that learners develop and achieve competence at different rates.

Figure 3

Variation, both warranted and unwarranted, can occur at multiple points in a single patient-learner encounter. Unwarranted variation experienced in the earlier stages of the encounter may be carried forward in the latter stages of the encounter and post-encounter, potentially and negatively affecting quality and safety for patients and learner development.

Variable approaches to observation and supervision appropriately aligned with learners’ developmental stage is logical, but misaligned and ineffective approaches can lead to significant deficiencies in both clinical care and education outcomes. Furthermore, failure to identify unwarranted variation or frank clinical errors negatively impacts the learner’s own development, such as entrenching suboptimal skills or experiencing the emotional impact of a consequential error. It is essential that faculty work to acquire high levels of competence in both the competencies of health care and teaching and assessment, as clinical care and education are highly interdependent (see Supplemental Digital Appendix 1, at http://links.lww.com/ACADMED/B248).

Suggested Steps to Address Unwarranted Variation in Medical Education

Activism as a philosophy is now needed to create meaningful change in medical education. Medical education must assume the tenets of CQI and innovation to methodologically address prevailing and emerging challenges in educational programs. With this foundational purpose, medical education could be a prime mover to help achieve a transformed future state for health care.

There are several changes to UME and GME curricula and assessment at the institutional, program, and individual learner levels that could reduce unwarranted variation while concomitantly improving health care and medical education. At the institutional level, hospitals and clinics should systematically involve learners in identifying and reducing quality gaps and unwarranted variation by targeting those measures with substantial clinical impact. [55] Learners, being frontline care givers, are ideal sources of information to detect and then address unwarranted variation as part of institutional CQI. GME programs can work with their institutional leadership to incorporate learners into QIPS initiatives or support learners in specialty-specific QIPS projects. [56]

Seven curricular and assessment targets focused on programs and their faculty can and should be implemented on a wide scale to accelerate improvements (see Table 2). Implementing these recommendations will cause or exacerbate numerous tensions such as standardization versus flexibility, learner-centeredness versus patient-centeredness, and formative versus summative assessment. Navigating these tensions requires a “both/and” polarity management approach. [57] Polarity thinking requires medical educators to examine tensions as “poles” that each have benefits (“upsides”) and also disadvantages (“downsides”). Unwarranted variation, by definition, is unwanted and harmful and does not represent a viable upside choice or pole. As programs and institutions struggle with these tensions, warranted and unwarranted variation should serve as a set of lenses for making curricular and assessment choices.

Table 2

Recommendations to Accelerate Improvements in Medical Education by Reducing Unwarranted Variation

Finally, a co-production mindset and approach can support CQI and further reduce unwarranted variation. Co-production in health care, as defined by Batalden and colleagues, is “the interdependent work of users and professionals to design, create, develop, deliver, assess and improve the relationships and actions that contribute to the health of individuals and populations.” [58] Co-production means that patients and learners must be critical partners with faculty, program, and institutional leadership in reducing unwarranted variation in health care and medical education. [58] Involving learners in co-production creates additional dividends by better supporting professional development, self-regulated learning, and reflective practice.

Both medical schools and GME programs can advance CQI, co-production, and reduce unwarranted variation by adopting the following practices:

Engaging learners as partners in curricular and educational system design;

Individualizing learning pathways for learners where possible by incorporating effective science-informed educational practices;

Enabling learners to “own” their own performance portfolio that includes quality and safety measures that support learning and growth;

Optimizing opportunities for learners to develop meaningful, longitudinal relationships with patients, peers, and teachers;

Engaging and supporting learners in guided reflection and self-assessment; and

Engaging patients in formal teaching roles that involve direct attention to issues of warranted and unwarranted variation. [59]

It will be exceedingly difficult to advance competency-based medical education if programs do not involve learners in addressing variation within the program.

Conclusions

Medical education institutions and programs have a long history of embracing uniqueness and honoring the nuances, or art, of patient care. Some of this variability is necessary, justified, and even valuable; but much is not and represents unwarranted variation that can lead to suboptimal outcomes for both patients and learners. One of the primary goals of medical education is to contribute to improving health and health care; reducing unwarranted variation in outcomes is an essential component so that learners are equipped with the competencies essential to practice and can function in our often fragmented health care delivery system. Further, ensuring that graduates can effectively practice CQI and reduce unwarranted variation is part of the solution for fixing an imperfect system. We must pay more attention to determining what is acceptable and unacceptable variation in approaches to patient care and medical education, and we must illuminate the interdependencies between the 2. [60] To start, confronting unwarranted variation in clinical care, poor quality of care practices, and patient safety problems is a logical target for all medical schools, residencies, and fellowships that will have direct educational benefits. Implementing science-informed educational changes and interventions can help concomitantly improve patient care and our learners’ and faculty’s professional development. The time is now to address and tackle the problem of unwarranted variation in medical education. Doing so could ultimately improve the educational outcomes of our learners and, most importantly, the health and outcomes of our patients and their communities.

Acknowledgments:

The authors would like to thank Paul Batalden, MD, and Jung Kim, PhD, for their thoughtful review of the manuscript.