Using the Residency Matched Method and Intent to Practice Method to Estimate Primary Care Workforce Production.

Introduction: Many medical schools overestimate the percentage of their graduates who enter the primary care workforce based on the "first-certificate" residency their graduates enter. To rectify this problem, Deutchman and colleagues proposed a new method of estimation. The objective of this study was to compare results from the traditional residency match and Deutchman methods to the actual percentage of University of Kansas School of Medicine (KUSM) graduates who practice primary care after completing medical school and all residency and subspeciality fellowship training.
Methods: A retrospective study was conducted using a convenience sample of KUSM graduates from 2003-2014. Percentages of graduates classified as primary care by the traditional Residency Match Primary Care Method (RMPCM) and the percentages of graduates identified as primary care by Deutchman's Intent to Practice Primary Care Method (IPPCM) were compared with the actual percentage of graduates who eventually entered the primary care workforce.
Results: Of the 1,944 KUSM graduates identified during the study period, the RMPCM predicted a 48.1% primary care output rate. The Deutchman's IPPCM predicted a 22.8% primary care output rate. The actual known percentage of graduates practicing primary care was 34.2%. Conclusions: Neither the RMPCM nor the Deutchman's IPPCM performed well in predicting the percentage or number of KUSM graduates who eventually practiced primary care. Due to predictions for the shortage of primary care physicians, there is a need to identify a method that more accurately predicts the medical schools' contribution to the primary care workforce.

INTRODUCTION

Public policy leaders have requested an increase in accountability from medical schools and hospitals for their use of Medicare Graduate Medical Education (GME) funding used to train resident-physicians.1 The Medicare program provides two types of extra funding to hospitals with residency programs. The first intends to offset the direct costs of residency programs, such as faculty salaries and administrative costs. The second provides a financial enhancement to a hospital’s Medicare Prospective Payment System reimbursement that is intended to make up for additional patient care costs associated with inefficiencies in training resident-physicians.2 The magnitude of these two payments is a driving force in the development of the physician workforce in the United States. The design of these payments may contribute to the national shortage of primary care physicians. It has been proposed that the current Medicare GME funding system incentivizes hospitals to preferentially support subspecialty training at the expense of primary care training.1

A source of controversy in determining accountability is the method of measuring workforce outcomes from each training program. One measure counts the number of medical school graduates who match into primary care residency programs; primary care generally is defined as family medicine, internal medicine, pediatrics, and medicine-pediatrics. This is known as the RMPCM.3 However, because of the various subspecialty fellowships that residency graduates may enter after completion of their “first certificate” residency program, the RMPCM measurement magnifies the actual number of medical school graduates who eventually enter primary care practice at the conclusion of all their GME. For example, half of pediatrics residents and well over half of internal medicine residents enter subspecialty fellowships at the conclusion of their “first certificate” residency training.4,5 Some have called this quandary the “Dean’s lie”; double counting a significant number of medical school graduates as both primary care physicians at entry into residency and subspecialty physicians at the conclusion of their fellowship training.6,7

Understanding the final outputs of the entirety of the medical education process is important to meeting physician shortage and mal-distribution problems. In 2013, Chen et al.1 demonstrated a new way of determining the ultimate workforce specialty mix at the conclusion of GME training. Using the AMA Masterfile and data from the National Residency Matching Program (NRMP), this method measured a medical school’s outputs five years after graduation, when primary care practices generally are established and subspecialty fellowship training has been completed.

Predicting the future primary care workforce and setting public policy on the use of funding for medical education is an important goal. An accurate predictive model based on NRMP match statistics would be useful and efficient and avoid the need to track all graduates of a medical school and determine their specialty-of-practice after completion of all their GME, years after they complete medical school.

In 2020, Deutchman and colleagues proposed another method of GME output measurement by limiting the determination of a medical school’s primary care output to the match categories of family medicine, medicine-primary, pediatrics-primary, and medicine-pediatrics.3 Deutchman and colleagues compared this method (IPPCM) to the traditional RMPCM, then used online resources to confirm the physicians’ actual practice specialty at the conclusion of their medical school, residency, and subspecialty fellowship training. Deutchman and colleagues concluded that the new “Intent to Practice Primary Care Method more accurately predicted a medical school’s actual primary care output than the Residency Match Primary Care Method”.

This study applied the traditional RMPCM, as well as the new IPPCM, to determine the primary care production of graduates from the University of Kansas School of Medicine (KUSM) and compared them to the actual percentage of graduates who practice primary care.

METHODS

A retrospective study was conducted of the RMPCM and the IPPCM using a convenience sample of KUSM graduates. To allow time for completion of residency and entry into practice, physicians who had graduated from medical school between 2003 and 2014 were included, the same time period included by Deutchman and colleagues3 in their study. The information available to us included the graduate’s name, medical school graduation/match year, match specialty, city and state of the matched residency program, and current practice specialty. To determine the actual practice specialties of the physicians at the conclusion of all GME training, several methods were used, including internet resources such as Doximity, LinkedIn, and Google/Yahoo, as well as a residency graduate database available in the Department of Family and Community Medicine on the KUSM-Wichita campus. This was called the Actual Primary Care Method (APCM).

Table 1 outlines the specialties that Deutchman and colleagues3 defined as primary care for the RMPCM, the IPPCM, and the APCM. Table 1 also includes those specialties that Deutchman and colleagues specifically defined as not primary care. The Deutchman IPPCM calculates primary care output using medical school match categories of family medicine, medicine-primary, pediatrics-primary, and medicine-pediatrics. Board certification in family medicine, internal medicine, and pediatrics through the American Board of Medical Specialties is sometimes referred to as a “first certificate”.8 Physicians with a “first certificate” may indicate that they are board-certified in one of those three primary care specialties. Many subspecialty fellowships require a “first certificate” for subsequent medical subspecialty board certification. In our study, the Deutchman and colleagues’ definitions of primary care (Table 1) were used in our calculations of RMPCM and IPPCM.

Table 1

Definitions of primary care used in this study.

Definitions Used at Entry into Residency after Medical School Graduation		Definitions Used at Time of Entry into Practice after Residency Completion
Residency Match Primary Care Method	Intent to Practice Primary Care Method	Actual Primary Care	Not Primary Care
Internal medicine (categorical) Medicine-primary Family medicine Pediatrics (categorical) Pediatrics-primary Medicine-pediatrics	Medicine-primary Family medicine Pediatrics-primary Medicine-pediatrics	Family medicine General internal medicine General pediatrics Medicine-pediatrics Geriatrics	Any medical or surgical subspecialty hospitalist Emergency medicine Urgent care Hospice/palliative care

Note: Reprinted with permission from Deutchman et al.3 All rights reserved.

Standard descriptive statistics were used to describe primary care for the RMPCM, the IPPCM, and the APCM. Chi-square tests were used to compare the proportion of graduates matched to primary care defined by RMPCM, the IPPCM, and APCM. Microsoft Excel and the IBM SPSS (Statistical Package for the Social Sciences; Armonk, NY) version 26 were used for these analyses. The University of Kansas Medical Center Institutional Review Board reviewed and granted an exempt status to the study.

RESULTS

Results of the Residency Match Primary Care Method

As Table 2 shows, data on 1,944 KUSM graduates were included in the study. The RMPCM yielded primary care match rates that ranged from 43.3% to 52.5% per year and an overall 12-year average of 48.1% (935 of 1,944) for the entire study cohort of KUSM graduates (Table 2).

Table 2

Residency Matched Primary Care Method, Intent to Practice Primary Care Method, and Actual Primary Care Graduates, 2003–2014 (N = 1,944).

Year	Total Graduates	Percentage of all Graduates Identified as Primary Care by Residency Match Primary Care Method	Percentage of all Graduates Identified as Primary Care by Intent to Practice Primary Care Method¥	Percentage of All Graduates Who Actually Practice Primary Care§	Percentage of Residency Match Primary Care Method Graduates Who Actually Practice Primary Care€	Percentage of Intent to Practice Primary Care Graduates Who Actually Practice Primary Care	Percentage Primary Care Status Missing
2003	164	72/164 = 43.9	38/164 = 23.2	49/164 = 29.9	49/72 = 64.8	36/38 = 94.7	4.9%
2004	152	72/152 = 47.4	40/152 = 26.3	53/152 = 37.5	53/72 = 72.2	33/40 = 82.5	4.6%
2005	167	79/167 = 47.3	46/167 = 27.5	57/167 = 34.1	57/79 = 71.6	40/46 = 87.0	6.6%
2006	172	84/172 = 48.8	48/172 = 27.9	59/172 = 34.5	59/84 = 66.7	37/48 = 77.1	3.5%
2007	160	79/160 = 49.4	35/160 = 21.9	51/160 = 32.5	51/79 = 73.8	29/35 = 82.9	3.1%
2008	155	67/155 = 43.2	29/155 = 18.7	43/155 = 27.7	43/67 = 63.2	25/29 = 86.2	3.2%
2009	152	67/152 = 44.1	26/152 = 17.1	47/152 = 30.9	47/67 = 67.1	23/26 = 88.5	5.9%
2010	163	83/163 = 50.9	30/163 = 18.4	60/163 = 37.4	60/83 = 74.1	28/30 = 93.3	4.3%
2011	160	77/160 = 48.1	43/160 = 26.9	52/160 = 33.1	52/77 = 60.4	30/43 = 69.8	1.9%
2012	162	85/162 = 52.5	41/162 = 25.3	64/162 = 39.5	64/85 = 74.1	40/41 = 97.6	1.9%
2013	155	75/155 = 48.4	29/155 = 18.7	56/155 = 36.8	56/75 = 66.2	26/29 = 89.7	1.3%
2014	182	95/182 = 52.2	38/182 = 20.9	73/182 = 40.1	73/95 = 72.6	36/38 = 94.7	1.1%
Total	1,944	935/1,944 = 48.1	443/1,944 = 22.8	664/1,944 = 34.2	664/935 = 71.0	383/443 = 86.5	3.5%

The difference between the overall proportion of graduates matched to primary care based on the Residency Match Primary Care Method vs the Intent to Practice Primary Care Method was statistically significant (χ2, p < .001).

The difference between the overall proportion of graduates matched to primary care based on the Residency Match Primary Care Method vs the graduates who actually practice primary care was statistically significant (χ2, p < .0001).

The difference between the overall proportion of graduates matched to primary care based the Intent to Practice Primary Care Method vs the graduates who actually practice primary care was statistically significant (χ2, p < .001).

Results of the Intent to Practice Primary Care Method

The new Deutchman and colleagues3 IPPCM yielded primary care practice rates ranging from 17.1% to 27.9% per year and averaged 22.8% (443 of 1,944) for the entire study cohort of KUSM graduates over 12 years (Table 2).

Results of the Actual Primary Care Method

The practice specialty-of-choice of each KUSM medical school graduate between 2003 and 2014 was identified after completion of their graduate medical education “first certificate” residency and any subspecialty training. Actually practicing primary care was defined as those physicians who met the definition “Actual Primary Care” and who did not meet the definition of “Not Primary Care” (Table 1). Six hundred and sixty-four graduates were identified who ultimately practiced primary care based on the APCM, constituting 34.2% of the entire study cohort of 1,944 KUSM graduates (Table 2). The APCM practice rates ranged from 27.7% to 40.1% per year. We could not identify the primary care status of 3.5% of the entire study cohort.

Comparison of Results from the Three Methods

Twenty-nine percent ([935 – 664] = 271 of 935) of KUSM graduates labeled as primary care by the RMPCM actually were not practicing primary care, overestimating primary care output by 271 physicians or 13.9% (271 of 1,944) of the entire study cohort. Four hundred and forty-three KUSM graduates were labeled as primary care by the Deutchman and colleagues3 IPPCM, underestimating primary care output by 221 (664 – 443 = 221) physicians or 11.4% (221 of 1,944) of the entire KUSM study cohort. As Table 2 shows, there were statistically significant differences between the overall proportion of graduates matched to primary care based on the RMPCM versus the IPPCM, the RMPCM versus the APCM, and the IPPCM versus the APCM.

Results of “First Certificate” Match Specialties of Physicians Who Actually Practice Primary Care

Table 3 shows the percentage of medical students matching into each NRMP primary care specialty who actually ended up practicing primary care at the conclusion of all of their residency and subspecialty medical education. Family medicine had the highest percentage (93.6%) of “first certificate” residents who actually practiced primary care, based on the APCM (Table 3). Table 4 shows the percentage of each specialty’s contribution to the total primary care workforce produced by KUSM over the 12-year study period. A total of 336 family medicine “first certificate” physicians actually practice primary care, constituting 50.6% of the 664-total number of primary care physicians produced by KUSM during the study period (Table 4).

Table 3

Percentage of “first certificate” residency match specialties of physicians who actually practice primary care, 2003–2014.

Match Specialtya	Proportion of Match Specialty who Actually Practice Primary Care (N = 935)
Family Medicine	93.6% (336 of 359)
Pediatrics (categorical)	75.8% (141 of 186)
Medicine-Pediatrics	57.4% (31 of 54)
Internal Medicine (categorical)	46.5% (152 of 327)
Medicine-Primary	44.4% (4 of 9)
Pediatrics-Primary	0.0%
Medicine-Family Medicine	0.0%

Number of graduates who matched in “first certificate” residency programs.

Table 4

Contribution of residency match specialties of physicians who actually practice primary care, 2003–2014.

Match Specialty	Contribution of Match Specialty to Actual Primary Care Physician Workforce (N = 664)
Family Medicine	336 (50.6%)
Internal Medicine (categorical)	152 (22.9%)
Pediatrics (categorical)	141 (21.2%)
Medicine-Pediatrics	31 (4.7%)
Medicine-Primary	4 (0.6%)
Pediatrics-Primary	0 (0.0%)
Medicine-Family Medicine	0 (0.0%)

Nearly 76% of pediatrics (categorical) “first certificate” residents actually practice primary care (Table 3), making up a total of 141 (21.2%) physicians contributing to the primary care workforce (Table 4). Almost 47% of internal medicine (categorical) “first certificate” residents actually practice primary care (Table 3), making up a total of 152 (22.9%) physicians (Table 4). Medicine-pediatrics and medicine-primary together contributed less than 6% to the total primary care workforce. Interestingly, no one from a pediatric-primary or medicine-family medicine residency program practicing primary care was identified because few, if any, graduates of KUSM matched into these two graduate medical education programs. When combined, family medicine (50.9%), internal medicine categorical (22.9%), and pediatrics categorical (21.2%) programs contributed nearly 95% of the total primary care workforce produced by KUSM.

Table 3 demonstrates that 24.2% of pediatrics (categorical) graduates, 42.6% of medicine-pediatrics, and 53.5% of internal medicine (categorial) graduates at KUSM did not practice in primary care. These drop-offs account for the inaccuracy of the RMPCM. On the other hand, not including internal medicine (categorical) programs and pediatrics (categorical) programs in the IPPCM calculation results in an underestimation of primary care output from KUSM. Furthermore, though the total numbers were small, only 44.4% of medicine-primary residents actually practice primary care (Table 3), making up only 0.6% of the primary care workforce in the sample (Table 4).

DISCUSSION

This study aimed to apply two methods of approximating the number and percentages of graduates from KUSM who actually practice primary care after graduation from medical school and after completing all of their GME. By tracking KUSM graduates and determining their specialty-of-practice using internet resources, 34.2% of KUSM graduates actually practiced primary care. The RMPCM overestimated primary care practicing physicians who graduated from KUSM by 13.9% (48.1% vs. 34.2%) while the new IPPCM underestimated the institution’s primary care output by 11.4% (22.8% vs. 34.2%). The difference between both the RMPCM and the IPPCM was statistically different from the APCM. Our conclusion was that neither method of approximation provided accurate estimates for our institution.

A total of 93.6% of KUSM graduates who matched into family medicine residency programs actually practiced primary care. The finding that family medicine comprising the greatest percentage (50.6%) of the primary care workforce was in line with findings from previous studies.1,3

Data from KUSM were not included in the Deutchman and colleagues3 original study. Comparing our institution’s workforce output to that of the 14 medical schools in the Deutchman and colleagues’ study, KUSM would rank first in the percentage of medical school graduates actually practicing primary care. In Chen’s analysis of all graduate medical education programs in the country, the University of Kansas School of Medicine-Wichita campus alone ranked sixth nationally in the production of primary care physicians.1

This study confirmed that the “double-counting” of the traditional RMPCM contributed to an overestimation of a medical school’s contribution to the primary care workforce. On the other hand, the IPPCM underestimated KUSM’s contribution to primary care. This latter finding supported results from the Deutchman and colleagues3 study, which underestimated the primary care workforce production at the majority of the 14 institutions studied. It would appear that including only internal medicine-primary and pediatric-primary residents while excluding internal medicine (categorical) and pediatrics (categorical) residents from the IPPCM was the major reason this method significantly underestimated the primary care workforce production from the KUSM.

A 2016 analysis by the U.S. Department of Health and Human Services estimated a shortage of over 5,000 primary care physicians for the Midwest region by 2025.9 The need for accurate medical school graduation data is essential to enhance the prediction of the number of physicians who will practice primary care in the future and improve outcomes of GME funding policy decisions. The GME funding policy decisions could drive the necessary increase of the primary care physician workforce to meet societal needs. Unfortunately, neither prediction method used in this study provided an accurate estimation for our institution’s contribution to the primary care workforce.

Additional steps are needed to develop new methods of accurate estimation of the future primary care physician workforce. Reliance on a single methodology may be inadequate. Perhaps an average of different methodologies will be required. Our findings showed that the RMPCM provided an overestimation of our institution’s production by 13.9%, and the IPPCM underestimated our institution’s production by 11.4%. The average of these two methodologies ([{935 + 443}/2] = 689/1,944 = 35.4%) more closely estimated the actual percentage (34.2%) of graduates from our institution practicing primary care. Alternatively, as Deutchman and colleagues3 suggested, medical schools might develop an adjusted formula for the prediction of their future contributions to the primary care workforce based on historical averages of their institutions.

A limitation of this study was that it examined data over a 12-year period from 2003 through 2014. Repeat analysis that includes more data from more recent years, as it becomes available, will better reflect the percentage and number of graduates actually practicing primary care. Another limitation of the study was the accuracy of the assigned practice specialties that physicians actually were practicing using the search methodologies employed. Additional limitations were that the career pathways of medical students who entered subspecialty residency programs and switched to primary care at some point during their GME were not followed, nor was the ultimate career pathways of residents who entered transitional or preliminary GME programs tracked, but these probably would have contributed little to overall primary care production.

Tracking the ultimate practice specialty of each graduate from a medical school or residency program is cumbersome and time intensive. An option to determine KUSM primary care production would be to use the AMA Masterfile and data from the NRMP, similar to the Chen methodology1, but the AMA Masterfile is expensive to access and has limitations of its own. Ultimately, a simple and reproducible method of predicting workforce production would help to determine how institutions are meeting the primary care needs of the country.

CONCLUSIONS

Several methods have been proposed to estimate the percentage and number of medical school graduates who eventually will practice primary care. Our study compared the new IPPCM of Deutchman and colleagues3 and the traditional RMPCM to the actual percentage of graduates who practice primary care for our institution. Neither method succeeded at closely estimating the primary workforce. More work is needed to find an accurate way to estimate primary care workforce production at KUSM.