Assessment of residency program outcomes via alumni surveys.

BACKGROUND: One trend in medical education is outcomes-oriented training. Outcomes usually refer to individuals' acquisition of competencies, for example, during training in residency programs. However, little is known about outcomes of these programs. In order to fill this gap, human resource (HR) data were analyzed and alumni of a pediatric residency program were surveyed at the Department of Pediatrics, Bern University Hospital, Switzerland.
METHODS: Residency program outcomes (demographics, career choices, part-time or full-time work status, competencies, feedback) were assessed through in-house HR databases, publicly available data on the Internet (physician directory and practice homepages), and 2 alumni surveys (S1, S2).
RESULTS: In all, 109 alumni met the inclusion criteria. Retention rate at the hospital was low (14%). Forty-six alumni (42%) in private practice were eligible for alumni surveys. Response rates were 87% (S1) and 61% (S2). Time intervals between 2 career decisions (selecting specialty of pediatrics vs selecting setting of private practice) varied widely (late-training decision to enter private practice). Mean employment level in private practice was 60% (range 20%-100%). Most valued rotation was emergency medicine; most desired competencies in future colleagues were the ability to work in a team, proficiency in pediatrics, and working economically.
CONCLUSION: A broadened view on outcomes - beyond individuals' competency acquisition - provides informative insights into a training program, can allow for informed program updates, and guide future program development.

Introduction

Medicine evolves rapidly in many areas. Training of physicians must similarly evolve. One trend in medical education is the focus on outcomes.1 Outcomes generally refer to the acquisition of competencies in individuals.1 Individuals are trained in residency programs. However, outcomes of these programs are rarely reported.

In order to fill this gap and broaden the scope from individual outcomes to program outcomes, human resource (HR) data were analyzed and alumni of a pediatric residency program were surveyed at the Bern University Hospital, Switzerland, to determine the demographics, career choices, workforce, competencies, and feedback and to guide the program in future development.

Hospital-based residency programs have to cover 3 important areas: teaching trainee doctors in a workplace setting, ensuring optimal patient care, and working economically soundly at the same time. Importantly, hospital-based residency programs also provide the supply of primary care physicians in private practices.2

In Switzerland, pediatric residency consists of 5 years of training.2 After 3 years of basic training, residents can choose between paths for hospital or ambulatory careers. Rotations through training hospitals category 3 or 4 (accredited for 3 and 4 years of residency training in pediatrics, respectively) are mandatory for both career options for a minimum time of 2 years. Board certification additionally requires passing the written as well as the oral/skills board exams.2

Residency curricula in Switzerland vary individually per resident, as switching between training sites as well as between hospital and ambulatory tracks is possible. On the one hand, the rotation system offers variable, individualized training opportunities and open career paths for residents. On the other hand, it creates a continuous challenge for residency program directors to coordinate training rotations and workforces in their hospitals. Multi-option, individualized curricula can lead to maximum freedom in choice but may also delay decisions and choices in career planning.3 Moreover, curricular fragmentation impedes outcomes data acquisition of training programs. Outcomes of residency programs are not routinely assessed.

There are 6 accredited category 4 pediatric teaching hospitals in Switzerland.4 The Department of Pediatrics at the Bern University Hospital is an accredited category 4 training hospital for pediatrics as well as an accredited training site for 10 of the 11 pediatric subspecialties.2 Accrediting bodies base standards and quality criteria for residency programs on national and global requirements as provided by the World Federation for Medical Education, and they explicitly ask programs to involve stakeholders including alumni into quality management processes.5

Resident surveys of medical societies can allow for benchmarking between different training sites, contribute to assessment of workforces and overall planning strategies in a given field of training.3,5–10 However, they cannot contribute to quality management of individual programs and their respective curriculum development. Detailed data are needed for improvements of individual programs.3,5,11 The concepts of quality management as well as a framework for curriculum development can help address this issue.11–14

Getting stakeholder feedback is 1 step in Kern’s curriculum development framework (targeted needs assessment) which acts in the interplay of, 1) problem identification and general needs assessment, 2) targeted needs assessment, 3) setting goals and objectives, 4) developing educational strategies, 5) implementation of changes, and 6) evaluation and feedback.13 Feedback loops are vital for assessing and assuring quality as well as for continuous quality improvement.12

Assessment of outcomes through residency program alumni surveys could add value to the program not only in terms of demographics but also by, 1) determining career paths (private practice, hospital, and university hospital) and time points when alumni made career choices, 2) predicting future workforce levels,15 3) identifying gaps between residency training and needs in work-life after graduation from the program,11 4) setting goals and objectives for program development, 5) providing basic data for future comparison,13 and 6) possibly fostering connections between graduates and educators in the hospital to enhance knowledge exchange, continuing medical education, and offer recruitment opportunities.

For this study on program outcomes, analyses of HR data were complemented with surveys of alumni working in private practice.

Methods

The institution is an accredited category 4 training hospital in the Swiss capital Bern, a city of 140,500 inhabitants (2015), serving a population of ~250,000 individuals <19 years of age from primary to tertiary care. It also trains physicians in 10 of 11 pediatric subspecialties.

Outcomes assessment of the pediatric residency program (number of trainees and graduates, their part-time or full-time work status, career choices, career decision time points, needs and perceived gaps between their training and their current work, competency acquisition, and feedback) was done through the analysis of HR databases, information from the Internet, and alumni surveys. Inclusion criteria for alumni surveyed in this study were defined as completion of ≥18 months full-time training in pediatrics at the institution and having left the pediatric residency program (completion of residency or end of rotation). One year of part-time employment in job sharing (ie, 0.5 full-time equivalents [FTE]) was counted as 6 months. Inclusion criteria had to be met within the 10-year period from January 1, 2003 to December 31, 2012.

Demographic data of the 109 included individuals were extracted from the HR databases. Work place and e-mail addresses were retrieved from publicly available data on the Internet (www.doctorfmh.ch; private practice homepages) and through personal communication.

Alumni in private practice (n=46) were assessed with 2 independent surveys (S1 and S2). The surveys contained different question types: single choice, multiple choice, 5-point Likert scales,16 and open questions. Both surveys were created by our team, tested for comprehensibility, and piloted in a voluntary test cohort. The pilot feedback was incorporated into the surveys before they were e-mailed to the identified individuals. Both surveys were sent as pdf documents with check boxes and free text fields.

Each questionnaire (S1 and S2) was sent twice within 1 month time and a personalized reminder was sent to non-respondents 6 weeks after the initial e-mail. Survey 1 was sent in February 2015. Workforce data in S1 were asked as of March 31, 2015. Survey 2 was sent in February 2016 and included responses received by April 15, 2016. Participation in the surveys and completion of the questionnaires were voluntary. No financial or other benefits were provided. Participants were informed with the letters attached to S1 and S2 regarding the analyses and planned publications of the gathered anonymized data. Voluntary completion and return of the surveys was deemed as consent. Both surveys could be opened from the e-mail attachment, filled in electronically, and e-mailed back as pdf attachments. Printing the e-mailed surveys and answering the questions by handwriting before sending them back by surface mail or as scanned documents through e-mail was according to the respondents’ preference.

Survey 1 collected data on current employment in private practice, whether alumni were working in single or group practices, number of weekly working hours, and additional patient care duties in hospitals and/or outpatient emergency on-call services. With the complementing survey S2, information was gathered about duration of training at the institution, time points of career choice for pediatrics and private practice, about work experiences in private practice prior to establishing a career in private practice, learning effects of different training rotations at the hospital, whether respondents would have welcomed a teaching outpatient practice operated by the hospital, and where they think future colleagues should acquire important competencies for private practice (wound care, ultrasound, laboratory medicine, medical law, communication skills, nutrition, team leading, medical economics, use of information technology, vaccinations, counseling adolescents, complementary medicine, developmental testing, time and practice management, billing,17 health promotion, and parental counseling).

Qualitative data acquisition in S2 was done through open-ended free text questions, 1) about which competencies an ideal new colleague the responding alumni would employ in their practice should have, 2) in which areas young colleagues in training show deficits when doing training rotations through their pediatric private practices, and 3) about perceived special trends for the future of outpatient pediatrics.

Returned S1 and S2 were printed, and the responses were filled into a password-secured Excel file by the authors. For analytic purposes and depiction in graphs, categorical data were converted into numerical data. Ratings of learning effects on a 5-point Likert scale in the questionnaire were converted to 1 (no), 2 (little), 3 (medium), 4 (high), and 5 (very high) effect. Mean and standard deviation (SD) were calculated (Figure 1). The 3-item response choices in the question about preferred competency learning sites were assigned a score of 1 (hospital), 2 (hospital and practice), or 3 (practice) (Figure 2). Data analyses included calculating frequencies and inferential statistics to determine tendencies (mean, SD, and p-values). A p-value of ≤0.05 was considered significant. Analyses were carried out question by question. All answered questions were included into analyses.

Figure 1

The reported educational effectiveness of residency training rotations (depicted as ranking based on respondent ratings) shows that most respondents rated emergency medicine as highly effective for private practice, whereas intensive care unit training was rated with a medium learning effect for private practice.

Notes: The vertical y-axis shows the rotation name as well as the number of respondents who completed and rated the rotation to the total number of respondents (n=28). The horizontal x-axis shows the 5-point Likert scale of: −− no learning effect (1), − little learning effect (2), +/− medium learning effect (3), + high learning effect (4), ++ very high learning effect (5). The bar graph markers represent mean scores; the bar graph range lines depict standard deviations.

Figure 2

Alumni reported that they would have preferred to learn competencies required for private practice pediatrics in different settings, depending on the subject.

Notes: Wound care, for example, would be preferentially learned in a hospital setting, whereas practice management would be best learned in a practice setting. The 3-item response choices were assigned scores of 1 (hospital), 2 (hospital and practice), or 3 (practice) for analytic purposes. The dots represent the mean of the combined responses of each competency.

Ethics approval was requested from the cantonal ethics committee office and judged not applicable for the surveys and data analyses according to the Swiss Human Research Act. The State Ethics Committee declared that this type of survey was outside the scope of its responsibility. The hospital’s HR and legal services allowed the use of these data provided they were used anonymously in public reports. The research was conducted in accordance with the Helsinki Declaration revised in 2013. Publicly available statistics software (http://vassarstats.net), Microsoft Word and Excel, Adobe Acrobat and Reader, as well as GraphPad Prism were used for documentation, presentation, and statistical analyses.

Results

Demographics

The number of individuals who met the inclusion criteria was 109. Of the 109 residents, only 15 (14%) were still working at the institution (73% female). Four alumni had left the field of pediatrics and 4 were lost to follow-up. The majority of alumni 98/109 (90%) was board certified in pediatrics (Foederatio Medicorum Helveticorum; FMH). Mean time from graduation from medical school to board certification in pediatrics was 8.1 years (8.6 years for alumni in private practice and 7.7 years for alumni working in hospitals) (Table 1). Response rates to the surveys sent to alumni in private practice (n=46) were 87% (S1) and 61% (S2).

Table 1

Summary of respondent data and demographics (S1, www.doctorfmh.ch, individual practice websites, and hospital human resources database)

Subgroup denominators				Number of alumni
Met inclusion criteria				109
Were board certified in pediatrics				98
With publicly available professional background data				95
Worked in hospital				55
Worked in private practice (eligible for S1 and S2)				46
Responded to S1				40
Responded to S2				28
Career choices (109 alumni)
	Individuals	Female alumni				% of all alumni
Private practice	46	35 (76%*)				42
Hospital	55	30 (55%*)				50
Lost to follow-up or left the field of pediatrics 8**						7
Professional background data (available from 95 alumni)
Time from graduation from medical school to board certification in pediatrics (years)		Mean 8.1 (SD 2.4; median 7)
		Gender					Female, mean 8.0; male, mean 8.2 (p=0.704)
		Career				Private practice, mean 8.5; hospital, mean 7.7 (p=0.089)
		Women/career				Private practice, mean 8.6; hospital, mean 7.3 (p=0.035)
Career choice and board certification among hospital pediatricians*** (55 alumni)
			Individuals		Female alumni			% of all alumni
Department of Pediatrics, Bern University Hospital			15		11 (73%)			14
Other university hospital in Switzerland			15		7 (47%)			14
Nonuniversity hospital in Switzerland			18		9 (50%)			15
Hospital abroad			7		3 (43%)			7
			Individuals		% of hospital pediatricians
Board certification in pediatrics			50		91
+1 Pediatric subspecialty			24		44
+2 Pediatric subspecialties			1		2
+ Another main specialty			7		13
Career choice and board certification among private practice pediatricians (46 alumni)
			Individuals			% of respondents
Single practice			4			10
Group practice			36			90
Providing additional hospital services			22			55
Providing urban/regional night calls			30			75
Employment
Independent practice (co)-owner						21 (66% female)
Employed by practice owner						19 (79% female)
Employment levels based on a 50-h week (mean; median)						60%; 55%
Employment levels, independent vs employed (mean)						67% vs 52% (p=0.031)
Working time per week (mean; median)						31 h; 28 h
Working time per week, independent vs employed (mean)						35 vs 26 h (p=0.040)
Workforce (FTE) generated by 40 physicians						23.9
			Individuals			% of practice pediatricians
Board certification in pediatrics			46			100
Plus 1 pediatric subspecialty			0			0
Plus another main specialty			1			2

Notes:

More women among private practice pediatricians (p=0.04, Chi-square with Yates correction);

4 alumni left the specialty of pediatrics, 4 lost to follow-up;

Data from www.doctorfmh.ch as per March 31, 2015.

Abbreviations: SD, standard deviation; FTE, full-time equivalents.

Workforce levels

Self-declared employment levels in private practice varied between 20% and 100% calculated as a 50-h week equaling 100% FTE. Mean employment level was 60%. Ninety percent of alumni were contracted in a group practice; only 10% (4/40) were working in a single practice. Data on career choices and employment levels are summarized in Table 1. Practices were geographically located in the Canton of Bern in 60% of cases, whereas 40% of alumni were working in practices outside the Canton of Bern. None of the respondents had work experiences abroad. Alumni who had left the program to work in private practice had a self-declared cumulative employment time, that is, training time of 110 years at the institution. This equals 110 cumulative years of training provided, which partly mirrors the program’s investment in the alumni; 81 years (74%) were declared by women and 29 years by men (26%), resulting in a mean cumulative employment time at the institution of 4 years in female and 3.6 years in male respondents. Twelve of the 28 S2 respondents reported job sharing during residency (45% of all female and 38% of all male residents). More than half of the alumni (55%) had worked in private practice during residency (65% of female and 25% of male residents).

Career choices

Forty-six alumni (42%) were working as pediatricians in private practice and were assessed through S1 and S2 (76% female). Women were equally represented in the 2 groups of respondents (S1 and S2) with 73% each. Respondent board certification rate in pediatrics (FMH) was 100%. Major career decision time points were the time at which the decision to become a pediatrician and the time at which the decision to get into private practice were taken. Intervals between these 2 career decision time points varied widely among alumni (ranges from elementary school to residency and from medical school before first contact with pediatrics to after completion of training in pediatrics) (Figure 3).

Figure 3

The time interval between 2 career decisions (selecting specialty of pediatrics vs selecting setting of private practice) per respondent (x-axis: − = before; + = after) is notable for the late-training decision to enter private practice.

Competencies

Reported educational effectiveness of different in-hospital residency training rotations is summarized in Figure 1. Respondents who had completed a rotation in emergency medicine rated this rotation as very effective for their current work in private practice. In-house consultant service rotations had been completed by a third of respondents and were rated as having a high educational effect as well. Intensive care unit rotations had been completed by 16 of 28 respondents and were rated as medium effective.

When the respondents were asked where (based on their experiences) future colleagues could best learn different competencies, they allocated perceived best learning environments (hospital, practice, or both) for future residents on private practice career tracks (Figure 2). The allocation of perceived best training sites varied from wound care, ultrasound, and medical law in hospitals to the use of information technology, vaccinations, and counseling adolescents in both hospital and practice to parental counseling and practice management in private practice. During residency, an outpatient training rotation in a practice associated with the teaching hospital would have been helpful, according to the majority of alumni in private practice (71%).

In S2, alumni in private practice were questioned about competencies they would wish to see in optimal future colleagues whom they would employ in their practice. From these free text questions, the 3 equally important top scored competencies were teamwork/ability to work in a team, professional expertise/proficiency in pediatrics, and economic efficiency/working efficiently. Communication skills and social skills scored equally as the second most desired competencies, followed by empathy, flexibility, and autonomy.

Alumni did not mention teamwork as a lacking skill in residents who engage to private practice during training. However, lack of different professional skills and competencies was mentioned in the areas of primary care pediatrics: nutrition, physiological development and development testing of children, prevention medicine, sleeping disorders, managing problems at school and during adolescence, as well as difficulties in differentiating the importance of one clinical problem over another. Often, time management and decision making were also scored as missing in residents trained in the hospital setting.

Alumni identified the following trends and increased needs in future training for private practice pediatrics: working in group practices versus single practices, caring for children with special health care needs, that is, in attention deficit hyperactivity syndrome (ADHS) and obesity, and caring for adolescent patients and patients with migration background. Ultrasound diagnostics was identified as an additional trend.

Discussion

The present study sought to assess outcomes of the pediatric residency program through alumni surveys and analysis of HR databases (demographics, career choices, workforce levels, competencies, and feedback) in order to complement the existing data and carry out a needs assessment for program development.

Demographic outcomes did not change over the 10-year period of analysis. No change was observed in gender distribution among the alumni. The residency training positions did include 2 job-sharing positions. This proportion might change in the future, if residents increasingly demand part-time work or job sharing.3 Since there is no central resident allocation system and no predefined number of residents who can go into a certain medical specialty in Switzerland, the stable ratio of 2/3 female and 1/3 male residents could reflect the overall gender distribution in this field, but it might also be influenced by the selection process of residents at the institution. Overall, there is a slight trend within the last years with an increased number of female and a decreased number of male physicians being board certified in pediatrics.6 The group of private practice pediatricians and the group of tertiary care hospitalists had a slightly higher percentage of female individuals than the group of pediatricians working in non-tertiary care hospitals. However, numbers of these subgroups were small and cannot necessarily be transferred to other cohorts.

Employment levels of the alumni were mostly <100%. Average employment levels in private practice were 60%, favored by a trend toward working in group practices where part-time working is possible. Only 1 in 10 physicians was working in a single practice at a 100% employment level. These results reflect the tendency described in the literature.15 Outcomes-wise this means that alumni would have to be considered as part-time workers. This has to be taken into account while calculating demands and supply of future workforce. Residency programs might have to increase training capacities, not only with a possible increase in demands of job sharing or part-time work during residency in the future, but also if the same or an increased need of workforce of pediatricians is predicted for the future.15 Moreover, interdependency of residents in job sharing would increase and require advanced planning and coordination of trainees in the program as well as advanced communication and hand-over skills in order to secure optimal patient care. This also means that possibly the number of staff who train the residents would also have to be increased to efficiently handle increased numbers of residents while maintaining and optimizing quality of teaching and patient care in an economically sound manner. Also, training modalities, teaching materials, and didactics would have to be adapted to increasing numbers of part-time residents.

Training of primary care pediatricians is as important as assuring and supporting a lively scientific community among residents and enhancing training in tertiary care. However, choices between one or the other career path (primary care pediatrics in private practice vs tertiary care pediatrics in a university hospital) are highly individual. In the cohort of this study, only 14% of alumni still worked in the hospital of this study. The extremely low retention rate of graduates from the residency program reflected the bottleneck in to academic careers and tertiary care hospital services beyond residency.3 More stringent residency rotation planning as well as early contacts with the field of pediatrics and early mentoring could be possible steps in optimizing outcomes by choosing and following one’s appropriate career in pediatrics.3,9,18

The duty of tertiary care university hospitals is to assure the training of skilled specialists and foster academic careers in the long run. Future analyses will show how tertiary care training centers can balance training for private practice physicians, hospitalists, and tertiary care specialists.3,15

The relatively late time points of career decisions in the cohort have to be looked at and followed in the future. The motivators of pursuing a career in ambulatory health care pediatrics might be influenced by different factors known from the literature, such as workload, flexibility, and family friendliness.3 Future analyses of career choices might not only look at working hours, family, or academic interests but also consider structures such as access to childcare, work satisfaction, new technologies, rewards in work, and sufficient payment.3,15 None of the trainees who went into private practice did a rotation abroad, which might reflect a less scientific focus and higher local attachment in that cohort, known from the literature.3

Residency programs have to be seen in the continuum of medical education. As medical school curricula evolve, residency programs have to make sure that they continue with emerging trends. For example, in recent years medical students at the University of Bern are trained in communication skills19 and exposed to primary care in longitudinal clerkships in order to raise interest in primary care specialties such as pediatrics in early stages of their careers.18 Since these programs started in late 2007, this study did not have alumni in the cohort who completed this training. Future residents will be equipped with different subsets of skills when entering a residency program.18,19 Residency programs will have to take into account these skills and optimize them with up-to-date training. The continuum of medical training from undergraduate to postgraduate to continuing medical education will benefit if the phases are closely linked and well informed about needs, trends, and changes made. Effects of both measures, primary care clerkships and communication skills trainings, will show many years later in time, reflecting the need for long-term perspectives and outcomes orientation of training programs in medical education.

The competencies for which adaptions are planned in the residency program based on this study will be part-time work options, a special track for ultrasound training, and communication training. A portfolio will also be assembled for the residents on private practice career tracks. Another measure will be cooperation enhancement with teaching practices outside the hospital in order to provide training positions in pediatric practices closely linked to the hospital. Moreover, a teaching practice will be included in the hospital group within the next years. Curricular-wise, plans are to increase exposure of residents on private practice career paths to training in medical areas such as ADHS, obesity, migrant health, and adolescent care. With these measures, it is believed that the residency program’s outcomes will be optimized for the best of the population cared for and served: children, adolescents, and their families.

To our knowledge, this is the only study analyzing the outcomes of a pediatric residency program in Switzerland, assessed through alumni surveys. By making individual residency program outcomes data publicly available, valuable information is added to physician statistics and for informed discussions, for example, on workforce planning and the topic of shortage of pediatricians in private practice.15

There are some limitations to this study. Including participants of a single university hospital might decrease generalizability, and the data provided might not be transferable to other residency programs. However, for several reasons, the chosen approach seems appropriate. First, the study contributes data of 1 of the 5 major pediatric residency training sites at university hospitals in Switzerland. Second, data on postgraduate training program outcomes in general are scarce, and thus, data of a single institution can serve public health interest, for example, in terms of numbers needed as a basis for calculation of future workforce. Third, content and feedback data provided here can serve as examples to other residency programs and their curriculum development. The study could not further assess currently unemployed alumni, but the 7% of alumni who had left the field of pediatrics or were lost to follow-up are unlikely to potentially bias the survey data, assuming that they did not pursue a career in private practice at the time of the surveys and thus would not have been eligible for S1 and S2. Response rates of 87% (S1) and 61% (S2) are likely to provide a good spectrum of information. Addressing exclusively alumni already working in private practice very likely underestimates the true output of the residency program for ambulatory pediatrics, as the chosen time period for inclusion ended only 15 months before data acquisition in S1. It is likely that more colleagues, who at this time point of S1 were still working in a hospital setting, will join the ambulatory sector within the upcoming years.

Of course, the data provided are not applicable or transferable to every setting. However, parts of this work could be transferred to other programs as new trends will emerge in various training hospitals as well. Continuous and repeated alumni surveys could be of benefit to detect timely emerging needs, problems, and trends in residency program outcomes of all medical specialties and allow for adaption, be it in the number of training positions, training modalities, or trainers who might get additional qualifications and benefit from modern teaching strategies.

Conclusion

Outcomes-based medical education can be expanded from the focus on individual outcomes to a broader view with a perspective on program outcomes. Residency program outcomes can be assessed through alumni surveys. Besides demographic data, information on career choices, and workforce, acquired competencies and alumni feedback can be used as needs assessment to guide informed program development. Residency program outcomes can help reflect the impact of the training program for the health care sector, be it in ambulatory or in hospital medicine. Medical training and population health care are closely linked and should be informed about the outcomes and needs of each other. Close interaction in these sectors is crucial to fulfill health care needs in the future. Thus, expanding the concept of outcomes in medical education from micro, that is, individual levels, to meso (program) levels, could help to further elucidate the impact and benefit for the population (macro level).