Do changing medical admissions practices in the UK impact on who is admitted? An interrupted time series analysis.

INTRODUCTION: Medical admissions must balance two potentially competing missions: to select those who will be successful medical students and clinicians and to increase the diversity of the medical school population and workforce. Many countries address this dilemma by reducing the heavy reliance on prior educational attainment, complementing this with other selection tools. However, evidence to what extent this shift in practice has actually widened access is conflicting. AIM: To examine if changes in medical school selection processes significantly impact on the composition of the student population. DESIGN AND
SETTING: Observational study of medical students from 18 UK 5-year medical programmes who took the UK Clinical Aptitude Test from 2007 to 2014; detailed analysis on four schools. PRIMARY OUTCOME: Proportion of admissions to medical school for four target groups (lower socioeconomic classes, non-selective schooling, non-white and male). DATA ANALYSIS: Interrupted time-series framework with segmented regression was used to identify the impact of changes in selection practices in relation to invitation to interview to medical school. Four case study medical schools were used looking at admissions within for the four target groups.
RESULTS: There were no obvious changes in the overall proportion of admissions from each target group over the 8-year period, averaging at 3.3% lower socioeconomic group, 51.5% non-selective school, 30.5% non-white and 43.8% male. Each case study school changed their selection practice in decision making for invite to interview during 2007-2014. Yet, this within-school variation made little difference locally, and changes in admission practices did not lead to any discernible change in the demography of those accepted into medical school.
CONCLUSION: Although our case schools changed their selection procedures, these changes did not lead to any observable differences in their student populations. Increasing the diversity of medical students, and hence the medical profession, may require different, perhaps more radical, approaches to selection. © Author(s) (or their employer(s)) 2018. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

We were able to assess if changes in selection processes for medical school resulted in increased diversity of the student population—a focus of the UK widening access strategy.

A strength was that over 24 000 admissions were considered across 18 UK medical schools, with linkage to admissions test scores. The use of the case study approach was advantageous as it allowed a more nuanced account of the process of medical selection than is available from aggregated data.

A limitation was that we were not able to consider any underlying secular trends that may have impacted on admissions.

Introduction

Selection into medicine is a complex process with multiple, potentially competing priorities. Medical schools want to select applicants who will be successful both in the short term, as medical students, and in the longer term, as practising clinicians.

However, in many countries, medical schools are also under strong political pressure to increase the matriculation of certain under-represented groups.1–5 The rationale for this is twofold. First, to address societal issues of social justice and social mobility in terms of encouraging people from all backgrounds into higher education rather than birth dictating one’s social and economic outcomes in life.6–8 Second, training a diverse healthcare workforce is considered essential to improving healthcare quality by ensuring doctors are as representative as possible of the society they serve (in order to provide the best possible care).9–11

There is clear evidence that significant under-representation of some social, cultural and ethnic groups in medical schools and medicine worldwide persists despite a variety of national initiatives (eg, quota systems and political imperatives) and local activities (eg, pipeline programmes) to ameliorate such under-representation.12–14

Moreover, these goals of predictive validity and increasing the diversity of the medical school population are potentially conflicting because prior academic attainment, which until relatively recently has been the main selection ‘tool’ for medical education, is strongly influenced by factors associated with demographic disadvantage, such as ethnicity and/or socioeconomic class.8 15–18 In other words, certain groups face inequalities in preuniversity education that then significantly limit their chances of obtaining the necessary grades/grade point average to be eligible for medical school. The precise groups that are educationally disadvantaged vary by country. In the UK, disadvantage related to socioeconomic background, status or ‘class’ is the main issue,19–21 whereas ethnicity/race is the foremost issue in other countries.12–14

Medical schools have tried to address this dilemma by redesigning their selection processes. In the UK context, most medical schools have shifted from relying solely on prior academic attainment as an indicator of capability to using combinations of different tools designed to assess a range of other cognitive and personal attributes.22–25 For example, all UK medical schools now include an admissions test as part of selection (eg, UK Clinical Aptitude Test (UKCAT)), and schools are also expected to make use of interviews in the selection process as well (see later).

Medical schools have chosen to increasingly use evidenced-based selection tools such as multiple mini interviews (MMIs; a format of many short independent assessments, typically in a timed circuit) rather than traditional interviews and decreased their use of personal statements in attempts to become fairer, more objective and transparent in their selection methods.26

Such a broader approach to selection seems, on face value, to address the dilemma of balancing predictive validity and widening access. However, it is critical to know if selection strategies, tools and processes are actually effective in terms of helping medical schools achieve the aim of increasing diversity.

Measuring this is not straightforward. While there is much research examining whether the tools used for selection into medical school measure what they claim to measure, and do so consistently,7 26 27 very little is known on whether selection practices support increasing diversity/widening access to medicine. What evidence is available is conflicting. For example, Tiffin et al 28 found that certain ways of using the UKCAT (an admissions test) were associated with a higher proportion of students from under-represented groups being admitted to UK medical schools. However, in a more recent longitudinal study, Mathers and colleagues29 failed to identify any consistent effect of different usages of the UKCAT on equity in selection processes. In another context (Denmark), O’Neill et al 30 showed that selection strategy (grade based or attribute based) had no effect on the social diversity of their medical student population.

There is an additional issue. As mentioned above, medical schools in the UK and many other countries use a combination of tools (such as prior academic attainment, MMIs, an aptitude test, references and personal statements).26 However, selection research has typically focused on the qualities of one particular tool or method in its own right, rather than whether various tools can be combined effectively. Where studies have looked at combining tools, the focus has typically been on examining the psychometric properties of doing so (ie, incremental validity).26 31 32 The few studies that have considered the impact of combining tools suggest that different weighting (eg, 50% for prior academic attainment, 30% aptitude test and 20% local assessment; or a hurdle model of ‘if over x, then through to the next stage’) may lead to different outcomes in terms of who is selected.28 33 However, yet again, the literature is conflicting, with certain combinations putting some groups at less of a disadvantage but biases remaining towards other groups.34 35

In short, while UK medical schools now typically use a combination of selection tools to discriminate between applicants, we do not know if this more systems-based approach to selection supports increasing diversity/widening access. Building on our previous work,7 26 36–38 we wished to examine if changes in medical school selection criteria or processes impact on the demographic composition of the student population. Specifically, we wanted to examine if a change in selection processes at medical school level impacts on the proportion of students admitted from certain groups/target populations for widening access to medicine initiatives in the UK context.21 39

Methods

This was a quantitative study using an interrupted time-series framework using segmented regression to examine if changes in medical school selection criteria during 2007 to 2014 impacted on the proportion of admissions in four target groups.

Data source

The most commonly used admissions test is the UKCAT, introduced in 2006 to help medical schools increase the diversity of medical students (www.ukcat.ac.uk). UKCAT provided data on student admissions to undergraduate programmes at 18 UK medical schools from 2007 to 2014, covering around 24 000 admissions. The data were accessed within the Health Informatics Centre Safe Haven (HIC), run through the University of Dundee, to ensure adherence to the highest standards of security, governance and confidentiality when storing, handling and analysing identifiable data. Received data were anonymised. The following datasets were provided and merged together to form a working data file. The most recent UKCAT score was used, and duplicates were removed.

Admissions: anonymised student ID, university, course and academic year.

Demographics collected by UKCAT: anonymised student ID, gender, ethnic group, year of birth, year of UKCAT test, school type (defined according to funding criteria, whether state funded or privately funded: see later), highest qualification (indicated by academic score or tariff (the weighting applied to academic results in the admissions process)), socioeconomic group based on parental occupation (derived from National Statistics Socioeconomic Classification: see later).

UKCAT test scores: anonymised student ID, year of test and scores for the five UKCAT subtests (verbal reasoning, decision making, quantitative reasoning, abstract reasoning and situational judgement test (SJT) and UKCAT total score. Candidates receive a scale score (300–900) for each of the first four subtests and a banding (1–4) for the SJT. Note that schools using the test in selection will have mainly relied on an aggregated ‘total score’ on the UKCAT, which typically ranges from 1200 to 3600.

Demographics

In line with UK widening access policies,21 39 we were interested in admission of applicants who were from lower socioeconomic status (SES) groups, who had attended non-selective secondary schooling, were non-white and/or were male. Each of these is explained below.

SES was determined by the widely used parental National Statistics Socio-Economic Classification (NS-SEC)40 where categories 4 and 5 of a 1–5 scale represent lower socioeconomic groups (group 1: managerial and professional occupations, group 2: intermediate occupations, group 3: small employers and own account workers, group 4: lower supervisory and technical occupations, and group 5: semiroutine and routine occupations). The proportion of the UK population in the five categories in 2011 was NS-SEC1: 41.4%, NS-SEC2: 12.7%, NS-SEC3: 9.4%, NS-SEC4: 6.9% and NS-SEC5: 25.2%.41

Non-selective secondary school is typically a state school (no entrance exam, not fee paying and based on residential catchment), with the comparison being independent, typically fee-paying schools, often with an entrance exam, which are attended by about 7% of UK school pupils overall.42 There is an overlap of some schools, but this categorisation was deemed appropriate for analysis given our knowledge of how this has been approached in UK studies from the wider field of education.

The dichotomy of white and non-white is typically used in UK studies looking at selection into medical school of UK students.31 38 43 44 Non-white is a broad categorisation that is likely to mean different things in different contexts. In the UK context, non-white participants are typically Indian or Asian, with very small numbers of black/Afro-Caribbean participants represented in this group. There is much research indicating that ‘non-white’ applicants, and medical students are disadvantaged in terms of performance on selection and at medical school.31 38 43–45

Finally, we also looked at male gender as females surpass males in high school examination performance in many countries including the UK.46 47 Whether related to this or not (the pattern of performance is different at the extremes),48 in the UK, the proportion of male medical students is significantly lower than female students. This is of relevance to this paper as there is much debate about the impact of this pattern on future healthcare delivery.49

Admissions processes

During the time period examined, most medical schools in the UK used a combination of prior academic attainment (eg, A Levels), a cognitive or aptitude test (eg, UKCAT), the personal statement and an interview for selection. Prior academic attainment and UKCAT scores were used by medical schools in one of two main ways: as a factor percentage in a decision to interview, offer a place or both and as a threshold score to select for interview or to make an offer, with a score typically between 1900 and 2800 used.28 50 An assessment of the personal statement and/or reference could also be used in this process as part of the factor weighting, but the use of the personal statement as part of the selection process decreased between 2007 and 2014 (RG, personal data). Information on selection criteria used as obtained from RG who receives this information on an annual basis as part of her employment at the UKCAT consortium (the information is not published as such).

We were specifically interested in whether the introduction of different usages or increases/decreases in the factor weightings or increases/decreases in the threshold score for invitation for interview led to changes in the proportion of admissions in the four target demographic groups.51 We anticipated that increased use of the UKCAT, shifting from traditional to MMIs and decreasing the use of traditional interviews and personal statements, would potentially increase the diversity of applicants invited to interview given the evidence base for the ‘fairness’ of each of these selection tools.25 35 We also anticipated that a ‘stronger’ use of the UKCAT score (as a factor or threshold) would be associated with increased odds of selecting entrants who were male, from a low socioeconomic background or a state school.28

Selection criteria

Previous work by RG provided information on the selection policies of each of the 18 schools included in this analysis and any changes from year to year (eg, change in factor weightings, change in threshold score of UKCAT, change in prior academic attainment, for example, requirement of AAA at A Level instead of AAB and introducing interviews). The information presented here is the selection criteria for use to decide whether to invite a candidate to interview or not, the last hurdle in the selection process that would ultimately lead to an applicant being admitted to medical school or not.

Analysis

Within the data safehaven (HIC), data were merged using STATA (V.14) after appropriate recoding. For each year of admission, total number of admissions for the 18 schools was obtained, and the proportion of those of low SES, attending a non-selective secondary school, non-white ethnicity and male was calculated across the whole sample. Due to the confidential nature of the data from schools, we have not presented this information for all schools separately, as it would potentially give away the location of the schools. The aim was to assess if the change in selection procedure impacted on the proportion of admissions. Again to maintain anonymity of the schools, we were unable to do this for every school. Thus, we selected four schools to act as individual case studies. These four schools were chosen to be representative of the 18 medical schools in terms of diverse geographical locations; student intake (ranging from 120 to 230 per year group), curricula (eg, case based or traditional learning) and age of school (from hundreds of years old, to one of the newest UK medical schools). In addition, the four chosen schools had known changes in policy that we could look at and sufficient data before and after the change to allow analyses. As the changes occurred at different times for different schools, we needed to analyse individually. For each case study, we looked at changes in selection criteria during 2007–2014 and whether any changes impacted on the proportion of admissions in the four target groups. This was undertaken within an interrupted time-series framework using segmented regression. The interruption was the year of admission in which the change had occurred (ie, selection year was one prior). In some cases, minor changes had occurred, but the interruption chosen was the biggest amendment to selection policy. It was important the change being tested was prespecified. Models were fitted with an interruption (level) effects, trend preintervention (slope) and trend postinterruption (slope) and a constant term. These coefficients represent the proportion at the start of the period, the slope in the preinterruption phase (pretrend), the change in level caused by the interruption and the slope of the postinterruption phase (post-trend). In addition to these effects, it was of interest to calculate the absolute and relative effects (rate of change) of the interruption. Statistical software R was used to calculate estimates of relative change (following the interruption). SEs for these estimates were generated using the method specified by Zhang and Wagner.52

Patient and public involvement

Patients and public were not involved in this research.

Results

Overall admissions

In total, across the 18 medical schools, there were 24 346 recorded admissions between 2007 and 2014 for which data were available. Table 1 shows the number of admissions in each year across the 18 medical schools and the proportion of admissions by the four target groups (lower socioeconomic group, non-selective secondary schooling, non-white and male) along with the number of schools that provided data for that year.

Table 1

Proportion of admissions for all institutions by year

Admission Year	Schools N	Total admissions	Low SES*		Non-selective school		Non-white		Male
			%	Missing n	%	Missing n	%	Missing n	%
2007	16	3181	0.84	680	53.13	606	32.61	47	41.91
2008	17	3994	3.40	526	52.17	580	27.82	55	42.44
2009	18	4186	3.50	555	50.16	685	30.05	50	43.45
2010	18	4141	3.88	382	49.97	623	28.56	45	44.24
2011	13	2923	3.41	193	51.25	556	26.04	262	44.41
2012	11	2623	2.99	181	50.37	465	29.45	256	44.22
2013	7	1729	4.04	121	52.43	327	29.78	198	45.17
2014	7	1569	4.37	104	53.48	406	34.78	186	47.23
Total	18	24 346	3.27	2742	51.3	4248	29.5	1099	43.78

*Categories 4 and 5 of the 1–5 scale.

Not all schools provided data at each year so there is some year to year variation in the number of total admissions. Percentages presented are the percentage of admissions for known values of the characteristics (ie, excluding missing data). Figure 1 displays the proportion of admissions by year and shows that, when the data from all 18 medical schools were combined, there were no obvious changes in the proportion of admissions from each of the target groups over the 8-year period of study.

Figure 1

Proportion of admissions in each target group from 2007 to 2014.

Case study A

Figure 2 shows the proportion of admissions by target group across each year for case study A. Case study A used UKCAT as a weighted factor along with prior academic attainment and personal statements in the decision to invite for interview. In 2012, a change was made to academic attainment (increase in As at A level) and the weighting for the UKCAT increased from 7% to 14%. Year 2013 saw the introduction of MMIs and 2014 the removal of personal statements from the decision to interview and, as a result, a much larger weighting placed on both the academic attainment and the UKCAT. A segmented regression using 2012 as the interruption year was carried out. Later changes could not be investigated as there were insufficient postinterruption time points (table 2). There were no obvious interruption effects, and the trends before and after the interruption were not statistically significant. Relative change estimates were not statistically significant for proportion low SES, proportion non-selective schooling and proportion non-white. The relative change for proportion male was −0.33 (95% CI −0.73 to −0.07) indicating the proportion of males per year decreased by this amount compared with what it would have been if the selection policy had not changed.

Figure 2

Proportion of admissions: case study A. SES, socioeconomic status; UKCAT, UK Clinical Aptitude Test.

Table 2

Model coefficients from the segmented regression and estimates of relative change following the interruption for each case study and demographic criteria

	Term	Coefficient	95% CI	P values	R2
Case study A
Low SES	Constant	1.61	–	–	9%
	Pretrend	0.15	(−1.48 to 1.80)	0.804
	Interruption	−1.13	(−10.0 to 7.75)	0.766
	Post-trend	0.46	(−3.56 to 4.48)	0.766
	Relative change	0.57	(−4.41 to 5.55)	–
Non-selective	Constant	64.4	–	–	70%
	Pretrend	−1.25	(−4.26 to 1.76)	0.313
	Interruption	−12.5	(−28.8 to 3.80)	0.1
	Post-trend	6.41	(−0.97 to 13.8)	0.073
	Relative change	0.48	(−0.02 to 0.97)
Non-white	Constant	9.65	–	–	70%
	Pretrend	1.88	(−0.16 to 3.94)	0.063
	Interruption	1.9	(−9.20 to 13.0)	0.66
	Post-trend	−4.17	(−9.19 to 8.85)	0.083
	Relative change	−0.93	(−9.49 to 7.62)	–
Male	Constant	35.6	–	–	72%
	Pretrend	1.64	(−0.17 to 3.44)	0.066
	Interruption	2.24	(−7.54 to 12.0)	0.56
	Post-trend	−3.05	(−7.47 to 1.37)	0.128
	Relative change	−0.33	(−0.73 to 0.07)	–
Case study B
Low SES	Constant	−0.83	–	–	42%
	Pretrend	2.66	(−2.82 to 8.17)	0.248
	Interruption	0.82	(−9.99 to 11.6)	0.844
	Post-trend	−3.41	(−9.42 to 2.62)	0.191
	Relative change	−0.62	(−1.27 to 0.03)	–
Non-selective	Constant	43.2	–	–	77%
	Pretrend	0.02	(−5.44 to 5.47)	0.993
	Interruption	10.4	(−0.35 to 21.1)	0.055
	Post-trend	−3.18	(−9.15 to 2.80)	0.214
	Relative change	−0.05	(−0.48 to 0.37)	–
Non-white	Constant	72.1			73%
	Pretrend	−5.16	(−15.6 to 5.33)	0.244
	Interruption	−13.4	(−34.1 to 7.20)	0.145
	Post-trend	9.62	(−1.87 to 21.1)	0.081
	Relative change	0.81	(−1.58 to 3.21)	–
Male	Constant	51.4	–	–	46%
	Pretrend	−1.78	(−5.93 to 2.35)	0.298
	Interruption	3.1	(−5.04 to 11.2)	0.35
	Post-trend	2.05	(−2.49 to 6.58)	0.278
	Relative change	0.3	(−0.23 to 0.85)	–
Case study C
Low SES	Constant	5.37			30%
	Pretrend	−0.51	(−2.25 to 1.23)	0.463
	Interruption	0.13	(−17.0 to 17.3)	0.984
	Post-trend	2.54	(−7.92 to 13.0)	0.538
	Relative change	11.8	Non-est*	–
Non-selective	Constant	66.7	–	–	58%
	Pretrend	−1.72	(−4.49 to 1.04)	0.159
	Interruption	0.66	(−26.6 to 27.9)	0.95
	Post-trend	0.52	(−16.1 to 17.1)	0.935
	Relative change	0.07	(−0.91 to 1.05)	–
Non-white	Constant	30.8	–	–	9%
	Pretrend	−0.84	(−4.81 to 3.13)	0.589
	Interruption	4.65	(−34.4 to 43.6)	0.757
	Post-trend	−0.19	(−24.0 to 23.6)	0.983
	Relative change	0.14	(−2.78 to 3.07)	–
Male	Constant	44.8	–	–	22%
	Pretrend	−0.24	(−5.09 to 4.60)	0.895
	Interruption	14	(−33.6 to 61.7)	0.46
	Post-trend	−4.98	(−34.5 to 24.1)	0.66
	Relative change	−0.37	(−3.82 to 3.09)	–
Case study D
Low SES	Constant	3.85	(−1.43 to 1.78)	–	41%
	Pretrend	0.19	(−1.40 to 1.78)	0.758
	Interruption	−2.5	(−11.1 to 6.12)	0.466
	Post-trend	1.5	(−2.40 to 5.40)	0.346
	Relative change	1.21	Non-est*
Non-selective	Constant	62.8	(41.7 to 83.9)	–	26%
	Pretrend	−1.55	(−7.92 to 4.81)	0.535
	Interruption	2.4	(−32.1 to 36.7)	0.856
	Post-trend	−0.48	(−16.1 to 15.1)	0.936
	Relative change	−0.01	(−1.10 to 1.08)
Non-white	Constant	31.1	(22.0 to 40.1)	–	87%
	Pretrend	−0.36	(−3.08 to 2.37)	0.736
	Interruption	−5.36	(−20.1 to 9.41)	0.371
	Post-trend	7.75	(1.07 to 14.4)	0.032
	Relative change	1.46	(0.37 to 2.54)	–
Male	Constant	38.3	(−21.6 to 54.9)	–	11%
	Pretrend	0.16	(−4.87 to 5.19)	0.936
	Interruption	2.44	(−24.8 to 29.7)	0.816
	Post-trend	−0.05	(−12.4 to 12.3)	0.991
	Relative change	0.05	(−1.03 to 1.14)	–

*Non-est, not estimable due to unrealistic variance because of small sample size.

Case study B

Figure 3 shows the proportion of admissions for case study B. This institution used both convenience (ie, set to select the required number of candidates for interview) and an actual UKCAT threshold up to 2013, with both increases and decreases in required scores. In 2009, the required academic attainment was increased. In 2013, the university continued to use a UKCAT threshold score but also added in a large factor weighting for both academic tariff and UKCAT to select for interview; thus, 2013 was used as the year of interruption in analysis (table 2). The only evidence of the interruption having an effect was for the proportion from non-selective school, where there was a jump up, although the trend before and after was not statistically significant. There was a jump up in the proportion of non-white at the same time but not found to be statistically significant (table 2). None of the estimates of relative change were significant, indicating that the proportions observed postinterruption were not obviously different from what they would have been if the change in selection policy had not occurred.

Figure 3

Proportion of admissions: case study B. SES, socioeconomic status; UKCAT, UK Clinical Aptitude Test.

Case study C

Figure 4 shows the proportion of admissions for case study C. This school increased their academic attainment requirements in 2011 and used both a UKCAT threshold and a factor % for the UKCAT. The year of interruption for the analysis was taken as 2013 where the UKCAT factor weighting and threshold value were increased (factor percentage to 50% and threshold was additional 500 points). The factor percentage for personal statement was reduced. The interrupted time series (table 2) did not yield any statistically significant results for the trends before or after the interruption and at the 2013 interruption. None of the estimates of relative change were statistically significant, indicating the proportions observed postinterruption were not obviously different from what they would have been if the change in selection policy had not occurred.

Figure 4

Proportion of admissions: case study C. SES, socioeconomic status; UKCAT, UK Clinical Aptitude Test.

Case study D

Figure 5 shows the proportion of admissions for case study D. This institution used a combination of personal statement scoring and UKCAT ‘trade-off’ (ie, candidates with higher UKCAT scores being considered favourably) approach up until 2010 and switched to factor weighting in 2012 (UKCAT and academic attainment). In 2014, the factor weighting for academic attainment was increased at the expense of the UKCAT. The year of interruption investigated was in 2012.

Figure 5

Proportion of admissions: case study D. SES, socioeconomic status; UKCAT, UK Clinical Aptitude Test.

There were no significant trends or intervention effects for male, non-selective or low SES students for case study D (table 2). However, there was a significant postinterruption increasing trend for proportion of non-white student (p=0.032), although there were only two additional postinterruption time points. This translated into a relative change of 1.46 (95% CI 0.37 to 2.54) showing the rate of change in non-white was increased following the change in policy. Estimates of relative change were not statistically significant for the other three widening access criteria.

Discussion

Our aim was to examine if changes in medical school selection criteria or processes leading to invitation to interview impact on the demographic composition of the student population. At an aggregate level, across our whole sample of 18 medical schools with over 24 346 admissions, despite significant changes in the use and weighting of selection criteria, there was little change in the proportions of students accepting a place who were from lower socioeconomic groups, non-selective schools, were non-white and/or male. Yet, our case study data show that all four example schools changed their admissions practices over the time period of the study. Some schools changed their admissions practices frequently over the 8-year study period, sometimes changing multiple things in the same year. This within-school variation made little difference locally, or overall in terms of increasing the diversity of medical students: changes in admission practices to practices that seemed ‘fairer’25 did not lead to any discernible change in who was accepted into medical school.21

There are numerous possible reasons for this. First, none of the adjustments we observed were particularly radical. A set of selection criteria results in a ranking of applicants. Small changes to weightings (such as school A increasing the weighting for the UKCAT from 7% to 14%) would not radically alter that ranking, especially when gaining a place remains largely determined by prior academic attainment in all cases. Second, historical evidence suggests that academic and cognitively oriented assessment tools, which encompass school exams and cognitive ability tests such as those used as part of the UKCAT, tend to favour ‘traditional’ applicants to medicine, that is, white and high social class individuals.28 53 54 Thus, it is possible that the common usage of prior academic attainment and an aptitude test comprising cognitive ability tests as the first two hurdles within the selection process sequence may select appropriately in terms of predictive validity yet at the same time actually ‘narrow’ rather than widen access.55 However, to change from this practice would require medical schools to, for example, move the assessment of personal values or attributes from its typical position as the last hurdle to the forefront of the process. Our previous work suggests that this would not be embraced by medical schools, many of which struggle to see how widening access can fit with their culture, ethos and aspirations.36 On a pragmatic note, many need the first stages of selection to help them screen large numbers of applications for a much smaller number of interview places and ultimately medical school places. Third, we do not know why schools changed their processes. Did school A, for example, bring in MMIs and remove personal statements from the selection process with the explicit aim of opening the doors to a wider range of applicants? For example, what was the rationale for school A doubling the UKCAT weighting in 2012 (and why was it 7% originally and 14% thereafter)? What did they hope to achieve by this change in practice? We do know that the timeframe of the study was one where widening access to medicine was extremely topical within the UK, and schools were expected to provide evidence to the regulator as to how they were addressing this issue. Was change enacted for accountability purposes only, seeking improved validity or embraced as a means of really making a difference in terms of widening access?56 Fourth, different selection processes may attract different applicants, which would change the nature of the applicant pool and hence influence the changes observed. However, our data did not indicate that the applicant pools differed notably pre and post changes. Fifth, our outcome measure of invitation to interview is not equivalent to accepting a place. It may have been that changes in the selection processes did have the intended effect on diversity, but the interview process mitigated the impact of change. However, what little evidence does exist is conflicting in terms of the medical school interview introducing further social bias into the selection system.57 58 Further work is needed to explore the potential social bias into the selection system. Sixth, we do not know how independent or otherwise the various selection tools are from each other. What is the relationship between, for example, UKCAT and MMI scores? Do different selection tools overlap in terms of what they measure? In terms of the latter, the few studies that look at this conclude modest–moderate relationships between different selection tools, even those that measure the same qualities.59 60 Finally, an alternative, or additional explanation for this finding, is to consider the nature of applicants. Modifying selection processes is unlikely to have major impact on widening access if there is a very small pool of people from certain backgrounds applying. However, while this may be the case in the UK in terms of applicants from lower socioeconomic groups,61 there are plenty of applicants to medicine who are male, non-white and/or who have attended non-selective schools.15 38 This suggests there may be biases inherent in the current selection processes that need further exploration28 33 35 and that medical schools need to increase their focus on encouraging pupils from diverse backgrounds to apply for medicine.61

In addition, a potential limitation of an interrupted time-series approach is that there may be other secular trends occurring as well as any identified interruption. Thus, it is possible that other background trends may have swamped any signal from the changes in admission policy implemented by the medical schools. For example, as the UK economy weakened, those from under-represented groups may have been further discouraged from applying to university due to fees.

This work joins an ongoing conversation in the literature related to selection into medicine and how best to widen access to medical education.1–5 7 22 25 The particular approach used in this paper (time series analysis of case studies) provides a more nuanced account of the processes of medical selection than is available from large-scale studies looking solely at aggregate data.18 26 28 31 32 38 53 62

The time series analysis allowed for detailed analysis of individual school admission information and shows there was individual variation but, as a whole combined across medical schools, the picture is unchanged. However, the UKCAT now incorporates two different components: the cognitive ability tests that are the component referred to throughout this study and the new SJT component that measures a range of personal attributes. Recent research suggests that the SJT may not favour those from more privileged educational backgrounds.34 To alter the demography of those offered a place would mean weighting the SJT component of the UKCAT more heavily than the cognitive ability components and/or use the interview component of selective differently. Medical schools may not be ready to do this. Moreover, in short, the implication from the current study is not just that the combination, weighting and sequencing of selection methods need rethinking but that schools need to decide what they want to achieve via selection. For example, are they trying to attract particular groups to their medical schools? Depending on the priority, to achieve their aims, the weighting and sequencing of selection methods may be different. Indeed, a framework has previously been proposed to guide how selection can be optimised in order to maintain entrant quality while minimised the adverse impact on disadvantaged groups.63 It may also be that schools need to completely rethink their approach to increasing diversity, to depend less on comparing the performance of diverse applicants on standardised selection tools, and shift more towards an individualised approach, which gives consideration to applicant background and life experiences, and what they can bring to medicine.64 65

Addressing calls in the literature, we focused on widening access in terms of socioeconomic background and examined other potential dimensions of disadvantage, such as gender, ethnicity and schooling. Our methodological approach also avoids the issues associated with single-site, cross-sectional work by comparing across schools and over time. We need to know more about why schools change their admissions policies and what they hope to achieve by doing so. A qualitative methodology would be appropriate to explore this question.36 Also, on a more practical note, we urge schools to collect and scrutinise their own data at a granular level, akin to the approach taken in our case studies, as this information is essential to assess the status quo (baseline) and evaluate the impact of any change.

The four medical schools were chosen from the original 18 for which data were available because they represented the diversity of the sample. The commonalities were that they all used the UKCAT, the most widely used admissions test in the UK, and offered traditional 5-year programmes. However, we took care in how we presented the (historical) data to minimise the chance of individual schools being identified by ‘insiders’. We do not know if the within-school and between-school diversity we identified is also found in UK schools using other admissions tests, or in accelerated or extended medical programmes; this remains to be explored. This work was carried out in one context, the UK, and hence the findings may not be generalisable across contexts. However, the admissions ‘tools’ combination of prior attainment (whether school exit examinations, grade point average or specific knowledge-based examinations for medical admissions (eg, Medical College Admission Test)), some sort of aptitude test and either a traditional or MMI is typical of many countries.7 26 Moreover, our study does not focus on the tools themselves but rather on changing admissions processes. Given that medical schools across the world are constantly reviewing and changing their admissions practices, our messages will resonant across contexts.

In conclusion, UK medical schools now have a political mandate to increase the diversity of medical students and a time line in which to achieve certain goals.39 However, this study suggests that current selection processes will not help the medical profession ‘throw open its doors to a far broader social intake than it does at present’.21 If we wish to increase the diversity of the medical profession in the future, we suggest that medical schools should be better supported to take a more radical and less risk averse approach to selection.