Ten years of NIHR research training: who got an award? A retrospective cohort study.

OBJECTIVE: In 2017, the National Institute for Health Research (NIHR) academy produced a strategic review of training, which reported the variation in application characteristics associated with success rates. It was noted that variation in applicant characteristic was not independent of one another. Therefore, the aim of this secondary analysis was to investigate the inter-relationships in order to identify factors (or groups of factors) most associated with application numbers and success rates.
DESIGN: Retrospective data were gathered from 4388 applications to NIHR Academy between 2007 and 2016. Multinominal logistic regression models quantified the likelihood of success depending on changes in the explanatory factors; relative risk ratios with 95% CIs. A classification tree analysis was built using exhaustive χ2 automatic interaction detection to better understand the effect of interactions between explanatory variables on application success rates.
RESULTS: 936 (21.3%) applications were awarded. Applications from males and females were equally likely to be successful (p=0.71). There was an overall reduction in numbers of applications from females as award seniority increased from predoctoral to professorship. Applications from institutions with a medical school had a 2.6-fold increase in likelihood of success (p<0.001). Classification tree analysis revealed key predictors of application success: award level, type of programme, previous NIHR award experience and applying form a medical school.
CONCLUSION: Success rates did not differ according to gender, and doctors were not more likely to be successful than applications from other professions. Taken together, these findings suggest an essential fairness in how the quality of a submitted application is assessed, but they also raise questions about variation in the opportunity to submit a high-quality application. The companion qualitative study (Burkshaw et al. (2021) BMJ Open) provides valuable insight into potential candidate mechanisms and discusses how research capacity development initiatives might be targeted in the future. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This is the first in-depth evaluation of trends and determinants of funding success of a national health research funding programme in the UK.

Previous research has focused on specific predictors of success, such as gender, or specific professional backgrounds. This study was broader ranging and employed two different analytical approaches (logistic regression and a classification tree) to minimise the likelihood of misinterpreting the findings.

Key predictors of application success were (1) the award level (ie, as awards get more senior they become more competitive), (2) type of training programme applied to, and (3) the applicant’s previous experience of having had an award.

Gender was not a predictor of application success; the success rate for males vs females at each award level was not significantly different. However, as award seniority increased from predoctoral to research professorship the numbers of applications from females reduced.

Applications from institutions with, or associated with, a medical school were more likely to be successful; however, some training programmes were only available to applicants associated with a host medial school.

Introduction

In the UK, the National Institute for Health Research (NIHR) was established in 2006 and is the largest national funder of health and social care research in Europe. A corner stone of the NIHR mission is to support research training for the most promising biomedical and applied health research professionals. In 2016, the NIHR Academy conducted a 10-year strategic review of training with the aim of reviewing existing funding for trainees and informing its future vision. The subsequent report1 described substantial growth in the number of awards overall funding over the ten year period; in the financial year 2006/2007 the NHIR Academy managed 450 awards which had increased to 2296 by 2015/2016. The report authors noted that application numbers had also grown during this period, while the success rate (about one in five) had stayed fairly constant over time.1

The NIHR training programmes were developed to build capacity quickly by working in response to changing health needs and prioritisation through commissioned research calls to shape the emerging research portfolio. NIHR Academy funding has focused on health and care research with close links to other funding agencies such as the Medical Research Council and Wellcome Trust. The intention has been to drive up the quality of health and care research through competitive processes that respond to the changing health needs of the population. One of the reasons for the 10-year strategic review of training1 was an awareness that research capacity development has to anticipate changing needs for research in response to changing needs of healthcare. In the current paper, we present a secondary analysis of the application and award data to explore patterns in the funding of academy trainees over the decade 2007–2016. Our aim was to identify factors associated with successful and unsuccessful applications, and to consider the implications of any patterns which, if unaddressed, might perpetuate inequalities or lead to underprovision in key areas of future research need.

To our knowledge, this is the first in-depth evaluation of trends and determinants of funding success of a national health research funding programme in the UK. In 2016, the Canadian Institutes of Health Research published data on their funding programmes.2 The authors identified a number of factors associated with grant success, including applicant’s gender and professional background as well as the host organisation’s size.2 Gender inequality is a recognised feature of academic careers in medicine and research.3 For example, Burns et al identified gender differences in the success of applications for both research grants and personal awards in topic areas ranging from cancer to health services and policy research.4 Waisbren et al showed that the gender disparity in medical research funding was largely explained by gender differences in academic seniority, but when seniority was accounted for the gender disparity was equalised.5 However, this study did not focus on personal awards, such as fellowship. While the NIHR database did not contain the necessary information for research topic to be included in the present analysis, in respect of applicant characteristics, there was information on gender, professional background and Academy award schemes (which can be used as a proxy measure for academic seniority). In addition, we could examine the characteristics of the host institutions to evaluate whether any particular institutions or group(s) of institutions were more likely to support successful applications; that is, research intensive universities, or institutions linked to a medical school. In classifying host institutions for analysis purposes, we were mindful that a description based on geographic region would not enable us to test some pervasive beliefs about institutional status (ie, research income). Therefore, we used publicly available information to classify each host institution under a number of additional headings as detailed below (see the ‘Data preparation’ section).

It had already been established that both application numbers and success rates varied according to applicant, host institution and award scheme characteristics.1 However, many of these characteristics are not independent of each other, for example, the Integrated Academic Training (IAT) programme is only open to doctors and dentists. Therefore, the analyses reported in this paper took the inter-relationships into account in order to identify the most important factors associated with application numbers and success rates.

Methods

Study population

Retrospective data were gathered from 4420 applications to the NIHR academy (formerly trainees coordinating centre) between 2007 and 2016. Data consisted of applications to the personal awards schemes operated by the NIHR academy:

IAT programme: in-practice fellows and clinician scientists awards.

Integrated Clinical Academic (ICA) training programme: clinical doctoral fellowships, clinical lectureships, senior clinical lectureships.

Fellowships: doctoral research fellowships, transitional research fellowships, postdoctoral fellowships, career development fellowships, senior research fellowship, research professorships, clinical trials fellowships, knowledge mobilisation research fellowship.

It should be noted here that data are based on applications, not individual applicants. Data were pseudonymised prior to analysis, therefore, individual applicants were not identifiable.

Patient and public involvement

Patients and public were not involved in this project.

Data preparation and management

All applications to personal awards within the IAT, ICA and fellowships training programmes between 2007 and 2016 were eligible. Data for personal awards are more complete than data for the institutional awards (ie, academic clinical fellowships) which are held by local partnerships. Therefore, only data for personal awards were used in this analysis. The following data were extracted from the management system in April 2016:

Application outcome (successful/rejected).

Gender.

Professional background (medic, nurse, dentist, allied health professional (AHP), midwife, other HP, not a HP).

Whether applicant previously held an NIHR award (yes/no).

Academy programme (IAT, ICA, Fellowships).

Award level categorised as predoctoral, doctoral, early postdoctoral, late postdoctoral, senior lecturer/prechair, professor).

Year application submitted 2007–2016 (also categorised into one of three cohorts: 2007–2010, 2011–2013, 2014–2016).

Host institution type (university, National Health Service (NHS) trust, other).

Region of England of host organisation (based on the NIHR Research Design Service (RDS) regions).

Whether the host institution was associated with a medical school (yes/no).

Host institution name (eg, University of Leeds).

Using the host institutions’ name, each application was classified as having come from one of three higher education groups: Golden Triangle (GT), Russell Group (RG), other.

Host institution factors

These higher education groups were classified using published lists for Universities within the GT6 and RG.7 In the UK, the RG is a formal grouping of 24 research intensive universities which are globally renowned for their success in capturing competitive research income.7 The GT is an unofficial, but widely recognised, grouping of research intensive universities within the cities of Cambridge, London and Oxford. London universities included within the GT are King’s College London, University College London, London School of Economics and Imperial College London.6 The decision to classify host institution in this way was undertaken to respond to questions regarding equity of access to NIHR Academy funding for applicants applying from host institutions with traditionally higher research incomes (ie, institutions within the GT or RG) compared with those with lower research incomes. In this analysis, RG institutions located in the ‘GT’ were classified as ‘GT’.

Data analysis

Of the 4420 applications available for analysis, 32 were excluded due to missing gender data. All subsequent analyses are based on 4388 (99.3%) applications with complete data. The analysis focused on identifying factors associated with applications that were successful (ie, awarded) compared with those which were rejected (at any stage). The data were summarised using frequency (proportion) or median (IQR) as appropriate.

The primary focus of this analysis was to identify whether any of the explanatory factors were associated with an increased or decreased likelihood of a successful application. An a priori assumption was that there may have been a clustering effect of individual ‘host institution’ which may have warranted a multilevel modelling approach. Therefore, to guide the modelling, a simple null logistic model was fitted which had as the outcome success/reject of an application. The logistic regression was rerun as a two-level model with a random intercept included for host institution. Comparison of interclass correlation coefficients of the two models demonstrated that only 2.62% of the variation in success rate was associated with the host institution. This process of model selection showed two things; first, that there was little benefit using multilevel modelling and therefore subsequent analyses used single level multi-nominal logistic regression models. Second, that there was very little clustering of successful applications within individual host institutions, that is, 97.3% of the variation in success was explained by factors other than the host institutions.

Therefore, multinominal logistic regression models were used to understand whether the likelihood of success varied depending on changes in the explanatory factors. The data from these models are presented as relative risk ratios (RRR) with 95% CI. Significance was set at p<0.05. Three sets multinominal logistic regression models were performed. First univariate models were used to calculate the RRR of application success for each individual explanatory factor. Second, a fully adjusted multivariate model included all explanatory variables. Third, a model was created containing only those explanatory factors that were significant in the second fully adjusted model.

As explained previously, many of the characteristics associated with application success are not independent of each other. Therefore, we expected there to be a high degree of interaction and collinearity between the explanatory factors. To further investigate the inter-relationships between the explanatory factors and application success, we used a classification tree analysis. This is a simple and stable method for dealing with collinearity and for interpreting interactions between variables. The classification tree method groups applications together based on common characteristics and identifies the factors most commonly associated with an application being successful. The classification tree splits all the applications into groups based on application characteristics where the groups separate by success rate (awarded vs rejected). This is done step by step at branching points.

The classification tree used exhaustive χ2 automatic interaction detection (CHAID) to identify factors most strongly associated with application success (awarded/rejected).8 9 The exhaustive CHAID model was chosen as it examines all possible splits for each predictor variable and chooses the independent (predictor) variable with the strongest interaction with the dependent (outcome) variable.8 10 The exhaustive CHAID model was specified based on a pragmatic approach to produce a robust yet simple classification tree that can be easily interpreted. Our approach was informed by published literature,8 9 11 as well as the authors’ prior experience (as per Ziegler et al12), on the selection of CHAID model parameters to produce a classification tree with interpretable summaries of the data. Therefore, the following criteria were selected for the tree’s construction: all variables were entered into the exhaustive CHAID model; the tree was limited to a maximum of eight nodes (levels); parent node size was set to a minimum of 200 applicants; child node size was set to a minimum of 100 applicants; the significance level for splitting nodes and merging categories was adjusted using the Bonferroni method with threshold set at 0.001.

For all logistic regression and classification tree models, p values less than 0.001 were considered statistically significant (two tailed). Analysis was conducted using STATA V.15 (StataCorp, Release 15, StataCorp) and IBM SPSS Statistics for Windows, V.23.0 (IBM).

Results

Application numbers and overall success rate for personal awards

Between 2006 and 2016, 4388 applications were submitted for personal awards to the NIHR Academy with complete data. Of these 936 (21.3%) were awarded and 3484 (78.7) were rejected. Figure 1 shows that the application success rate has remained stable since 2006/2007 at around 20%. From 2006, the number of programmes and schemes funded by the Academy, as well as trainees in post, has increased year on year. In 2012, the numbers of trainees managed by the academy reached a plateau, from which time the number of trainees in post has been in steady state (figure 1).

Figure 1

Personal awards applications, awards funded and success rate by financial year data include complete cases of applications and awards funded to the personal Award schemes from 2007 to 2016.

The data summarised in table 1 show that 61% applications were from females, 39% from males. More than one-third of applications (37%) were from medics, and a quarter (24%) of applications came from non-healthcare professionals (eg, applied health and social sciences researchers), followed by AHPs (14%), other HPs (12%) and nurses/midwives (11.5%). Just under 20% of applications (n=845) were from people who had previously held an NIHR award, of which 70% (n=601) were applications from medics. Just under 80% of applications came from Higher Education Institutions (HEIs) (ie, universities) and 71% of applications were from institutions with a UK medical school. About half of all applications were for doctoral level fellowships, and over one third of applications came from institutions within London. Seventy-eight per cent of applications came from institutions within either the GT, 37% or RG, 41% and just over 20% of applications came from Other (non-GT or RG) institutions.

Table 1

Summary of descriptor variables for all applicants to NIHR academy personal awards between 2006/2007 and 2015/2016

Variable	All applicants
	N=4388 (%)
Gender
Female	2659 (60.6)
Male	1729 (39.4)
Professional background
Medic	1609 (36.7)
Dentist	72 (1.6)
Allied HP	601 (13.7)
Nurse/midwife*	505 (11.5)
Not HP	1052 (24)
Other HP	549 (12.5)
Previous NIHR award holder
No	3534 (80.5)
Yes	854 (19.5)
Award level
Predoctoral	159 (3.6)
Doctoral	2204 (49.9)
Early postdoctoral	1198 (27.3)
Late postdoctoral	513 (11.6)
Senior/prechair	146 (3.3)
Chair	168 (4.3)
Programme
ICA	800 (18.1)
NIHR fellowship	3174 (72.5)
IAT	414 (9.4)
Cohort (year of submission)
2007–2010	1224 (27.9)
2011–2013	1541 (35.1)
2014–2016	1632 (37)
Host organisation type
HEI	3441 (78.4)
NHS trust	865 (19.7)
Other†	82 (1.9)
Region
East Midlands	252 (5.8)
East of England	256 (5.8)
London	1555 (35.4)
North East	206 (4.7)
North West	476 (10.9)
South Central	439 (10.0)
South East Coast	77 (1.8)
South West	283 (6.4)
West Midlands	412 (9.3)
Yorkshire and Humber	432 (9.8)
Medical school
Yes	3120 (71.2)
No	1268 (28.8)
Higher education group
GT	1615 (36.7)
RG	1793 (40.9)
Other	980 (22.4)

*Nurses n=443, midwives n=64

†Other host organisation=charity (4 applications), Clinical Commissioning Group (16 applications), other non-NHS (38 applications), other NHS (24 applications).

GT, golden triangle; HEI, higher education institution; HP, health professional; IAT, Integrated Academic Training; ICA, Integrated Clinical Academic; NHS, National Health Service; NIHR, National Institute for Health Research; RG, russell group.

Factors associated with successful applications

Applicant factor: gender

The multinominal logistic regression models presented in table 2 show that there was no main effect of gender either before (model 1) or after adjusting for other factors (model 2 and model 3), that is, applications from males and females were equally likely to be successful. There was, however, a clear pattern in the numbers of applications as shown in figure 2: as award seniority increased from pre-doctoral to Research Professorship, the number of applications from females reduced. There was an equal number of male and female applications for the predoctoral awards (figure 2) and a 60/40 split in favour of females for doctoral applicants. This evened out over the postdoctoral and senior fellowship awards, and became a 60/40 split in favour of males for the research professorship applications. A univariable multinominal logistic regression model confirmed this trend showing that, compared with applications for the predoctoral awards, applications to the doctoral and early postdoctoral awards were significantly less likely to be from male applicants (RRR 0.55 (95%CI 0.39 to 0.76), p<0.001; 0.62 (95%CI 0.44 to 0.87), p<0.01). While for the research professor awards, applications were almost twice as likely to come from males applicants (1.87 (95%CI 1.20 to 2.92) p>0.01). Thus, although the number of successful applications from males was higher than that from females at senior award levels (eg, n=19 vs n=12 at professorship level), this was due to the fact that more males submitted applications for these awards. In fact, the success rate for males versus females at each award level was not significantly different. Taken together, these data show that when award seniority and application rates are accounted for applications from males and females are equally likely to be successful.

Table 2

Outcome of application (rejected/awarded)

Variable	Rejected	Awarded	Model 1			Model 2			Model 3
	3484 (78.8%)	936 (21.2%)
			RRR	95% CI	P value	RRR	95% CI	P value	RRR	95% CI	P value
Gender
Female	2107 (79.2)	552 (20.8)	–	–	–	–	–	–
Male	1345 (77.8)	384 (22.2)	1.09	0.94 to 1.26	0.25	1.03	0.88 to 1.21	0.71
Professional background
Medic	1235 (77)	374 (23)	–	–	–	–	–	–
Dentist	44 (61.1)	28 (38.9)	2.1	1.29 to 3.42	0.003	1.47	0.85 to 2.50	0.15
Allied HP	456 (75.9)	145 (24.1)	1.05	0.84 to 1.31	0.66	1.01	0.74 to 1.37	0.92
Nurse/midwife	415 (82.2)	90 (17.8)	0.72	0.55 to 0.92	0.01	0.72	0.52 to 1.01	0.06
Not HP	875 (83.3)	177 (16.7)	0.67	0.55 to 0.81	<0.001	0.94	0.66 to 1.08	0.64
Other HP	427 (77.8)	122 (22.2)	0.94	0.75 to 1.19	0.62	1.11	0.83 to 1.49	0.47
Previous NIHR award holder
No	2834 (80.2)	700 (19.8)	–	–	–	–	–	–	–	–	–
Yes	618 (72.4)	236 (27.6)	1.55	1.31 to 1.83	<0.001	1.74	1.41 to 2.13	<0.001	1.77	1.45 to 2.14	<0.001
Award level
Predoctoral	77 (47.8)	84 (52.2)	4.31	3.11 to 5.98	<0.001	4.73	2.54 to 6.68	0.0001	4.79	2.96 to 7.74	<0.001
Doctoral	1749 (79.4)	455 (20.6)	–	–	–	–	–	–	–	–
Early postdoctoral	953 (79.1)	252 (20.9)	1.02	0.85 to 1.21	0.79	1.09	0.85 to 1.33	0.38	1.08	0.89 to 1.31	0.41
Late postdoctoral	422 (81.9)	93 (18.1)	0.85	0.66 to 1.08	0.21	0.98	0.77 to 1.27	0.87	0.99	0.76 to 1.28	0.94
Senior/prechair	125 (85.6)	21 (14.4)	0.64	0.41 to 1.04	0.07	0.64	0.41 to 1.04	0.07	0.65	0.40 to 1.06	0.08
Chair	158 (83.6)	31 (16.4)	0.87	0.51 to 1.12	0.49	0.97	0.62 to 1.47	0.87	0.99	0.65 to 1.49	0.97
Programme
ICA	581 (72.5)	220 (27.5)	1.74	1.45 to 1.08	<0.001	2.2	1.69 to 2.87	<0.001	2.09	1.68 to 2.59	<0.001
NIHR fellowship	2634 (82.2)	569 (17.8)	–	–	–	–	–	–	–	–	–
IAT	269 (64.7)	147 (35.3)	2.52	2.02 to 3.14	<0.001	1.08	0.76 to 1.53	0.65	1.14	0.82 to 1.59	0.43
Cohort
2007–2010	971 (79.5)	253 (20.4)	–	–	–	–	–	–	–	–	–
2011–2013	1187 (77.3)	354 (22.7)	1.14	0.95 to 1.37	0.15	1.09	0.90 to 1.33	0.36	1.1	0.91 to 1.34	0.29
2014–2016	1294 (79.8)	329 (20.2)	0.97	0.81 to 1.12	0.79	0.79	0.64 to 0.96	0.02	0.81	0.66 to 0.98	0.04
Host organisation type
HEI	2721 (79.3)	720 (20.7)	–	–	–	–	–	–	–	–	–
NHS trust	669 (77.4)	196 (22.6)	1.11	0.93 to 1.32	0.26	2.08	1.37 to 3.21	0.001	2.21	1.45 to 3.33	<0.001
Other*	62 (75.6)	20 (24.4)	1.22	0.73 to 2.03	0.45	2.09	1.08 to 4.05	0.03	2.08	1.07 to 4.01	0.03
Region
East of England	205 (84.9)	39 (15.1)	0.98	0.69 to 1.38	0.34	0.78	0.54 to 1.15	0.23	0.76	0.53 to 1.10	0.23
East Midlands	211 (81.8)	47 (18.2)	0.78	0.53 to 1.11	0.15	1.05	0.71 to 1.61	0.76	0.88	0.62 to 1.26	0.76
London	1268 (81.1)	295 (18.9)	–	–	–	–	–	–	–	–	–
North East	159 (76.4)	49 (23.6)	1.33	0.94 to 1.88	0.16	1.63	1.09 to 2.51	0.02	1.4	0.98 to 1.99	0.06
North West	382 (79.2)	100 (20.7)	1.13	0.87 to 1.45	0.41	1.36	0.97 to 1.92	0.08	1.15	0.89 to 1.49	0.29
South Central	344 (77.8)	98 (22.2)	1.23	0.95 to 1.58	0.21	1.26	0.96 to 1.65	0.09	1.22	0.94 to 1.59	0.15
South East Coast	63 (80.8)	15 (19.2)	1.03	0.57 to 1.82	0.84	1.24	0.65 to 2.41	0.52	1.09	0.59 to 2.04	0.76
South West	198 (70)	85 (30)	1.83	1.38 to 2.43	<0.001	2.2	1.56 to 3.25	<0.001	1.87	1.39 to 2.51	<0.001
West Midlands	326 (78.9)	87 (21.1)	1.14	0.88 to 1.51	0.32	1.23	0.88 to 1.74	0.26	1.03	0.78 to 1.36	0.82
Yorkshire and Humber	314 (72.2)	121 (27.8)	1.66	1.30 to 2.12	<0.001	1.98	1.43 to 2.82	<0.001	1.66	1.29 to 2.13	<0.001
Medical school
No	1028 (80.8)	245 (19.2)	–	–	–	–	–	–	–	–	–
Yes	2456 (78)	691 (22)	1.19	1.01 to 1.40	0.05	2.5	1.64 to 3.82	<0.001	2.61	1.77 to 3.84	<0.001
Higher education group
GT	1295 (79.8)	372 (20.2)	–	–	–	–	–	–
RG	1388 (76.8)	420 (23.2)	1.2	1.02 to 1.42	0.02	1.25	0.95 to 1.65	0.11
Other	801 (808.9)	189 (19.1)	0.94	0.77 to 1.14	0.55	1.04	0.81 to 1.35	0.73

Bold values indicate significant RRR at p<0.05

*Other organisation=charity (4 applications, 1 awarded), Clinical Commissioning Group (16 application, 15 awarded, other non-NHS (38 applications, 3 awarded), other NHS (24 applications, 1 awarded).

GT, golden triangle; HEI, higher education institution; IAT, Integrated Academic Training; ICA, Integrated Clinical Academic; NHS, National Health Service; NIHR, National Institute for Health Research; RG, russell group; RRR, relative risk ratio.

Figure 2

Applications by gender and award level for personal award schemes data include all applications to personal awards schemes. *Predoctoral awards represent the in-practice fellowship scheme only. **Chair represents the research professors scheme.

Applicant factor: professional background

There was variation in the success rate across the different professional backgrounds of applicants (model 1, table 2). Compared with medics, dentists were twice as likely to be successful, while nurses/midwives and non-healthcare professionals were 28% and 33%, respectively, less likely to be successful (table 2, model 1). However, these differences did not remain significant when all other factors were adjusted for (table 2, model 2). As with gender differences described above, additional information helped clarify where the difference between professional groups did and did not, lie. Considering only medics and nurses/midwives, a total of 2114 applications were received, of which 1609 (76.1%) were from medics. The figures for successful awards were not dissimilar: approximately four times as many awards were made to medics (80.6% of awards) than to nurses/midwives (19.4%). The slightly greater disparity in awards compared with applications was reflected in the univariate comparison of success rates between professional groups, which became non-significant once other factors had been taken into account. Figure 3 shows that this ratio has increased from 3:1 in 2007/2010 to 5:1 in 2014/2016. Again, this reflects application numbers rather than a disparity in application success rates.

Figure 3

Ratio of successful applications from Medics compared with nurses data include applications from medics and nurses to personal award schemes.

Applicant factor: previous NIHR award

The third applicant factor considered was whether the applicant had previously held an NIHR award. Table 2, model 1, shows that having had a previous NIHR award was associated with a 55% increase in the chance of an application being successful. The improved chance of success for applicants with a previous NIHR award increased after adjusting for other factors to almost twofold increase (1.77 (95% CI 1.45 to 2.14) p<0.0001).

NIHR academy factors

When compared with doctoral level awards, there was some variation in success rates for the other levels of award (table 2); however, the most pronounced difference lay in the much higher success rate of the pre-doctoral schemes, an effect which was maintained in the fully adjusted model (4.79 (95% CI 2.69 to 7.74), p<0.001).

There was also variation in success rates between the three types of programme: 17.8% of applications to the Fellowships programme were funded compared with 27.5% of IAT applications and 35.3% of ICA applications In unadjusted model 1, applications made to both ICA and IAT programmes were more likely to be successful compared with fellowship applications (table 2). However, when adjusting for other factors in the fully adjusted model 3, only applications to the ICA programme remained significantly associated with higher success rates compared with the fellowship programme (2.09 (95% CI 1.68 to 2.59), p<0.001).

In the unadjusted model 1 (table 2), there were no difference in success rate across the three cohorts; however, in the adjusted models 2 and 3 applications submitted in the 2014/2016 cohort had slightly reduced chance of success compared with the 2007/2010 cohort (0.81 (95% 0.66 to 0.98), p=0.04).

Institution factors

In the unadjusted model 1 (table 2), application success rates were comparable across the three host organisation types (HEIs, NHS Trusts, other institutions). Taking other factors into account (models 2 and 3, table 2), success rates were higher in NHS trusts and other institutions compared with HEIs (2.21 (95% CI 1.45 to 3.33), p<0.001; 2.08 (95% CI 1.07 to 4.01), p=0.03); although, applications from NHS trusts and other institutions accounted for less than 17% of all applications (n=731).

There was limited regional variation in application success rate across the RDS region in England; however, there were some exceptions to the pattern. When compared with applications from London institutions, applications from institutions in the South West and from Yorkshire and Humber were more successful (1.87 (95% CI 1.39 to 2.15), p<0.001; 1.66 (95% CI 1.29 to 2.13), p<0.001).

In the unadjusted analysis model 1 (table 2), applications from institutions with a medical school were slightly more likely to be successful than those applying from institutions without a medical school (1.19 (95% CI 1.01 to 1.40), p=0.05). When all other had been taken into account (model 2 and model 3, table 2), this effect strengthened to 2.61 times increased likelihood of a successful application from an institutions associated with a medical school (2.61 (95% CI 1.77 to 3.85), p<0.001).

Modest differences in success rate between GT, RG and other institutions seen in the unadjusted model 1 (table 2) were no longer apparent once other factors were taken into account (model 2, table 2).

Classification tree

To better understand the interactions between explanatory variables and the effect of these interactions on application success rates, we undertook a classification tree analysis. Our analysis protocol generated a classification tree with three levels shown in figure 4. Within these three levels we found six important predictive factors associated with application success: award level, previous NIHR award, programme, professional background, medical school. The categories for each predictor variable as determined by exhaustive CHAID, including cut-points for continuous variables, are indicated above each node in figure 4.

Figure 4

Classification tree of application success. HP, health professional.

Level 1: Across the sample as a whole, the most important factor associated with a successful application was the award level applied for (χ2=101, p<0.001). Just over half (52.8%) of applications for the predoctoral awards were successful, compared with 20.8% of doctoral and early-postdoctoral applications and 17.5% of the more senior award applications (figure 3).

Level 2: For applications to the doctoral and early-postdoctoral awards, type of programme applied to was the strongest predictor of application success. The success rate was 26.7% for ICA/IAT applications vs 18.4% for applications to Fellowship programmes (χ2=29.1, p<0.001).

Subclassifications beyond this level differed between the fellowship and ICA/IAT award groups.

Level 3: Doctoral and early-postdoctoral applications for NIHR fellowships. For this subgroup, the most important factor associated with obtaining an award was whether or not the application came from an institution with, or associated with, a medical school. The success rate was 20.6% for applications which did have such an association and 11.4% for those which did not (χ2=25.1, p<0.001).

Level 3: Doctoral and early-postdoctoral applications for ICA/IAT awards. Unlike applications to the Fellowship scheme, for applications to these early career professional schemes, the distinction between doctoral and early postdoctoral awards was the most important factor associated with success. Here, doctoral applications were less successful (23.6%) than those to early postdoctoral schemes (31.9%, χ2=8.13, p<0.004).

Level 2: For applications to the late postdoctoral, senior and chair awards having had a previous NIHR award was the strongest predictor of application success. The success rate was 25.5% for those who had had a previous NIHR awards vs 15.2% for those who had not (χ2=10.77, p<0.001).

Level 3: Late postdoctoral, senior and chair applications from people without a previous NIHR award. For this subgroup, an applicants’ professional background was the most important factor associated with obtaining an award. The success rate was 24.5% for allied HPs and other HP vs 16% for medics and 9.1% for dentists, nurses/midwives and non-HPs (χ2=18.32, p=0.009).

Discussion

Principal findings

The key predictors of application success were: the award level and type of programme applied to, and the applicant’s previous experience of having had an award. One characteristic of the host institution was important: applications from institutions with, or associated with, a medical school were more likely to be successful. After adjusting for factors such as having a medical school, applications from GT and RG institutions were not found to be more successful than those from other institutions, and applications from HEIs were found to be less successful than those from NHS Trusts and other organisations. Differences in success rate according to the professional background of the applicant were only apparent in the classification tree modelling: in the subgroup of applications for senior awards from people who had not previously held an NIHR award, AHPs and other HPs fared better than medics, who in turn fared better than nurses/midwives, non-HPs and some other groups. It was notable that gender was not a predictor of success in any of the analyses.

Strengths and limitations

This paper reports analyses of a well-maintained administrative dataset which contained information on a number of factors potentially associated with award success. In interpreting the study results, it is important to be aware that the dataset necessarily reflects the eligibility characteristics of the award schemes in operation over the 10-year period: some awards (especially at predoctoral level) were only open to some types of applicant; some awards (especially those combining research with clinical practice) could only be held in certain types of institution; and not all schemes were open for the full 10-year duration of this study. These eligibility criteria particularly affect the interpretation of the classification tree results, and merit further consideration because of their potential policy implications. For example, particular care is needed when eligibility criteria constrain the factors which might differentiate within one subgroup but not within another; for example, within the large group of early career researchers, where association with a medical school appears to be less relevant to success for applicants to the ICA and IAT schemes than for applicants to the NIHR Fellowship scheme. Such a conclusion would, however, be unjustified because nearly all the ICA and IAT awards—but not the NIHR Fellowship awards—are held in institutions associated with a medical school, reducing the potential for within-group distinction on that basis.

Over and beyond formal eligibility considerations, potential predictive factors were found not to be independent of each other; but this was expected and addressed to a substantive extent by the regression modelling and classification tree. However, more subtle selection effects (eg, who got put forward for which award and by what kind of host institution) are also likely to have been in operation and these cannot be examined using routinely collected administrative data. Other published research (eg, Burns et al4) which uses a similar kind of data source has indicated the same kinds of interpretive limitations.

The inherent variability of classification tree modelling should also be considered when interpreting the potential predictive factors identified in our analyses. As a form of multivariable analysis, the classification tree method is dependent on the parameters selected and the values of the input variables. Selection of different parameters and input variables may result in a slightly different tree structure; there is an interdependence between the input variable, model parameters and the output variables.10 The parameters selected in our classification tree analysis were based on a pragmatic approach to obtaining robust and interpretable summaries of the data. As each node and branch of the tree is an element of knowledge about the relationship between the output variable (award success) and the input (predictor) variables, our intention was to construct a tree with sufficient detail to identify the main effects, without over complicating the interpretation of the results.

Previous research has tended to be designed around specific potential predictors of success, such as gender, or specific subgroups of applicants such as clinical academics. For example, Waisbren et al,5 Brown et al3 and Burns et al4 all focused on gender differences in grant funding. This study was broader ranging and employed two different analytic approaches to minimise the likelihood of misinterpreting the findings. By contrast with other published work, including Burns et al,4 in this dataset and using these analytical approaches, no direct association was found between gender and award success. The success rate for male and female applications was equitable (22% vs 21%, table 2). However, in similarity with Waisbren et al5 we observed that the numbers of applications made by males and females explained differences in the numbers of awards made at different seniority levels.

The distribution of medical schools helped explain differences in success rates between types of host institution often considered to be of higher or lower status. It has long been assumed research intensive universities, such as those within the RG or GT, are at an advantage at securing NIHR funding.1 However, our modelling has shown that the disparity in research funding across HIE groups is explained by the presence of an associated medical school. This is reasonable given that the premise of the NIHR is to support applied health and care research within the UK National Health Service (NHS).

Finally, the available dataset was limited in terms of the available applicant demographic data. Applicant data on ethnicity, age, full time/part time, and use of RDS support is not routinely collected at application stage and was therefore not available for analysis. Similarly, applications are judged individually and are therefore not routinely categorised into topic areas and types of research at the application stage. While the findings of this report suggests that funding applications for NIRH Academy personal awards are treated equitably, further scrutiny of application success by additional applicant factors and topic areas would support the development of future funding strategies.

Implications of the findings

Overall, the evidence suggests the NIHR Academy succeeded in its objective of treating all applications equitably. Factors associated with type of award (seniority level and type of programme) were the most important predictors of success, together with a specific characteristic of the host institution (ie, association with a medical school). Success rates did not differ according to the gender of the applicant, and applications from doctors were not more likely to be successful than applications from other professions.

There were some specific circumstances in which combinations of personal characteristics did seem to be relevant: among applicants who had not held a previous award, the success rate of applications for the more senior awards differed according to the applicant’s professional background, AHPs/other HPs being more successful than the other groups. Taken together, these findings suggest an essential fairness in how the quality of a submitted application is assessed, but they also raise questions about variation in the opportunity to submit a high-quality application.

Nevertheless, it must be acknowledged that there are likely to be factors associated with funding success that we did not have access to. For example, this study cannot address the question of how the presence of a medical school improves the likelihood of success of an application. We also cannot know to what extent host institutions provide support and mentoring for preparing an application. The companion study, however, by using a qualitative methodology, does identify some candidate mechanisms, and taken together, the two sets of findings provide valuable insight as to how research capacity development initiatives might be targeted in the future.

Further research

Other studies have found that some topic areas and types of research are more likely to find favour with funders than other areas. In addition, we were unable to evaluate variation in application success by applicant factors such as ethnicity, age or institution factors such as access to RDS support. It was not possible to examine the role of these variables in the present study, and they represent a gap which future research should seek to rectify. The relationship between applicant gender, award seniority and application outcome (funded/rejected) should be further explored and evaluated within the context of national gender distribution by seniority to identify the inflection point where applications for personal fellowship awards from women drop below the median. In the longer term, the impact of any initiatives prompted by the present study findings would need to be evaluated.