The impact of a streamlined funding application process on application time: two cross-sectional surveys of Australian researchers.

OBJECTIVE: To examine if streamlining a medical research funding application process saved time for applicants.
DESIGN: Cross-sectional surveys before and after the streamlining.
SETTING: The National Health and Medical Research Council (NHMRC) of Australia. PARTICIPANTS: Researchers who submitted one or more NHMRC Project Grant applications in 2012 or 2014. MAIN OUTCOME MEASURES: Average researcher time spent preparing an application and the total time for all applications in working days.
RESULTS: The average time per application increased from 34 working days before streamlining (95% CI 33 to 35) to 38 working days after streamlining (95% CI 37 to 39; mean difference 4 days, bootstrap p value <0.001). The estimated total time spent by all researchers on applications after streamlining was 614 working years, a 67-year increase from before streamlining.
CONCLUSIONS: Streamlined applications were shorter but took longer to prepare on average. Researchers may be allocating a fixed amount of time to preparing funding applications based on their expected return, or may be increasing their time in response to increased competition. Many potentially productive years of researcher time are still being lost to preparing failed applications. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.

This is the first research to examine whether simplifying a research funding application process saves time for applicants.

Time spent preparing applications may be driven by competition or the expected financial return rather than the number of pages to complete.

Many potentially productive years of researcher time are still being lost to preparing failed applications.

The two surveys were not random samples and responses from the same participant were not linked.

Introduction

Applying for research funding is time-consuming and often ends in failure. Our previous research found that over five centuries of researchers’ time went into just one funding round for the major health and medical research scheme in Australia,1 and that for most researchers the process is stressful and impacts on work and family life.2 As success rates are often between 10% and 20%, this means that many productive research days are lost, which adds to the research community's sense of frustration.

The lost productivity of Australian researchers was picked up the two major Australian political parties and cutting the application time became official policy of the current government.3 The National Health and Medical Research Council (NHMRC) of Australia began streamlining their Project Grant application process in 2014. The number of data fields in the online form was reduced from 180 to 68.2 Applications were around 50 pages long for the streamlined system compared with 100 pages for the old system.

To see if the streamlined application process saved time we repeated our 2012 survey of Australian health and medical researchers in 2014. Our survey was only concerned with Project Grants which are the biggest national source of funding for new medical research in Australia. In 2014 there were 3810 applications with a budget of $458 million (£246 million).4

Methods

Survey methods

We used an online cross-sectional survey in 2012 before the streamlining and repeated the time spent questions from this survey in another online cross-sectional survey in 2014 just after the streamlined application round. In March 2012 and March 2014 (just after the funding round closed) email invitations to participate in the surveys were distributed to the Australian research community through existing networks from previous studies.1 2 The target group was researchers who had applied for NHMRC Project Grant funding. Researchers responded from March to May 2012 and March to April 2014, and took 10–20 min to complete the survey depending on how many additional comments they provided.

Our surveys were emailed via distribution lists as privacy considerations prevented the construction of a sampling frame from official sources. The only publicly available information is the list of previous Project Grant winners, but this would be biased towards more successful researchers.

Survey questions

In both surveys researchers were first asked how many applications they had submitted. Then for each application they were asked: their position on the application (first, second, etc), how much time they spent (in working days), and whether the application was new or a resubmission. Researchers were asked to estimate the time in full working days of 7.5 h, and to include time spent on weekends. This time could include: background reading, analysing pilot data, writing the application and the budget, attending meetings about the application, and entering data into the online submission form.

In the first survey we asked researchers to give their salary (in a range) so that we could cost their time. Owing to large amounts of missing data for the salary question we changed this in later surveys to academic position, from which salary can be estimated. To compare our two surveys we assume a salary above $140K indicates a professor.

In the second survey we asked researchers if they thought the new streamlined process was better than the old system. We also asked researchers to comment on their experience with the streamlined process.

Statistical methods

To investigate representativeness we compared our sample of researchers to the wider population of all researchers who applied using location and success rate.

We used a bootstrap procedure to estimate the mean time per application and the mean time for all applications.5 We used the bootstrap in order to convert the estimated times of individual researchers into estimated times per application involving multiple researchers as illustrated in figure 1. For each bootstrap application we first randomly generated the number of researchers from 1 to 10 using the observed frequency distribution from successful applications in 2013. We then randomly generated whether the application was a resubmission or not using a Binomial distribution based on data from our 2012 survey. We then randomly sampled researcher times conditional on the position of the researcher and whether it was a resubmission. We created times for 3727 applications, which was the number submitted in 2012. We calculated the average time per application and the time for all 3727 applications. We repeated the simulation 10 000 times and generated means, bootstrap CIs and bootstrap p values for the time per application and overall times.

Figure 1

Illustration of the bootstrap procedure for estimating total application time from individual researcher times where n is the total number of responses from researchers and P is the total number of applications submitted to the NHMRC.

The number of applications in 2014 was 3810 (2% more than 2012), but we use the 2012 number for both years so that the overall times before and after streamlining are independent of the increase in application numbers.

In our previous analysis we used a formula to estimate the time for all applications.1 We used the bootstrap here instead of a formula because it provides a distribution free method for estimating the uncertainty in the mean and hence comparing the mean times before and after streamlining. The estimated mean times from the bootstrap and the formula differed by only 0.5%.

To graphically compare the time spent on applications we smoothed the density of times before and after streamlining using a binned kernel density estimate with a bandwidth of 5 days.6 To simplify the comparison we only used the times of the lead researcher, and these researchers usually contribute the most amount of time. To compare the time spent by all researchers we used boxplots of time before and after streamlining by researcher position (first, second, third, etc). We tabulated the mean times by resubmission and researcher position, and estimated the mean difference and 95% CI using a regression model with a random intercept to adjust for multiple responses from the same researcher.

For the satisfaction question, we present mean percentages together with 95% CIs assuming a Binomial distribution.

The R package (V.3.0.2) was used for all analyses.

Results

Sample characteristics

The characteristics of the researchers and their applications are in table 1. Our sample was under-representative of the Go8 (eight leading universities in Australia), but the success rate of our sample in 2012 was the same as the observed success rate. The two samples were reasonably similar in terms of salary or academic level, and the percentage resubmissions. The average number of applications per researcher was 2.2 in 2012 and 3.1 in 2014, an increase of 0.9 (95% CI 0.7 to 1.1). The percentage of lead researchers decreased by 7% between 2012 and 2014 (95% CI 0.8% to 13.1%).

Table 1

Characteristics of surveyed researchers and their applications, and information from the National Health and Medical Research Council (NHMRC) where available

	Before streamlining (2012)n=446	After streamlining (2014)n=236	All NHMRC applications (2012)
Researchers
Number of applications, mean (SD)	2.2 (1.3)	3.1 (1.3)
Salary $140K+, n (%)	81 (18%)	NA
Academic level=professor, n (%)	NA	47 (20%)
Location=Go8*, n (%)	NA	95 (40%)	2374 (62%)
Location=research Institute, n (%)	NA	45 (19%)	578 (15%)
	n=685	n=440
Applications
Lead researcher, n (%)	387 (56%)	218 (50%)
Resubmission, n (%)	256 (37%)	180 (41%)
Success, n (%)	21%	NA	21%

Go8, Group of Eight university; NA, not available.

Application time

The average time spent on applications increased after streamlining. The average time per application increased from 34 working days (95% CI 33 to 35) to 38 working days (95% CI 37 to 39; 4-day increase, bootstrap p value <0.001). The overall time spent was 547 working years (95% CI 535 to 559) before streamlining and 614 working years after streamlining (95% CI 600 to 629). This is an increase of 67 working years after streamlining.

The densities of time spent are similar (figure 2), but after streamlining there was a smaller peak around 15 days and a larger probability around 27 days. The boxplots of time by position show that the first researcher usually did the majority of the work (figure 3). For most other researchers the time spent after streamlining was slightly larger. The summary statistics of time spent according to resubmission status and lead researcher in table 2 show that the mean times were fairly similar. The only decrease in time spent after streamlining was for resubmissions for the lead researcher.

Figure 2

Smoothed density of application time for the lead researcher in each survey. The lines show a binned kernel density estimate with a bandwidth of 5 days.

Figure 3

Boxplots of application times by researcher position and survey. Positions 6–10 are combined due to small numbers (under 10) when examining individual positions. The y-axis has been truncated at 60 days in order to focus on the majority of the data (99% of times were 60 days or under).

Table 2

Summary statistics on the average time spent on applications in working days by round, resubmission status and researcher position

Resubmission	Lead researcher	Before streamlining		After streamlining		Difference (after–before)
		n	Mean (SD)	n	Mean (SD)	Mean	95% CI
No	No	183	6.4 (7.7)	130	8.7 (14.7)	3.0	−0.1 to 6.1
	Yes	219	27.2 (23.1)	129	28.7 (20.6)	2.4	−3.1 to 7.9
Yes	No	103	4.0 (4.0)	91	6.5 (8.2)	3.1	1.0 to 5.3
	Yes	153	21.0 (13.4)	89	17.7 (9.5)	−3.0	−6.2 to 0.3

The estimated mean difference is from a regression model with a random intercept to adjust for multiple responses from the same researcher.

Most researchers agreed that the streamlined system was better with 75% (95% CI 69% to 81%) agreeing that it was better than in previous years.

Discussion

Streamlining the application system has not had the desired effect of reducing applicants’ time. Instead the amount of time spent has increased by an estimated 67 working years. This could be because the application time is independent of the length of the application and is instead dependent on the potential financial return or the competitiveness of the process.

Financial return

Researchers may be allocating a fixed amount of their year for preparing grant applications. This fixed allocation would be in proportion to the potential funding available. In 2013 the average awarded grant was AUD $650K (GBP £350K), and a success rate of 20% gives an expected return of $130K (GBP £70K). Allocating 1–2 months for this return is therefore a rational decision for those who believe they have an average chance of success. Pressure to apply also comes from colleagues and the university, and for researchers who are dependent on grant funding to continue working, there is no alternative than to dedicate large amounts of time to applications.

If researchers spend a fixed amount of time on applications then streamlining the system is still worthwhile because it means that they will spend more time on the sections that require the most thought (such as the scientific plan) and less time on ancillary and bureaucratic information. This is supported by many comments from our surveyed researchers, including:

I think I ended up spending more time on the science of the application itself because I had less ‘other’ parts to write.

The amount of time researchers spent logged on to the online submission system decreased by 50% in 2014 (personal communication, NHMRC staff member). As researchers primarily log-in simply to enter data then this reduction suggests that less time was spent on bureaucratic activities. Most researchers also preferred the streamlined system, with 75% of our survey respondents agreeing that it was better.

Response to competition

Researchers may be increasing their preparation time—regardless of the application process—because the competition is getting stronger. When the funding total is fixed, it is the relative performance of the applicant that determines success. Success rates for the Project Grant scheme have steadily declined from 23.5% in 2010 to a low of 16.9% in 2013. In a similar fashion to elite athletes whose relative effort in training determines success, increased competition for research funds means that success depends on even more carefully crafted applications requiring more time, or that more applications are submitted in order to increase the chance of getting some funding. Our respondents increased their average number of applications from 2.2 per researcher in 2012 to 3.1 in 2014 (0.9 increase, 95% CI 0.7 to 1.1).

A simulated study found that in competitive funding systems there was spiralling competition with researchers spending an average of 55% of their time writing applications.7 The simulated system became over-competitive which then inflated preparation time and decreased application quality.7 If competition is driving application time then saving researchers’ time will only be achieved by reducing competition. Reducing competition is achievable by: (1) increasing the funding pool, which is unlikely in the current economic climate or (2) decreasing the number of applications, which is possible via unpopular methods such as cooling off periods for unsuccessful applicants8 or levying a submission fee.9

Another potential approach to save time is to use a lottery to select applications that attain a certain standard.10 If this was adopted the incentive on the researcher would change significantly, as applications could be judged solely on the grounds that they are considered fundable, and the random allocation of funds amongst these fundable applications reduces the incentive to expend additional preparation effort. The New Zealand Health Research Council is using exactly this approach for its Explorer grants,11 and so there are opportunities to study how applicant behaviour changes with this approach.

More streamlining needed

The streamlining may not have been severe enough. Most time is spent on the scientific plan and track records, which were sensibly not cut. Time may only be saved if these key sections are reduced as part of the further planned streamlining.12

Another potential cause of the increase in time was that other changes were made to the application process, including formatting changes and changes to track records and references. These changes may have offset the benefits of streamlining as described by one researcher:

Any changes that were made to make it shorter were counteracted by the time spent adjusting the application to conform to the new structure and guidelines.

This may explain the increase in time after streamlining for non-lead researchers shown in figure 3, as they had to update their two-page track records to conform to the new system.

Limitations

Our survey participants were not randomly selected and therefore our results may not be an accurate representation of the wider population. We were limited in our ability to compare our sample with the wider population. We were under-representative of the leading eight universities, but we were representative in terms of success rate in 2012.

Our findings are based on retrospective self-reported times spent preparing proposals, and we could not verify these times. Participants completed our survey soon after the NHMRC closing date for submissions which should have reduced recall bias. Ideally participants would have prospectively completed a diary of their time and what they were doing, which would also help identify what parts of the application process take the most time. Researchers may have responded to both surveys, but we did not link their data as we did not collect any identifying information. This was a deliberate choice in order to make the survey anonymous and hence increase the number of responses.

Conclusions

Researchers invest lots of time in preparing applications and many potentially productive years of research are currently being lost on failed applications. Preventing this lost research time is a high priority in Australia, but streamlining applications may not be the answer. Answers may be found by spending some of the research budget on research into research,13 including experiments and pilot projects that examine alternative funding systems.14

Reducing wasted time is not just an issue in Australia. A survey of 29 funding agencies in 22 countries found that 12 reported that there were too many applications in the system, and 11 reported that the administrative burden was worse than 5 years ago.