Directing research funds to the right research projects: a review of criteria used by research organisations in Australia in prioritising health research projects for funding.

OBJECTIVES: Healthcare budgets are limited, and therefore, research funds should be wisely allocated to ensure high-quality, useful and cost-effective research. We aimed to critically review the criteria considered by major Australian organisations in prioritising and selecting health research projects for funding.
METHODS: We reviewed all grant schemes listed on the Australian Competitive Grants Register that were health-related, active in 2017 and with publicly available selection criteria on the funders' websites. Data extracted included scheme name, funding organisation, selection criteria and the relative weight assigned to each criterion. Selection criteria were grouped into five representative domains: relevance, appropriateness, significance, feasibility (including team quality) and cost-effectiveness (ie, value for money).
RESULTS: Thirty-six schemes were included from 158 identified. One-half of the schemes were under the National Health and Medical Research Council. The most commonly used criteria were research team quality and capability (94%), research plan clarity (94%), scientific quality (92%) and research impact (92%). Criteria considered less commonly were existing knowledge (22%), fostering collaboration (22%), research environment (19%), value for money (14%), disease burden (8%) and ethical/moral considerations (3%). In terms of representative domains, relevance was considered in 72% of the schemes, appropriateness in 92%, significance in 94%, feasibility in 100% and cost-effectiveness in 17%. The relative weights for the selection criteria varied across schemes with 5%-30% for relevance, 20%-60% for each appropriateness and significance, 20%-75% for feasibility and 15%-33% for cost-effectiveness.
CONCLUSIONS: In selecting research projects for funding, Australian research organisations focus largely on research appropriateness, significance and feasibility; however, value for money is most often overlooked. Research funding decisions should include an assessment of value for money in order to maximise return on research investment. © Author(s) (or their employer(s)) 2018. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

The first critical review of research project selection criteria from a funder perspective in Australia.

A comprehensive review of available funding schemes, selection criteria and scoring weights to prioritise research proposals.

The recommendations provided will help research organisations streamline funding to worthy projects to maximise return on research investment.

The review takes an Australian perspective, but the findings and recommendations maybe applicable to other jurisdictions.

Introduction

Research is vital to generate evidence to guide medical decision making and improve health. Therefore, the Australian Government and various research organisations allocate considerable resources to fund clinical trials and other health research. The total expenditure on health research in Australia was around $A5.4 billion in 2014.1 Recently, the Australian Government has announced the establishment of the $A20 billion Medical Research Future Fund, which aims to improve health, contribute to a sustainable health system and provide significant economic benefits.2 There has been an emerging interest in Australia and internationally to maximise value and reduce waste in healthcare research.3–5 Although research value should be ensured throughout the continuum (ie, from research question development to implementation of the findings), directing research funds to the right research projects in the first place is key to optimise health and economic benefits from healthcare research. This is typically achieved at two levels: 1) selecting strategic research areas or topics (eg, indigenous health or cancer) to guide overall research activity and commissioning and 2) selecting specific research projects for funding from proposals put forward by researchers.6 7

Most research projects in Australia are investigator-initiated and researchers must seek financial support for their proposals through research funding organisations (eg, the National Health and Medical Research Council (NHMRC)). However, the overall funds available for research are limited compared with the number of submissions and decisions have to be made about the best way to distribute research funds. Thus, funding organisations need to have a transparent and systematic way to evaluate and prioritise research projects for funding.5 8 9 This is often done based on the assessments of the merits of the submitted proposals according to the judgements of experts sitting on funding panels.8 10 In this process, submitted proposals are assessed and scored against predefined criteria with each criterion, or group of criteria, being assigned a weight reflecting its relative importance. Such practice corroborates with the recommendations of many international initiatives for setting research priorities where the use of explicit and comprehensive criteria is encouraged to ensure that important considerations are not overlooked during the selection process.9 11–14 In general, these criteria may include burden of the disease, equity, scientific rigour, research team capabilities, innovation and impact of research results. The choice of criteria and the scoring system may differ, depending on the needs of stakeholders involved in this exercise.9 11–14 Literature examples on prioritising research topics using explicit criteria are abundant9 11–14; however, there is a dearth of articles that provide a clear critical insight on the criteria used to select research projects from research proposals competing for funding.15

While health research funding decisions in Australia rely heavily on the ability of research proposals to meet selection criteria, it is unknown what criteria are more commonly used by research funders, how these criteria and their weights vary across funding organisations, and whether these criteria are comprehensive enough to capture all important considerations to ensure high quality and value for money research. This knowledge is important to assess the current approach of selecting and funding research projects, and to guide future efforts to optimise health research funding mechanisms in the country. Therefore, the aim of this paper was to critically review the criteria considered by major Australian research organisations in their selection of health research projects for funding.

Methods

Patient and public involvement

Patients and public were not involved.

We reviewed all research funding schemes listed on the Australian Competitive Grants Register (ACGR), which provides a comprehensive list of funding schemes that have been approved by the Australian Government as being competitive research grants.16 The identified schemes were included if they were health related, active in 2017 and had clear selection criteria which were publicly available on the funders’ websites. Health research refers to research with human health or medical purpose, including research on the aetiology, diagnosis or management of disease, mental condition or behaviour in human. To focus on schemes for funding research projects and programmes, research schemes dedicated solely to training, capacity building, equipment or infrastructure were excluded. These include fellowships, awards and scholarships as well as research and training centres.

Data extracted included scheme name, year first implemented, funding organisation, selection criteria and the relative weight assigned to each criterion. Selection criteria were grouped into five representative domains: relevance (ie, why should we do it? including the burden of disease and level of existing knowledge), appropriateness (ie, should we do it? including scientific rigour and suitability to answer the research question), significance of research outcomes (ie, what will we get out of it? including impact and innovation), feasibility (ie, can we do it? including team quality and research environment) and cost-effectiveness (ie, is the proposed research potentially good value for money?).9 12 The domains were selected based on the lists of criteria and categories suggested in comprehensive tools for research prioritisation including the Essential National Health Research Approach (relevance, appropriateness, feasibility and significance),12 Child Health and Nutrition Research Initiative (answerability, effectiveness, deliverability and impact)11 and the Checklist for Health Research Priority Setting (benefits, feasibility and cost-effectiveness).9 Disagreements related to assigning criteria to their representative domains were either resolved by discussion or the involvement of a third reviewer who was provided with the full assessment or selection criteria for consensus decision-making. A domain was counted under a given scheme if at least one criterion within that domain is reported in the selection criteria of that scheme. Table 1 provides a description of the representative domains.

Table 1

Description of domains and relevant criteria9 12

Domain	Definition
Relevance	The key question for this domain is “why should we do it?” The proposed research is pertinent to the health problems of interest. It takes into consideration burden of disease, equity, alignment with national/organisational objectives and the level of existing knowledge in relation to the intervention.
Appropriateness	The key question for this domain is “should we do it?” The proposed research is well suited to answer the decision problem (ie, answerability). It takes into consideration ethical, moral and legal acceptability, and scientific rigour.
Significance	The key question for this domain is “what will we get out of it?” It represents the benefit of implementing/translating the research results. It takes into consideration the impact on health, innovation and ability to foster capacity building and collaboration.
Feasibility	The key question for this domain is “can we do it?” The focus is on the chances of research success. It considers team quality (track record) and capability, research environment and the research plan.
Cost-effectiveness	The key question for this domain is ‘is the research cost-effective?’ This theme focuses on the value for money of the research proposal and budget justification. It considers the costs and expected benefits of conducting research.

Results

Thirty-six schemes met our inclusion criteria from 158 schemes listed on the 2017 ACGR. Figure 1 summarises the review process.

Figure 1

Flow chart of the review.

One-half of the schemes were under the NHMRC. Five schemes (14%) were specifically for research in cancer, four (11%) for dementia, four (11%) for mental health and three (8%) for diabetes. A summary of the selection criteria considered by each scheme is presented in table 2. Further details on the selection criteria and scoring weights are provided in online supplementary file.

Table 2

Selection criteria domains for schemes and funding organisations

Organisation and scheme name	Relevance			Appropriateness			Significance				Feasibility				Cost-effectiveness
	Burden	National/ Organisational priorities	Existing knowledge	Scientific quality	Answerability	Ethical/ Moral	Innovation	Impact	Translation/ Implementation	Capacity/ Collaboration	Research team quality	Environment	Stakeholders involved	Research plan	Budget justification	Value for money
NHMRC
Boosting Dementia Research Grants32	-	✔	-	✔	-	-	✔	✔	✔	-	✔	-	✔	✔	✔	✔
Dementia Research Team Grants33	-	✔	✔	✔	✔	-	✔	✔	✔	✔	✔	-	✔	✔	-	-
Development Grants34	✔	✔	-	✔	-	-	✔	✔	✔	-	✔	✔	-	✔	-	-
Global Alliance for Chronic Diseases—Chronic Lung Disease35	-	✔	✔	✔	✔	-	✔	✔	✔	-	✔	-	✔	✔	-	-
Global Alliance for Chronic Diseases—Mental Disorders35	-	✔	✔	✔	✔	✔	✔	✔	✔	-	✔	-	✔	✔	-	-
Global Alliance for Chronic Diseases—Type 2 Diabetes Countries35	-	✔	✔	✔	✔	-	✔	✔	✔	-	✔	-	✔	✔	-	-
National Institute for Dementia Research Grants36	-	✔	-	✔	✔	-	✔	✔	✔	-	✔	✔	✔	✔	✔	✔
NHMRC/NSFC—Prediction and Treatment of Type 2 Diabetes37	-	✔	-	✔	✔	-	-	✔	✔	✔	✔	✔	-	✔	-	-
Northern Australia Tropical Disease Collaborative Research Programme38	-	-	-	✔	✔	-	✔	✔	✔	✔	✔	-	✔	✔	-	-
Partnership Projects39	-		-	✔	✔	-	✔	✔	✔	-	✔	-	-	✔	-	-
Programme Grants40	✔	-	-	✔	-	-	✔	✔	✔	✔	✔	✔	-	✔	-	-
Project Grants41	-	-	-	✔	-	-	✔	✔	✔	-	✔	-	-	✔	-	-
Targeted Call—Engaging Young Adults to Improve Eating Behaviours and Health Outcomes42	-	✔	-	✔	✔	-	-	✔	✔	-	✔	-	✔	✔	-	-
Targeted Call—Mental Health: Suicide Prevention in Aboriginal and Torres Strait Islander Youth43	-	-	-	✔	✔	-	-	✔	✔	-	✔	-	✔	✔	-	-
Targeted Call—Fetal Alcohol Spectrum Disorder among Aboriginal and Torres Strait Islander Peoples44	-	-	✔	✔	✔	-	-	✔	-	-	✔	-	✔	✔	-	-
Targeted Call—Preparing Australia for the Genomics Revolution45	-	✔	-	✔	✔	-	-	✔	✔	-	✔	-	✔	✔	-	-
Targeted Call—Wind Farms and Human Health46	-	✔	-	✔	✔	-	-	✔	✔	-	✔	-	✔	✔	-	-
Translational Research Projects for Improved Healthcare47	-	✔	-	-	-	-	-	✔	-	-	✔	-	-	✔	-	✔
Cancer Australia
Priority-driven Collaborative Cancer Scheme48	-	✔	-	-	-	-	-	✔	✔	✔	✔	-	✔	-	-	-
Support for Cancer Clinical Trials Programme—Existing National Cooperative Oncology Groups49	-	-	-	✔	-	-	-	-	✔	✔	-	-	-	✔	✔	✔
National Breast Cancer Foundation
Accelerator Research Grant50	-	✔	-	✔	-	-	-	✔	✔	-	✔	-	-	✔	-	-
Innovator Grant51	-	✔	-	✔	-	-	✔	✔	-	-	✔	-	-	✔	-	-
Department of Foreign Affairs and Trade
Tropical Disease Research Regional Collaboration Initiative52	-	✔	-	✔	✔	-	-	✔	✔	✔	✔	-	-	✔	✔	✔
Alzheimer’s Australia Dementia Research Foundation
Dementia Grants Programme53	-	-	-	✔	✔	-	✔	✔	-	-	✔	-	-	✔	-	-
Australian and New Zealand College of Anaesthetists
ANZCA Research Grants Programme54	-	-	-	✔	-	-	-	-	-	-	✔	-	-	✔	-	-
Australian Rotary Health
Mental Health Research Grants55	-	-	-	✔	-	-	✔	✔	-	-	✔	-	-	✔	-	-
Bupa Foundation
Bupa Health Foundation56	-	-	-	-	-	-	-	✔	-	-	✔	-	-	✔	-	-
Cure for MND Foundation
Translational Research Grants57	-	-	-	✔	-	-	-	✔	✔	-	✔	✔	-	✔	-	-
Diabetes Australia Research Trust
General Grants58	-	-	✔	✔	-	-	✔	✔	✔	-	✔	-	-	✔	-	-
Healthway (Western Australian Health Promotion Foundation)
Health Promotion Intervention Research Grants59	-	✔	✔	✔	-	-	-	✔	✔	✔	✔	-	✔	✔	✔	✔
HCF Research Foundation
Health Services Research Grants60	✔	✔	✔	✔	-	-	-	✔	✔	-	-	-	-	✔	-	-
Motor Neurone Disease Research Institute of Australia
Motor Neurone Disease Research Grants61	-	✔	-	✔	-	-	-	-	-	-	✔	-	-	-	-	-
Multiple Sclerosis Research Australia
Research Grants62	-	✔	-	✔	✔	-	✔	✔	-	-	✔	-	-	✔	-	-
National Heart Foundation of Australia
Vanguard Grants63	-	✔	-	✔	-	-	-	✔	✔	-	✔	✔	-	✔	-	-
Prostate Cancer Foundation of Australia
New Concept Grant64	-	-	-	✔	-	-	✔	✔	-	-	✔	✔	-	✔	-	-
The Movember Group and beyondblue
Australian Mental Health Initiative65	-	✔	-	✔	✔	-	✔	✔	✔	-	✔	-	✔	✔	-	-

ANZCA, Australian and New Zealand College of Anaesthetists; HCF, The Hospitals Contribution Fund of Australia; MND, motor neuron disease; NHMRC, National Health and Medical Research Council; NSFC, National Natural Science Foundation of China.

The most commonly used criteria were research team quality and capability (94%), research plan clarity (94%), scientific quality of the proposal (92%) and research impact (92%). Criteria considered less commonly were existing knowledge (22%), fostering collaboration (22%), research environment (19%), budget justification (17%), value for money (14%), disease burden (8%) and ethical/moral considerations (3%). When selection criteria were grouped into relevant domains, all schemes considered feasibility criteria, 94% of the schemes considered significance, 92% considered appropriateness, 72% considered relevance and only 17% considered cost-effectiveness. Only five schemes (14%) considered all five domains namely, NHMRC National Institute for Dementia Research Grants, NHMRC Boosting Dementia Research Grants, Cancer Australia Clinical Trials Programme, the Department of Foreign Affairs and Trade Tropical Disease Research, Health Promotion Intervention Research Grants. Figure 2 depicts the distribution of selection criteria.

Figure 2

Overall research criteria and their representative domains.

When reported, the relative weights for the selection criteria also varied across schemes with 20%–75% for feasibility, 20%–60% for each appropriateness and significance, 15%–33% for value for money and 5%–30% for relevance criteria.

Discussion

Using a predefined set of selection criteria is a transparent approach to select and prioritise high-quality research projects for funding. Typically, the relevance of research proposals is gauged with criteria that are mostly related to the project’s ability to advance knowledge7; however, these criteria should also reflect the mandate of the funding organisation and the purpose of the funding scheme.9 12 A broad range of criteria were reported in the included schemes with a clear focus on the quality of the research team, research plan, scientific rigour, impact and translation/implementation potential. The identified schemes, within the same organisation and across organisations, had variable selection criteria and scoring weights. When grouped into representative domains, funding organisations in Australia appear to focus on research relevance, appropriateness, significance and feasibility; however, cost-effectiveness of research projects was largely overlooked. The observed variation in criteria and scoring weights in our review may be justified by the different emphasis placed on certain aspects to achieve the outcomes sought under each scheme. For instance, collaborative and partnership schemes focused on partnership strengths, collaborative gains and team integration.

The only aspect that was considered by all schemes was research feasibility with a clear emphasis on the quality of the research team. Team quality and capability (often based on past performance) is vital to ensure that the funded research projects can be effectively conducted within the time and budget specified; nevertheless, over relying on this criterion may result in giving disproportionate share of funding to established teams at the expense of more novel and innovative projects. This bias can be reduced by introducing initiatives that fund innovative research ideas with high impact potential such as the ‘Grand Challenges’ initiative and the NMRC ‘Ideas Grants’.5 6 Notably, there are important criteria that were not considered by most of the included schemes. Equity considerations were not explicitly mentioned as a selection criterion, and ethical/moral considerations were only considered in one scheme. This might be explained by the implicit assumptions that all submitted proposals will be approved by ethics committees and that equity is addressed by targeted research grants (eg, Research in Aboriginal and Torres Strait Islanders) or considered during final deliberations to select proposals for funding. However, without clarity about where the responsibility for ethical and equity considerations lies there is the potential for these criteria to be overlooked.

There has been a shift away from exclusive technical merit-review of proposals towards relevance of research funding as judged by multiple stakeholders.7 17 18 Research organisations, particularly if publicly funded, are increasingly expected to make the best use of taxpayers’ money to increase value and reduce waste when research priorities are set.19 20 For example, Chalmers et al recommended to engage potential users of research in research prioritisation, justify additional research by systematic reviews to show what is already known and to periodically monitor and analyse impact of funded research.4 5 21 Around 40% of the schemes in our study considered stakeholders’ involvement (ie, consumers and/or clinicians); however, the level of stakeholders’ engagement and influence on funding decisions was unclear. Considering the needs and inputs of various stakeholders such as patients, caregivers, clinicians and decision makers is essential to fund research that is useful to solve real-life problems. The experience of the Patient-Centred Outcomes Research Institute in the USA suggested that involving patients and stakeholders alongside scientists in reviewing research applications is influential in panel discussions and merit review outcomes.18 22 Despite its importance to avoid research duplication, only 22% of the identified schemes considered existing knowledge, but none of the schemes explicitly required a systematic review of literature to demonstrate knowledge gaps. Although conducting systematic reviews to identify knowledge gaps may not be required when responding to targeted research calls or when research is commissioned, since evidence review is often conducted by the commissioning organisations, showing what is already known should be required by researchers submitting investigator-initiated proposals. Our results echo the findings of a review of the extent to which 11 international organisations.3 In that review, only one organisation required reference to relevant systematic reviews in all funding applications and four funders required systematic reviews for funding clinical trials.3

Another important value aspect is the impact of funded research. Research impact broadly refers to generated benefits in terms of knowledge production, informing policy, capacity-building, health benefits and broader social and economic benefits.19 23 The presence of multidimensional benefits reflects how the definition of impact varies with the perspectives of different stakeholders as patients, clinicians, government, industry and academia.19 23 24 The majority of the schemes in our review considered research impact as a funding criterion with elements including advancing knowledge, improving health outcomes and scientific publications; however; the schemes did not specify how these benefits should be measured and presented in funding proposals. Greenhalgh et al have reviewed established approaches to measure impact (eg, Research Impact Framework, Canadian Academy of Health Sciences and UK Research Excellence Framework).19 They have concluded that approaches to impact assessment differed according to the circumstances and purpose of funding, they also noted that the most robust approaches are complex and labour-intensive and called for research on research impact.19 Of note, most of these approaches were designed to measure research impact retrospectively, that is, to evaluate the benefits of particular research programmes that have already been conducted; notwithstanding, funding organisations and researchers need prospective approaches to infer the benefit of new research to support research funding decisions. Incorporating impact evaluation frameworks into the priority-setting processes is a necessary requirement that should be studied.7

Importantly, funding organisations may implicitly assume that selecting high impact projects would ensure value for money; nevertheless, value for money cannot be established without explicitly comparing the costs and expected benefits of proposals competing for funding.6 25 26 This is because research budgets are finite and decisions must be made about how to allocate these funds (ie, which research proposals should be funded) to maximise benefits. Failure to consider this aspect brings the risk of funding research projects where the costs of conducting research outweigh the expected research benefits (ie, research projects that are not cost-effective). This would result in ‘opportunity cost’, which is the benefit forgone elsewhere by adopting suboptimal choices.27 Interestingly, none of the schemes that required demonstration of value for money provided guidelines on how the cost-effectiveness of research projects should be performed and presented. Of note, there are rigorous analytical methods to prospectively quantify the expected benefits of research on improving health outcomes, the key analytical approaches are the ‘prospective payback of research’ (a similar approach to return on investment) and the value of information approach.6 8 Under the payback approach, the value of a research study is typically inferred from its ability to result in a beneficial change in clinical practice.28 The value of information approach, on the other hand, considers the uncertainty in the relevant available evidence (eg, from systematic reviews and meta-analyses) and the consequences of this uncertainty (eg, implementing a suboptimal intervention).25 29 Research benefits calculated by these approaches are scaled up by considering the population expected to benefit from research results over time, and these benefits are compared with research budget to inform cost-effectiveness.26 30 It should be acknowledged; however, that assigning monetary value to research benefits and conducting economic evaluation of research proposals may not be acceptable or feasible (eg, due to capacity considerations) by some jurisdictions. Therefore, the decision making context and the availability of resources to conduct such analyses should be carefully considered before incorporating these approaches into priority-setting processes. Furthermore, it should be emphasised that the cost-effectiveness criterion should not be the only consideration when making research funding decisions. It is recommended that cost-effectiveness be used to supplement (ie, in combination with) other considerations that deemed important to the stakeholders.8 23

A limitation to our work is that we only reviewed active grant schemes listed on the ACGR; and therefore, some grant schemes may not have been included in our review; however, the ACGR is a comprehensive registry of major research grants by leading funding organisations in Australia. Additionally, it is noted that selection criteria, and schemes, change over time to meet political and administrative objectives. For example, the NHMRC is revising grant schemes as well as the selection criteria and processes for a new series of grants to commence funding in 2019.31 In addition, our review was limited by the amount of publicly available information for each scheme, and thus, we could not extract some important elements as scheme budgets and the knowledge generation and translation frameworks adopted by various funding organisations. The next step for this research would be to engage with funding organisations to gain further insights on their approaches to prioritise research proposals for funding.

In conclusion, healthcare research is vital to improve health; however, there is a need to ensure that funded research is relevant and value for money. In selecting research projects for funding, Australian research funding organisations focus on research appropriateness, significance and feasibility; nevertheless, other important criteria should not be overlooked such as equity and stakeholders’ engagement. Importantly, research funding decisions should include an assessment of value for money in order to maximise return on research investment.