The clinical relevance and newsworthiness of NIHR HTA-funded research: a cohort study.

OBJECTIVE: To assess the clinical relevance and newsworthiness of the UK National Institute for Health Research (NIHR) Health Technology Assessment (HTA) Programme funded reports. STUDY
DESIGN: Retrospective cohort study.
SETTING: The cohort included 311 NIHR HTA Programme funded reports publishing in HTA in the period 1 January 2007-31 December 2012. The McMaster Online Rating of Evidence (MORE) system independently identified the clinical relevance and newsworthiness of NIHR HTA publications and non-NIHR HTA publications. The MORE system involves over 4000 physicians rating publications on a scale of relevance (the extent to which articles are relevant to practice) and a scale of newsworthiness (the extent to which articles contain news or something clinicians are unlikely to know). MAIN OUTCOME MEASURES: The proportion of reports published in HTA meeting MORE inclusion criteria and mean average relevance and newsworthiness ratings were calculated and compared with publications from the same studies publishing outside HTA and non-NIHR HTA funded publications.
RESULTS: 286/311 (92.0%) of NIHR HTA reports were assessed by MORE, of which 192 (67.1%) passed MORE criteria. The average clinical relevance rating for NIHR HTA reports was 5.48, statistically higher than the 5.32 rating for non-NIHR HTA publications (mean difference=0.16, 95% CI 0.04 to 0.29, p=0.01). Average newsworthiness ratings were similar between NIHR HTA reports and non-NIHR HTA publications (4.75 and 4.70, respectively; mean difference=0.05, 95% CI -0.18 to 0.07, p=0.402). NIHR HTA-funded original research reports were statistically higher for newsworthiness than reviews (5.05 compared with 4.64) (mean difference=0.41, 95% CI 0.18 to 0.64, p=0.001).
CONCLUSIONS: Funding research of clinical relevance is important in maximising the value of research investment. The NIHR HTA Programme is successful in funding projects that generate outputs of clinical relevance.

The strength of this study is that a rigorous, extensive and independent approach to assessing the clinical relevance and newsworthiness of funded research provided by MORE was applied to the National Institute for Health Research (NIHR) Health Technology Assessment (HTA) reports.

A limitation of this study was that the data were reliant on funded research teams self-reporting their publications. It is most likely that the number of publications is under-reported from NIHR HTA-funded studies.

Raters contributing to McMaster Online Rating of Evidence are voluntary and are therefore likely to be interested in evidence-based medicine. Therefore, they may not be representative of all physicians.

Introduction

Commissioning new research in areas of high clinical relevance is an important part of minimising research waste. Chalmers and Glasziou1 in their paper on avoidable waste in research argue that selecting unimportant research questions is a major contributor to wasted research funds. Studies have shown, for example, that only moderate correlations exist between publicly funded research and burden of disease.2 Similarly, research conducted on patient or clinician research priorities has revealed potential mismatches between clinical priorities and funded research. Tallon et al,3 for example, examined the research priorities of patients with osteoarthritis of the knee and their clinicians and reported that their priorities differed from randomised controlled trials, which were typically focused on drug evaluations.

One proxy measure of the clinical relevance of funded research is the assessment of the relevance of the final publication. McMaster University has established a rigorous approach to assessing the clinical relevance and newsworthiness of outputs from health research.4 The McMaster approach involves selecting relevant papers from 120 high-impact journals selected on the basis of suggestions from librarians, clinicians, editors and editorial staff, Science Citations Index impact factors, a systematic examination of journal contents and an ongoing assessment of the number of articles that meet inclusion criteria of quality. Articles from selected journals are assessed by trained staff using inclusion criteria on methodological quality and whether the article has at least one clinically important outcome.4 Articles passing these criteria are entered onto the Critical Appraisal Process (CAP) system before being transferred to the McMaster Online Rating of Evidence (MORE) system, which has in excess of 4000 physician raters in over 60 physician disciplines. Articles are sent to at least four raters for each matched discipline/specialty, who rate papers on two seven-point scales (seven—highest score; table 1). The first scale of relevance assesses the extent to which the article is relevant to practice. If relevance is rated three or above, a second scale is used assessing the extent to which the article contains news or something that clinicians are unlikely to know. Articles that average three or above for a discipline are then transferred to a Premium LiteratUre Service (PLUS) database, and are then used to alert subscribers via Evidence Updates, ACP JournalWise and other current awareness services. Ratings, which are collected at the time of journal article publication, have been shown to be predictive of subsequent citation counts.5

Table 1

The McMaster Online Rating of Evidence (MORE) criteria for clinical relevance and newsworthiness

Score	Relevance to clinical practice in your discipline—criteria	Newsworthiness to clinical practice in your discipline—criteria
No score	Beyond my personal area of expertise but may be of interest to my discipline	I don't know if this is newsworthy
7	Directly and highly relevant	Useful information; most practitioners in my discipline definitely do not know this (unless they have read this article)
6	Definitely relevant	Useful information; most practitioners in my discipline probably do not know this
5	Probably relevant	Useful information; most practitioners in my discipline possibly do not know this
4	Possibly relevant: likely of indirect or peripheral relevance at best	Useful information; most practitioners in my discipline possibly already know this
3	Possibly not relevant	Useful information; most practitioners in my discipline probably already know this
2	Probably not relevant: content only remotely related	It probably does not matter whether they know this or not
1	Definitely not relevant: completely unrelated content area	Not of direct clinical interest

The McMaster approach to assessing clinical relevance and newsworthiness has enabled interesting comparisons to be made between research and publication types. One study explored the relevance and newsworthiness of high-quality original articles and systematic reviews from over 110 clinical journals and the Cochrane Database of Systematic Reviews.6 The study revealed that systematic reviews had a significantly higher rating for relevance than original articles, but a significantly lower rating for newsworthiness.6 In addition, clinicians accessed reviews more frequently than original articles.6

The National Institute for Health Research (NIHR) Health Technology Assessment (HTA) Programme was established in 1993 to commission research that is useful to clinical practice and NHS decision-makers.7 Since its inception, significant effort has been made to identify and prioritise important topics from a range of stakeholders and to assess applications on the basis of scientific merit, feasibility and value for money. In the 10 years since 2002, 98% of NIHR HTA-funded studies have completed with a final report in the HTA Journal8 and the journal has always been included in the McMaster's top 120 journals. The reports (also referred to as monographs) are peer reviewed, are available freely in the public domain and contain a full record of the study. Unlike typical peer reviewed journals, the HTA Journal has a large word guide of approximately 50 000 words and unlimited appendices, thus enabling more detail to be included than standard peer review publications. Given the emphasis on identifying and commissioning important topics and having easily accessible and usable final reports, this study aimed to assess the clinical relevance and newsworthiness of NIHR HTA reports compared with non-NIHR HTA publications assessed by the MORE process.

Methods

Selection of reports and publications

Of 314 studies published as full reports in the NIHR HTA Journal in the period 1 January 2007−31 December 2012, 311 were selected for the study. Three NIHR HTA reports were excluded from the analysis as they were updates of earlier Cochrane reviews, and therefore ratings of relevance and newsworthiness of these reports were not based on the first time these reviews were made available to the public. PubMed Identification numbers were identified for each report, which were obtained by hand searching PubMed.

Publications arising from NIHR HTA-funded studies publishing outside the HTA Journal in the period 1 January 2007−31 December 2012 were identified using research management databases maintained by the NIHR HTA Programme. Again, PubMed Identification numbers were obtained for each publication and these data were used to remove NIHR HTA Programme-funded research outputs from the database of articles from the 120 selected journals that had passed CAP and MORE criteria. A subgroup was identified for publications from studies reporting in the HTA Journal in the period 2007–2012. This enabled a direct comparison between reports in the HTA Journal and related publications from the same studies publishing outside the Journal.

All data were assessed by a second researcher to ensure that there were no duplications and that the publication and PubMed details were accurate. Data were held in an Excel spreadsheet, which was given to McMaster. The PubMed identification numbers were used to determine which articles were reviewed for and passed the CAP process, and then to automatically retrieve relevance and newsworthiness ratings from the MORE database.

NIHR HTA-funded studies publishing as full reports in HTA in the period 1 January 2007–31 December 2012.

Publications in the period 1 January 2007–31 December 2012 from NIHR HTA-funded studies publishing outside HTA for those studies included in category A above.

Publications from non-NIHR HTA-funded publications publishing in the period 1 January 2007–31 December 2012 from Journals included by McMaster and passing CAP and MORE criteria.

Units of analysis, hypotheses and statistical analyses

Two principal hypotheses were tested

Hypothesis 1: Reports published in HTA have a higher MORE clinical relevance and newsworthiness rating than non-NIHR HTA publications. The rationale for this hypothesis is that the NIHR HTA Programme aims to ensure that funded research responds to clinical need and has a rigorous process of identification and prioritisation to reflect clinical priorities. We therefore wanted to see if outputs from NIHR HTA funded research rated higher in terms of clinical relevance and newsworthiness.

Hypothesis 2: Reports published in HTA have a lower mean average MORE clinical relevance and newsworthiness rating than publications from the same research published outside the HTA Journal. The rationale for this is that funded research teams are encouraged to publish outside the HTA Journal, and therefore prior publication may affect the relevance ratings of the report if study findings are already in the public domain. Previous research had reported that some journal publications did precede publication in the HTA Journal, although it was recognised that this was not always the case and that for some studies, non-HTA Journal publications occurred well after the end of the project.9

Subgroup analysis was undertaken for hypotheses exploring differences in clinical relevance and newsworthiness ratings between original studies and review articles. Original studies are defined as ‘Any full text article in which the authors report first-hand observations’. Review articles are defined as ‘any full text article that was bannered “review, overview, or meta-analysis” in the title or in a section heading, or it was indicated in the text of the article that the intention was to review, summarise, or highlight the literature on a particular topic’.10 Previous research had identified that original research studies were more newsworthy but less clinically relevant than review articles.6 We wished to explore whether this was also the case for NIHR HTA reports.

As discussed elsewhere,6 article ratings were treated as continuous variables and were analysed by calculating mean averages and proportions and comparing CIs of computed results. SPSS V.20 was used for all analyses; differences were compared using p values.

Results

Proportions of reports and publications meeting CAP and MORE criteria

Of 311 NIHR HTA reports published in the period 2007–2012, 286 (92%) were included in the initial CAP assessment. Twenty-five reports were not included in the analysis due to a technical error, 18 of which were published in the period 20 February 2012–8 May 2012. Of the 274 HTA publications publishing outside the HTA Journal in the period 2007—2012, 118 (43.1%) were assessed for CAP. Non-inclusion of NIHR HTA-funded publications was due to the publications appearing in journals not included in the 120 journals assessed by McMaster.

Table 2 reveals that 67.8% (n=194) of NIHR HTA reports passed initial screening criteria of methodological rigour and relevance, compared with 57.6% (n=68) of other NIHR HTA-funded publications and 10.6% (n=20 194) of non-NIHR HTA publications included on the McMaster database. A total of 67.1% (n=192) of NIHR HTA reports passed MORE inclusion (ie, scoring ≥ 3/7 for both relevance and newsworthiness for at least one discipline), compared with 57.6% (n=68) of other NIHR HTA-funded publications and 10.5% (n=19 921) of non-NIHR HTA-funded publications.

Table 2

Proportion of publications passing CAP and MORE ratings of relevance and newsworthiness

	A: number (%) NIHR HTA reports 2007–2012	B: number (%) Publications 2007–2012 from NIHR HTA studies included in A	C: number (%) non-NIHR HTA publications2007–2012
All publications	286	118	190 391*
Number (%) passing CAP	194 (67.8)	68 (57.6)	20 194 (10.6)
Number (%) passing MORE in at least one discipline	192 (67.1)	68 (57.6)	19 921 (10.5)

*Estimated on PubMed searches of the number of articles published in all read journals, 2007–2012 (excluding editorials, letters).

CAP, critical appraisal process; HTA, health technology assessment; MORE, McMaster Online Rating of Evidence; NIHR, National Institute for Health Research.

Clinical relevance ratings in MORE

NIHR HTA reports had an average mean relevance rating of 5.48 (95% CI 5.36 to 5.59), which was higher than the mean relevance rating for non-NIHR HTA publications passing CAP and MORE criteria, which was 5.32 (95% CI 5.31 to 5.33) (mean difference=0.16, 95% CI 0.04 to 0.29, p=0.01; table 3). NIHR HTA-funded publications publishing outside HTA had an average mean relevance rating of 5.59 (95% CI 5.42 to 5.75), which was slightly higher than the rating for NIHR HTA reports (mean difference=0.11, 95% CI −0.10 to 0.32, p=0.419).

Table 3

Mean average MORE clinical relevance and newsworthiness ratings across publication types

	Number of publications	Average relevance (95% CI)	Average newsworthiness (95% CI)
A: NIHR HTA reports 2007–2012	192	5.48 (5.36 to 5.59)	4.75 (4.60 to 4.86)
B: publications 2007—2012 from NIHR HTA studies included in A	68	5.59 (5.42 to 5.75)	4.93 (4.73 to 5.13)
C: non-NIHR HTA publications 2007–2012	19 921	5.32 (5.31 to 5.33)	4.70 (4.69 to 4.71)

HTA,health technology assessment; NIHR,National Institute for Health Research.

The mean relevance rating of NIHR HTA original research reports was 5.54 (95% CI 5.34 to 5.74), which was higher than non—NIHR HTA original research publications, which had a mean relevance rating of 5.27 (95% CI 5.26 to 5.29) (mean difference=0.27, 95% CI 0.03 to 0.51, p=0.03; table 4). The mean relevance rating of NIHR HTA original research reports was similar to publications from the same studies publishing outside the HTA Journal, which had a mean average relevance rating of 5.57 (95% CI 5.36 to 5.78) (mean difference=0.03, 95% CI −0.32 to 0.26, p=0.875; table 4).

Table 4

Mean average clinical relevance and newsworthiness ratings for original research by publication type

	Number of publications	Average relevance (95% CI)	Average newsworthiness (95% CI)
A: NIHR HTA reports 2007–2012	54	5.54 (5.34 to 5.74)	5.05 (4.88 to 5.22)
B: publications 2007—2012 from NIHR HTA studies included in A	46	5.57 (5.36 to 5.78)	4.96 (4.69 to 5.23)
C: non-NIHR HTA publications 2007–2012	13 302	5.27 (5.26 to 5.29)	4.77 (4.75 to 4.79)

HTA,health technology assessment; NIHR,National Institute for Health Research.

The mean relevance rating of NIHR HTA-funded review reports was 5.46 (95% CI 5.32 to 5.60), which was similar to that for non-NIHR HTA-funded publications, which had a mean relevance rating of 5.41 (95% CI 5.39 to 5.43) (mean difference=0.05, 95% CI −0.09 to 0.19, p=0.46; table 5). NIHR HTA-funded review reports had a slightly lower mean relevance rating than publications from the same studies publishing outside the HTA Journal, which had a mean relevance rating of 5.62 (95% CI 5.38 to 5.87) (mean difference =0.16, 95% CI −0.052 to 0.20, p=0.41; table 5).

Table 5

Mean average clinical relevance and newsworthiness ratings for review research by publication type

	Number of publications	Average relevance (95% CI)	Average newsworthiness (95% CI)
A: NIHR HTA reports 2007–2012	138	5.46 (5.32 to 5.60)	4.64 (4.51 to 4.77)
B: publications 2007—2012 from NIHR HTA studies included in A	22	5.62 (5.38 to 5.87)	4.87 (4.56 to 5.17)
C: non-NIHR HTA publications 2007–2012	6619	5.41 (5.39 to 5.43)	4.56 (4.54 to 4.58)

HTA,health technology assessment; NIHR,National Institute for Health Research.

NIHR HTA-funded original research reports had a similar mean average relevance rating to NIHR HTA-funded review reports with a mean rating of 5.54 (95% CI 5.34 to 5.74) compared with 5.46 (95% CI 5.32 to 5.60) (mean difference=0.08, 95% CI −0.33 to 0.17, p=0.54).

Newsworthiness ratings in MORE

NIHR HTA-funded reports had a mean newsworthiness rating of 4.75 (95% CI 4.60 to 4.86), which was similar to that for non-NIHR HTA publications, which had a mean newsworthiness rating of 4.70 (95% CI 4.69 to 4.71; mean difference=0.05, 95% CI −0.18 to 0.07, p=0.402; table 3). NIHR HTA-funded publications publishing outside the HTA Journal had a higher average mean newsworthiness rating than NIHR HTA reports, with a rating of 4.93 (95% CI 4.73 to 5.13; mean difference=0.18, 95% CI −0.03 to 0.39, p=0.110; table 3).

The mean newsworthiness rating of NIHR HTA-funded original research was 5.05 (95% CI 4.88 to 5.22), which was higher than that for non-NIHR HTA original research publications, which had a mean newsworthiness rating of 4.77 (95% CI 4.75 to 4.79; mean difference=0.28, 95% CI 0.03 to 0.53, p=0.03; table 4). The mean newsworthiness rating for NIHR HTA-funded original research reports was similar to publications from the same studies publishing outside the HTA Journal, which had a mean average newsworthiness rating of 4.96 (95% CI 4.69 to 5.23; mean difference=0.09, 95% CI −0.04 to 0.22, p=0.63; table 4).

The mean average newsworthiness rating of NIHR HTA-funded review reports was 4.64 (95% CI 4.51 to 4.77), which was similar to that for non-NIHR HTA-funded reviews, which had a mean newsworthiness rating of 4.56 (95% CI 4.54 to 4.58; mean difference=0.08, 95% CI −0.05 to 0.21, p=0.24; table 5). The mean average newsworthiness rating of NIHR HTA-funded review reports was lower than that for the same NIHR HTA-funded reviews publishing outside the HTA Journal, which had an average rating of 4.87 (95% CI 4.56 to 5.17; mean difference=0.23, 95% CI −0.11 to 0.57, p=0.21; table 5).

NIHR HTA original research reports had a higher mean newsworthiness rating than NIHR HTA review reports with a mean rating of 5.05 (95% CI 4.88 to 5.22) compared with 4.64 (95% CI 4.51 to 4.77; mean difference=0.41, 95% CI 0.18 to 0.64, p=0.001).

Discussion

This study found that hypothesis one was correct in part: NIHR HTA studies reporting in HTA are rated as statistically more clinically relevant than non-NIHR HTA articles publishing in the 120 high-impact journals included by McMaster that pass CAP and MORE criteria. However, ratings of newsworthiness were similar between NIHR HTA studies reporting in HTA and non-NIHR HTA publications. The second hypothesis was not correct: ratings of clinical relevance and newsworthiness were slightly lower for NIHR HTA reports in HTA than publications from the same studies publishing outside the journal, although these differences were not statistically significant. NIHR HTA-funded original research reports publishing in the HTA Journal were seen to be more newsworthy than NIHR HTA-funded review reports.

The study also found that the proportion of NIHR HTA reports and other publications from NIHR HTA-funded research passing initial screening criteria was higher than that of non-NIHR HTA-funded publications (67.8% and 57.6%, compared with 10.6%, respectively). This provides a further indication of the comparatively high relevance and methodological rigour of NIHR HTA-funded studies.

It is interesting to note that subsequent to the completion of this study, another project has been completed that suggests that the assumptions behind the second hypothesis were not correct. Chinnery et al assessed the time to publication of projects funded by the NIHR HTA Programme publishing in the HTA Journal and as a journal article in the wider biomedical literature.11 They examined NIHR HTA projects with a planned submission date for the draft final report on or before 9 December 2011 and reported that for primary research the median time to publication in the HTA Journal was faster than that in other non-HTA Journal publications (26.5 and 35.5 months, respectively). This indicates that further exploration of the clinical relevance and newsworthiness of NIHR HTA reports compared with publications from the same studies publishing outside the HTA Journal is warranted.

The strength of this study is that a rigorous, extensive and independent approach to assessing the clinical relevance and newsworthiness of funded research provided by MORE was applied to NIHR HTA reports. However, there were limitations to the study. First, the study used internal research management data that were reliant on funded research teams self-reporting their publications. It is likely that the number of publications is under-reported from NIHR HTA-funded studies and therefore some NIHR HTA publications may be included in the non-NIHR HTA publication dataset. Second, the study compared NIHR HTA reports with non-NIHR HTA publications passing CAP and MORE. NIHR HTA-funded studies are typically randomised controlled trials, systematic reviews or Technology Assessment Reports (which identify, assess and synthesise research evidence from a number of healthcare interventions, providing estimates of the relative effectiveness and cost effectiveness of a range of interventions). While subgroup analyses matched original and review research, it could be more effective to match publications by specific research designs (eg, randomised controlled trials) or by similar research funders. Third, the MORE scales were treated as interval scales and therefore mean average ratings were obtained. However, the data may be viewed as ordinal, in which case median or mode averages would be more appropriate. Fourth, raters contributing to MORE do so on a voluntary basis and are likely to be interested in evidence-based medicine, and therefore may not be representative of all physicians. Fifth, no subgroup analyses were conducted comparing articles from NIHR HTA-funded studies published elsewhere prior to publication in the HTA Journal and those published after the HTA report. Conducting such subgroup analyses will provide interesting data on the impact of prior publication on clinical relevance and newsworthiness ratings.

Findings from this study contribute to the relatively under-researched area of clinical relevance of health research. Commissioning research topics in areas of low relevance or in areas where the research evidence is already known is a significant contributing factor to research waste.1 The MORE process has previously been used to provide exploratory assessments of clinical research by research type. McKinlay et al,6 for example, reported that systematic reviews were rated significantly higher than original articles for relevance, but significantly lower for newsworthiness. This study reflected these findings in part with NIHR HTA-funded original research reports having a similar mean average relevance rating to review reports, but a statistically higher mean newsworthiness rating than review reports. The lower newsworthiness rating of reviews is to be expected, given that all the original articles which inform the reviews are already published and thus are in the public domain.6

The high clinical relevance rating of NIHR HTA reports in relation to non-NIHR HTA-funded research could be explained by the NIHR HTA Programme's identification and prioritisation processes, which are specifically designed to ensure that important knowledge gaps are identified. Topics are generated via a wide range of stakeholder groups and sources, such as web submissions, workshops with clinicians, academics and patient and public representatives and outputs from the James Lind Alliance Priority Setting Partnerships (http://www.lindalliance.org/). These are screened by specialists and an ‘overlap check’ is undertaken to ensure that there is no duplication of current or completed research internally and externally, after which topics are developed and prioritised by panels comprising independent experts and patient and public contributors. The higher clinical relevance rating of NIHR HTA reports may therefore be attributed to the effectiveness of these processes.

An additional factor may be due to the fact that the NIHR HTA Programme has its own journal HTA, which enables fuller descriptions of research than most other journals allow. The accessibility and structure of the reports in HTA may be a contributing factor to enhancing assessments of relevance. However, all NIHR HTA-funded studies are contractually obliged to report in HTA regardless of the study outcome, resulting in a high publication rate of funded studies.8 The HTA Journal is therefore not as likely to be affected by publication bias as other health research literature, which could therefore limit ratings of relevance and newsworthiness when compared to other journals.

This study presents an exploratory assessment of clinical relevance and newsworthiness by assessing the outputs of one research funder. While this provides an initial assessment of relevance and newsworthiness in relation to highly rated publications in high-impact journals, further research is recommended to confirm the observations from the study. In particular, it is recommended that further analyses are undertaken on outputs that are matched by research type (eg, RCTs, systematic reviews), research funders or specific journals. Further research should also be conducted on assessing the degree to which prior publication of NIHR HTA-funded studies influences ratings of clinical relevance and newsworthiness of HTA reports.

Commissioning or publishing research of low-clinical relevance is an important factor in the waste of research funds. It is recommended, therefore, that careful consideration is given to clinical relevance when funding and disseminating research.

Conclusion

Research commissioned by the NIHR HTA Programme and published in the HTA Journal is more clinically relevant than highly rated articles publishing in the 120 journals included in MORE. Reports publishing in the HTA Journal had similar ratings of newsworthiness. NIHR HTA-funded original research reports had a similar mean average relevance rating to review reports, but had a significantly higher newsworthiness rating.

The clear and full accounts provided by the HTA Journal and the rigorous topic identification and prioritisation processes adopted by the NIHR HTA Programme may be a factor in generating the high clinical relevance rating. This study illustrates the importance of considering clinical relevance when commissioning and disseminating research in an effort to minimise waste in research funding.