What is the impact of innovation on output in healthcare with a special focus on treatment innovations in radiotherapy? A literature review.

OBJECTIVE: To analyse how often innovations in healthcare are evaluated regarding output, especially in radiotherapy. Output was defined as either survival, toxicity, safety, service, efficiency or cost-effectiveness.
METHODS: A systematic literature review was conducted, using three search strategies: (1) innovations in general healthcare; (2) radiotherapy-specific innovations, i.e. organizational innovations and general implementation of innovations; (3) innovations per tumour group/radiotherapy technique. Scientific levels were classified according to the system used in European Society for Medical Oncology guidelines. Finally, we calculated the percentage of implemented innovations in Dutch radiotherapy centres for which we found evidence regarding output in the literature review.
RESULTS: Only 94/1072 unique articles matched the inclusion criteria. Significant results on patient outcome, service or safety were reported in 65% of papers, which rose to 76% if confined to radiotherapy reviews. A significant technological improvement was identified in 26%, cost-effectiveness in 10% and costs/efficiency in 36% of the papers. The scientific level of organizational innovations was lower than that of clinical papers. Dutch radiotherapy treatment innovations were adequately evaluated on outcome data before implementation in clinical routine in a minimum of 64-92% of cases.
CONCLUSION: Only few studies report on output when considering innovations in general, but radiotherapy reviews give a reasonably good insight into innovation output effects, with a higher level of evidence. In Dutch radiotherapy centres only small improvements are possible regarding evaluation of treatment innovations before implementation. Advances in knowledge: This study is the first of its kind measuring how innovations are evaluated in scientific literature, before implementation in clinical practice.

Introduction

Radiotherapy centres and other healthcare providers have the task of simultaneously improving patient outcomes, patient safety, patient service and cost-effectiveness.[1] In the past decades, the implementation of innovations, from new products, services and technologies to systematic changes, has helped make great progress in coping with this complex task. For example, in radiotherapy, technological innovations have improved the precision of radiation therapy, resulting in improved patient outcomes.[2,3] Despite medical advances, the literature on innovation also shows that some innovations that are not yet well evaluated and therefore have not been proven to be effective nevertheless diffuse rapidly. By contrast, other innovations with high potential sometimes show a slow uptake in practice.[4,5] In both cases, this may lead to disappointments or even damaging effects for care outcomes and patients. It is not clear whether this is also the case in radiotherapy. On the one hand, this medical discipline is increasingly evidence based, with many large randomized trials of high methodological quality.[6] On the other hand, however, in radiotherapy innovative technologies are also sometimes rapidly introduced into clinical practice without sufficient patient data, based on the presumption that dosimetric advantages will eventually lead to better treatment outcomes.[7]

Thousands of studies on innovation in healthcare have looked into the effect of one single innovation intervention or a series of related interventions, on one or a few possible output dimensions. The limited scope of these studies makes it hard to draw scientific conclusions about the relationship between innovation and output in general and/or over time.

To prevent damaging effects caused by implementing innovations too rapidly or too slowly, and to improve cost effectiveness, it is necessary to get a better understanding of how innovation by care providers and especially radiotherapy centres affects performance. For example, the majority of the increase in healthcare spending is attributed to technological innovations.[8] In radiotherapy, too, innovative technologies often involve substantial investments in terms of equipment, quality assurance and additional training of staff, while it is often uncertain to what extent the innovations will translate into better patient outcomes.[7] The literature does not always provide a clear picture of how to evaluate innovations, for example, because scientists argue that the generation of innovations is not suited to strict evaluation or that innovations are constantly evolving.[9-14] However, from a societal perspective, it is necessary to gain better insight into the expected benefits of a proposed innovative intervention in clinical practice versus the expected additional costs.

It is therefore desirable to evaluate innovations not on just one dimension of output, but on the total performance. In the literature, this is defined as a combination of effectiveness and efficiency.[15] Here, effectiveness refers to external criteria used to evaluate the products and services; from a patient’s perspective, we relate this in our study to patient outcomes and/or patient safety and/or service to the patient. Efficiency refers to how successful these products, services or treatments are in relation to their costs.

In healthcare, and thus also in radiotherapy, specific cost-effectiveness (economic analysis that compares relative cost and outcomes, commonly measured in quality adjusted life years -QALYs-) is also a dimension that should be included in the evaluation of performance, because in the Netherlands, for example, the Dutch Health Care Institute increasingly requires cost-effectiveness studies to substantiate their decision regarding reimbursement of new treatments.[16]

The general aims of this study are:

To gain more insight into the extent to which studies on innovations evaluate output in general healthcare and, specifically, in radiotherapy.

To investigate if implemented innovations in Dutch radiotherapy centres are supported by adequate evidence in scientific literature.

Ad A: We carried out a review of the literature to answer the following three specific research questions, both with respect to general healthcare and more specifically for radiotherapy.

Does the results section of included studies report:

statistically significant results on one or more output dimensions (patient outcome, survival, toxicity, patient safety, patient service, efficiency and cost-effectiveness) related to innovation, and what is the scientific level of the study? (RQ1)

statistically significant technological or process improvements without patient-related output or efficiency/cost-effectiveness output? (RQ2)

statistically significant results on multiple or all dimensions of output (outcome, service/safety or efficiency/cost-effectiveness)? (RQ3)

Ad B: Subsequently, our fourth research question was:

Which percentage of innovations in treatment in Dutch radiotherapy centres are supported by adequate evidence in scientific literature before they are implemented in clinical routine? (RQ4)

Methods and Materials

We used three literature search strategies to find an answer to our first three research questions. The first two searches were on general innovations in general hospital care, and more specifically in radiotherapy; the last search focused on innovations related to tumour groups and radiotherapy techniques.

A 5-year period was used for the general searches. For practical reasons (a 5-year period resulted in 1923 articles) we used a shorter period (2015/2016) for the specific tumour/technique innovations.

Only articles written in English were included, because the international scientific language in western countries is English. Because of feasibility, we only included papers that were accessible on Web of Science through our university’s library subscriptions or freely accessible on PubMed.

Search strategy

General search strategy regarding innovations in general healthcare

We searched the databases PubMed and Web of Science. We looked for relevant English language articles with an abstract from 2011 to 26 September 2016. We selected the following terms: hospital, innovation, continuous improvement, outcome, survival, toxicity, safety, efficiency, service, output, cost-effectiveness and patient satisfaction. An overview of the applied search terms is presented in Table 1.

Table 1.

Applied search terms for general search

PubMed

In PubMed we introduced the following limitation: written in English, and free full text available.

Web of science

We used the same terms as with PubMed. We refined the search by language (English). Furthermore, the following Web of Science categories were included: healthcare sciences, computer science interdisciplinary applications, management, medical informatics, planning development, business, computer science information systems, economics, communication, business finance, operation research management science and oncology.

Search strategy specifically for radiotherapy

We also performed a search specific to radiotherapy, using the databases PubMed and Web of Science. We looked for relevant English-language articles with an abstract from 2011 to 2016. We looked for relevant articles and selected the following MESH terms: radiotherapy, radiation oncology, diffusion of innovation and organizational innovation. We also searched on radiotherapy, radiation oncology and innovation. A summary of the applied search terms is shown in Table 2.

Table 2.

Applied search terms for the search specifically for radiotherapy

Search strategy per tumour group

Finally, we also conducted a search in the database PubMed on reviews regarding tumour groups and radiotherapy techniques. We looked for relevant articles in the period 1 January 2015 through 17 August 2016 and selected the following terms when searching per cancer care path: radiotherapy, radiation oncology and breast cancer/head and neck cancer/neuro oncology/skin cancer/lung cancer/gastrointestinal oncology/gastrointestinal cancer/gynaecologic oncology/prostate cancer/sarcomas. The search strategy is presented in Table 3.

Table 3.

Applied search terms for the search per tumour group

3DCRT, 3D conformal radiotherapy; DGRT, dose-guided radiotherapy; IGRT, image-guided radiotherapy; IMRT, Intensity-modulated radiotherapy; IORT, intraoperative radiotherapy; SABR, stereotactic ablative radiotherapy; SBRT, stereotactic body radiotherapy; VMAT, volumetric modulated arc therapy.

Each term was a separate search.

Search strategy per radiotherapy technique

We searched in the database PubMed for relevant articles from 01 January 2015 to 17 August 2016 using the following terms (based on the websites of the Dutch Society for Radiation Oncology and the National Institute of Health and a brainstorm by the authors which generated not a 100% complete search result but a sample we considered to be representative: radiotherapy, radiation oncology and intensity modulated radiotherapy (IMRT)/image guided radiotherapy (IGRT)/3D conformal radiotherapy/dose guided radiotherapy/stereotactic body radiotherapy (SBRT)/volumetric modulated arc therapy/brachytherapy/ stereotactic/protons/ stereotactic ablative radiotherapy (SABR)/tomotherapy/ particle therapy/intra operative radiotherapy/respiratory gating/in vivo/adaptive radiotherapy/stereotactic radiosurgery. An overview of the applied search terms is presented in Table 3.

For search strategies 3a and 3b, we introduced the following limitations in PubMed: review, free full text, must contain an abstract and be written in English.

Study selection and data extraction

We reviewed all articles resulting from searching the databases on the basis of their title and abstract. We considered any study in which innovation and output were discussed. We included these studies only if the output of the innovation was reported in the abstract. We decided to include all possible study designs for the first two searches. For the search on tumour group/radiotherapy techniques, only reviews were included.

Table 4 lists the inclusion and exclusion criteria for all the search strategies.

Table 4.

Inclusion and exclusion criteria for all three searches

Inclusion criteria	Exclusion criteria
Containing an abstract	Only concerning innovation and no output effects
Written in English	Case report, letter to the editor, comment or opinion
Focus on healthcare organizations	Not concerning an innovation
Reports concerning effects of innovations on performance	Specifically for the general search:– Not a hospital or hospital care – Not in Europe or North-America
Original scientific study
Specifically for the general search:– Concerning Europe or North America – Concerning hospitals (hospital care) – Publication between 01 January 2011 and 26 September 2016
Specifically for the search per tumour group and radiotherapy technique:– Review – Focus on radiotherapy or radiation oncology – Publication between 01 January 2015 and 17 August 2016
Specifically for the radiotherapy search: Publication between 2011 and 2016

Two reviewers (MJ and RS) independently scanned titles and abstracts to select studies for consideration. Initial disagreements on study selection were resolved by reaching consensus. Publications were selected for further assessment of the full text if inclusion criteria were met or if it was impossible to determine this based on the abstract. We listed all results in Appendices A, B and C.

The level of evidence was assessed by two reviewers (RS and JS). The assessment was based on the classification of level of evidence and grades of recommendation which is used by European Society for Medical Oncology (ESMO) in their guidelines and is based on the Infectious Diseases Society of America—United States Public Health Service Grading System.[17] An overview of the different levels of evidence is provided in Table 5.[18]

Table 5.

Level of evidence and grades of recommendation (adapted from the Infectious Diseases Society of America—United States Public Health Service Grading System)[

Level of evidence
I	Evidence from at least one large randomized, controlled trial of good methodological quality (low potential for bias) or meta-analysis of well-conducted randomised trials without heterogeneity
II	Small randomized trials or large randomized trials with a suspicion of bias (lower methodological quality) or meta-analysis of such trials or of trials with demonstrated heterogeneity
III	Prospective cohort studies
IV	Retrospective cohort studies or case-control studies
V	Studies without control group, case reports, expert opinions
Grades of recommendation
A	Strong evidence for efficacy with a substantial clinical benefit, strongly recommended
B	Strong or moderate evidence for efficacy but with a limited clinical benefit, generally recommended
C	Insufficient evidence for efficacy or benefit does not outweigh the risk or the disadvantages (adverse events, costs, …), optional
D	Moderate evidence against efficacy or for adverse outcome, generally not recommended
E	Strong evidence against efficacy or for adverse outcome, never recommended

Senkus E, Kyriakides S, Ohno S, Penault-Llorca F, Poortmans P, Rutgers E et al. Primary breast cancer: ESMO Clinical Practice Guidelines for diagnosis, treatment and follow-up. Annals of Oncology. 2015; 26(suppl 5):v8–v30. (Permission for reproduction granted by Annals of Oncology).

Comparing review results with results from a study on degree of innovation in dutch radiotherapy centres

To answer RQ 4, we first selected the treatment (product) innovations from a previous study on innovation in Dutch radiotherapy centres in the years 2011–2013. We compared these innovations with the results which we found using search strategies 3a and 3b of this study.[19] Subsequently, we counted the implementation frequency of these treatments in different centres, and calculated the percentage of significant innovations with proven patient-related output. We verified whether all innovations were evaluated before implementation in 2011–2013. We also counted implemented innovations which were significantly better technologically and should be implemented according to the ALARA/P (as low radiation as reasonable achievable/practicable) principle. We used a 95% confidence interval to determine the uncertainty of the calculated percentages as estimated for all innovations.

Results

Number of articles included

Figure 1 shows the results of the three searches. The general search identified 396 unique articles. During the screening of the titles, 71 articles were excluded. After screening the abstracts and conclusions of these articles, 219 articles were excluded because they did not meet the inclusion criteria. We reviewed the remaining 106 articles in detail. This resulted in the exclusion of another 83 articles, leaving 23 papers for this study.

Figure 1.

Selection process per search.

In the search for radiotherapy-specific articles, 335 unique articles were identified. While screening the abstracts and conclusions, 266 articles were excluded because they did not meet the inclusion criteria. The remaining 69 articles were reviewed in detail. This resulted in the exclusion of 52 articles, leaving 17 articles for analysis.

In the search per tumour group, 202 unique articles were identified. After screening the titles and abstracts, 141 articles were excluded. The remaining 61 articles were reviewed in detail, resulting in the exclusion of another 37 articles. This left 24 articles to include in this review. The search per radiotherapy technique identified 181 articles; 42 articles were excluded because they were already included in the search per tumour group, leaving 139 unique articles. After screening these articles on title and abstracts, 65 articles were excluded because they did not meet the inclusion criteria. The remaining 74 articles were reviewed in detail. This ultimately led to the exclusion of 44 articles, leaving 30 papers for this review.

Taking all three literature searches combined, a total of 94 articles were included in this review out of 1072 unique articles (8.7%). Figure 2 gives an overview of the included number of articles and those with significant evidence or without significant evidence and those with no significant output but a technological improvement with significant evidence (research questions 1 and 2).

Figure 2.

Overview of articles and the type of output.

General review regarding innovations in healthcare

The 23 studies in the general review included technological, treatment or organizational innovations. In Appendix A, the output of the innovations and the scientific level of the studies are described. 10 out of 23 studies reported output with significant results, 6/23 articles reported output without significant results and 6/23 articles were technological improvements with significant evidence, of which five have no reported patient outcomes. Furthermore, 2/23 articles reported mixed output (some significant results and some results not significant). In 2 of the 23 papers, cost-effectiveness of the innovation was mentioned. Costs/efficiency was mentioned in 17 papers. 12/23 papers were organizational innovation, while 11/23 were clinical papers. The level of evidence of the organizational papers was 3x level I evidence, 2x level IV and 7x level V. For the clinical papers this was 4x level I, 1x level III, 5x level IV and 1x level V.

Review specific to radiotherapy

From the 17 studies specific to radiotherapy, 16 described product or technological innovations. From these 16 studies, 9 reported significant output (outcome, service, safety, efficiency or cost-effectiveness). There were three studies which mentioned output but where the results were described as “promising”, “may potentially reduce treatment planning time and effort”, “potential to improve local control and toxicity”, “potential to improve outcomes of patients”. Six studies reported a technological improvement with significant results, of which four articles report no significant patient output.

One article reported both significant and not significant output. There were two studies that mentioned the cost-effectiveness of the innovation. Costs/efficiency was mentioned in seven papers. There was one paper on organizational innovation. An overview of the studies is provided in Appendix B. The level of evidence of the clinical papers was 9x level I, 1x level III, 1x level IV, 5x level V. For the organizational paper this was level III.

Review per tumour group and per radiotherapy technique

From the 54 papers in this search, 40 articles reported patient-related output with significant results (of which, 31 articles report only significant output, the other nine report mixed output), while 8 reported only output without significant results. Furthermore, twelve studies report a technological improvement with significant results; five of these reported no significant patient outcomes. The review included 10 studies that reported mixed output. Only 5 papers contained information on cost-effectiveness. Costs/efficiency was mentioned in 10 papers.

Appendix C offers an overview of all innovations. The level of evidence of these papers was 25x level I, 1x level II, 9x level III, 2x level III/IV, 11x level IV and 6x level V.

Results for research questions 1 and 2

Summarized, the scientific level of papers on organizational innovation was lower than clinical papers (23% vs 47% level I; 54% vs 15% level V).

Table 6 summarizes the results of our study in relation to effectiveness (patient outcome, patient service and patient safety), efficiency and cost-effectiveness. Statistically significant outcome results were reported in 59%. For safety results this was 1% and for service 5%. (total 65% which rose to 76% for radiotherapy reviews). A significant technological improvement was identified in 26% of papers, cost-effectiveness in 10% of papers and costs/efficiency in 36% of the papers.

Table 6.

Results in relation to effectiveness/efficiency/cost-effectiveness

Review	Number articles	Outcome Sig./not sig.	Safety Sig./not sig.	Service Sig./not sig.	Costs/ efficiency	Technological improvement	Cost-effectiveness
General	23	8/1	1/3	3/2	17	6	2
Radiotherapy	17	8/6	0/0	0/0	7	6	2
Tumour group/technique	54	40/18[a]	0/1	1/0	10	12	5

There are only 54 articles included in this review; however some articles report both significant and not significant outcomes.

Results for research question 3

Regarding our third research question (to what extent are significant results reported on all output dimensions), we found the following (Table 7):

Table 7.

To what extent are significant results reported on all output dimensions

Review	Articles with significant output on effectiveness and costs/efficiency	Articles with significant output only on effectiveness	Articles with significant output only on costs/efficiency	Cost-effectiveness
General review	7	2	3	2
Review radiotherapy	2	6	2	2
Review per tumour group/radiotherapy technique	4	36	0	3

In the general review, 7/12 articles (58%) with significant output combine effectiveness and efficiency, two studies report only on effectiveness and three article only on efficiency. The most common dimension of effectiveness is patient outcome. Service and safety are underrepresented.

In the radiotherapy review, only 2/10 articles (20%) with significant output combine effectiveness and efficiency, six studies report only on effectiveness and two articles only on efficiency. The most common dimension of effectiveness is patient outcome. There are no articles in this review reporting on patient service and patient safety.

In the review specific to radiotherapy per tumour group and per technique, only 4/40 articles (10%) with significant output report on a combination of effectiveness and efficiency; 36/40 articles only report on effectiveness; no articles report only on efficiency. The articles reporting on effectiveness do so only on one dimension, namely patient outcome. There was only one article that mentions efficiency, patient outcome and patient service.

Cost-effectiveness was reported on in two articles with significant output in the general review. This number is similar for the radiotherapy review. The review per tumour group/radiotherapy technique counted three articles.

Results on research question 4

After comparing the results of this review per tumour group/radiotherapy technique with the study on implemented product or treatment innovations in Dutch radiotherapy centres in 2011–2013, we found 32 times an innovation that was also included in the reviews per tumour group/technique (Table 8). In 17/32 cases (59%), these innovations were reported in this review with significant patient results. Furthermore, one product innovation, deep inspiration breath hold, was implemented in eight centres. In accordance with the ALARA principle this innovation should be implemented despite the fact that patient outcome data are not available, especially because research showed that the actually delivered dose to the target volume is similar in case of DIBH compared to free breathing.[20] This means that in 25/32 cases innovations were adequately evaluated with regard to patient-related output before implementation. Using a 95% confidence interval, it can be concluded that a minimum of 64–92% of all innovations, 168 times implemented in Dutch radiotherapy centres, were adequately evaluated on proven patient output before implementation in Dutch centres.

Table 8.

Number of innovations implemented in the Dutch radiotherapy departments in 2011–2013, according to innovations with or without report on outcome, as found in the current review

Innovations	Innovations with report on output with significant evidence patient outcomes	Technological improvements with significant evidence, no significant patient outcomes reported	Level of evidence
DIBH		8	I/V
VMAT prostate	2		I
SBRT pancreas		1	III
SBRT lung	4		I/III/IV/V
Brachytherapy prostate	2		I
Brachytherapy skin	1		V
Hypofractionation breast	3		I
Hypofractionation prostate	1		I
IMRT lung		4	III/IV
IMRT gynaecology		2	IV
SBRT intra- and extracranial	1		IV
IORT	1		I
IMRT anus	1		I
IMRT prostate	1		I/III

DIBH, deep inspirational breath hold; IMRT, intensity modulated radiotherapy; IORT, intraoperative radiotherapy; SBRT, stereotactic body radiotherapy; VMAT, volumetric modulated arc therapy.

Discussion

It is necessary to implement innovations in healthcare and radiotherapy to improve effectiveness (patient outcomes, patient safety and patient service), costs/efficiency or cost-effectiveness. However, the literature shows that innovations are not always properly evaluated before implementation. Our literature review aimed to gain insight into the extent to which innovations are evaluated on output in scientific studies. Considering innovations in healthcare in general, we found that not many studies report statistically significant outputs. Radiotherapy reviews, on the other hand, offer good insight into output effects of innovations, with a higher level of evidence.

Furthermore, we found that studies on output effects of innovation often do not report on all relevant dimensions of output such as effectiveness (patient outcome, service and safety), costs/efficiency or cost-effectiveness. Furthermore, we found that between 64 and 92% of treatment innovations that were implemented in Dutch radiotherapy centres from 2011 to 2013 were adequately evaluated regarding patient output before implementation.

Innovation studies and reported output

These figures show that the general search term “innovation” results in a low number of papers describing output effects as a result of innovation. From the few papers in our search covering a 5-year period (not concerning tumour group and radiotherapy technique reviews) that met the inclusion criteria (respectively 23 and 17 out of 731 = 5%), the effect on output was significant in only 20/40 papers (50%). For the papers on tumour group/radiotherapy techniques this percentage is higher (24 and 30 out of 341 = 16%), for a 1.5-year period. Out of these 54 papers, 41 (76%) report significant output.

To illustrate the method and findings the example of lung cancer is described. Several innovations were found in the review. Innovations, which result in a significant positive effect on survival and/or toxicity, were SBRT (3 publications, 17 citations, level of evidence II and III), higher dose radiotherapy and SABR (2 publications, 12 citations, level of evidence I and IV).

Intensity modulated radiotherapy (IMRT) (3 publications, 12 citations, level of evidence III and IV) showed improved survival and/or toxicity, however without significant evidence. Dosimetric studies showed that this is a technological improvement with significant evidence. However, there were no randomized controlled trials published in 2008–2011 which compare the clinical outcomes of 3D-CRT and IMRT in lung cancer. Nevertheless, this innovation was implemented in 4 out of 20 Dutch radiotherapy centres in that period.

The low number of papers using the term “innovation” that met the inclusion criteria suggests that most papers on innovation do not study output effects in general, or do not use the term “innovation” even if they do, in fact, deal with innovations. This is not what we expected. Already over 10 years ago, Porter et al introduced a famous model for value-based healthcare, in which they challenged healthcare organizations to increase value for patients by incorporating the outcomes of interventions that matter to patients in their organization strategy, in relation to the costs of achieving those outcomes. They also stressed the importance of measuring, reporting and comparing those outcomes.[21,22] Healthcare is among the best-endowed of all industries in the richness of its scientific base.[23] However, with regard to the relation between innovation and output, this seems apply much more to medical papers than to papers on innovation in general. There are likely many innovations that are not described with the term “innovation”, and were therefore missed in our review. For example, the introduction of an electronic health record (EHR) is definitely an organizational innovation, but we did not find it with our search strategy. As a try-out, we did a search on the number of papers on EHR and output as defined above, and found a large number of hits, but these often had such a limited scope, that it was not possible for us to draw firm conclusions about the relationship between an EHR and output. The same probably holds true for numerous other innovations, which causes the fragmented picture described in the introduction. In medical disciplines, we expect that the problem that innovations are not described as “innovation” can be solved by systematically investigating reviews. In management studies this is much more difficult, and if at all possible, also time-consuming because the field is very broad and a system to include all applicable reviews is lacking.

Level of evidence

Papers on organizational innovation had a lower level of evidence than radiotherapy-specific medical papers. New treatments and (sometimes) new technology in healthcare can be tested with extensive Phase I, II and III trials, which have a higher level of evidence than the kind of research which is usually conducted in management practice. In management practice, it is usually not feasible to conduct randomized controlled clinical trials, so observational studies or research designs with less evidential value are mostly used.

An evidence base is often created by conducting the research several times under different circumstances.[24,25] Although in the past two decades a large number of studies have been conducted with the intention to provide a solid evidence base for management practice, most insights in management are still based on the personal experience of experts. Furthermore, in management practice researchers and practitioners mostly operate in different worlds, with the consequence that research results are regularly not fully understood and supported by the practitioners, and therefore not implemented in practice.[24,25]

Output dimensions

In our study, we investigated the output of innovations not only on one dimension of output but also on the total performance. We think this is important to get a complete view. For example, an innovation with a very small advantage for patients at very high cost must be considered carefully before taking the decision to introduce or reject it in clinical routine. As defined in the introduction of this study, with respect to total performance we distinguished the following dimensions: effectiveness (patient outcome, patient service and patient safety), efficiency and cost-effectiveness. A combination of effectiveness and/or efficiency and/or costs/cost-effectiveness in articles with statistically significant results was reported in 7/23 general papers (30%), in 2/17 radiotherapy papers (12%) and in 4/40 papers on tumour group/radiotherapy technique (10%). We think the last two figures can be explained by the fact that the papers per tumour group/radiotherapy technique and radiotherapy-specific innovations are almost all written by professionals in the field of radiotherapy, who are logically most interested in patient outcomes because of the nature of their profession. Managers and policymakers in the field of healthcare and radiotherapy have to fill the gap regarding the other dimensions of output performance. Of course, professionals also have a clear responsibility to ensure efficiency and cost-effectiveness, but managers and policymakers have a greater responsibility to initiate research on these dimensions.

Evaluation of output before implementation of innovations

In our introduction, we motivated our interest in this research by mentioning the consequences of implementing poorly evaluated innovations or not implementing already proven innovations (the so-called research implementation gap”). In previous research in which we investigated the degree of innovations implemented in Dutch radiotherapy centres, we found that radiotherapy centres quickly adopt innovations within their discipline and are very dynamic and innovative. In the period from 2011 to 2013, radiotherapy centres implemented 525 innovations (168 of which were treatment innovations).[19] When comparing the implemented treatment innovations with this literature review study, we found no reported significant effect on patient outcome (survival/toxicity) for breath-hold technique in breast cancer, SBRT for pancreatic cancer, IMRT for lung cancer and IMRT for gynaecological cancers. The breath-hold technique was found to be a significant technological improvement, however. It reduces the radiation dose to the heart significantly, and following the ALARA principle (as low as reasonably achievable), this innovation should be implemented. Since the reproducibility of this technique has been shown to be as high as radiotherapy during free breathing, the risk that tumour cells are missed is considered to be absent.[20] Although IMRT for lung cancer and SBRT for pancreatic cancer are also better from a technological point of view, it is desirable to have data on patient outcomes before implementation, because otherwise the effects on tumour control and normal tissue injury are not completely clear. Furthermore, IMRT for gynaecological cancers was implemented in 2012 while, according to a review, this therapy was proven to improve patient outcomes only a year later, in 2013. As reported, 64–92% of implemented innovations in Dutch centres were properly evaluated regarding patient outcomes before implementation. It is debatable whether this is a good score. For example, medical oncology will probably never introduce a new drug in routine practice without one or several randomized trials. They will have a near 100% score. However, this can partly be explained by the fact that in oncology new drugs are given to patients in addition to the existing practice, while in radiotherapy most new technologies improve the precision of dose delivery. In this case, the ALARA principle can justify implementation if research shows that no tumour cells will be missed. Furthermore, in other medical disciplines we also see innovations (e.g. the DaVinci robot) being introduced before they are properly evaluated, while they have no decision-supporting information like in silico studies in radiotherapy. Nevertheless, we conclude that small improvements are still possible to improve evaluation before implementation (research question 4), as is also demonstrated by the current ordering of MR-linacs before the clinical outcome is precisely known.

It is possible that the implementations of IMRT for lung cancer and SBRT for pancreatic cancer took place in a multicentric study and that the two- or 5-year outcome results are not yet known. However, in the Netherlands we found no such studies (also not for IMRT for gynaecological cancers) in that period on http://www.trialregister.nl and http://clinicaltrials.gov. We think it is important that innovations which have not been proven yet but are promising are implemented as much as possible in a study setting, unless the ALARA principle is applicable and the innovation is cost-effective (ALARP principle), especially when the technology is expensive as for example in the case of particle therapy. This is possible via well-designed clinical trials, but sometimes this is not considered possible due to practical and ethical limitations, since clinical trials are only possible if the treatment options being compared look like acceptable trade-offs between risks and benefits.[26] Methods from health technology assessment and health economics are useful complements to standard methods from evidence-based medicine.[6] Another option to evaluate effectiveness is to use a model-based approach, as is currently used for the introduction of proton therapy (preferably in existing centres) in the Netherlands.[19] In the case of particle therapy, reduction in secondary cancers and other late-toxicities is difficult to evaluate through randomized clinical trials, since the improvement in treatment techniques will evolve dramatically over time and the benefits/harms take so long to materialize. We do feel that the relationship between dose and complications is known for many toxicities and can therefore be used to predict the benefit of particle therapy, although there is some uncertainty in these models. Therefore, in the model based approach in the Netherlands thresholds will be used to define clinical benefit based on the grade of toxicity, including secondary cancers and late toxicities. Protons are only allowed if patients are expected to have a significant clinical benefit in comparison with photon therapy taking into account these thresholds for significance in normal tissue complication probability.[27]

Limitations

The main limitation of this review is that we found it impossible to develop a search strategy that guaranteed finding all reports of innovation in the literature. This means that organizational innovations are probably underrepresented. However, we think these organizational studies will likely have a limited scope and not address organizational innovations in a broader sense.

Furthermore we chose to only include papers for which we had full text access either through our university or because they were freely available on PubMed. Especially in search strategies 3a and 3b (where we looked for reviews on innovations per tumour group and radiotherapy technique) this may have led to underreporting as these were solely based in PubMed and not on Web of Science. Specifically, in search 3a and 3b, 44% of all papers meeting the search strategy in the specific period were freely available. A limitation of the study is the assumption that this is a representative sample of reviews that describe the relation between innovation and output and that that this relation is not different in freely accessible and not freely accessible review papers.

Another limitation is the fact that the table used to rank scientific level as used by ESMO was developed for use in medical science. This means, for example, that it does not say anything about cost-effectiveness for which we should record utilities (EuroQoL), and it does not distinguish between expert opinion papers and studies with measurements at baseline and one or 2 years after intervention.

Conclusion

Only few studies report on output when considering innovations in general, but radiotherapy reviews give a reasonably good insight into the output effects of innovation, with a higher level of evidence. In Dutch radiotherapy practice, clinical innovations are properly evaluated before implementation in daily routine. There is room only for small improvements.

We recommend that managers in radiotherapy initiate research on organizational innovation with a more rigorous methodology to provide more high-level evidence on the output of organizational innovation. Organizational innovations should be further tested and evaluated, and not directly implemented.

In clinical radiotherapy, we recommend improving studies by evaluating innovations more systematically on all dimensions of output, including costs or cost-effectiveness. Furthermore, we recommend introducing medical innovations in clinical routine only after a thorough evaluation (through well-designed trials or model-based approaches) prior to implementation.