How efficient is translational research in radiation oncology? The example of a large Dutch academic radiation oncology department.

OBJECTIVE: : To study the efficiency of research implementation in a large radiotherapy institute, in either an internal review board-approved clinical trial or clinical routine.
METHODS: : Scientific publications of the institute were listed. We asked clinicians from tumour expert groups whether the study had been implemented yet in a clinical trial or in clinical practice and which facilitators or barriers were relevant. An independent investigator verified all results. We calculated the implementation rates and the frequency of mentioned facilitators and barriers.
RESULTS: : Resident researchers had published 234 studies over the past 4 years. Overall, 70/234 (30%) technical or preclinical studies were tested or implemented in a clinical environment in either trials or routine. In total, 45/234 (19%) studies were routinely implemented; in the 61 clinical studies, this percentage was higher: 38% (23/61). The main facilitator was the level of evidence and the main barriers were workload and high complexity.
CONCLUSION: : We were able to calculate the implementation ratio of published research into clinical practice and set benchmark figures for other radiotherapy clinics. Level of evidence was an important facilitator, while workload and high complexity of the new procedures were important barriers for implementation. Recent articles suggest that academic entrepreneurship will facilitate this process further. ADVANCES IN KNOWLEDGE:: This study is the first of its kind calculating implementation rates of published studies in the clinical environment and can contribute to the efficiency of translational research in radiotherapy. We propose to use this metric as a quality indicator to evaluate academic departments.

INTRODUCTION

Innovation in radiotherapy has resulted in remarkable progress in the quality of care and outcomes owing to the growing ability to identify and target tumours with a high accuracy and precision.[1,2] In order to innovate continuously in the face of future challenges and opportunities, translating research findings into clinical practice is very important. The pace of the translation of scientific discoveries into clinical practice is not well known in radiotherapy and is described as slow in healthcare in general.[3,4] The literature frequently reports on the gap between the findings in published empirical literature and the actual use of this evidence in clinical practice.[5-9] However, calculated implementation rates of research findings into clinical practice are lacking. Figures about the gap are based, for example, on published research regarding the integration of evidence-based interventions within clinical practice relative to basic research, the number of patients receiving recommended care or the percentage of basic scientific findings licensed for clinical use.[6,7,10,11]

Because of the importance of the continuous translation of research into clinical radiotherapy practice, we want to investigate the real implementation rates. Our aim was to study the efficiency of research implementation, in either clinical routine or clinical trials in a large radiotherapy institution in the Netherlands over a period of 4 years and to provide other radiotherapy centres figures for benchmarking.

The institution that is the subject of our study has stated in its policy plan that it has an integrated strategy for research, technology transfer and patient care (i.e. the same focus in each area with strong alignments between these areas) and facilities such as a data centre for clinical trials and a software development team.

The main research questions are threefold:

What is the rate of clinical testing of published preclinical (laboratory) and technological findings?

What is the percentage of published findings routinely implemented in clinical practice?

What are the facilitators of and barriers to the implementation process in the clinical environment?

For the second research question, we also analyzed the impact of national and international collaboration on implementation efficiency, since research collaboration has been reported to be related to research productivity and we want to know whether this is also the case for implementation.[12] Finally, we investigated whether the type of funding is related to implementation rates because research funding agencies are held accountable for the public money they spend.[13,14] Clinical use of research findings legitimizes research expenditures. Companies also provide funding with the aim to make research findings available in the near future. Our long-term ambition is to identify quality indicators of academic departments and key variables to improve the efficiency of innovation implementation.

METHODS AND MATERIALS

Data collection

The scientific publications of researchers at the institute, as included in its annual report, were listed for the period from 2008 to 2011 (4 years). Radiotherapy department figures are listed in Table 1.

Table 1.

Radiotherapy department 2011 figures

Characteristics of department	Number
Number of treatments	3802
Number of patients treated	3015
Number of accelerators	7
Number of brachytherapy suites	1
Radiation oncologists in the clinic (FTE)	14
Physicists (FTE)	8
Radiation oncologists in training (FTE)	8
Physicists in training (FTE)	2
Technologists (FTE)	56
Researchers (FTE)	20

FTE, full time equivalent.

Each author was asked to place his or her study (or studies) into one of the following categories: clinical research study (retrospective, prospective cohort, clinical trial); preclinical research; technical research (physics, information communication technology/computer science, imaging); other categories (reviews, case reports, cost–benefit analysis, in silico trial), as shown in Table 2.

Table 2.

All publications from 2008 to 2011, categorized according to type of research. The columns represent the total number of studies; the number of studies implemented in routine clinical practice; negative findings; studies with patient material/patient data; clinical trials; mean implementation time in months; and the range of the implementation time

Type of research	Number of publications	Implemented in clinical routine	Negative findings	Studies with patient material/data	Tested in prospective clinical trial (IRB approved)	Mean implementation time (months)	Implementation time (range) (months)
Clinical research	61 (26%)	23 (10%)	1 (0.4%)	0	0	4	0–18
Retrospective	11	2		0	0
Cohort	24	6		0	0
Trials	26	15		0	0
Preclinical research	57 (24%)	0	0	12 (5%)	4 (2%)	Studies 1a,b	0–4
Technical research	65 (28%)	15 (6%)	0	43 (18%)	11 (5%)	Clinical routine 15cStudies 19a,d	0–470–54
Physics	34	7		30	4	19c	0–47
ICT/computer science	2	1		2	1	39c	–
Imaging	11	7		11	6	8c	0–31
Other categories	51 (22%)	7 (3%)	1 (0.4%)	0	0	4	0–18
Reviews	34	4		0	0
Case reports	6	0		0	0
Cost–benefit analysis	8	1		0	0
In silico trials	3	2		0	0
Total	234 (100%)	45 (19%)		55 (24%)	15 (6%)

ICT, information communication technology; IRB, internal review board.

Owing to a lack of a comprehensive registration, not all information is available.

Based on 4 of 12 studies.

Based on all publications implemented in clinical routine.

Based on 16 of 43 studies.

Subsequently, we asked (in 2015) 15 clinicians (all of whom were either radiotherapy oncologists or physicists) from tumour expert groups at our institute to judge whether the published study had been implemented in the institute in (1) daily routine clinical practice before or after the study; (2) studies with patient material or patient data; and (3) an internal review board-approved prospective clinical trial. This has been verified by an independent investigator who was not involved in the data-analysis. Studies which evaluated the outcome of a previously implemented innovation in the treatment process, but did not result in an adjustment of the treatment as a consequence of the evaluation, were not counted as implemented. Also, studies which added knowledge for the physicians/physicists (e.g. knowledge on certain risk factors for toxicity), but did not result in an adjustment of guidelines/local protocols, were not included in the implementation figures. Finally, we inquired about the time between publication and implementation and whether there were publications with negative findings.

On the basis of the literature, we listed the facilitators and barriers in the implementation process in clinical routine.[5,11,15-17] We listed the perceived level of evidence, relative advantage (the degree to which the implementation of the scientific finding is perceived as being better than the existing practice), compatibility (the degree to which the finding is perceived as consistent with the existing values, past experiences and needs of the clinicians), complexity (the degree to which a scientific finding is perceived as relatively difficult to understand and use), trial ability (the degree to which an innovation resulting from a scientific finding may be experimented with on a limited basis), observability (the degree to which the results of the implementation of scientific findings are visible to others), interorganizational connections (implementing the scientific findings together with other organizations), workload and researchers–clinicians gap (no common vision and no alignment between researchers and clinicians). We then asked the clinicians from the tumour expert groups which facilitators or barriers regarding implementation in clinical routine were applicable for studies concerning findings that could potentially be implemented in clinical practice. We also asked whether other barriers or facilitators than the ones we had listed were present. We put the number of times the item was mentioned by the clinicians as a barrier or a facilitator on a numerical scale.

Finally, all studies were categorized according to single-centre, national multicentre or international multicentre research and according to national, international and corporate funding. In 45% of the studies, there was no funding source mentioned in the study, probably in most cases because the research was “internally” funded, i.e. the scientists conducted the research within their regular working schedule paid by the clinic or university.

Data analysis

We calculated the implementation rate by dividing the number of publications implemented in clinical trials and/or in daily routine clinical practice by the total number of publications. In addition, we scored the frequency of attributes which facilitate innovations if findings were implemented or barriers which block innovation.

To study the impact of the national or international setting of the studies and the type of funding on the implementation rate, we used a χ2 test. In these tests, we calculated the expected implementation rates by combining the observed average implementation rate with the frequency distribution of the setting (single-centre vs multicentre national vs multicentre international) and the frequency distribution of the funding (national/international/company/mixed). Subsequently, we calculated the sum of the squares of observed values minus the expected values divided by the expected values using SPSS® (IBM Corp., New York, NY; formerly SPSS Inc., Chicago, IL).

RESULTS

Categorization of publications

Resident researchers published 234 articles in 4 years, of which 61 (26%) articles were judged to be clinical studies, 57 (24%) studies were preclinical, 65 (28%) studies were technical and 51 (22%) studies were other categories (Table 2).

Implementation rate

In total, 100/234 (43%) studies were implemented in a clinical environment (Table 2).

45/234 (19%) studies were implemented in clinical routine; in the 61 clinical studies, this percentage was even higher: 38% (23/61).

55/234 (24%) technical or preclinical (laboratory) studies were tested in a clinical environment, mostly in the context of a research project. Of these technical or preclinical studies, 15 studies were tested in a prospective clinical trial (15/122 = 12%).

Facilitators and barriers

The radiation oncologists/physicists (15) interviewed most often mentioned “perceived level of evidence” as a facilitator of the implementation of the published scientific findings in clinical routine. The barriers for not implementing the published research findings in clinical routine mentioned most often were workload and high complexity of the new procedure (Figure 1).

Figure 1.

Overview of the mentioned facilitators of and barriers to publications which concern findings that could potentially be implemented in clinical practice.

Impact of collaboration and funding on implementation rate

The implementation rate of international multicentre studies was significantly lower than the observed average implementation rate in our study (11% vs 19%), whereas single-centre studies (28% vs 19%) or national multicentre studies (30% vs 19%) had much higher implementation rates than the observed average implementation rate (Pearson χ2 p = 0.003) (Table 3)

Table 3.

Cross-tabulation study setting and implementation counts

Study setting	(Expected) Count	Implemented	Not implemented	Total
International multicentre	Count	14.0 (11%)	112.0 (89%)	126.0
	Expected count	24.2 (19%)	101.8 (81%)	126.0
National multicentre	Count	16.0 (30%)	38.0 (70%)	54.0
	Expected count	10.4 (19%)	43.6 (81%)	54.0
Single-centre local	Count	15.0 (28%)	39.0 (72%)	54.0
	Expected count	10.4 (19%)	43.6 (81%)	54.0
Total	Count	45.0 (19%)	189.0 (81%)	234.0
	Expected count	45.0 (19%)	189.0 (81%)	234.0

In terms of type of funding, the observed implementation rate was higher than the average observed implementation rate in our study for studies with national funding (32% vs 14%) and was lower than the average for studies with funding from mixed sources (0% vs 14%). Implementation rates for international or company funding were not significantly different from the observed average implementation rate (Pearson χ2 p = 0.003) (Table 4).

Table 4.

Cross-tabulation funding and implementation counts

Funding	(Expected) Count	Implemented	Not implemented	Total
Company funding	Count	4.0 (19%)	17.0 (81%)	21.0
	Expected count	3.0 (14%)	18.0 (86%)	21.0
International funding	Count	5.0 (11%)	39.0 (89%)	44.0
	Expected count	6.2 (14%)	37.8 (86%)	44.0
Mixed funding	Count	0.0 (0%)	35.0 (100%)	35.0
	Expected count	4.9 (14%)	30.1 (86%)	35.0
National funding	Count	9.0 (32%)	19.0 (68%)	28.0
	Expected count	3.9 (14%)	24.1 (86%)	28.0
Total	Count	18.0 (14%)	110.0 (86%)	128.0
	Expected count	18.0 (14%)	110.0 (86%)	128.0

DISCUSSION

Dissemination and implementation of research findings in practice are necessary to improve the outcomes of radiotherapy treatment and also to achieve a return on investment for research expenses. Investigating a large radiotherapy department in 2015, we found that 19% of studies published in 2008–2011 were implemented in routine clinical practice, another 6% studies in clinical trials and another 24% studies in studies with patient material/data. In addition, national studies or studies using national funding had higher implementation rates than international studies or studies funded by companies.

Negative findings were presented in only 2 out of 234 articles. This could be explained by the well-known publication bias: studies with statistically significant results are more likely to be published than studies with non-significant results.[18-21]

The implementation time in clinical practice was 4 months for clinical research and 15 months for technical research. It was possible in only 20 of the 55 cases to discover the date on which a study was continued as part of further studies in the clinical setting. The implementation times for preclinical studies and technical studies were 1 month (n = 4) and 19 months (n = 16), respectively.

The main facilitators of implementation were high level of evidence, low complexity and high compatibility and observability, whereas the main barriers were high complexity, high workload and a large gap between researchers and clinicians.

As far as we know, we are the first to report implementation figures like these; so, we cannot compare our figures with the literature. To get a rough indication of implementation rates, other researchers tend to examine, for example, the types of articles appearing in the peer-reviewed literature or studies about the number of patients receiving recommended care.[6,7,10,11] Despite this lack of comparable studies, much of the literature states that implementation rates are low, but that it is not completely clear why implementation rates are low, and that disseminating new evidence and treatments into clinical practice is a slow and poorly understood process.[22] It is common to refer to the “pipeline” from research to practice, where the “leakage” or loss of medical clinical research is described at each stage from completed research to the ultimate implementation.[23]

In addition, in radiotherapy, the interval from the development of new technologies to their application as clinical tools can be long as well (e.g. ≥10 years).[24] Looking closer at the implementation rates in the department in our study, the following remarks can be made.

Clinical research (n = 61) had the highest implementation rate, 38% (23/61), and a short implementation time (4 months), which can be explained by the fact that this research is performed by clinician scientists, is well known by other members of the medical staff and has already been shown to be compatible with daily practice.

Among the preclinical studies (n = 57), 21% (12/57) studies were tested in patients and 7% (4/57) studies resulted in a clinical trial. When investigating implementation rates of studies published in 2008–11, in 2015, high implementation rates cannot be found because the pipeline of this kind of preclinical research is usually longer than 4–7 years. Previous research even found that it takes an average of 17 years for only 14% of new scientific discoveries to enter day-to-day clinical practice.[7] Taking into account this long pipeline, the above-mentioned scores of 21% and 7% cannot be qualified as low.

Among the technical studies (n = 65), 23% (15/65) studies were implemented in clinical routine, 66% (43/65) studies were tested in patients and 17% (11/65) studies resulted in a clinical trial. These high implementation rates of technical studies are probably owing to the fact that radiation therapy is situated at the interface between many disciplines and relies heavily on physics (including imaging). The department in our study is pioneering in the fields of dose-guided radiotherapy (DGRT) and image-guided radiotherapy, resulting in commercialization of the developed software. The high percentage of technical research is in line with this focus on DGRT and image-guided radiotherapy.

Possibly, the integrated strategy for research, technology transfer and patient care and the availability of a data centre for clinical trials including a software development team also influenced the high implementation rates of technical studies. We cannot prove this statement scientifically because we did not compare clinics with and without this strategy, but we hypothesize that the collaboration between clinician scientists, medical staff and basic researchers in networks based on common interests was one of the main factors for the high implementation rates, as mentioned in the literature.[25]

In conclusion, to the best of our knowledge, implementation rates in clinical practice have not been investigated before. Further research in clinical practice elsewhere is necessary to make statistical comparisons. Our results can serve as a benchmark for such a comparison.

Collaboration and funding

We found higher implementation rates for national studies and for studies with national funding than the averages observed in our study (Tables 3 and 4). This may be explained by the fact that many of these studies have been initiated by our own clinicians or researchers. We cannot rule out that results of international studies, conducted by other researchers, have been implemented in one of the collaborating institutes or elsewhere, since we only looked at implementation in our own clinic.

Barriers, facilitators and possible interventions

The literature has identified the following barriers to be effective in the dissemination of (research-based) innovations: researchers not being oriented to practical problems, practitioners not having useful solutions to their practical problems, perceived level of evidence and organizational barriers such as workload, lack of resources, workplace culture, poor implementation planning and ineffective leadership.[5,15,16] Radiotherapy-specific barriers are mentioned as well, such as the aggressive marketing of industrial partners promoting products that are premature and/or not really innovative for further development in a research contract.[24] Furthermore, it is known that the attributes of a potential innovation resulting from research affect the rate of adoption.[15] These attributes are the relative advantage, compatibility, complexity, trial ability and observability.[15]

The most frequently observed barrier in our study was workload/lack of resources followed by high complexity and a gap between researchers and clinicians. Workload is a barrier that is generally known and most frequently identified, especially in healthcare.[5] Therefore, it is necessary to regard research implementation not as something that comes on top of normal workload. Time management is an important skill for the individual worker; nevertheless, support from administration and healthcare funders is required to manage workload for each staff member and provide opportunities to invest time in research implementation.[5] In order to close the gap between researchers and clinicians, the clinic in our study takes further actions to involve both clinicians and researchers from the start of a new project. Only one barrier which we had not listed was mentioned: switching to another vendor. Because the research was performed on and based on the equipment of the specific vendor, the results could not be implemented when new accelerators of another vendor were introduced in the clinic.

The facilitators of implementation most frequently mentioned were a high level of evidence, low complexity and high compatibility and observability. If clinicians perceive a high level of evidence, there is a high likelihood for successful adoption. Conversely, a low perceived level of evidence can be an obstacle. In addition, there is often a relationship with reimbursement. Lack of reimbursement hinders implementation.[26] Insurance companies in the Netherlands increasingly require cost-effectiveness studies to substantiate their decisions regarding reimbursement of new treatments, although not yet systematically. If such a study is required, implementation of scientific findings may be delayed at the centre finding the results. However, once cost effectiveness has been proven, it can be a facilitator of fast implementation of the findings at many other centres. In our study, reimbursement was not mentioned by clinicians. Low complexity is a facilitator, but can obviously be a barrier if it is high. Unfortunately, reducing complexity is difficult. Compatibility and observability were mentioned 17 times as a facilitator and only once as a barrier. In the literature, it is stated that compatibility can be further enhanced through multiple iterative trials that refine the intervention to meet the needs of practice, with results that are readily observable.[17] From the literature, we know that observability cannot really be enhanced by “passive” dissemination methods. These are, for example, educational material and distributions of recommendations for clinical care including guidelines, audio-visual material, electronic publications or lectures. Consistently effective interventions related to observability, and thus efficiency of translation, are, for example, interactive workshops, educational outreach visits, reminders (manual or computerized) or multifaceted interventions.[8,9]

Finally, a new trend where academic institutions invest in intellectual property management, academic entrepreneurship and technology transfer is becoming apparent, which is suggested to facilitate the implementation process.[27-29] Academic entrepreneurship offers an incredible potential for the commercialization and implementation of research discoveries. The setting up of technology transfer offices, which offer professional support for contacts with companies, can enhance technology transfer and translation of scientific findings into practice.[30,31] In our study, the above-mentioned technology transfer of DGRT is a good example.

Most frequently mentioned barriers, facilitators and possible interventions are summarized in Table 5.

Table 5.

Most frequently mentioned barriers, facilitators and possible interventions

Barriers and facilitators	Interventions
Workload	Support from administration and healthcare funders to provide time for research
Researchers–clinicians gap	Involvement of clinicians and researchers from the start of a new project
Complexity	No general interventions but adjusted to specific situation
Compatibility	Multiple iterative trials
Observability	NOT ONLY: educational material, distributions of recommendations for clinical care including guidelines, audio-visual material, electronic publications and lectures.BUT ALSO: interactive workshops, educational outreach visits, reminders (manual or computerized) and multifaceted interventions

Strengths and limitations

In our view, the key strength of this study is that we are the first to provide actual data of true clinical implementation of research results. However, the disadvantage of being the first is that we cannot compare our results with those in the existing literature.

The main limitation of this study is its single-centre character, which may affect the generalizability of the results.

In addition, we should take into account that implementation rates calculated in a limited period do not reflect clinical impact. A study can be complex and thus poses a barrier to clinical implementation; but, once implemented, it can have a high impact and vice versa: an implemented study can have a low impact.

A final limitation is that it is known from the literature that self-assessments of adherence to guidelines are overestimated.[32] Our study does not concern adherence to guidelines but implementation of own scientific findings. Still, it remains a self-assessment, with the proviso that we asked clinicians from tumour expert groups to answer our questions, who are not always the same people as the scientists who have published the studies. In addition, an independent investigator verified the results.

CONCLUSION AND FURTHER STUDIES

We were able to calculate the implementation rates of published research from a large academic radiotherapy department in their own clinical practice and set benchmark figures for other radiotherapy clinics. Level of evidence was an important facilitator, whereas high workload and complexity were important barriers. The literature suggests some specific interventions to overcome these hurdles. Taking actions to improve implementation rates is an important task for the management of the institute, because research implementation is a key for improving outcomes, service, safety and efficiency in radiotherapy further. The next step will be to investigate implementation rates at national and international level and in other centres. We propose that the rate of clinical implementation of published research findings, in clinical routine or in trials, should be a quality indicator for organizations whose activities are both research and patient care, such as a comprehensive cancer centre.