Can the completeness of radiological cancer staging reports be improved using proforma reporting? A prospective multicentre non-blinded interventional study across 21 centres in the UK.

OBJECTIVES: Following a diagnosis of cancer, the detailed assessment of prognostic stage by radiology is a crucial determinant of initial therapeutic strategy offered to patients. Pretherapeutic stage by imaging is known to be inconsistently documented. We tested whether the completeness of cancer staging radiology reports could be improved through a nationally introduced pilot of proforma-based reporting for a selection of six common cancers.
DESIGN: Prospective interventional study comparing the completeness of radiology cancer staging reports before and after the introduction of proforma reporting.
SETTING: Twenty-one UK National Health Service hospitals. PARTICIPANTS: 1283 cancer staging radiology reports were submitted. MAIN OUTCOME MEASURES: Radiology staging reports across the six cancers types were evaluated before and after the implementation of proforma-based reporting. Report completeness was assessed using scoring forms listing the presence or absence of predetermined key staging data. Qualitative data regarding proforma implementation and usefulness were collected from questionnaires provided to radiologists and end-users.
RESULTS: Electronic proforma-based reporting was successfully implemented in 15 of the 21 centres during the evaluation period. A total of 787 preproforma and 496 postproforma staging reports were evaluated. In the preproforma group, only 48.7% (5586/11 470) of key staging items were present compared with 87.3% (6043/6920) in the postproforma group. Thus, the introduction of proforma reporting produced a 78% improvement in staging completeness . This increase was seen across all cancer types and centres. The majority of participants found proforma reporting improved cancer reporting quality for their clinical practice .
CONCLUSION: The implementation of proforma reporting results in a significant improvement in the completeness of cancer staging reports. Proforma-based assessment of cancer stage enables objective comparisons of patient outcomes across centres. It should therefore become an auditable quality standard for cancer care. © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2018. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

This is the first study to measure the performance of cancer reporting by radiologists. This quality improvement initiative has shown that, prior to the intervention of proforma reporting, crucial information needed to make therapeutic decisions were missing from 52% of radiology cancer reports. This improved by 78% when proforma reporting was implemented

The trial was a non-blind study and consequently there may have been some observer (Hawthorne) effect which may have also contributed to the improvement. If so, this effect could benefit healthcare in future through similar audits of the quality of cancer radiology reporting.

A minority of centres were unable to implement proforma reporting and thus failed to improve cancer staging; this was due to a combination of personnel and information technology related barriers. Therefore future efforts and resources should be directed towards rolling out the models of care in centres that achieved cancer reporting improvements

Introduction

Once a patient is diagnosed with cancer, the next steps in patient care are crucial and result in life changing treatment decisions such as intensity and radicality of therapies that can result in long term side effects. Such decisions hinge on the accuracy and completeness of cancer staging provided to the clinical teams and patient.1 2 The majority of initial cancer treatment plans are almost entirely based on radiological assessment of both the cancer prognostic stage and anatomic distribution of disease. Thus, clear documentation of imaging-derived staging is required of radiologists to facilitate multidisciplinary team (MDT)-based decisions. In most cancers, radiological staging assessment is used to guide radiotherapy and surgical planning, and to select which patients require preoperative (neoadjuvant) chemotherapy. For example, in studies of patients with rectal cancer, preoperative radiological staging and MDT discussion increased the proportion of patients receiving neoadjuvant treatment, curative surgery rates and local disease control.3 4

Despite the importance of preoperative radiology assessment, very few prospective audits of imaging reports for cancer have been performed but all have shown significant deficiencies in documented staging information. A single-centre study found tumour resectability status in rectal cancer, which informs the decision for preoperative chemoradiotherapy, was missing in 40/55 (73%) of free-text radiology reports whereas proforma reporting reduced this to 4%.5 An audit of practice by Ontario Cancer Care showed similar findings with missing data noted in 40% (51/128) of rectal cancer staging reports submitted by radiologists.6

Assessment of cancer after surgery has taken place is better documented. Pathologists developed the concept of the "minimum dataset" which set the standard for information that should be included in all cancer staging histopathology reports. Using a proforma-based systemfor cancer reporting by pathologists is now well established.7–10 Previous audits of histopathology reporting of cancer stage have shown an increase in minimum staging data documented in histopathology reports from 31% to 100% in colorectal cancer following the introduction of proforma reporting.11 12 Similar improvements in data completeness have been seen in pathology reporting of other cancers, such as pancreas, prostate and melanoma, following standardisation.13–19 The impact on patient outcomes was demonstrated by a study showing that patients with incomplete staging reports with dataset items missing had worse cancer survival.20 The link between proforma reporting and improvement in survival is through enabling the correct identification of high-risk patients who can be offered further therapy. 21 As a consequence, minimum dataset reporting of prognostic histopathological data following surgery for cancer has become a global standard of care.11 22

Current guidelines for cancer care mandate structured reports for histopathology staging after surgery has been undertaken but there is no such standard for radiology cancer reporting prior to treatment by surgery or preoperative therapy. At present, there is neither any evidence of a national standard for cancer radiology reporting nor data on whether there is potential for improvement using proformas. This national quality improvement initiative was therefore launched through the Royal College of Radiologists to test whether the completeness of radiological cancer assessment could be improved through a nationally introduced pilot of proforma-based reporting for a group of common cancers (CAncer Staging using ProformA Reporting in Radiology, (CASPAR)).

Methods

The project was jointly initiated by the Royal College of Radiologists (RCR) and the National Cancer Intelligence Network (NCIN) and was designed in consultation with representatives from the Royal College of Physicians, Royal College of Surgeons of England and the Royal College of Pathologists. This collaborative proposal was jointly funded by the Academy of Medical Royal Colleges and the RCR.

This study did not require Research Ethics Committee approval as only anonymised patient data (MDT radiology reports) and National Health Service (NHS) staff interview/questionnaires were used.23 The requirements of the Data Protection Act 1998 and the clinician’s common law duty of confidentiality were met by the preanonymisation of all patient records by clinical care staff. Only centres that obtained written approval from the Trust Data Protection Officer (Caldicott Guardian) to release anonymised radiology reports to the CASPAR team for analysis were included. One centre did not obtain Caldicott agreement and was excluded from the study.

To compare the minimum datasets of prognostically and therapeutically important staging data from radiology reports before and after adoption of proforma-based reporting.

To determine:

how pilot centres implemented proforma reporting and any areas of difficulty;

the usefulness of support workshops and guidelines;

the clinical impact of proformas from the radiology MDT lead and end-users (core MDT members).

Secondary objectives

The project was conducted in the UK NHS hospitals by radiologists reporting newly diagnosed lung, prostate, endometrial, cervical, colon and rectal cancer working within their respective MDTs. Expressions of interest were sought from the UK Radiology departments via the RCR website and an email invitation to all RCR Regional Chairs, the leads of all Special Interest Groups (SIG) and members of the NCIN Site-Specific Clinical Reference Groups (SSCRG). Participating centres were selected by the CASPAR Steering Group to represent a spectrum of the UK NHS hospitals, to maximise participation from the 2012 strategic health authority regions, ensuring the ratio of non-teaching to teaching hospitals was weighted proportionately.

Based on the criteria above, 21 centres were selected to take part in the evaluation. Sample size estimate allowed for an expected 10%–15% drop-out rate.

A workshop was held to launch the project, this provided a project overview and demonstrated the six pilot proformas (lung, prostate, endometrial, cervical, rectal and colon) (see online supplementary appendix 1). The pilot proformas were designed by the tumour site leads, with input and feedback from the relevant SIG and SSCRG. Breakout groups were held for each tumour site, where the individual proformas and guidance were explained in greater detail. Participants were requested to complete feedback forms. A follow-up teleconference was held to answer remaining queries.

This was an interventional ‘before and after’ study. In order to reduce the risk of bias in reporting standards preproforma introduction, reports were submitted from 3 months prior to and following the introduction of proforma reporting. To account for differences in the estimated cancer-specific diagnosis rates between centres, the specific periods were modified for recruiting site and tumour type.

Pretreatment MDT radiology cancer staging reports for the six cancer types were eligible for inclusion. For pelvic malignancies, this included local staging pelvic MRI reports and CT assessment for metastatic disease. For lung and colon cancers, this included CT report for both primary and metastatic disease staging. Only tumour staging reports as documented by the radiologist (either MDM radiology report, report addendum following MDM or staging cancer report) were acceptable. Annotations made by the clinical teams or MDT coordinators during MDT discussions were not accepted. Imaging reports submitted not fulfilling the above criteria were excluded.

Cohort 1 (preproforma (free-text) reporting): consecutive patients for whom a cancer staging radiology report was submitted prior to implementation of proforma reporting.

Cohort 2 (postproforma reporting): consecutive patients for whom a cancer staging radiology report was submitted following implementation of proforma reporting.

The radiology reports were completed by consultant radiologists. The study was non-blind, radiologists were aware of participation in the study in the preproforma and postproforma cohorts.

The following staff were asked to provide feedback on the use of the proforma reports:

Radiologists who had completed at least one proforma report.

Clinical end-users (MDT core members) who had used at least one proforma report for decision-making.

MDT radiology reports and staff feedback questionnaires were collected between March 2012 and April 2013. The project was extended from the original 3-month preproforma and 3-month postproforma duration to allow for differences in the rates of cancer incidence and to allow time for implementation of proformas into the radiology information systems (RIS).

The key minimum staging items considered essential to making clinical treatment decisions were defined by consultation with the NCIN SSCRG comprising lead specialist multidisciplinary representatives. Cancer-specific proforma report templates were produced to include these key data items considered clinically important for cancer treatment and prognosis (see online supplementary appendix 2). These were approved by the respective UK SIG and the NCIN SSCRG. The completeness of reports was assessed using scoring/coding forms (designed by project leads) that listed the presence or absence of the predetermined key staging data (see online supplementary appendix 3). Staging items that were not applicable to a particular case were deducted from the ‘total’ count to produce a ‘total needed’ count.

All free-text (preproforma) report scoring was carried out by experienced members of the project team. All proforma report scoring was carried out by an independent data analyst team and queries were referred to the project team.

Standardised questionnaires were used to solicit staff feedback on the usefulness of proformas in reporting imaging findings (radiologists) and facilitating clinical decision-making (end-users).

Data analysis

A project database was developed by the independent data analyst team. The database was checked by the independent data analyst team for completeness and checked against the data collection form, any missing data were identified and corrected as appropriate. A 10% sample of coded and source reports were sent to the independent data monitoring committee (DMC) to assess quality and fairness of coding of preproforma and proforma reports (see online supplementary appendix 4). The DMC also checked that recruitment was adequate to meet the number needed based on the three scenario power calculations (table 1).

Table 1

Power calculations

Proportion difference	Power (%)	Significance (%)	Sample size needed
0.10	90	5	518
0.20	90	5	124
0.30	90	5	51

Statistical analysis for the primary end point

Hypothesis: the introduction of proforma reporting improved the completeness of reporting in the cancers tested by an expected 20% with an expected completeness rate preproforma of 50% (based on a previous internal audit). A difference in the percentage of completed data items between proforma and non-proforma reports of at least 20% following proforma introduction required a sample size of 124 cancer reports per cancer type prior to and after the introduction of proforma reporting, with 90% power and 5% significance.

Sample size calculations with variable proportion differences in completeness of reports to achieve at least 90% power and 5% significance were calculated as mentioned in table 1.

Thus, a total of 248 (124 free-text and 124 proforma) cancer reports per cancer type were required to show an increase of 20% completeness of reports between preintervention and postintervention cohorts.24

Primary objective

Differences in completeness of reporting of the predefined minimum staging data were calculated before and after proforma implementation. The data were analysed for the whole sample and stratified by tumour site and reporting hospital. The 95% CIs for proportions of completed data items were calculated by the method of Wilson.25 Differences in proportions of completed data items preproforma and postproforma reporting were calculated and CIs for these differences calculated using method 10 of Newcombe.26

Secondary objective

A qualitative analysis through questionnaire responses was undertaken to evaluate the secondary objectives.

Results

The study flow and landmarks are summarised in figure 1. A total of 36 radiology departments expressed an interest in taking part in the evaluation. Twenty-one centres attended the launch meeting workshop and enrolled to participate in the project.

Figure 1

Study flow and landmarks. RCR, Royal College of Radiologists.

Primary end point

Two centres (5 and 16) failed to supply any data, 62 preproforma and 3 proforma reports did not comply with the inclusion criteria, and were excluded.

Nineteen centres provided preproforma free-text reports for inclusion in the study (table 2). Of these, four centres provided preproforma reports only (centres 6, 8, 14 and 21). In total, 15 of the 19 centres provided both preproforma and postproforma reports for at least two tumour types (table 2). The total number of reports provided by cancer type is summarised in table 3.

Table 2

Percentage of data fields completed by centre

Centre	Pre				Post				Proportion difference in completeness	95% CI
	Total number of reports	Number of data items completed	Total needed	Total % completeness	Total number of reports	Number of data items completed	Total needed	Total % completeness
1	62	401	920	43.6	34	312	440	70.9	0.27	0.22 to 0.32
2	18	109	265	41.1	30	390	433	90.1	0.49	0.45 to 0.55
3	40	225	523	43.0	18	226	240	94.2	0.51	0.45 to 0.56
4	52	373	717	52.0	52	672	718	93.6	0.42	0.37 to 0.46
5	0	-	-	-	0	-	-	-	NA	NA
6	12	127	201	63.2	0	-	-	-	NA	NA
7	84	516	1210	42.6	45	559	702	79.6	0.37	0.33 to 0.41
8	56	447	899	49.7	0	-	-	-	NA	NA
9	32	268	508	52.8	56	884	917	96.4	0.44	0.39 to 0.48
10	20	126	295	42.7	23	274	352	77.8	0.35	0.28 to 0.42
11	57	495	836	59.2	45	507	586	86.5	0.27	0.23 to 0.32
12	41	317	602	52.7	27	391	419	93.3	0.41	0.36 to 0.45
13	43	347	600	57.8	36	432	460	93.9	0.36	0.31 to 0.40
14	45	252	648	38.9	0	-	-	-	NA	NA
15	61	452	879	51.4	44	440	550	80.0	0.29	0.24 to 0.33
16	0	-	-	-	0	-	-	-	NA	NA
17	72	500	1053	47.5	20	238	272	87.5	0.40	0.35 to 0.45
18	36	224	519	43.2	27	279	302	92.4	0.49	0.44 to 0.54
19	14	69	186	37.1	16	203	210	96.7	0.60	0.52 to 0.66
20	20	106	281	37.7	23	236	319	74.0	0.36	0.29 to 0.43
21	22	232	328	70.7	0	-	-	-	NA	NA
Total	787	5586	11 470	48.7	496	6043	6920	87.3	0.39	0.37 to 0.40

NA, not available.

Table 3

Percentage of data fields completed by tumour type

	Lung cancer		Prostate cancer		Cervical cancer		Endometrial cancer		Colon cancer		Rectal cancer		Overall cancer
	Pre	Post	Pre	Post	Pre	Post	Pre	Post	Pre	Post	Pre	Post	Pre	Post
No. of proformas	125	84	156	108	117	46	112	59	142	88	135	111	787	496
Staging item completed	1509	1236	885	871	918	596	823	921	707	1049	744	1370	5586	6043
Staging items needed	1969	1359	1944	1086	1877	720	2005	1059	1775	1132	1900	1564	11 470	6920
AP totals	76.6%	90.9%	45.5%	80.2%	48.9%	82.8%	41.0%	87.0%	39.8%	92.7%	39.2%	87.6%	48.7%	87.3%
Proportion difference	0.14		0.35		0.34		0.46		0.53		0.48		0.39
95% CIs	0.12 to 0.17		0.30 to 0.37		0.30 to 0.37		0.43 to 0.49		0.50 to 0.55		0.46 to 0.51		0.37 to 0.40
Mean completed	76.6%	90.9%	45.4%	80.1%	48.2%	82.4%	41.0%	87.0%	39.9%	92.7%	39.3%	87.5%	48.1%	86.9%
Median completed	76.5%	93.8%	41.7%	90.9%	47.1%	88.2%	44.1%	94.4%	38.5%	92.3%	40.0%	92.9%	46.2%	92.9%
SD	19.8%	10.4%	19.1%	23.4%	17.5%	15.7%	13.5%	13.7%	14.9%	8.8%	17.4%	14.7%	21.4%	16.2%
Min	25.0%	56.3%	0.0%	33.3%	11.8%	41.2%	0.0%	66.7%	7.7%	69.2%	0.0%	41.7%	0.0%	33.3%
Max	100.0%	100.0%	92.3%	100.0%	88.2%	100.0%	77.8%	100.0%	76.9%	100.0%	93.3%	100.0%	100.0%	100.0%
IQR 1	60.0%	87.5%	30.8%	63.6%	35.7%	70.6%	33.3%	72.2%	30.8%	84.6%	27.9%	78.6%	33.3%	78.6%
IQR 3	100.0%	100.0%	58.3%	100.0%	58.8%	94.1%	50.0%	100.0%	46.2%	100.0%	50.0%	100.0%	60.0%	100.0%

The total number of preproforma reports for cervical and endometrial cancer and postproforma reports for all of the tumour types was less than 124. However, following review of the data by the DMC, the CASPAR group were advised to stop collecting data as the sample size had met the endpoint of showing a significant improvement in staging items for all cancer types including cervical cancers.

A total of 787 preproforma and 496 postproforma staging reports met inclusion criteria for analysis. The proportion of completed staging data from 787 preproforma staging cancer reports were 5586 of 11 470 staging items (48.7%), compared with 6043 of 6943 staging items using proforma reports (87.3%). The improvement in cancer staging achieved by proforma reporting amounted to an absolute increase of 38.6% (95% CI 37% to 40%). Thus, the overall improvement was significant and surpassed 30%. A significant improvement in completeness of reporting was not only observed for each cancer type but was also seen in each one of the 15 centres that submitted both pre- and post-proforma reports (table 1 andtable 2).

An improvement in completeness was seen across all tumour types, and the improvement was >30% for four of the six tumour types (table 3). For lung cancer however, the percentage improvement was lower but still significant at 14% (95% CI 12% to 17%), this probably relates to the high percentage completeness of the preproforma lung cancer staging reports (76.6%). For cervical cancer, the improvement in completeness was still significant at 34%; for the 46 patients assessed.

The distribution of elements of staging data by cancer site is summarised in online supplementary appendix 5, tables 1-6). For lung cancer, two staging items (differentiation from consolidation and metastases) were less complete on the proforma reports compared with free-text reports, but the difference was small: 3% and 7%, respectively. There were no other instances of a decrease in the completeness of staging items when proforma reports were compared with preproforma reporting.

For lung cancer staging, significant improvements in 2/17 minimum data cancer staging items were observed. There was a notable improvement in the documentation of endobronchial and pleural disease using proformas. Prostate proforma introduction saw 30% or greater improvement in 9/13 staging items, of particular clinical relevance was the improvement in documentation of local invasion and tumour, node, metastases (TNM) stage. Proforma reporting of endometrial cancer produced a 30% or greater improvement in reporting of 12/18 staging items. The most striking improvements were in involvement of the serosa and pelvic organs, all crucial to surgical decision-making and prognosis. For cervical cancer, an improvement of >30% was seen in 9/17 staging items following proforma reporting. One of the greatest improvements was for pelvic side wall invasion, a predictor of pelvic nodal involvement. For rectal cancer staging proforma reports, improvements were seen in 13/15 staging items including extramural spread and extramural vascular invasion. Both are important prognostic markers and guide selection for neoadjuvant therapy. Marked improvement in 10/13 staging items was seen by the use of the colon cancer proforma reports. The greatest improvements were for peritoneal infiltration and resectability—both critical for surgical success.

A wide range of percentage completeness in individual reports was seen, before and, to a lesser degree, after the introduction of proformas. For example, the range of completeness of lung cancer report was 25%–100% (preproforma) and 56%–100% (postproforma) and for prostate was 0%–92% (preproforma) and 33%–100% (postproforma). This probably, at least in part, reflects the difference in reporting style between individual radiologists. The effect of proforma reporting was not studied in individual radiologists. The range of percentage completeness reduced and the mean completeness increased for all cancer types after the introduction of the proforma. However, it is noted that even in the postproforma cohort, there were incomplete reports. It is unclear, without further assessment, the reasons for this. Possibilities include difficulties in using the proforma, inexperience or uncertainty in evaluating certain parameters or it could reflect limitation of the imaging modality.

Secondary end points

Some queries raised regarding the lung staging proforma were resolved by teleconference. For the remaining cancer-specific workshops, 100% of the attendees agreed that ‘the presentation given in this session was very clear’ and 80%–100% agreed that ‘they can see how (they) can use this proforma in clinical practice’. There was an average of 67% agreement among the workshop attendees that ‘(they) feel confident to explain the use of this proforma to colleagues’.

During the study, six sites reported problems encountered with implementation of the proforma into their RIS systems. These included unavailability of the software upgrade within the project timeframe. For one site, the RIS system did not use voice recognition so paper versions of the template were manually completed.

Feedback was received from 11 of 21 centres participating in the launch meeting. All sites indicated moderate to strong agreement that the proformas were self-explanatory, included all key items and improved report quality. Feedback from those centres unable to submit proforma reports is summarised in box 1. Suggestions for improving proforma design included: mechanisms to document equivocal findings, reduce the time taken document negative findings and to include incidental findings. For three sites, inability to engage colleagues and time pressure were cited as limiting factors and four sites indicated that lack of IT support from RIS suppliers resulted in failure to implement the proformas. Technical barriers to integration of proforma report templates into existing RIS is clearly an important obstacle to implementation.

Include a section for documenting other findings (site 3).

An alternative approach might be to follow an algorithm only specifically mentioning positive findings as they are observed, rather than producing a report characterised by a long list of negative findings (site 17).

Very comprehensive many more items included than normally explicitly mentioned in my usual reports.

Comprehensive but much more time consuming than our current (site 17, 21).

More detailed guidance would have been helpful(site 4).

Ability to state equivocal findings. Proforma does not work well in cases which are not definite cancers or where there is uncertainty (site 4, 17, 21).

Although unable to implement the proforma, this is considered important to standardise the reporting of cancer without missing many important or relevant findings. In some respects, they are a good template for primary reporting, not just for reviews. Proforma reporting in principle is a good idea (site 4, 17).

The reporting format should be made available to RIS/PACS all over National Health Service and should be mandatory (site 4).

Heavy workload. Have lost colleagues. Concerns over prescriptive proforma-based reporting (site 3).

Cannot force colleague radiologists to do it (site 4).

One to one conversations and email reminders to colleagues. Most colleagues made one attempt to complete a proforma report and abandoned it due to the amount of time required compared with unstructured reporting. Not prepared to reconsider despite attempts to persuade them (site 17).

RIS not supportive of proforma. We explored possibility of setting up a template, but given the potential difficulties, we went for a pragmatic solution of manually filling in proformas alongside radiology report (site 3).

The forms had to be scanned on CRIS—not ideal. In support of the concept but the only way it can work is if it is tightly integrated into CRIS so the radiologist can electronically tick the boxes as images are reported. HSS have still not incorporated the proformas into CRIS for digital reporting; if they had, I feel we could all be persuaded to continue to use the proformas whenever possible/routinely. Early implementation in a PACS/CRIS friendly format is what I look forward to. Enthusiasm was very high in our department but the lack of integration into CRIS has meant that participation will not be ongoing until we can integrate (site 4).

Sunquest RIS did not have ability for e-form, but we did put equivalent of proformas on VRS for endometrium, cervical and prostate. The RIS system was complicated and the reports produced were not user-friendly. The report produced in our RIS system looked very cluttered and found them very difficult to follow (site 21).

End-user feedback was received from 35 MDT participants (across seven centres), including surgeons, medical and clinical oncologists and CNS (figure 2). Most respondents, 27/35 (77%), found proforma reports contributed positively to cancer staging, 27/35 (77%) and 28/35 (80%) agreed they improved MDT efficiency and data collection, respectively. Interestingly, 15/35 (43%) end-users felt that proforma reports had no impact on diagnosis, this maybe because diagnosis is often multifaceted, that is, also based on clinical examination and histological information. Feedback was received from 32 MDT lead radiologists (figure 3), 26/32 (81%) respondents found it a worthwhile exercise and 16/32 (50%) felt proforma reporting improved the quality of their reports, whereas 5/32 (16%) respondents did not feel it improved quality and 9/32 (34%) were neutral. Eighteen of 32 (56%) radiologists reported no technical difficulties completing the form. However, of 28 responses, the majority, 20/28 (71%) found proforma report took longer to complete than free-text reports (figure 4).

Figure 2

MDT end-user rating of impact of proforma reporting (n=35). MDT, multidisciplinary team.

Figure 3

Multidisciplinary team lead radiologist’s rating of proforma reporting (n=32).

Figure 4

Radiologists’ feedback on time taken to complete proforma reports (stratified by number of reports completed (n=28)).

Discussion

Main findings

The study has shown that proforma-based reporting was successfully implemented in 15 of the 21 centres with 1283 cancer staging reports submitted. The implementation resulted in a significant global improvement in the proportion of prognostic and therapeutically important cancer imaging features reported by radiologists—from 48.7% completeness using free-text reports to 87.3% using proformas, showing a 78% improvement in staging completeness. Improvements were seen across all the cancer types and all 15 centres. Since the quality of this information drives preoperative cancer treatment decisions, this has profound implications for the quality of care in patients with newly diagnosed cancer. Proforma reports also improved the consistency of completeness of cancer staging data.

Of the preproforma report cohort, lung cancer had the greatest completeness (75%). This was the only cancer type that did not have a >30% improvement following proforma reporting (a 14% improvement was still observed). A possible explanation for this is that lung cancer is the the most common cancer in the UK; furthermore, the TNM staging system is very clear and comprehensive and is the only classification that is included in the core curriculum for radiology trainees.27 Thus, most radiologists, whether they attend the MDT or not, will be familiar with staging lung cancer and have a practised approach to reporting.

Study feedback reflected high acceptability of structured reports. The clinical teams that make treatment decisions based on radiological assessment of cancer found proforma reports helpful for treatment planning and MDM efficiency. A few centres reported inability to deploy the template proformas into RIS systems as a major barrier. The majority of radiologists considered proforma reporting more time consuming than free-text reporting. Highlighting once again that one of the perceived major obstacles to uptake by radiologists is increased time needed to complete a proforma report. Arguably, free-text reporting is faster when there is so much missing information and would take much less time to produce compared to more detailed structured reports. If it is accepted that a radiology cancer staging report should include all the prognostic information to manage a patient with cancer, then it is logical to conclude that a prepopulated template with the required information set out will be much faster to complete than a free-text report. On the other hand, it seems likely that a structured template would be time saving for those radiologists already undertaking comprehensive free-text reporting. To produce radiology cancer staging reports of a better quality and standard may take more radiology time, but given the importance of cancer imaging assessment in treatment decision making providing such levels of detail routinely would inevitably improve patient care.

Our audit has revealed that if preproforma reports had been used in MDMs they would not have met the national standards for MDM working. Thus, when staging items are missing on cancer staging reports, the radiologist taking the MDT must provide this information. The extra time taken to do this, which will be proportional to the amount of missing data, is rarely acknowledged.

Proforma reports also provide an educational resource, especially for radiologists and trainees who do not regularly attend the relevant cancer MDM and so may not appreciate the staging items pertinent to clinical decision-making.

Progress in cancer treatment has been paralleled by developments in imaging technology that enable more accurate and detailed radiological evaluation. Despite the increase in the complexity and amount of information that needs to be interpreted and conveyed by the radiologist, the reporting style has largely remained unchanged from its original free prose format. While the deficiencies in some reports may be rectified on MDM review, this is not a reliable or efficient method and is inconsistently documented. Currently, only clinical T, N and M data are recorded for the cancer registries. Consequently, it may not always be possible to determine the basis on which treatment decisions for patients were made.

Strengths and weaknesses in relation to other studies

Previous studies have highlighted deficiencies in cancer staging information from free-text reporting for various cancer types.28 29 Furthermore, studies have shown structured reports to improve completeness and clarity.13–19 30–32 A study of radiological assessment of pancreatic cancer showed proforma reporting improved assessment of resectability and confidence in treatment decisions.29

The management options in cancer treatment are ever increasing, and there is now an established evidence base for the selective use of preoperative treatment to improve outcomes in many cancers.21 However, there remain wide variations in cancer care and outcomes in the UK, as demonstrated for lung cancer management in a recent large UK study.33 Radiology proforma reporting could improve cancer staging data available for national cancer statistics, which in turn could be used to identify the causes of variation in cancer care.

The pathology model has shown that a structured report template provides an effective conduit for capturing and storing data, which in turn is easier to extract and view.14 34 Structured radiology cancer reporting provides high-quality and more complete information that is more conducive to data gathering. With the increasing emphasis on healthcare systems to demonstrate regular and robust quality assessment followed by improvement, the structured format is well suited to audit and research. It also facilitates the development of ‘bioregistries’ and tumour databanks.

Strengths and weaknesses of this study

Four of the 19 participating sites failed to show an improvement in cancer reporting by proforma implementation. Feedback indicates that these four sites struggled to overcome barriers to proforma implementation – such as technical difficulties with integration into RIS and poor uptake/time pressures by reporting radiologists. However, with similar constraints, 78% of hospitals not only managed to implement the proforma reporting but also demonstrated an overall 80% improvement in the completeness of cancer reports. Therefore, it should be possible to roll-out and overcome technical and personnel difficulties by modeling the successes achieved in pilot sites. The difficulties with integration of proforma templates into IT systems will need to be addressed by commercial RIS providers. They will need to ensure there is an effective user-template interface so that using templates in regular reporting practice is easy and efficient. Despite using proformas, some staging information was still incomplete, even in users that volunteered to participate in the study. We hope that in future this would be corrected by improvements in radiology user interface software which will not permit a report to be signed off unless all fields have an entry.

Undoubtedly, the advances in software through artificial intelligence and machine learning will enhance the prospects of delivering comprehensive cancer staging by radiologists at diagnosis and in time for initial critical therapeutic decision.

A further limitation of our study is that while improving the content and quality of the report through measuring completeness, it was beyond the scope of this study to assess the accuracy of individual reports and indeed the greater task of whether this translates into improved outcomes. However, the the accuracy and limitations of these modalities in cancer staging have already been extensively evaluated and staging items included in proforma reports have already been previously validated in terms of prognostic accuracy.2 Therefore, we could infer that by their routine inclusion in staging reports, it is likely that treatment stratification, and thus patient outcome, would be improved.

The implementation of the structured reporting template was a non-blinded intervention, thus the degree of report completeness, including in the preproforma cohort, may have been inflated by the process of this as an audited measure (a Hawthorne effect). Arguably, an ongoing “Hawthorne effect” could also be an advantage of introducing standardised proforma reporting as a nationally audited quality measure of excellence in cancer care.

Implications for doctors and policy makers

The Royal College of Radiologists CASPAR working group have shown that minimum dataset cancer staging radiology reports, like pathology minimum dataset reports, should be a mandatory standard for patients with newly diagnosed cancers. This model of proforma reporting is amenable to modifications, and could be expanded to other cancer types, developed with the input of relevant cancer specialists. In the future, the aim should be towards developing evidence-based validated reporting templates with a standardised structure and content including expert consensus agreed essential reporting elements.

Structured proforma reporting clearly improves the information available that is needed for patient care. To facilitate proforma template implementation and utilisation on a national scale, support through education, training and IT infrastructure improvements will be needed. This will require collaboration between RIS providers and the RCR. Manufacturers need to improve functionality to enable easier integration of proforma report templates into RIS/IT systems to ensure that proforma reporting can be implemented efficiently without becoming burdensome or time consuming for radiologists.

Sufficient resource will be necessary to test and maintain radiologists’ competence in such a crucial component of cancer care to safeguard the consistency of standards. Measuring the quality and accuracy of radiology reports against pathology (where available) and outcomes will contribute to this.

Unanswered questions for future research

Clinical research has already established that items recorded on proforma are of prognostic significance. A consistent system of recording cancer features as seen on imaging and collating this nationally will be an effective means of driving improvements in cancer care. The accuracy, consistency and completeness of radiological assessments by individual radiologists could be measured against patient outcomes and detailed analysis of data retrieved from radiology proformas may help us to better understand the wide variations currently observed in cancer patient outcomes