A multicentre randomised controlled trial of reciprocal lung cancer peer review and supported quality improvement: results from the improving lung cancer outcomes project.

BACKGROUND: Results from the National Lung Cancer Audit demonstrate unexplained variation in outcomes. Peer review with supported quality improvement has been shown to reduce variation in other areas of health care but has not been formally tested in cancer multidisciplinary teams. The aim of the current study is to assess the impact of reciprocal peer-to-peer review visits with supported quality improvement and collaborative working on lung cancer process and outcome measures.
METHODS: English lung cancer teams were randomised to usual care or facilitated reciprocal peer review visits followed by 12 months of supported quality improvement. The primary outcome was change in the following national audit indicators; mulitdisciplinary team discussion, histological confirmation, active treatment, surgical resection, small-cell chemotherapy and specialist nurse review. Patient experience was measured using a new lung cancer patient questionnaire in the intervention group.
RESULTS: Thirty teams (31 trusts) entered the intervention group and 29 of these submitted a total of 67 quality improvement plans. Active treatment increased in the intervention group (n=31) by 5.2% compared with 1.2% in the control group (n=48, mean difference 4.1%, 95% CI -0.1 to 8.2%, P=0.055). The remaining audit indicators improved similarly in all groups. Mean patient experience scores in the intervention group did not change significantly during the study but a significant improvement was seen in the scores for the five teams with the worst baseline scores (0.86 to 0.22, P<0.001).
CONCLUSIONS: Reciprocal peer review with supported quality improvement was feasible and effective in stimulating quality improvement activity but resulted in only modest improvements in lung cancer treatment rates and patient experience.

RCT Entities:   Population Interventions Outcomes

Lung cancer is the commonest cause of cancer death in England and Wales with around 38 000 cases diagnosed each year and ∼35 000 deaths. Data from the National Lung Cancer Audit (NLCA) demonstrate significant variation in process and outcome measures across England. In 2009 there was a three-fold difference in survival and active treatment rates, which persisted following case mix adjustment (Beckett ). Furthermore, reported lung cancer outcomes in the UK are worse than other comparable European countries (Walters ) and have improved little in recent years (Khakwani ). It has been estimated that if survival rates were increased to that of the best in Europe, around 1300 lives could be saved each year in the United Kingdom (Abdel-Rahman ).

Variation in health care is not unique to lung cancer and addressing unwarranted variation is challenging (Wise, 2010). Although external regulation may have a role in some areas, this approach is more difficult to apply to the complex pathways involved in lung cancer diagnosis and treatment. Peer review with supported quality improvement offers a promising alternative but the evidence for its effectiveness is limited. The Washington State's Surgical Care and Outcomes Assessment Program utilised a peer support programme to share the best practice which led to a significant reduction in post-operative complications (Kwon ). Within the United Kingdom, the national COPD resources and outcomes project demonstrated that reciprocal peer-to-peer review led to only limited quantitative differences in the quality of services offered (Roberts ). A qualitative analysis of this study identified a number of barriers to improvement including difficulties in establishing effective working relationships, funding changes and service re-design. In 2003, the Institute for Healthcare Improvement described the collaborative model to achieve a breakthrough improvement (Institute for Healthcare Improvement, 2003). Collaboratives allow teams working on the same issue to share good practice and innovation permitting others to take these ideas and implement them in the context of their own organisation, resources and case mix. Pronovost successfully employed this collaborative approach, together with supported quality improvement, to implement five evidence-based interventions on the intensive care unit resulting in the reduction in catheter-related bloodstream infections to zero. These studies offer a persuasive proof of concept, but the absence of a control group or of patient-specific outcomes measures limits their implementation in other disease areas such as cancer.

The aim of the current study is to determine whether a programme of reciprocal peer-to-peer review visits with supported quality improvement and collaborative working can significantly improve lung cancer process and outcome measures, and thus reduce unwarranted variation in outcomes.

Materials and methods

Study design

We conducted a prospective randomised controlled trial.

Study population

One hundred and sixty-two English NHS trusts were identified from the 2008 NLCA annual report. Centres only providing treatment (not diagnostics), orthopaedic hospitals and ambulance trusts were excluded. Invitations to participate were sent to the remaining 152 trusts. Trusts who agreed to participate and who had 2008 NLCA case ascertainment rates of > 50% expected were paired before randomisation on the basis of contrasting results for four key indicators from the NLCA. The indicators were active treatment rates, surgical resection rates, median survival and the proportion of patients assessed by a clinical nurse specialist. Each trust was colour coded for each indicator, red if below the national average and green if above. By placing each trust with its colour-coded indicators on a map, we were able to pair trusts on the basis of a contrasting mixture of red and green indicators and a travel time between centres of around 2 h. On the basis of data from the national COPD resources and outcomes project, we determined that we would be able to complete 30 peer review visits during the lifetime of the project thus allowing 30 lung cancer multidisciplinary teams (15 pairs) to be randomised into the intervention arm. Randomisation was performed in a blinded fashion by assigning a random number to each pair of trusts and then allocating pairs numbered 1–15 to the intervention group. The remaining trusts formed either the control group (if they had agreed to participate) or the non-participant group and had no further contact with the study team but continued to submit data to the NLCA as usual.

Intervention

The study timeline is shown in Figure 1. Following introductory workshops the multidisciplinary teams within each pair undertook facilitated reciprocal site visits. The visits consisted of observation of the host team's multidisciplinary team meeting, three discussion sessions focusing on the functioning of the mulitdisciplinary team meeting, the host team's NLCA data and patient experience questionnaire results. The final session aimed to identify the focus of improvement work to be undertaken by the host team. The quality improvement facilitator introduced a structured template for the quality improvement plans and provided a short introduction to using the model of improvement to guide implementation of the plans. Over the next 12 months the quality improvement facilitator provided support via electronic mail, telephone and follow-up visits, where required. Teams within the intervention group supported each other via mini-collaboratives in the form of web-based teleconferences and two face-to-face workshops.

Figure 1

Study timelines.

Outcomes

Changes in process and outcome were assessed using data from local quality-improving plans and the following indicators from the NLCA: the proportion of patients discussed at a multidisciplinary team meeting, histological confirmation rate, active treatment rate, surgical resection rate, the proportion of patients with small-cell lung cancer receiving chemotherapy and the proportion of patients seen by a lung cancer nurse specialist. Patient experience was assessed in the intervention group using a new lung cancer-specific patient experience questionnaire designed in collaboration with the Roy Castle Lung Cancer Foundation. The questionnaire included 11 questions selected with permission from the previously validated 2004 national cancer patient survey. The questions covered the following domains: communication, privacy, respect and dignity and three free text questions (see Appendix I). Participating teams were asked to distribute 30 questionnaires to patients recently seen in their services. The clinical nurse specialists distributed the questionnaires to patients who anonymously returned them to the Royal College of Physicians. An independent qualitative ethnographic evaluation of the study was undertaken by the Social Science Applied to Healthcare Improvement Research Group at the University of Leicester.

Statistical methods

Data were tested for normality using the Shapiro–Wilk test. Baseline NLCA indicators were taken from the 2009 NLCA report and the intervention, control and non-participant groups were compared using a χ2- test. The changes in NLCA indicators from 2009 to 2011 were compared using an independent t-test. Patient experience questionnaire responses for each question were labelled and re-coded to separate them into the worst patient experience category (score 1) vs all other responses (score 0). These scores were then summated to create a domain and a total patient experience score with a possible range of 0–11, whereby a higher score indicates a worse patient experience. Analyses were performed using the statistical software package SPSS (International Business Machines Corp., Armonk, NY, USA).

Funding and ethics

The study was funded by a ‘Closing the Gap' grant from the Health Foundation. The National Research Ethics Service confirmed that the study was service evaluation and quality improvement and did not require ethical review.

Results

One hundred trusts (66%) replied to the invitation to participate and 91 (61%) agreed to participate in the study. Eighty-one trusts had 2008 NLCA data of sufficient quality to allow pairing. Two trusts provided a joint multidisciplinary team allowing 40 pairs of multidisciplinary teams to be created. One pair agreed to act as a pilot and was excluded from further analysis. Of the remaining 39 pairs, 15 pairs (31 trusts) were randomised to the intervention group. The remaining 24 pairs formed the control group. During the study, two trusts in the control group amalgamated to form one trust so the total number of trusts in the control group was 47 (Figure 2).

Figure 2

Consort diagram, disposal of eligible trusts including screening, randomisation and follow-up.

Quality improvement plans

Two hundred and thirty medical professionals from 31 trusts participated in the review visits. Twenty-nine teams submitted a total of 67 quality improvement plans. The issues identified in the quality improvement plans are shown in Table 1. Eighteen teams collected local data to measure impact. An example of such data is shown in Figure 3. This trust identified small-cell lung cancer chemotherapy as an area for improvement. They introduced a number of changes to their diagnostic and treatment pathways including prioritisation of small-cell pathology reporting, faxing of the results to the multidisciplinary team coordinator and lung nurse specialist to allow early booking of oncology appointments. These changes were monitored using a run chart that demonstrated a reduction in the time from multidisciplinary team meeting to chemotherapy treatment and an increase in the proportion of small-cell lung cancer patients receiving chemotherapy from 60% in 2009 to 71% in 2011.

Table 1

Quality improvement plan themes

Quality improvement plan theme	Number of plans
Multidisciplinary team effectiveness	31
Diagnostic pathways	13
Treatment pathways	9
Access to clinical nurse specialists	8
Clinical trial recruitment	4
Patient experience	2

Figure 3

Run chart showing the waiting times from the multidisciplinary team meeting to the first treatment for 10 consecutive small-cell lung cancer patients following the implementation of the quality improvement plan at one trust in the intervention group.

National lung cancer audit indicators

Baseline (2009) NLCA indicators for the intervention, control and non-participant groups were similar (Table 2). The mean change for each NLCA indicator from baseline to 2011 in the intervention and control group is shown in Figure 4. The proportion of patients receiving active anti-cancer treatment in the intervention group increased by 5.2% compared with 1.2% in the controls (mean difference 4.1%, 95% CI −0.1 to 8.2%, P=0.055). The remaining NLCA indicators improved similarly both in the intervention and control groups.

Table 2

Baseline (2009) national lung cancer audit indicators

	Control (n=47)		Intervention (n=31)		Excluded (n=67)		P-value
	Mean (%)	s.e.m.	Mean (%)	s.e.m.	Mean (%)	s.e.m.	Control vs intervention vs non-participant	control vs intervention
Case ascertainment	158.1	38.6	122.0	7.2	107.4	3.6	0.220	0.455
Discussed at the MDT meeting	95.2	0.7	93.7	1.7	90.9	1.9	0.155	0.370
Histological confirmation rate	75.7	1.2	76.4	1.8	78.4	1.6	0.409	0.739
Active treatment	59.5	1.2	55.9	2.2	59.5	1.5	0.305	0.131
Surgery (all cases)	13.4	0.6	13.0	0.8	14.2	0.7	0.469	0.648
SCLC (chemo)	65.1	2.2	66.5	3.9	63.3	2.7	0.746	0.733
Seen by CNS	70.3	3.8	76.6	3.2	58.3	4.2	0.007	0.243
CNS present diagnosis	44.0	3.8	49.4	5.4	38.7	3.8	0.237	0.403

Abbreviations: CNS=clinical nurse specialist; MDT=mulitdisciplinary team; SCLC=small-cell lung cancer.

Data are shown as mean and s.e. proportion of patients.

Figure 4

Mean change in national lung cancer audit metrics from baseline (2009) to 2011. P=0.055 active treatment—intervention vs controls. Intervention n=31 trusts, control n=47 trusts and non-intervention (control and non-participants combined) n=66 trusts. Abbreviations: CNS, clinical nurse specialist; MDT, multidisciplinary team; SCLC, small-cell lung cancer.

Patient experience

In the intervention group, patient experience questionnaires were returned by 438 patients from 30 multidisciplinary teams at baseline (return rate 49%) and 372 patients from 27 trusts following the intervention (return rate 41%). Baseline total scores were low (0–1.31) indicating high levels of patient satisfaction with the care received, although there was a statistically significant (P<0.001) variation in results by the multidisciplinary team (Figure 5). In particular, the proportion of patients responding yes to the question ‘did you find that the person who told you about your diagnosis did so with sufficient sensitivity/care?' varied significantly by 57%–100% (P<0.001).

Figure 5

Total patient questionnaire scores by the multidisciplinary team in the intervention group at baseline (pre) and at the end of the study (post). A low score indicates better experience. Each symbol represents the mean score for each trust in the intervention group. The maximum possible score for the questionnaire is 11.

The total questionnaire scores did not change significantly during the study (0.22–0.17, P=0.377), however, the variation by the multidisciplinary team reduced (Figure 5). Given that the study aimed to bring the standard of the lower performing trusts to that of the best, we performed a post hoc analysis for the five trusts with the worst baseline patient experience scores. This demonstrated that the mean total score improved significantly for these trusts from 0.86 to 0.22, P<0.001. The biggest improvement in this group was seen in the proportion of patients responding yes to the question ‘did you find that the person who told you about your diagnosis did so with sufficient sensitivity/care?' which increased from 75% to 90% (P=0.05). One multidisciplinary team in this group achieved this improvement by using their baseline questionnaire results as a lever to encourage attendance at an advanced communications skills course.

The questionnaire domain-specific scores did not change significantly during the study. Of the individual questions, a significant improvement was seen in the rating of the quality of information provided as excellent, which rose from 53%–59%, P<0.05.

Qualitative evaluation

Participants' experiences were overwhelmingly positive. The reciprocal peer-to-peer visits with supported quality improvement were seen as a strong driver to change. The method of pairing multidisciplinary teams was important. In particular, pairing teams with different results, not just ‘good' with ‘bad', and allowing teams to visit each other's sites to ensure a two-way sharing of best practice. The independent quality improvement facilitator role was seen as crucial to ensure the visits remained focussed and that the engagement with quality improvement plans was maintained. Finally, the involvement of senior managers was crucial to the successful implementation of the quality improvement plans. The detailed findings from the independent evaluation of this project have been reported elsewhere (Aveling ).

Discussion

Lung cancer outcomes remain relatively poor and reducing unexplained variation is an attractive proposition to promote improvement. There are a number of ways that clinical teams may share best practice and innovative service delivery models, however, studies formally evaluating their impact are limited. To our knowledge, this is the first study to formally test a national quality improvement strategy, which aimed to bring the standard of all lung cancer teams to that of the best. We have demonstrated that reciprocal peer-to-peer review with supported quality improvement is both feasible and effective at stimulating local quality improvement activity but had a relatively modest and somewhat disappointing impact on process and outcome measures as measured by NLCA indicators and a new lung cancer patient experience questionnaire.

The facilitated reciprocal visits represented a new and unique opportunity for all members of a lung cancer team to exchange ideas in a supported environment, and to formally design then implement quality improvement plans. Nearly two-thirds of lung cancer multidisciplinary teams in England agreed to take part in the study and reassuringly baseline NLCA indicators did not differ significantly between participants and non-participants, suggesting that the willingness to participate in quality improvement activity is not related to baseline performance. There were a wide range of areas identified for improvement, but nearly half of the teams identified multidisciplinary team meeting effectiveness as a key issue. This is not surprising given that these meetings are pivotal in the lung cancer pathway. Live observation of each multidisciplinary team meeting followed by facilitated feedback proved to be a strong driver to improve on problems such as ensuring weekly presence of all the treatment specialists, as well as more simple issues such as room layout. The need to streamline diagnostic and treatment pathways was also identified as a common problem. Recent NICE guidance on the management of lung cancer (National Institute for Health and Care Excellence, 2011) recommended a paradigm shift in the diagnostic algorithm from performing multiple diagnostic and staging investigations to performing a single test that will provide both diagnostic and staging information. A number of teams within our study were able to introduce such pathways and demonstrate impressive reductions in diagnostic times and more prompt treatment. This, together with more effective multidisciplinary team working, may have led to the small increase in the active anti-cancer treatment rates seen within the intervention group. However, an alternative explanation for the improvement is regression to the mean, given that treatment rates in the intervention group were lower at baseline, and overall, the lack of significant improvement across the range of NLCA indicators in the intervention group was disappointing. One possible explanation for this is the challenge that some participating teams encountered converting enthusiastic quality improvement plans into tangible improvements for patients over a relatively short time period. The qualitative evaluation confirmed that participants often underestimated the time and energy required to implement and sustain change, and highlighted the importance of early engagement with hospital managers to maintain momentum (Aveling ). Alternatively, other national lung cancer initiatives implemented at the time of the study may have driven coexistent improvements in the control group. For example, the drive to encourage all lung cancer patients to be referred for clinical nurse specialist support has subsequently been shown to increase the probability that a lung cancer patient receives active treatment.

Although even small improvements in lung cancer treatment rates are very welcome, it is recognised that undergoing investigation for suspected lung cancer generates high levels of patient anxiety and many patients will remain too unwell to benefit from currently available drugs. The assessment of patient experience is therefore of particular importance in lung cancer. This has proved challenging in detailed national cancer surveys owing to the advance in age, poor health and short median survival of lung cancer patients. The response rate to our short questionnaire was relatively high at 41–49% compared with the 2011 national survey in which only 7% of lung cancer patients responded (Department of Health, 2012) but still represents the views of less than half of lung cancer patients and is a relative limitation in terms of generalisability of the results. It was reassuring to note that, at entry to the study, patients in the intervention group generally rated their experience as highly satisfactory. This may explain the low number of teams who specifically identified patient experience as an area for quality improvement. In terms of assessing the impact of the reciprocal peer-to-peer review visits and supported quality improvement on patient experience, it is likely that this high-baseline satisfaction, and the lack of patient experience data for the control group, limited our ability to detect a significant change. However, our results suggest that those teams with poor scores may be able to use patient experience data to promote significant improvements, particularly in areas such as communication skills. Further work is required to develop a lung cancer patient experience measure that is both acceptable to patients and able to detect small but clinically important changes in experience.

Although similar in name to the national cancer peer review process, there are a number of important differences between the reciprocal peer-to-peer review and supported quality improvement process employed in the current study and national cancer peer review. The latter predominantly performs a quality assurance role, ensuring that cancer teams meet a minimum standard via compliance with a number of process measures. Support with quality improvement is not provided and site visits are now rarely performed. The qualitative evaluation of our study highlighted the importance of an independent quality improvement facilitator to the success of the peer review visits and the subsequent implementation of the quality improvement plans. Integration of facilitated reciprocal peer-to-peer review and supported quality improvement into national cancer peer review, both for lung cancer and other tumour sites, is an attractive proposition and requires further study. However, our results suggest that this strategy alone is unlikely to have a major impact on lung cancer treatment rates. This phenomenon is not new in lung cancer, for example, the introduction and NICE approval of gefitinib treatment for the first-line treatment of lung cancer in 2010 was associated with only a 1% increase in active anti-cancer treatment rates over the following year (Health and Social Care Information Centre, 2012). Achieving a stepwise increase in lung cancer treatment rates and survival is likely to require a multi-targeted approach including earlier diagnosis, streamlined lung cancer pathways, new treatments and a reduction in unexplained variation via supported quality improvement programmes.