Knowledge or noise? Making sense of General Practitioners' and Consultant use of 2-week-wait referrals for suspected cancer.

BACKGROUND: Early diagnosis and treatment of cancer is the goal of the 2-week-wait referral pathway (2WW). Variation exists between General Practice use of 2WW and rates of consultant reprioritisation of GP referral from routine to 2WW (Consultant Upgrade). We investigated variation in General Practice and Consultant Upgrade 2WW referral activity.
METHODS: Data from 185 000 referrals and 29 000 cancers recorded between 2011 and 2013 from the Northern Ireland Cancer Waiting Time database (CaPPS) were analysed to ascertain standardised referral rate ratios, detection rate (DR) (=sensitivity) and conversion rate (CR) (=positive predictive value) for Practice 2WW referrals and Consultant Upgrade 2WW. Metrics were compared using Spearman's rank correlation co-efficients.
RESULTS: There was consistency in Practice and Consultant Upgrade 2WW referral rates over time, though not for annual DR (Spearman's ρ<0.37) or CR (Spearman's ρ<0.26). Practice 2WW referral rates correlated negatively with CR and positively with DR while correlations between DR and CR were restricted to single-year comparisons in Practice 2WW. In Consultant Upgrade, 2WW CR and DR were strongly correlated but only when the same cancers were included in both rates.
CONCLUSIONS: Results suggest 'random case mix' explains previously reported associations between CR and DR with more 'hard to detect' cancers in some Practices than in others in a given year corresponding to lower DR and CR. Use of Practice and Consultant Upgrade 2WW referral metrics to gauge General Practice performance may be misleading.

Earlier diagnosis and treatment of cancer is a crucial step towards improved cancer survival (Richards, 2009). Therefore, considerable resources and research effort are directed toward identifying and eliminating the causes of delay in a patient’s pathway to diagnosis. Several studies have attributed a significant proportion of the time from symptom onset to diagnosis and treatment to General Practice (Allgar and Neal, 2005; Lyratzopoulos ) and secondary care delays (Jensen ).

In the United Kingdom, an urgent General Practice referral system was established whereby a referral for suspected cancer triggers priority investigation with the aim that cases of suspected cancer should be investigated within 2 weeks of referral. In addition, consultants in secondary care can upgrade routine symptomatic referrals to 2WW (the 2-week-wait referral pathway) or downgrade 2WW to routine referral status. In Northern Ireland (NI), current targets require Health & Social Care Trusts to commence treatment of at least 95% of such referred patients within 62 days of referral (National Health Service, 2000). Targets are also in place for all urgent suspected breast cancer patients to have a first assessment by a breast cancer specialist within 14 days of referral. Trusts are also mandated to treat at least 98% of all cancer patients within 31 days of a decision to treat. However, there are no specific targets in place to expedite investigation and diagnosis of routinely referred cancer patients in secondary care (Department of Health, Social Services & Public Safety, 2015).

The choice of cancer referral route is an important predictor of delay in the United Kingdom (Larsen ; Jensen ). Concern exists that inappropriate non-urgent referral by General Practitioners (GPs) of patients subsequently proven to have cancer may contribute to diagnostic delays (Baughan ). Variation in the rate and accuracy of the 2WW system in General Practice exists. This has been taken to indicate inconsistency in referral guideline application by GPs, suggesting potential opportunities for improvement (McCoubrey ; Meechan ; Howell ). Understanding this variation has the potential to illuminate the quality of patient care.

In addition to Practice 2WW referrals, upgrades of routine Practice referrals by consultants in secondary care account for a significant proportion of the total volume of 2WW referrals (South Eastern Health & Social Care Trust, 2014) and the total volume of cancers detected via the 2WW (Bannon ; Cairnduff ). Because these consultant upgrades (CUs) aim to provide a failsafe mechanism to ensure appropriate fast tracking of suspected cancers not identified by the GPs, they may provide insights into referral quality in General Practice as upgrades reflect a variation between Practice and consultant appraisal. It may be hypothesised that high rates of Consultant Upgrade 2WW following Practice referral may be an indicator of poor General Practice referral decisions and the detection of more cancers may highlight poor categorisation of suspected cancer cases in General Practice.

A 2012 cross-sectional study of the use of 2WW referrals for all cancer diagnoses in England in 2009 (Meechin ) reported three main measures of 2WW referral activity.

The standardised referral rate ratio (SRR), defined as the ratio of observed to expected referrals for a practice of a defined size and age–sex population structure;

The conversion rate (CR), defined as the proportion of 2WW referrals that result in a cancer diagnosis and is the equivalent to the positive predictive value and;

The detection rate (DR), this is the proportion of cancers diagnosed that were 2WW referrals and is equivalent to sensitivity.

The authors observed significant correlations between the SRR and the CR (negatively correlated) and the SRR and the DR (positively correlated) as well as a significant positive correlation between the DR and the CR. The positive correlation between DR and CR was unexpected owing to their contrasting relationship with the SRR. Consequently, this relationship was interpreted as a potential indicator of the quality of clinical practice, with Practices recording higher CR and DR assumed to be making better referral decisions than those with low CR and DR. The results were swiftly included in cancer services policy, and Practices in England and Wales were ranked on the basis of their annual DR and CR for their use of 2WW referrals. Practice profiles were subsequently published as a measure of General Practice performance (Public Health England, 2016).

More recently, a study by Murchie suggested that the use of DR, CR and 2WW referral rates as measures of Practice performance fails to account for important differences between cancers and for the effect of ‘random case-mix’ – the fact that some cancers are easier to diagnose than others. Their analysis of routine data on ‘urgent-suspected cancer’ referrals between 2006 and 2013 from the NHS Grampian Cancer Care Pathway database suggested random case-mix was apparent in single-year observations due to small samples within General Practice, but this was addressed by combining years to increase sample size and reduce case-mix variability. In these larger samples of cancer patients in each Practice, no association between DR and CR was observed. Despite this, annual Practice Profiles continue to be published and publicised in England, and their introduction has been proposed elsewhere in the United Kingdom. It is therefore important to investigate elsewhere within the United Kingdom to determine whether the findings reported by Murchie can be replicated.

This study investigated variation of Practice and Consultant Upgrade 2WW referral profiles in a UK region with a view to understanding the role they can play as indicators of care quality for cancer diagnosis in General Practice. The results could have implications for cancer services performance measurement in General Practice in the United Kingdom and other health-care systems that use similar policies.

Materials and methods

Suspected cancer referrals by age, sex and Practice were generated from the region-wide Cancer Patient Pathway System (CaPPS) database. The database is an administrative system used by multi-disciplinary teams in secondary care to manage suspected cancer patients and is also used to monitor compliance with cancer waiting time targets in NI (Northern Ireland Assembly, 2011). The Office for Research Ethics Committee Northern Ireland (ORECNI) granted ethical approval (Reference: 12/NI/0034) in March 2012.

Inclusion criteria

The database for the study included all referrals for suspected cancer, at any site, recorded on CaPPS between 1 January 2011 and 31 December 2013 for all General Practices in NI.

Exclusion criteria

Duplicate records were omitted from the study. These included records with the same Health and Care Number (a unique identifier), date of referral and referral priority. Records with unknown referral priority or unknown Practice were also excluded.

Statistical analysis

An anonymised extract of the CaPPS database was loaded into Stata 11.2 (StataCorp, 2011) to generate the study database. Individual GP codes were identified for each patient using Practice Cipher numbers, and where these were unrecorded, they were manually identified using the HSC Webview system in the secure environment of the NICR.

The primary unit of analysis in the study was General Practice. The key measures in the study were the SRR, DR and CR. These measures are outlined below. These were calculated for both Practice 2WW referrals and for Consultant Upgrade 2WW referrals.

SRR was calculated using indirect age–sex standardisation. Demographic profiles (recorded in July of each year) by 5-year age–sex bands for each Practice in NI from HSC Business Services Organisation were used to calculate expected referrals in each Practice. The SRR was calculated as a comparison of the expected number of referrals against the observed number of referrals for each Practice.

Conversion rate

CR was defined as the proportion of 2WW referrals that had a subsequent cancer diagnosis recorded in CaPPS on either the 31- or 62-day patient pathway. These were calculated for each year and for each General Practice for the four most common suspected non-skin cancer sites (urology (C60–C68), lower GI (C18–C21), lung cancer (C33 and C34) and female breast cancer (C50)) as well as for all the suspected cancers combined.

Detection rate

DR was defined as the proportion of cancers that were detected in the 2WW pathway out of all the cancers diagnosed in that Practice population. These were also produced for the four most common suspected non-skin cancer sites as well as for all suspected cancers combined. For Consultant Upgrade 2WW referrals, sensitivity analysis was undertaken calculating DR excluding cancers from the denominator already referred via the 2WW.

Funnel plots (Spiegelhalter, 2002) were used to present Practice variation in the SRR for each year and all years combined with control limits set at three s.d. To investigate the relationship between 2WW SRR, CR and DR, Spearman’s rank correlation coefficients were produced for each year and all years combined for all cancers for both Consultant Upgrade and Practice 2WW referrals.

Sensitivity analysis

Sensitivity analysis was undertaken to investigate whether any associations between CR and DR were due to the fact that both metrics were derived from the same small number of events in each Practice rather than differences in referral behaviour between practices. This was undertaken by comparing CR and DR in different years to ensure that the same cancer cases are not included in the two metrics.

Results

There were 353 Practices in NI during the period 2011–2013. The median patient list size of these Practices was 4814 in 2011 with an interquartile range of 1700, the smallest Practice had <1300 patients and the largest had almost 15 000.

Following exclusion of duplicate and error records (n=62 472), there were 188 337 records of referral on the CaPPS database from 2011 to 2013 with 28 725 confirmed cancers recorded. There were 85 721 registered as 2WW referrals from a Practice and these were associated with 10 846 confirmed cancers with a median of 195 referrals and 23 cancers diagnosed in each practice annually. After exclusion of 1037 2WW referrals that were not attributed to any Practice, there were 84 684 2WW referrals and 28 443 cancers recorded across the 353 Practices in NI (see Figure 1).

Figure 1

Development of the Practice referral metrics database (inclusions and exclusions).

Patterns in referral, DR and CR varied significantly by Practice. Funnel plots for Practice and Consultant Upgrade 2WW referral rates are presented in Supplementary Figure. Practice and Consultant Upgrade variation differed by suspected tumour type and was lower for suspected lung cancer while being higher for suspected breast, lower GI and urological cancers (see Table 1). Median DR and CR by practice were higher for Practice 2WW referrals than for Consultant Upgrade 2WW referrals, with marked variation for both Consultant Upgrade and Practice 2WW referrals and variation by disease location. Specifically, Practice CRs were the highest for lung cancer, DRs were the highest for breast while lower GI was the lowest for both. Only among urological referrals did median CRs from CU exceed 10% (see Table 1).

Table 1

General Practice variation in SRR, CR and DR for the selected cancers (2011–2013)

	All cancers	Lung	Breast (female)	Lower GI	Urology
Practice 2WW
Referral rate (per 100 000)	896	34	219	170	144
SRR % in the expected range	26%	82%	50%	54%	63%
CR centile
25th	8%	14%	7%	2%	14%
50th	10%	25%	13%	5%	22%
75th	12%	36%	20%	8%	33%
DR rate centile
25th	22%	13%	17%	13%	17%
50th	28%	25%	31%	22%	25%
75th	33%	34%	42%	32%	35%
Consultant Upgrade 2WW
Referral rate (per 100 000)	493	6	342	37	106
SRR % in expected range	40%	95%	52%	52%	61%
CR centile
25th	4%	0%	0%	0%	0%
50th	5%	0%	4%	0%	13%
75th	7%	33%	7%	0%	23%
DR rate centile
25th	6%	0%	7%	0%	5%
50th	8%	0%	17%	0%	13%
75th	12%	55%	29%	6%	22%

Abbreviations: CR=conversion rate; DR=detection rate; GI=gastrointestinal; SRR=standardised referral rate ratio; 2WW=2-week wait.

To determine whether referral practices among General Practices were consistent over time, annual Practice ranking of their SRR were compared using Spearman’s rank correlation coefficient. There was a strongly positive correlation in the SRR for both Consultant Upgrade and Practice 2WW referrals. The same analysis was applied to annual DR and CR. Unlike referral rates, the correlation between annual Practice CR and DR were weak for both Consultant Upgrade and Practice 2WW referrals (see Table 2).

Table 2

Spearman’s rank correlation coefficient matrices for annual SRRs for all the cancers combined (2011–2013)

	Standardised referral rate ratio			DR			CR
Year	2011	2012	2013	2011	2012	2013	2011	2012	2013
Practice 2WW
2011	1.00			1.00			1.00
2012	0.81a	1.00		0.31a	1.00		0.30a	1.00
2013	0.65a	0.77a	1.00	0.22a	0.28a	1.00	0.18a	0.24a	1.00
Consultant Upgrade 2WW
2011	1.00			1.00			1.00
2012	0.61a	1.00		0.20a	1.00		0.09a	1.00
2013	0.56a	0.72a	1.00	0.13a	0.29a	1.00	0.03a	0.21a	1.00

Abbreviations: CR=conversion rate; DR=detection rate; SRR=standardised referral rate ratio; 2WW=2-week wait.

Significant at the 0.05 level.

The relationship between SRR, CR and DR

All cancers

Scatter plots and Spearman’s rank correlation coefficients were used to compare the three metrics. Practice SRR ratios were positively correlated with DR and negatively with CR. This relationship was consistently observed each year and for all years combined (see Table 3). Similar to Practice 2WW referrals, Consultant Upgrade 2WW rates were positively correlated with DR, though the correlation between SRR and CR observed for Practice referrals was not observed for Consultant Upgrade referrals.

Table 3

Spearman’s ρ for referral metrics by year of suspected cancer referral and for all years combined

	2011	2012	2013	Total
Practice 2WW
SRR vs DR	0.51a	0.50a	0.46a	0.58a
SRR vs CR	−0.35a	−0.30a	−0.44a	−0.53a
CR vs DR	0.43a	0.44a	0.35a	0.16ns
Consultant Upgrade 2WW
SRR vs DR	0.35a	0.38a	0.46a	0.49a
SRR vs CR	0.01ns	0.05ns	0.11a	−0.01ns
CR vs DR	0.87a	0.81a	0.86a	0.78a

Abbreviations: CR=conversion rate; DR=detection rate; SRR=standardised referral rate ratio; 2WW=2-week wait.

Statistically significant at the 95% level/NS (not statistically significant).

Regarding the relationship between CR and DR, Figure 2 illustrates how the relationship between Practice CR and DR, apparent for a single-year comparison, diminish when all years are combined. In contrast, for Consultant Upgrade referrals, a strong association between CR and DR was observed in the full data set as well as in the single-year comparisons.

Figure 2

Scatter plots depicting the relationships between SRR and CR, SRR and DR and CR and DR in NI General Practices for a single year (2012) and for the period 2011–2013.

By cancer site

Similar correlations were observed by cancer site as for all the cancers combined, though there was no negative association between SRR and CR for lung cancer. The same patterns were observed for Consultant Upgrade referrals though a weak positive correlation between SRR and CR was observed for suspected female breast and lower GI cancers (see Table 4).

Table 4

Spearman’s for referral metrics for the selected cancers (2011–13)

	Lung	Female breast	Lower GI	Urology
Practice 2WW
SRR vs DR	0.35a	0.58a	0.20a	0.36
SRR vs CR	0.05	−0.21a	−0.21a	−0.23
CR vs DR	0.54a	0.33a	0.47a	0.33
Consultant Upgrade 2WW
SRR vs DR	0.26a	0.48a	0.41a	0.56
SRR vs CR	0.07a	0.17a	0.35a	0.16
CR vs DR	0.74a	0.66a	0.60a	0.66

Abbreviations: CR=conversion rate; DR=detection rate; GI=gastrointestinal; SRR=standardised referral rate ratio; 2WW=2-week wait.

Statistically significant at the 95% level.

Sensitivity analysis

To assess the role of case-mix confounding in explaining the relationship between CR and DR, they were compared for different years, thereby separating the cancer cases included in the two metrics. Any correlations observed would therefore be more likely due to the referral practices rather than the random case-mix of disease characteristics. The results of these comparisons are presented in Supplementary Table. There was no evidence of significant correlation between CR and DR except for those calculated for the same year, which include the same cancer case-mix in both metrics.

Discussion

The 2WW referral system has been in place in the United Kingdom since 2000 (National Health Service, 2000). Although there has been variation in Practice referral patterns reported since the establishment of the system, few studies have investigated robustly this variation including potentially attributable quality indicators. This study improves our understanding of variation in use of the 2WW referral system in Practice and, in the context of previous studies, indicates the extent to which measures of General Practice and Consultant Upgrade 2WW referral, CR and DR may be used as a measure of practitioner performance with a view to identifying and spreading best practice.

Key findings

Variation in Practice and Consultant Upgrade SRRs was highly dispersed and variation was unexplained by age and sex distribution of patient populations. Similar to previous studies Meechin ; Murchie ), Practice referral rates correlated positively with DR and negatively with CR. Lower referral rates suggested the application of a higher threshold for suspicion before referral and hence higher predictive power (CR) and lower sensitivity (DR), while a higher referral rate was suggestive of a lower threshold of suspicion before referral and consequently lower predictive power and higher sensitivity. There was also significant variation by disease site with particularly low CR and DR for lower GI cancers reflecting the low predictive power of its symptoms.

DR and CR as performance metrics

Consistent with Murchie , the positive relationship between CR and DR, previously suggested by Meechin as a quality indicator for referral, was observed in the analysis of single-year data but was not apparent when comparing several years combined. This pattern may be explained by a combination of small numbers of cancer cases in each Practice and case-mix and, specifically, the ease with which a cancer is diagnosed. A Practice with a group of more easily diagnosed patients one year will have both higher CR and higher DR; similarly, the same Practice in another year may have a larger number of more difficult-to-diagnose cancers and consequently have both low DR and CR. As these rates are derived from the same events, they correlate, particularly at a Practice level where there are few cases of cancer diagnosed annually. So while studies using very large administrative data sets of cancer referrals may appear statistically robust, the units of comparison are dependent upon a small number of cancers in each Practice where comparisons are sensitive to case-mix confounding. We therefore require a number of years to accumulate a sufficient number of cancer records to address this case-mix confounding. Murchie used simulation modelling to estimate that approximately 25 cancers must be observed in a Practice to address this case-mix. In this study, the average annual number of cancers in each General Practice was 23, with an average of 80 over the 3-year period. The absence of correlation in the larger 3-year data set suggests that these relationships may be explained by case-mix rather than by the quality of referral activity. Although correlation between CR and DR was observed over the entire study period for individual cancer sites, this pattern was also likely to be related to a small number of cancers in General Practices over the study period. General Practice comparisons of referral rates for even the most common cancers may require up to a 10-year period to accrue a sufficient number of cancers in each practice and overcome case-mix influences.

CU activity as indicators of performance

Consultant Upgrade 2WW referrals showed a strong correlation between CR and DR including over a 3-year period, though it was only apparent when comparing CR and DR from the same years and was not observed when CR for one year was compared with the DR of another year. This finding may suggest that the observed correlations occurred because the variables are bonded by the same handful of cancer patients in each Practice and the specific characteristics of these cancers rather than because of the quality of the referral behaviour in General Practices. The suggestion that measures of referral rate activity may be used as quality indicators assumes that there is temporal consistency in the characteristics of General Practice referral activity over time. Although there appeared to be a strong relationship in the referral rates over time, there was no evidence to suggest any temporal consistency in CR or DR. Again, this analysis suggests that case-mix variation has a significant bearing on these measures, with no evidence to suggest that the CR or DR in a Practice in any one year could be predictive of those in successive years. The evidence here strongly suggests that use of CR and DR as quality indicators for Practice referral is methodologically weak and that several years of referrals are required to gather robust evidence of General Practice referral activity and to determine whether there is any temporal consistency when case-mix confounding is addressed.

Overall CR and DR were lower for Consultant Upgrade referrals compared with Practice referrals; the lower CR and DR among Consultant Upgrade referrals may reflect the fact that consultants were selecting from patients previously screened in General Practice who had already identified the more easily diagnosed cancers leaving only the ‘margin calls’ with a low index of suspicion to be identified by the Consultants. This clearly highlights that even with expert opinion the clinical diagnosis of cancer will involve the investigation of large patient volumes for every patient diagnosed owing to the low predictive power of symptoms rather than GP decision-making.

Strengths and weaknesses

This study enhances our understanding about variation in referral metrics in General Practice in a UK region using a regional database that includes referrals from all General Practices. The inclusion of 3 years of referral data enabled investigation of the risk of misinterpreting results due to case-mix confounding and analysis of temporal trends in referral metrics. Moreover, temporal separation of CR and DR help us understand the limited role that Practice CR and DR can have as measures of quality of cancer care. It is noteworthy, however, that the main findings here were observed previously in Scotland (Murchie ) and NI where average Practice list size almost 20% smaller than compared with England and Wales (Health & Social Care Information Centre, 2015). Consequently, the average number of cancers diagnosed in General Practice is also lower (Public Health England, 2016). Although the longer time period for accrual of cancer patients ensured sufficiently large samples, it would still be useful to confirm these findings elsewhere.

Implications

This study investigated the validity of a measure of referral quality that has been widely reported in England (Public Health England, 2016). This study, with large General Practice samples, provided no evidence to support the application of such quality metrics and suggests that these performance measures misinform service users and stakeholders. We require a much better understanding of the 2WW referral system and its implication for practitioner behaviour and for patient outcomes on a population level before developing measures of General Practice performance for cancer referral. Given the relationship with DR, SRRs should not be ignored as a potentially useful indicator of quality, although their relationship with patient outcomes requires further investigation. Although this study dispelled the suggestion of a consistent pattern in CR and DR, it also identified groups of General Practices in the 3-year cohort that had both high DR and CR and other groups with both low CR and DR. There is a need to understand whether these groups can identify General Practices with better and worse referral practices and if these have a relationship with patient outcomes. In particular, further research is necessary to determine the extent to which this variation may be explained by access to and use of diagnostic tools such as chest X-ray to diagnose lung cancer and PSA testing in the profiling of patients with symptoms of prostate cancer. Previous studies have examined the extent to which variation in chest X-ray rates among General Practices were associated with stage and mortality (O’Dowd ); similarly, studies of General Practice variation of PSA testing have examined their role in cancer mortality and incidence (Hjertholm ). However, it may be hypothesised that Practices with higher usage of these diagnostic tools may be more accurate in their referrals, hence improving risk profiling of patients, expediting diagnosis and reducing burden on secondary care. Further work ought to be undertaken to investigate whether these Consultant Upgrade and Practice referral metrics are associated with cancer mortality and survival.

The significant variation in 2WW SRR, CR and DR by disease site also reinforces the view that a one size fits all system for referral of suspected cancer in General Practice is inappropriate. The DRs of <25% and CRs of <5% ensure that the 95% of patients referred on the 2WW system who are not subsequently diagnosed with cancer are prioritised ahead of >75% of the cancer patients who were not referred via this route. This has the potential to lead to the unintended consequence of delaying diagnosis of cancer patients. On balance, lower GI cancer patients may be better served by a higher diagnostic threshold for 2WW referral and associated waiting time targets for 2WW routine referral. Further research examining waiting times for both routine and 2WW referrals is required to inform policies and targets for these harder-to-diagnose cancer types.

Conclusion

There is a current desire to produce measures of General Practice activity that can be used to influence practitioner behaviour with a view to improving cancer outcomes through earlier diagnosis. Given that most cancers are diagnosed clinically in primary care (Elliss-Brookes ), this should be a policy area with potential to improve outcomes. However, the relatively few cancers diagnosed by any individual Practitioner each year not only presents challenges to practitioners in identifying the cancer patient who is a ‘needle in the haystack’ but also to statisticians and researchers who must attempt to find meaning in data that deals with what are relatively rare events even for the most common cancers. Current use of Practice and Consultant Upgrade referral rate metrics as measures of General Practice performance is inappropriate and ought not to be considered owing to case-mix confounding.