Identifying patients with undetected colorectal cancer: an independent validation of QCancer (Colorectal).

BACKGROUND: Early identification of colorectal cancer is an unresolved challenge and the predictive value of single symptoms is limited. We evaluated the performance of QCancer (Colorectal) prediction model for predicting the absolute risk of colorectal cancer in an independent UK cohort of patients from general practice records.
METHODS: A total of 2.1 million patients registered with a general practice surgery between 01 January 2000 and 30 June 2008, aged 30-84 years (3.7 million person-years) with 3712 colorectal cancer cases were included in the analysis. Colorectal cancer was defined as incident diagnosis of colorectal cancer during the 2 years after study entry.
RESULTS: The results from this independent and external validation of QCancer (Colorectal) prediction model demonstrated good performance data on a large cohort of general practice patients. QCancer (Colorectal) had very good discrimination with an area under the ROC curve of 0.92 (women) and 0.91 (men), and explained 68% (women) and 66% (men) of the variation. QCancer (Colorectal) was well calibrated across all tenths of risk and over all age ranges with predicted risks closely matching observed risks.
CONCLUSION: QCancer (Colorectal) appears to be a useful tool for identifying undetected cases of undiagnosed colorectal cancer in primary care in the United Kingdom.

Colorectal cancer is a major health burden with 1.24 million new cases diagnosed worldwide in 2008 (Ferlay ; Jemal ). It is a major cause of cancer deaths with an estimated 608 000 deaths attributed to colorectal cancer in 2008, accounting for 8% of all cancer deaths (Ferlay ). In the United Kingdom, nearly 40 000 new cases were identified in 2008 (info.cancerresearchuk.org). In the United Kingdom, the National Awareness and Early Diagnosis Initiative (NAEDI) has been established to improve the public’s awareness of signs and symptoms of cancer, and to seek advice earlier that will ultimately prevent avoidable deaths (Richards, 2009). It is estimated that if survival rates in England matched best rates found in Europe, then about 1700 deaths could be avoided per year (Abdel-Rahman ).

The diagnostic utility of individual so-called ‘red-flag’ alarm symptoms (i.e., rectal bleeding) are limited and not necessarily specific to colorectal cancer (Jones ). Current guidance in the United Kingdom from the National Institute for Health and Clinical Excellence (NICE) recommends the decision for urgent referral to a specialist based on a series factors such as age, menopausal status, change of bowel habit, stool frequency, rectal bleeding, abdominal mass and anaemia (National Institute for Health and Clinical Excellence, 2005). However, there are suggestions that current guidance from NICE can only identify (the minority) high-risk symptomatic patients, and that the small improvement in survival in the United Kingdom (which falls behind much of Europe (Berrino )) implies the current approach is not working (Hamilton, 2009a, 2009b). While we do not necessarily share these views in the absence of evidence, there are clearly opportunities and a need to provide a more consistent and objective approach to assist primary care physicians in identifying patients with undiagnosed colorectal cancer.

While the usefulness of individual symptoms is limited (Ford ; Hamilton ; Hamilton ; Olde Bekkink ; Adelstein ; Astin ), a more appropriate approach would be to base the decision for referral using a combination of multiple risk factors, by means of a multivariable prediction model. Existing approaches based on multivariable prediction models have recently been derived that indicate improved performance over single symptoms can be achieved (Hamilton, 2009a; Marshall ). QCancer (Colorectal) are a pair of multivariable prediction models (one for men and one for women) that have recently been developed to predict the risk of having undiagnosed colorectal cancer (Hippisley-Cox and Coupland, 2012).

QCancer (Colorectal) was developed and internally validated on a large cohort of 3.6 million patients from the QRESEARCH (www.qresearch.org) database (Hippisley-Cox and Coupland, 2012). The QRESEARCH database is a large database comprising over 12 million anonymised health records from 602 general practices throughout the United Kingdom using the EMIS computer system. QCancer (Colorectal) was developed on 2.35 million patients aged between 30 and 84 years, contributing 4798 incident cases of colorectal cancer from 4 110 382 person-years of observation between 01 January 2000 and 30 September 2010. The final prediction models included seven risk factors for women and nine risk factors for men (Table 1). Open source code to calculate the QCancer (Colorectal) scores are available from www.qcancer.org/colorectal/ released under the GNU Lesser General Public Licence, version 3. The performance of the QCancer (Colorectal) was assessed on a separate sample of 1.2 million patients from the same QRESEARCH database with good discriminative ability and calibration (Hippisley-Cox and Coupland, 2012).

Table 1

Risk factors in QCancer (Colorectal)

Women	Men
Age (years)	Age (years)
Family history of gastrointestinal cancer (yes/no)	Family history of gastrointestinal cancer (yes/no)
Currently consulting a GP with first onset of abdominal pain (yes/no)	Currently consulting a GP with first onset of abdominal pain (yes/no)
Currently consulting a GP with first onset of current appetite loss (yes/no)	Currently consulting a GP with first onset of current appetite loss (yes/no)
Currently consulting a GP with first onset of current rectal bleeding (yes/no)	Currently consulting a GP with first onset of current rectal bleeding (yes/no)
Currently consulting a GP with first onset of current weight loss (yes/no)	Currently consulting a GP with first onset of current weight loss (yes/no)
Anaemia, defined as recorded haemoglobin <11 g dl−1 in past 12 months (yes/no)	Anaemia, defined as recorded haemoglobin <11 g dl−1 in past 12 months (yes/no)
	Change in bowel habit in previous 12 months (yes/no)
	Alcohol consumption (non-drinker, trivial drinker, light drinkerand moderate/heavy drinker)

Abbreviation: GP=general practitioner.

The aim of this article is to describe the results from an independent evaluation of QCancer (Colorectal) on a large dataset of general practice records in the United Kingdom not used to derive the prediction model.

Materials and methods

Cohort selection

Study participants were patients registered between 01 January 2000 and 30 June 2008, and recorded on the THIN database (www.thin-uk.com). The same exclusion criteria as the original development paper were adopted (Hippisley-Cox and Coupland, 2012). Patients were excluded if they had a prior diagnosis of colorectal cancer, were registered <12 months with the general practice, had invalid dates, were under the age of 30 years or were aged 85 years or over. Entry to the cohort was defined, as for the original development study (Hippisley-Cox and Coupland, 2012), defined as the latest of (1) the study start date, (2) date the patient registered with the practice and (3) for those patients with symptoms (abdominal pain, appetite loss, rectal bleeding, weight loss, anaemia or change in bowel habit), the date of the first recorded onset of any symptom within the study period.

Outcome measure

The primary outcome was diagnosis of colorectal cancer, which was defined as incident diagnosis of colorectal cancer during the 2 years after study (using relevant UK diagnostic Read codes). Patients without the study outcome were censored at the earliest of the date of death, date of leaving the practice study or 2 years of follow-up.

Statistical analysis

The 2-year predicted risk of colorectal cancer for every patient in the THIN cohort was calculated using QCancer (Colorectal) risk score (www.qcancer.org/colorectal). Multiple imputation using all predictors plus the outcome variable was used to replace missing values for alcohol consumption for men. This involves creating multiple copies of the data and imputing the missing values with sensible values randomly selected from their predicted distribution. Ten imputed data sets were generated and results from analyses on each of the imputed data sets were combined using Rubin’s rules to produce estimates and confidence intervals that incorporate the uncertainty of imputed values.

Predictive performance of the QCancer (Colorectal) risk score on the THIN cohort was assessed by examining measures of calibration and discrimination. Calibration refers to how closely the predicted 2-year colorectal cancer risk agrees with the observed proportions with a diagnosis of colorectal cancer within 2 years. This was assessed for each tenth of predicted risk, ensuring 10 equally sized groups, and each 5-year age band by plotting observed proportions vs predicted risk.

Discrimination is the ability of the risk score to differentiate between patients who experience an event during the study period and those who do not. This measure is quantified by calculating the area under the receiver operating characteristic curve c-statistic; a value of 0.5 represents chance and 1 represents perfect discrimination. We also calculated the D-statistic (Royston and Sauerbrei, 2004) and R2statistic (Royston, 2006) that are measures of discrimination and explained variation, respectively, and are tailored for censored survival data.

We used decision curve analysis (accounting for censored observations) to describe and compare the clinical effects of QCancer (Colorectal) (Vickers and Elkin, 2006). Briefly, the net benefit of a model is the difference between the number of true positives and the number of false positives weighted by the odds of the selected threshold for high-risk designation. It is interpreted as the additional number of colorectal cancer cases identified without increasing the number treated unnecessarily across a range of clinically relevant treatment/referral thresholds. QCancer (Colorectal) is assessed by comparing the gain of the model over a strategy of ‘investigate all’ (i.e., assume all high risk) and ‘investigate none’ (i.e., all low risk). On the resulting curve, the ‘investigate all’ strategy is depicted as the line that crosses the x axis at the value that corresponds to the incidence rate of the outcome, while the ‘investigate none’ is depicted by a horizontal line at 0.

All statistical analyses were carried out in R (version 2.14.1) (R Development Core Team, 2011) and the ICE (multiple imputation) procedure in Stata (version 11.2) (StataCorp, 2009).

Results

Between 01 January 2000 and 30 June 2008, 2 135 540 eligible patients from 364 general practices in the United Kingdom were registered in the THIN database. The 2 135 540 eligible patients contributed 3 701 761 person-years of observation (median follow-up was 2 years), among whom there were 3712 cases of colorectal cancer (1676 women and 2036 men). Table 2 details the characteristics of eligible patients.

Table 2

Characteristics of participants aged 30 to 84 years in the QRESEARCH development and THIN validation cohorts

	QRESEARCH		THIN (external validation)
Risk predictor	Development (n=2 351 052)	Internal validation (n=1 236 601)	Women (n=1 075 775)	Men (n=1 059 765)
Median age (IQR)	50.1 (SD=15.0)	50.1 (SD=14.9)	49 (38, 63)	47 (38, 60)
Alcohol status, n (%)
None	510 179 (21.7)	275 152 (22.3)	—	127 360 (12.0)
Trivial (<1 unit per day)	656 450 (27.9)	355 654 (28.8)	—	161 033 (15.2)
Light (1–2 units per day)	492 318 (20.9)	257 381 (20.8)	—	70 187 (6.6)
Moderate or heavy (⩾3 units per day)	175 953 (7.5)	93 075 (7.5)	—	58 980 (5.6)
Alcohol status not recorded	516 152 (22.0)	255 339 (20.6)	—	642 205 (60.6)
Medical history n (%)
Family history of gastrointestinal cancer	29 483 (1.3)	17 672 (1.4)	1380 (0.13)	999 (0.1)
Current symptoms and symptoms invthe preceding year, n(%)
Current abdominal pain	224 880 (9.6)	125 816 (10.2)	143 797 (13.4)	102 192 (9.6)
Current appetite loss	9959 (0.4)	5358 (0.4)	3295 (0.3)	2481 (0.2)
Current weight loss	25 113 (1.1)	14 065 (1.1)	15 398 (1.4)	12 891 (1.2)
Current rectal bleeding	52 453 (2.2)	29 118 (2.4)	27 811 (2.6)	28 423 (2.7)
Change in bowel habit	3153 (0.1)	1821 (0.1)	—	1670 (0.2)
Haemoglobin <11 g dl−1 recorded in the last year	31 330 (1.3)	16 985 (1.4)	13 659 (1.3)	4466 (0.4)

Abbreviation: IQR=interquartile range.

Complete data on all risk predictors were available for all women, while complete data on alcohol consumption were available for 39.4% of men (n=417 560). Compared with the original development cohort, the THIN cohort had more patients reporting abdominal pain, fewer patients reporting a family history of gastrointestinal cancer and more patients with missing data on alcohol consumption (22% in the development cohort compared with 60% in the THIN cohort).

Table 3 reports the age–sex incidence rates of each symptom included in the QCancer (Colorectal) prediction models. All the symptoms apart from abdominal pain tended to become more common with age. During the follow-up, the crude rate of colorectal cancer was 100 per 100 000 person-years of observation with 90 per 100 000 person-years for women and 111 per 100 000 person-years for men. As per the original development cohort, incidence rates of colorectal cancer increased sharply with age.

Table 3

Incidence rates of appetite loss, weight loss, rectal bleeding and abdominal pain per 100 000 person-years by sex and age in the THIN cohort

	Women		Men
Symptom and age range (years)	No. with symptom	Incidence rate (95% confidence interval)	No. with symptom	Incidence rate (95% confidence interval)
Appetite loss (all ages)	3295	50 (49–52)	2481	39 (38–41)
<35	118	16 (13–19)	85	11 (9–13)
35–44	406	25 (22–27)	292	17 (15–19)
45–54	366	26 (23–28)	303	21 (18–23)
55–64	397	34 (30–37)	381	33 (30–36)
65–74	740	81 (77–87)	626	79 (73–85)
75–84	1268	197 (187–208)	794	192 (179–206)
Weight loss (all ages)	15 398	234 (231–238)	12 891	203 (199–207)
<35	655	87 (81–94)	439	55 (50–60)
35–44	2417	146 (141–152)	1819	106 (101–111)
45–54	2406	168 (161–175)	2029	138 (132–144)
55–64	2544	215 (206–223)	2618	225 (216–234)
65–74	3054	336 (324–348)	2921	369 (355–382)
75–84	4322	672 (653–693)	3065	740 (714–767)
Rectal bleeding (all ages)	27 811	423 (418–428)	28 423	447 (442–453)
<35	1651	220 (210–231)	1650	207 (197–217)
35–44	5716	346 (337–355)	6830	399 (390–409)
45–54	5743	401 (391–412)	6420	436 (426–447)
55–64	6134	517 (505–531)	5917	508 (495–521)
65–74	4667	513 (499–528)	4653	587 (570–604)
75–84	3900	607 (588–626)	2953	713 (688–739)
Abdominal Pain (all ages)	143 797	2189 (2177–2200)	102 192	1609 (1599–1619)
<35	14 268	1904 (1873–1935)	6976	874 (854–895)
35–44	39 869	2415 (2391–2439)	25 701	1502 (1483–1520)
45–54	29 526	2062 (2038–2085)	22 228	1510 (1491–1530)
55–64	26 356	2223 (2197–2250)	21 107	1813 (1788–1837)
65–74	19 825	2180 (2149–2210)	16 385	2067 (2036–2099)
75–84	13 953	2171 (2135–2207)	9795	2365 (2319–2413)

Performance data for QCancer (Colorectal) from the original development cohort (Hippisley-Cox and Coupland, 2012) and the THIN cohort (multiple imputation (for men) and complete-case) are presented in Table 4. The R-statistics (percentage of explained variation) and the D-statistic were marginally higher in the THIN cohort, 65.8% and 2.84 (women) and 68.3% and 3.00 (men), compared with those reported in the original development paper, 64.8% and 2.78 (women) and 66.7% and 2.90 (men) (Hippisley-Cox and Coupland, 2012). The high values of the D-statistic and also the c-statistic 0.91 (women) and 0.92 (men) indicate very good discrimination properties of QCancer (Colorectal). The usefulness of using individual symptoms to identify patients with colorectal cancer is compared with QCancer (Colorectal) and is reported in Table 5. Using a cutoff (0.4% women and 0.5% men) that identified the 10% of women and men at the highest risk, QCancer (Colorectal) clearly outperformed with 71 and 74% of all new colorectal cancers identified for women and men, respectively. Using the presence of rectal bleeding, only 34 and 40% of new colorectal cancers were identified in women and men, respectively. Similarly, using only abdominal pain, only 36% of colorectal cancer cases in women and 31% in men were identified.

Table 4

Performance data in the original development cohort and the external validation (THIN) cohort

	QRESEARCH (Hippisley-Cox and Coupland, 2012)		THIN (external validation)
	(internal validation)		Multiple imputation ( m =10)	Complete-case
	Women (n=616 361)	Men (n=620 240)	Men (n=1 059 765)	Women (n=1 075 775)	Men (n=417 560)
R2 (95% CI)	64.8 (63.2–66.3)	66.7 (65.3–68.0)	68.32 (67.32–69.32)	65.81 (64.62–67.01)	65.30 (63.71–66.89)
D-statistic (95% CI)	2.78 (2.68–2.87)	2.90 (2.81–2.98)	3.00 (2.93–3.07)	2.84 (2.76–2.92)	2.81 (2.71–2.91)
c-Statistic (95% CI)	0.89 (0.88–0.90)	0.91 (0.90–0.91)	0.918 (0.913–0.923)	0.909 (0.903–0.915)	0.901 (0.892–0.910)

Abbreviation: CI=confidence interval.

Table 5

Comparison of strategies to identify patients of having a diagnosis of colorectal cancer in the next 2 years

Criterion	Risk threshold (%)	True negative	False negative	False positive	True positive	Sensitivity (%)	Specificity (%)	Positive-predictive value (%)	Negative-predictive value (%)
Women
Rectal bleeding	NA	1 046 859	1105	27 240	571	34.1	97.5	2.1	99.9
Abdominal pain	NA	930 900	1078	143 199	598	35.7	86.7	0.4	99.9
Appetite loss	NA	1 070 824	1656	3275	20	1.2	99.7	0.6	99.8
Weight loss	NA	1 058 792	1585	15 307	91	5.4	98.6	0.6	99.8
Anaemia	NA	1 060 613	1503	13 486	173	10.3	98.7	1.3	99.9
Top 10% risk score	0.4	970 241	491	103 858	1185	70.7	90.3	1.1	99.9
Top 5% risk score	0.8	1 021 128	739	52 971	937	55.9	95.1	1.7	99.9
Men
Rectal bleeding	NA	1 030 097	1245	27 632	791	39.9	97.4	2.8	99.9
Abdominal pain	NA	956 159	1414	101 570	622	30.6	90.4	0.6	99.9
Appetite loss	NA	1 055 272	2012	2457	24	1.2	99.8	1.0	99.8
Weight loss	NA	1 044 962	1912	12 767	124	6.1	98.8	1.0	99.8
Change in bowel habit behaviour	NA	1 056 108	1987	1621	49	2.4	99.8	2.9	99.8
Anaemia	NA	1 053 398	1901	4331	135	6.6	99.6	3.0	99.8
Top 10% risk score	0.5	953 979	528	103 751	1509	74.1	90.2	1.4	99.9
Top 5% risk score	0.9	1 006 421	787	51 307	1249	61.3	95.1	2.4	99.9

Abbreviation: NA=not applicable.

Calibration plots of QCancer (Colorectal) for women and men by tenth of risk are presented in Figure 1. Model calibration is very good with close agreement between predicted and observed colorectal cancer risks across all tenths of risk. Similarly, Figure 2 displays the calibration plots of QCancer (Colorectal) for women and men by 5-year age bands. Again, the QCancer (Colorectal) shows very good agreement across all age groups, with very slight overprediction in the older age groups in both sexes, though the overprediction is small.

Figure 1

Observed vs predicted colorectal cancer risks (triangles=predicted risk and circles=observed risk).

Figure 2

Observed vs predicted colorectal cancer risks by sex and age (triangles=predicted risk and circles=observed risk).

Figure 3 displays the net benefit curves for QCancer (Colorectal), which clearly show that QCancer (Colorectal) has higher net benefit compared with the ‘investigate all’ (line crossing the x axis at 0.2–0.25) and ‘investigate none’ (horizontal line at 0) strategies.

Figure 3

Decision curve analysis (net benefit=TP−wFP)/N; where TP=number of true positives, FP=number of false positive and w=ratio of harm to benefit).

Discussion

QCancer (Colorectal) is a new risk score to identify individuals with undetected colorectal cancer in a primary care setting. The risk score was developed and internally validated on a large primary care electronic database (QRESEARCH) of 3.6 million patients contributing 7401 cases of colorectal cancer between 01 January 2000 and 30 September 2008 (Hippisley-Cox and Coupland, 2012).

QCancer (Colorectal) was designed to be a prediction model based on risk factors that are recorded in patients’ health records or which patients themselves are likely to know. Thus, QCancer (Colorectal) has potential to be a useful tool in the primary care setting to identify patients who are at an increased risk of having undiagnosed colorectal cancer who would strongly benefit from further investigation.

In the United Kingdom, the CAPER score is an alternative prediction model that uses multiple symptoms to identify patients with undiagnosed colorectal cancer that is currently being considered by the Department of Health NAEDI framework (Hamilton ; Hamilton, 2009a). However, there are several methodological concerns about how the score was derived. The handling of missing data is questionable in that patients with missing data who appear to have been omitted, thereby suggesting a complete-case analysis that has been shown to be a methodologically flawed and biased in the development of prediction models (Clark and Altman, 2003; Burton and Altman, 2004; Moons ). Using data from a single location in the United Kingdom (Exeter), the authors screened 121 risk factors using univariate associations with colorectal cancer to reduce the number of predictors, a procedure that is not recommended (Sun ). Furthermore, with only 349 cases of colorectal cancer the number of events per variable (<3) (Hamilton ; Hamilton, 2009a) well below the rule-of-thumb of 10 is suggested to minimise the chance of over fitting. Lack of detailed reporting of how the score was derived and how to implement the model are also concerns (Khan, 2009). More importantly owing to the design of the study (a case–control study matched on age and sex), age and sex are not included as predictors in the CAPER score despite age being a major risk factor for colorectal cancer and the incidence of symptoms varying by age. Thus, an individual with or without symptoms will have the same predicted risk using the CAPER score regardless of age or sex. There is also currently insufficient performance (validation) data conducted by independent researchers to demonstrate the predictive performance of the CAPER score.

Our independent evaluation of the performance of QCancer (Colorectal) was carried out on the large separate database (THIN) of general practice records recording clinical data using the INPS Vision system that is used in 20% of UK general practices. The database comprised 2.1 million patients between 01 January 2000 and 30 June 2008 contributing 3.7 million years of observation and 3712 cases of colorectal cancer. The performance data presented in this article on the THIN cohort provide strong evidence to support the external validity of QCancer (Colorectal) in identifying patients with suspected colorectal cancer with marginal better performance as that seen in the internal validation data. QCancer (Colorectal) also clearly outperformed the use of using individual symptoms for identifying new cases of colorectal cancer.

A limitation of our study is that the recording of symptoms might be less complete, as individuals with mild symptoms may not visit their general practitioner or even report such mild symptoms if they do consult their general practitioner.

To date, the development, internal validation and our external validation of QCancer (Colorectal) has used 5.7 million patients contributing 10 million person-years of observation and 11 113 cases of colorectal cancer during the observation periods to develop and evaluate QCancer (Colorectal) to predict the risk of colorectal cancer in adults aged 30 to 84 years. In this study, we have provided an independent and external evaluation of the QCancer (Colorectal) risk score on a large cohort of patients in the United Kingdom. We have assessed the performance of QCancer (Colorectal) against performance metrics presented in the internal validation of QCancer (Colorectal), and have provided evidence to support the use of QCancer (Colorectal) in routine clinical practice.

Implementation of QCancer (Colorectal) or indeed any other risk prediction models used in the clinical decision making and patient managed should be evaluated by independent investigators using an appropriately large data set. Ideally, any prediction model should also be evaluated in an impact study to assess whether the prediction model changes clinician behaviour and ultimately improves patients outcomes (Moons ; Wallace ; Hamilton, 2012).

Conclusions

In this study, we have provided an independent and external validation of the QCancer (Colorectal) risk score on a large cohort of patients in the United Kingdom. The performance of QCancer (Colorectal) was similar to that in the internal validation with comparable results to indicating good predictive ability for identifying patients with undiagnosed colorectal cancer.

Ethical approval

Ethical approval was provided by Trent multicentre research ethics committee.