Validation of algorithms to detect distant metastases in men with prostate cancer using routine registry data in Denmark.

OBJECTIVE: Among patients with prostate cancer, diagnostic codes for bone metastases in the Danish National Registry of Patients have a sensitivity of 44%. In an attempt to improve the sensitivity of registry-based identification of metastases from prostate cancer, we tested a series of algorithms, combining elevated prostate-specific antigen (PSA) levels, use of antiresorptive therapy, and performed bone scintigraphy. PATIENTS AND METHODS: We randomly selected 212 men diagnosed with prostate cancer in 2005-2010 in the Central Denmark Region with prespecified PSA values, antiresorptive therapy, and bone scintigraphy who did not have a registry-based diagnostic code indicating presence of distant metastases. We defined three candidate algorithms for bone metastases: 1) PSA >50 μg/L and bone scintigraphy, 2) PSA >50 μg/L and antiresorptive therapy, and 3) PSA ≤50 μg/L with antiresorptive therapy or bone scintigraphy. An algorithm for distant metastasis site other than bone was defined as PSA >50 μg/L alone. Medical chart review was used as the reference standard to establish the presence or absence of metastases. Validity was expressed as a positive predictive value (PPV) or a negative predictive value, based on whether the algorithms correctly classified metastases compared with the reference standard.
RESULTS: We identified 113 men with evidence of metastases according to the candidate algorithms, and 99 men without evidence of metastases according to the candidate algorithm. The PPVs of PSA >50 μg/L were 0.10 (95% confidence interval [CI] 0.04-0.19) for bone metastases and 0.14 (95% CI 0.07-0.24) for nonbone metastases, regardless of receipt of antiresorptive therapy or presence of bone scintigraphy. The PPVs for any metastases were 0.16 (95% CI 0.06-0.32) for PSA >50 μg/L and 0.28 (95% CI 0.14-0.47) for PSA >50 μg/L with bone scintigraphy. Adding antiresorptive treatment to the algorithm did not improve PPV. All negative predictive values approached 1.00.
CONCLUSION: Algorithms based on elevated PSA, antiresorptive therapy, or bone scintigraphy are not suitable for supplementing diagnostic codes to identify additional cases of distant metastases among men with prostate cancer. However, it is possible that in this setting, medical chart review is not a gold standard to identify metastases.

Introduction

Almost 70% of patients with advanced prostate cancer have bone metastases.1 Utilization and safety of novel therapies for metastatic cancer can be monitored using registry-based data. The Danish National Registry of Patients (DNRP) is an administrative register of admissions to somatic hospitals in Denmark, including discharge diagnoses, selected therapies, and procedures.2 The specificity of the International Classification of Diseases (ICD)-10 code for bone metastases in the DNRP is 100% among patients with prostate cancer.3 In epidemiologic studies with bone metastases as an outcome of interest, relative measures of association should be unaffected by such misclassification, provided it is independent of the study exposure.4 The sensitivity of the DNRP-registered ICD-10 codes for bone metastases from prostate cancer is 44%,3 implying that the code fails to capture more than half of the patients with bone metastases. Depending on the role of the bone-metastasis variable in an epidemiologic study, low algorithm sensitivity will lead to underestimation of absolute risks, dilution of associations, or residual confounding.4,5

Elevated levels of serum prostate-specific antigen (PSA) in combination with bone scintigraphy and/or antiresorptive therapy may be a marker of distant metastases in men with prostate cancer,6–8 and the European Association of Urology recommends assessment of skeletal metastasis by bone scan among symptomatic patients with elevated PSA.9 In one study, 90% of prostate cancer patients with PSA in the range of 20–100 ng/mL were found to have bone metastases confirmed by bone scan. (There is a one-to-one correspondence between the PSA units ng/mL and μg/L).6 Both PSA levels and presence (although not the result) of the bone scans are identifiable in routine regional and/or national health registries in Denmark.2,10 Given the low sensitivity of the ICD-10 code and the potential predictive value of an elevated PSA and the subsequent bone scan, we tested a series of registry-based algorithms that could identify cases of distant metastases in patients with prostate cancer, in addition to cases identifiable via ICD-10 codes. The potential algorithm components were chosen based on their availability in routine records; the PSA cutoff was chosen to balance sensitivity against specificity of detection.

Patients and methods

Data sources

To conduct this study, we individually linked data from the following data sources.

The Danish Civil Registration System

The Danish Civil Registration System was established in 1968, and has since assigned a unique personal identifier to all Danish residents at birth or immigration. The system is updated daily, and it tracks births, deaths, and migrations. All data sources in Denmark are linkable via the personal identifier.11

The Danish National Registry of Patients

This registry was established in 1977, and has since then recorded all hospitalizations in somatic hospitals in Denmark.2 Starting in 1995, outpatient and emergency hospital visits became reportable to the DNRP. The registry contains data from discharge summaries, including the personal identifier, dates of admission and discharge, diagnoses, procedures, and some hospital-administered medications. The diagnoses were coded using the ICD-10 during the study period.

The Clinical laboratory Information System (the LABKA research database)

This database compiles results of laboratory tests, including PSA, performed at hospital-based laboratories, for residents of Central and North Denmark Regions.10 Laboratory tests are coded using the (International Federation of Clinical Chemistry and Laboratory Medicine–International Union of Pure and Applied Chemistry codes and local laboratory codes.

Source population and study population

Using the DNRP, we identified men with a prostate cancer diagnosis treated at Aarhus University Hospital between 2005 and 2010. Aarhus University Hospital serves the population of the Central Denmark Region. The region represents about a quarter of the total Danish population. At the start of the first quarter of 2008 (the middle of the study period), the adult male population of the region was 462,568 (http://www.statistikbanken.dk). During 2005–2010, 6,691 men had a discharge diagnosis, including 3,559 men discharged from Aarhus University Hospital. Because the aim was to identify algorithms in addition to the nominal ICD-10 codes, we excluded men whose registry record contained diagnostic codes for distant metastases on or after the date of prostate cancer diagnosis. To identify components of the potential algorithms, we used data from the DNRP and from the LABKA research database.10 From the DNRP, we identified records of bone scintigraphy and administration of antiresorptive therapy (bisphosphonates or denosumab). To investigate whether androgen-deprivation therapy (ADT) affects the validity of the algorithms, we separately sampled prostate cancer patients with and without a history of ADT. ADT was defined as receipt of two endocrine antineoplastic treatments no more than 9 months apart (median length of treatment was 88 days [quartiles 88–96 days])6 or a procedure code for bilateral orchiectomy.

We tested three candidate algorithms for bone metastases: 1) high PSA level (>50 μg/L) combined with bone scintigraphy, 2) high PSA level combined with antiresorptive therapy, and 3) low PSA level (≤50 μg/L) combined with either antiresorptive therapy or bone scintigraphy. The algorithm for nonbone metastases was defined as a high PSA level in the absence of antiresorptive therapy or bone scintigraphy (Figure 1). The index date was the date of fulfillment of the last component of a given algorithm. Absence of distant metastases was defined whenever all observed PSA measurements were ≤50 μg/L in the absence of a record of antiresorptive therapy or bone scintigraphy. For these patients, the index date was the date of the first PSA measurement ≤50 μg/L. All components of the algorithms had to be recorded on or after the diagnosis of prostate cancer. Table S1 lists the codes used in the algorithms.

Figure 1

Sampling of 212 patients with prostate cancer treated at Aarhus University Hospital, Denmark, 2005–2010.

Abbreviations: ART, antiresorptive therapy; ICD, International Classification of Diseases; PSA, prostate-specific antigen; TNM, tumor, node, metastasis.

Statistical analyses

We used information from patients’ hospital charts as the reference standard for confirming the presence and type of metastases. To locate the charts, we linked the registry records of the sampled prostate cancer patients with their charts at Aarhus University Hospital, using the unique civil registration number.12 The research nurse, blinded to the algorithm-determined metastasis status, reviewed the charts and recorded the presence of metastases, site of metastases (prostate, lung, lymph node, liver, other), and date of diagnosis. For each algorithm assumed to indicate the presence of metastases (Figure 1), we computed the positive predictive value (PPV) as the proportion of sampled potential cases that were confirmed by chart review. The negative predictive value (NPV) was computed for the algorithm, assuming that no metastases would be found in the absence of given algorithm components (Figure 1). Thus the NPV was calculated as the proportion of men with no confirmed metastases among men that met the criterion of the algorithm PSA ≤50 μg/L and no antiresorptive therapy or bone scintigraphy. In a post hoc analysis, we also computed NPVs for PSA ≤50 μg/L regardless of antiresorptive therapy or bone scintigraphy, assuming that this new algorithm (PSA ≤50 μg/L and no antiresorptive therapy or bone scintigraphy together with PSA ≤50 μg/L and antiresorptive therapy or bone scintigraphy) was not an indicator of bone metastases. The post hoc analysis was carried out in order to examine the proportion of patients with low PSA and with no confirmed bone metastases, provided that PSA was the only information available. We examined time windows of 30, 90, and 180 days before and after the index date, and as a sensitivity analysis, the entire period covered by the retrieved chart. We obtained overall estimates of PPVs and NPVs and estimates stratified on ADT. All estimates were reported with 95% confidence intervals (CIs).

Results

During the study period, we identified 113 men with potential metastases and 9 9 men without potential metastases using the sampling algorithms, from among all men diagnosed with prostate cancer diagnosis and treated at Aarhus University Hospital. Figure 1 shows the sampling scheme and frequencies. Mean age (standard deviation) at prostate cancer diagnosis was 71 (8.9) years for those with potential metastases and 70 (8.4) years for those without potential metastases. ADT was recorded for 64 of 113 (57%) men with potential metastases and for 50 of 99 (51%) men without potential metastases (Table 1).

Table 1

Descriptive characteristics of 212 men diagnosed with prostate cancer at Aarhus University hospital, Denmark in 2005–2010, according to presence of algorithm component for distant metastases

	Metastases algorithm component present (n=113)	Metastases algorithm component absent (n=99)
Age at prostate cancer diagnosis, years, mean (SD)	71 (8.9)	70 (8.4)
Androgen-deprivation therapy, n (%)	64 (57)	50 (51)
Year of prostate cancer diagnosis, n (%)
2005	14 (12)	12 (12)
2006	12 (11)	23 (23)
2007	23 (20)	12 (12)
2008	18 (16)	14 (14)
2009	24 (21)	19 (19)
2010	22 (19)	19 (19)

Abbreviation: SD, standard deviation.

Based on the chart review, 43 of 212 (20%) patients with prostate cancer and no ICD-10 codes for metastases were found to have metastases: 17 of 212 (8%) patients had bone metastases. Thirty of 212 (14%) patients had nonbone metastases, including metastases to lymph nodes (n=25), lung (n=1), and other sites (n=4). Patients could have metastases to more than one site. Four of the 30 patients had more than one type of metastases. The PPVs of all tested algorithms were low for confirmation of metastases within 30 days before or after the index date, and did not materially improve after broadening the confirmation period to 180 days before or after the index date (Table 2). The highest PPV (0.11, 95% CI 0.03–0.25) for bone metastases was observed for the algorithm “high PSA in the absence of antiresorptive treatment or bone scintigraphy”. Among 73 men with high PSA regardless of receipt of antiresorptive therapy, seven had confirmed bone metastases recorded in the medical chart within 180 days before or after the index date (PPV 0.10, 95% CI 0.04–0.19). When the entire available period was used for confirmation (range 2,322 days before and 462 days after the index date for the confirmed cases), eleven of 73 potential bone metastases were able to be confirmed using that algorithm (PPV 0.15, 95% CI 0.08–0.25).

Table 2

Positive predictive values of the potential algorithms to identify presence of distant metastases

Sampling algorithm	Total	Presence confirmed by medical record
		Bone metastases		Nonbone metastases		Any metastases
		n	PPV (95% CI)	n	PPV (95% CI)	n	PPV (95% CI)
		30 days before or after the index date
PSA >50 μg/L in the absence of ART or bone scintigraphy	37	1	0.03 (0.0007–0.14)	0	–	1	0.03 (0.0007–0.14)
PSA >50 μg/L and bone scintigraphy without ART	32	1	0.03 (0.0008–0.16)	3	0.09 (0.02–0.25)	4	0.12 (0.04–0.29)
PSA >50 μg/l and ART	4	0	–	0	–	0	–
PSA ≤50 μg/L and ART or bone scintigraphy	40	2	0.05 (0.006–0.17)	2	0.05 (0.006–0.17)	4	0.10 (0.03–0.24)
		180 days before or after the index date
PSA >50 μg/L in the absence of ART or bone scintigraphy	37	4	0.11 (0.03–0.25)	2	0.05 (0.01–0.18)	6	0.16 (0.06–0.32)
PSA >50 μg/L and bone scintigraphy without ART	32	3	0.09 (0.02–0.25)	6	0.19 (0.07–0.36)	9	0.28 (0.14–0.47)
PSA >50 μg/L and ART	4	0	–	0	–	0	–
PSA ≤50 μg/L and ART or bone scintigraphy	40	2	0.05 (0.01–0.17)	4	0.10 (0.03–0.24)	6	0.15 (0.06–0.30)

Abbreviations: ART, antiresorptive therapy; CI, confidence interval; PPV, positive predictive value; PSA, prostate-specific antigen.

The highest PPV for nonbone metastases was observed for the algorithm “high PSA and bone scintigraphy without antiresorptive therapy”, with six of 3 2 confirmed (PPV 0.19, 95% CI 0.07–0.36) within 180 days before or after the index date (Table 2). Of the 73 patients with high PSA regardless of receipt of antiresorptive therapy or bone scintigraphy, eight had a confirmed record of a nonbone metastasis within 180 days before or after the index date (PPV 0.14, 95% CI 0.07–0.24). When the entire available record was considered (from 77 days before to 311 days after the index date), 16 of 73 patients had confirmed nonbone metastasis (PPV 0.22, 95% CI 0.13–0.33).

The PPVs for identifying any metastases within 180 days before or after the registry record date were 0.16 (95% CI 0.06–0.32) for PSA >50 μg/L without antiresorptive therapy or bone scintigraphy and 0.28 (95% CI 0.14–0.47) for PSA >50 μg/L in combination with bone scintigraphy. The PPV for the algorithm “PSA ≤50 μg/L and antiresorptive therapy or bone scintigraphy” did not exceed 0.15, regardless of metastasis type and width of ascertainment window. All NPVs were close to 100% (Table 3). The estimates did not vary markedly by receipt of ADT (data not shown).

Table 3

Negative predictive value for absence of potential algorithm components within 30 days before or after the index date

Sampling algorithm	Total	Absence confirmed by medical record
		Bone metastases		Nonbone metastases		Any metastases
		n	NPV (95% CI)	n	NPV (95% CI)	n	NPV (95% CI)
PSA ≤50 μg/L and no ART or bone scintigraphy	99	99	1.00 (0.96–1.00)	99	1.00 (0.96–1.00)	99	1.00 (0.96–1.00)
PSA ≤50 μg/L regardless of ART or bone scintigraphy	139a	137	0.99 (0.95–1.00)	137	0.99 (0.95–1.00)	135	0.97 (0.93–0.99)

Notes:

Includes 99 patients with PSA ≤50 μg/L and no ART or bone scintigraphy and 40 patients with PSA ≤50 μg/L with ART or bone scintigraphy.

Abbreviations: ART, antiresorptive therapy; CI, confidence interval; NPV, negative predictive value; PSA, prostate-specific antigen.

Discussion

We proposed and tested algorithms to identify metastatic disease among men with prostate cancer, based on the knowledge of potential clinical or laboratory manifestations of metastases that are expected to be routinely recorded in the registry data in Denmark, including elevated PSA (>50 μg/L), use of antiresorptive therapy, or bone scintigraphy. No PPV exceeded 3 0%, while all NPVs were close to 100%, and the findings did not vary substantially according to receipt of ADT.

Nevertheless, the individual components of the tested algorithms had high PPVs in measuring what they purport to measure. Nearly 100% of DNRP records of hospital-administered bisphosphonates correspond to an intravenous administration of bisphosphonates.13 Completeness and PPV of prostate cancer diagnosis in the DNRP was nearly 90% when compared with the Danish Cancer Registry,14 and PPV for hormonal antineoplastic therapy in the DNRP exceeds 90%.15 Laboratory values recorded in the Labka database, although not formally validated, are considered to be highly accurate, and have been used as the gold standard to validate, for example, diagnoses of anemia.16

The low PPVs of the tested algorithms may be explained by several factors. Rising PSA levels may precede the development of bone metastases by years.6 Therefore, one potential explanation of the low PPVs is the long time periods that could elapse between fulfillment of all algorithm components and clinical diagnosis of metastases, placing the index date (the last component-fulfillment date) outside the range of the true diagnosis date. From a previous study, it is known that the serum PSA concentration alone provides limited information with regard to the presence of bone metastasis in patients with newly diagnosed cancer of the prostate, a factor that may have also contributed to our low PPVs.8 On the other hand, PSA-doubling time is a known important predictor of bone metastases, and therefore may have been a better proxy than PSA absolute value to examine in this algorithm.17,18 Finally, as the use of bone scintigraphy and computed tomography scans for screening have increased in Denmark in recent years, the algorithm based on bone scintigraphy may not distinguish between a screening and a diagnostic procedure.

Information recorded in the patient medical charts may itself be incomplete with respect to recording presence and/or exact onset of metastases. Many metastases are asymptomatic, and may not come to medical attention until routine checkups. Additionally, it is possible that recording of metastases in the medical charts is not prioritized for patients perceived to be close to the end of life. Antiresorptive therapy was expected to be a specific marker of bone metastases, although it can also be used to prevent osteoporosis among men with prostate cancer treated with hormonal therapy. The use of antiresorptive therapy itself may be incompletely recorded in the DNRP, despite the high PPV. Additionally, if a patient moved to another region of Denmark and received monitoring outside of Aarhus University Hospital, his metastasis diagnosis may not have been recorded in the charts from which we derived “true” metastasis status. Finally, PPV is highly dependent on the prevalence of the condition in the population being tested.19 After removing from the study population prostate cancer patients with registry-recorded evidence of metastases, the prevalence of metastases according to the chart review was 20%, contributing to the low PPVs we observed.

Conclusion

Among men with prostate cancer, the selected characteristics and their combinations yielded low PPVs in identifying metastatic disease, beyond the metastatic disease identifiable by diagnostic codes alone. Based on the results of this study, in their present form, algorithms based on PSA, antiresorptive therapy, or bone scintigraphy are not suitable for supplementing ICD-10-based algorithms for identification of additional cases of distant metastases among men with prostate cancer. It may be advantageous to define and explore alternative algorithms with adjusted components, cutoffs, or timing.

Table S1

Codes used in the study across databases in Denmark

Variable	Code type	Code
Prostate cancer	ICD-10	C61
Distant metastases	ICD-10TNM	C77–C79AZCD41
Bilateral orchiectomy	In-hospital procedure codes (NOMESCO classification)	KKFC10, KKFC11, KKFC13, KKFC15
Gonadotropin-releasing hormone agonist/antagonist therapy	In-hospital treatment codes	BJHKx antihormonal treatment, prostateBWHCx hormonal or antihormonal antineoplastic treatment
Skeletal scintigraphy	In-hospital procedure code	WKBGD19XX, WKBGS19XX, WKBGT19XX, WKBGW19XX, WKBPSFAXX, WKBSS19XX
Antiresorptive therapy	In-hospital treatment codes	BWHB40 treatment with bisphosphonatesBWHB40A treatment with zoledronic acidBWHB40B treatment with ibandronic acidBWHB40C treatment with pamidronic acidBWHB42 denosumab
Prostate-specific antigen	Labka database, IFCC–IUPAC codes and local laboratory codes	NPU21579P, NPU08669PLocal laboratory codes Central Region 110329; 1510914; 1610208; 1715095; 1815095; 276,13856; 1315230; 1415230

Abbreviations: ICD, International Classification of Diseases; TNM, tumor, node, metastasis; NOMESCO, Nordic Medico-Statistical Committee; IFCC, International Federation of Clinical Chemistry and laboratory Medicine; IUPAC, International Union of Pure and Applied Chemistry.