Registration in the Danish Regional Nonmelanoma Skin Cancer Dermatology Database: completeness of registration and accuracy of key variables.

OBJECTIVE: To validate a clinical database for nonmelanoma skin cancer (NMSC) with the aim of monitoring and predicting the prognosis of NMSC treated by dermatologists in clinics in the central and north Denmark regions.
METHODS: We assessed the completeness of registration of patients and follow-up visits, and positive predictive value (PPV), negative predictive value (NPV), sensitivity, and specificity of registrations in the database. We used the Danish Pathology Registry (DPR) (n = 288) and a review of randomly selected medical records (n = 67) from two clinics as gold standards.
RESULTS: The completeness of registration of patients was 62% and 76% with DPR and medical record review as gold standards, respectively. The completeness of registration of 1st and 2nd follow up visits was 85% and 69%, respectively. The PPV and NPV ranged from 85% to 99%, and the sensitivity and specificity from 67% to 100%.
CONCLUSION: Overall, the accuracy of variables registered in the NMSC database was satisfactory but completeness of patient registration and follow-up visits were modest. The NMSC database is a potentially valuable tool for monitoring and facilitating improvement of NMSC treatment in dermatology clinics. However, there is still room for improvement of registration of both patients and their follow-up visits.

Introduction

Nonmelanoma skin cancer (NMSC), including basal cell carcinoma (BCC) and squamous cell carcinoma (SCC), is the most common malignancy in western countries.1,2 The annual incidence of NMSC is rapidly increasing and NMSC has therefore become a significant challenge in terms of public health management and health care costs.3–5 However, high quality epidemiological data on NMSC is sparse due to incomplete registration of NMSC in cancer registries.4 While the Danish Cancer Registry (DCR), for example, contains basic information on incident cases, the completeness of BCC lesion registration is only 50%.1

Detailed data on NMSC and the treatment of NMSC in everyday clinical practice are important for surveillance, prediction of prognosis, improvements in quality of care and treatment, and for research purposes.4,6 Such data can be obtained from properly designed clinical databases, which are an attractive source for epidemiological research.6,7 However, the very high incidence of NMSC, as well as the common de novo occurrences of skin cancers seen among NMSC patients, poses a challenge to NMSC data collection. A result of this may be the incomplete registration of NMSC in population-based cancer registries throughout the world.2,8

In Denmark, most NMSCs are treated by dermatologists in dermatology clinics free of charge. Approximately two-thirds of all specimens registered in the Danish Pathology Registry (DPR) with a diagnosis of either BCC or SCC originate from patients diagnosed and treated by dermatologists.9 A Danish regional clinical database of NMSC diagnosed and treated by dermatologists was established in 2007. The database aims to examine the epidemiology of NMSC, Bowen’s disease, and keratoacanthoma treated by dermatologists in private clinics, and to examine the epidemiology of treatment procedures. The database will also be used to monitor the treatment effect and to facilitate improvement in the outcome of NMSC treatment in everyday clinical practice by providing data on predictors for disease outcome.

Validation of data is crucial to assess the usefulness of a clinical database as a valuable tool for answering clinical, administrative, and research questions.10 Thus, the aim of this study was to validate the data quality and completeness of NMSC cases in the Danish regional NMSC dermatology database.

Material and methods

The Danish Regional NMSC Dermatology Database

‘The Danish Regional NMSC Dermatology Database’ (The NMSC database) was initiated in cooperation with the Danish Dermatological Society in 2007. An online registration system was developed and registration in the database began as a pilot study in July 2007. In the pilot phase, two dermatology clinics in the Central Denmark Region (each with two full-time dermatologists) registered consecutive patients in the database. In August 2008, 15 of 19 private clinics in the two regions accepted affiliation with the database, and since October 2008, 10 of the clinics have registered data in the database.

In April 2009, the Danish National Board of Health approved the database as a regional clinical database. Since then, it has been mandatory for dermatologists in private clinics in the Central Denmark Region, and for certain clinics in the North Denmark Region, to register data in the database. By December 2009, only four of the 19 eligible clinics were still not affiliated with the database.

The database contains detailed information on type of NMSC, treatment, and prognosis after treatment in the private dermatology clinics. On the day of treatment, the treating dermatologist completes a questionnaire with detailed information on the tumor and patient-related factors, as well as performed treatments (Appendix 1). It is also mandatory for dermatologists to register NMSC cases in the DCR. To minimize the burden of registration, data from the NMSC database registration system can be transferred directly to the DCR on completion of the questionnaire. This procedure aims to avoid missing data in the NMSC database.

NMSC patients are scheduled for two follow-up visits, one at three months (0–6 months) and one at 12 months (6–15 months) after treatment. At these visits, information on recurrence, cosmetic result and complications are registered (Appendix 2).

Study population

In this validation study we included patients registered in the NMSC database from January 1 until June 30, 2008 (288 patients with 359 lesions). The period was chosen to obtain a follow-up period of a minimum of 15 months for all tumors registered. In the study period, two dermatology clinics registered data in the database; in each clinic, two to three dermatologists entered data daily. To calculate the completeness of registration of patients with one or more histologically verified tumor(s), the study population was restricted to patients with lesions verified histologically.

In order to include tumors which were not verified histologically, completeness of all tumor registration was assessed by reviewing medical records in the two clinics. Within the study period, three days from each of the six months were randomly selected for each clinic (a total of 36 working days). All contacts to the clinics on those days were reviewed via medical records in order to cover all contacts with patients with NMSC, Bowen’s diseases or keratoacanthoma. A total of 88 patients with 104 tumors were identified in the medical record review. Sixty-seven of the 88 patients from the medical record review were also registered in the NMSC database. This study population was used to calculate the completeness of the database and positive predictive values. When calculating completeness of registration of follow-up visits in the database, we excluded 13 of the 67 patients, since it is not mandatory for dermatologists to register follow-up visits for patients referred to hospital treatment, nor for patients with Bowen’s diseases or keratoacanthoma lesions.

Other data sources

The DPR contains information on histological examinations performed in Denmark. Since 2005, reporting to the DPR has been mandatory for both privately and publicly employed pathologists. The registry includes information on referring department, the performing department, the date of the pathology test, and the associated histological diagnoses. All diagnoses are coded according to Systematized Nomenclature of Medicine (SNOMED).11 Patients registered with a SNOMED code of NMSC, Bowen’s disease, and keratoacanthoma in the DPR during the study period, were identified in the DPR. (See Appendix 3 for the full list of SNOMED codes).

The Danish Civil Registration System (CRS) contains information on all Danish residents who are assigned a unique 10-digit personal civil registration number (CPR number) at birth or when immigrating into Denmark. Information on changes in vital status, such as emigration and death, is registered in CRS, and the unique CPR-number can be used to link information from different Danish registries.12 The CRS was used to obtain information on vital status for patients in the study population.

Medical records were reviewed from the two pilot dermatology clinics and included in the validation study. The medical records were systematically reviewed and information retrieved on history of skin cancer, size, location, treatment type, date of treatment, recurrences, cosmetic result, and complications related to treatment. Data were typed into EpiData 3.1 (The EpiData Association, Odense, Denmark).

Statistical analyses

Completeness of registration in the NMSC database

The completeness of registration of patients with a histologically verified tumor was assessed using the DPR as a gold standard. The completeness of registration was defined as the numbers of patients diagnosed with a tumor and registered in both the NMSC database and in the DPR, divided by the number of patients with the same diagnosis registered in the DPR.

To validate data on tumors which were not verified histologically, completeness of registration of all tumors was defined as the number of patients with a clinically diagnosed tumor registered in the NMSC database and in the medical record, divided by the number of patients recorded in the medical record with either type of diagnosis; any histologically verified diagnosis overruled a clinical diagnosis.

Completeness of registration of 1st follow-up visits was defined as the number of patients registered with at least one follow-up visit in the database between 0–6 months after initial treatment, divided by the number of patients registered with a follow-up visit in the medical records. The completeness of registration of a 2nd follow-up visit was defined as the number of patients registered with at least one follow-up visit in the NMSC database 6 to 15 months after initial treatment, divided by the number of patients registered with a follow-up visit in the medical records.

Accuracy

We evaluated the accuracy of the registrations in the NMSC-database using the information obtained in the medical record review as a gold standard. However, for the variable ‘histological diagnosis’ we used the DPR as the gold standard, as the DPR contains the pathologist’s registration of the histological diagnosis. For each of the registered variables, we estimated sensitivity, specificity, and positive and negative predictive values (PPV and NPV) where appropriate.13 We only included tumors registered in both the NMSC database and in the medical record. Please see Table 4 for detailed information on the calculation.

Table 4

Formulae to calculate the PPV, NPV, and sensitivity and specificity

	NMSC database	Medical record review
		yes	no
eg, First skin cancer ever	Yes	a	b	PPV = a/(a + b)
	No	c	d	NPV = c/(c + d)
		Sensitivity = a/(a + c)	Specificity = b/(b + d)

Notes: PPV, positive predictive value. The proportion of patients registered with ‘yes’ for a given variable in the database confirmed in the medical record (ie, the variable: diagnosis, according to the DPR) for variables: ‘diagnosis overall’, ‘treatment overall’, ‘size’, localization’ and ‘treatment date’ PPVs were the proportion of a given variable registered in the database confirmed in the medical record/DPR; NMSC, nonmelanoma skin cancer; NPV, The proportion of patients registered with ‘no’ for a given variable in the database confirmed in the medical record (ie, the variable; diagnosis, according to the DPR); Sensitivity, The proportion of patients registered with ‘yes’ for a given variable in the medical record, who were registered with a ‘yes’ in the database (ie, the variable diagnosis, according to the DPR); Specificity, The proportion of patients with ‘no’ for a given variable in the medical record, who were registered with a ‘no’ in the database (ie, the variable diagnosis, according to the DPR).

Abbreviations: DPR, Danish Pathology Registry; NMSC, nonmelanoma skin cancer; NPV, negative predicted value; PPV, positive predicture value.

Where the information registered in the NMSC database disagreed with that in the medical record, the medical record was reviewed again and any uncertainty was discussed and clarified with an independent dermatologist.

Tumor size was considered in agreement if there was no more than 2 mm difference between the sizes registered in the database and in the medical record. In the database, tumor localization was registered by marking the tumor on a drawing of the body. If this agreed with the written description of localization found in the medical record, the two were considered identical. Where the medical record had no information on complications, this was interpreted as ‘no complications’. In the NMSC database complications are recorded with a ‘yes’ or ‘no’.

Results

Table 1 lists characteristics of all registrations in the database until August 31 2009, characteristics of the study population (n = 288 patients), and characteristics of the patients registered in the database whose medical records were reviewed (n = 67 patients). Information on cosmetic results was not routinely recorded in the medical record and therefore was not retrieved. The study population and the sample selected for medical record review are representative of all registrations in the database. Furthermore, the database had very little missing data.

Table 1

Descriptions of patients and tumors in the nonmelanoma skin cancer database

	Registration from Jan 1 2008 – Aug 31 2009 in the NMSC database (%)	Study population for calculation of completeness: Registration from Jan 1 – Jun 30 2008 (%)	Medical record reviews used to calculate PPV and completeness (%)
Total numbers of patients	1,775	288	67
Male	892 (50)	136 (47)	31 (46)
Female	883 (50)	152 (53)	36 (54)
Clinics
Clinic no 1	457	178 (62)	39 (58)
Clinic no 2	296	110 (38)	28 (42)
Age < 55	264 (15)	49 (17)	16 (24)
Age ≥ 55 and <75	889 (50)	154 (53)	29 (43)
Age ≥ 75	622 (35)	85 (30)	22 (33)
History of previous skin cancer
No history of skin cancer	992 (56)	156 (54)	32 (48)
History of previous skin cancer	737 (42)	130 (45)	34 (51)
Unknown	45 (3)	2 (1)	1 (2)
Missing	1 (–)	0	0
Total numbers of tumors	2,400	359	78
Histological diagnosis
Basal cell carcinoma	2,090 (87)	328 (91)	72 (92)
Squamous cell carcinoma	123 (5)	10 (3)	2 (3)
Mb. Bowen	77 (3)	9 (3)	2 (3)
Keratoacanthoma	34 (1)	7 (2)	2 (3)
Keratoacanthoma/SCC	1 (–)	1 (–)	0
No pathology test performed	57 (2)	4 (1)	0
Histology inconclusive	10 (–)	0	0
Missing histology registration	8 (–)	0	0
Tumor registered is
A new primary tumor	2,173(91)	324 (90)	69 (88)
A previously treated tumor	221 (9)	35 (10)	9 (12)
Missing	6 (–)	–	–
Treatment modality
Curettage with or without cautery	1,919 (80)	294 (82)	65 (83)
Cryotherapy	38 (2)	3 (1)	0
Excision	157 (7)	9 (3)	1 (1)
Tangential excision with or without curettage	4 (–)	1 (–)	0
Photodynamic therapy	55 (2)	15 (4)	1 (1)
Imiquimod	42 (2)		1 (1)
Referred to hospital treatment	152 (6)	28 (8)	9 (11)
No treatment	17 (–)	1 (–)	0
Treatment variable missing	16 (1)	6 (2)	1 (1)

Abbreviations: PPV, positive predictive value; SCC, squamous cell carcinoma.

Completeness

The overall completeness of registration of patients in the database was 62% when using the DPR as a gold standard (Table 2), but differed according to clinic. The completeness in clinic 1 was 93%, whereas the completeness in clinic 2 was 40%. Based on the medical record review, the completeness of registration of patients in the database was 76% overall: 95% and 60% in clinic 1 and 2, respectively (Table 2).

Table 2

Completeness of registration of patients in the nonmelanoma skin cancer database

	Patients registered in database (no.)	Patients registered in the DPR (no.)	Degree of completeness (Evaluation: DPR) % (95% CI)	Patients registered in database (no.)	Patients registered in the medical records (no.)	Degree of completeness (Evaluation: medical records) % (95% CI)
Overall patients	288	452	62 (58–67)	67	88	76 (66–85)
Completeness for each clinic
Patients clinic no 1	178	182a	93 (88–94)	39	41	95 (84–99)
Patients clinic no 2	110	270b	40 (34–46)	28	47	60 (44–74)

Abbreviations: CI, confidence interval; DPR, Danish Pathology Registry.

Among the 54 patients eligible for registration of a follow-up visit, 39 were registered with a 1st follow-up visit and 22 were registered with a 2nd follow-up visit in the NMSC database, and a total of 45 of the 54 patients were registered with either 1st or 2nd follow-up visit. Overall, the completeness of registration of 1st follow-up visits in the NMSC database compared with registration in the medical record review was 85%, but varied according to clinic: 100% and 71% in clinic 1 and 2, respectively. The completeness of registration of 2nd follow-up visits was 69% overall: 74% and 63% in clinic 1 and 2, respectively. The medical record review revealed that 18 of the 54 patients did not participate in a 2nd follow-up visit. Based on the medical record review and data from the CRS there were a number of reasons for nonparticipation in follow-up visits. Four patients had a 2nd follow-up later than 15 months after treatment; two patients had a recurrence at 1st follow-up and were referred to hospital for treatment; one patient died during follow-up; one patient had a tumor clinically assessed to be a keratoacanthoma; 3 patients cancelled or failed to attend; while information on seven patients was missing (Table 3).

Table 3

Completeness of follow-up visit registration in the nonmelanoma skin cancer database

	1st follow-up visit (<6 months after treatment)				2nd follow-up visit (6–15 months after treatment)
	Patients registered in the database (no.)	Follow-up visit registered in database (no. of patients)	Follow-up visits registered in medical records (no.)	Degree of completeness of 1st follow-up visits: % (95% CI)	Patients registered in the database (no.)	Follow-up visit registered in database. (no. of patients)	Follow-up visits registered in medical records (no. of patients)	Degree of completeness of 1st follow-up visits: % (95% CI)
Overall	54	39	46	85 (71–94)	54	24	36	67 (49–81)
Clinic no 1	29	22	22	100 (85–100)	29	14	19	74 (49–91)
Clinic no 2	25	17	24	71 (49–87)	25	10	17	59 (33–82)

Abbreviation: CI, confidence interval.

Accuracy of the registered variables

PPV, NPV, sensitivity, and specificity, are given in Tables 5 and 6. The PPV, NPV, sensitivity, and specificity of histologically verified diagnoses was 100%, using the DPR as a gold standard (Table 5). The PPV, NPV, sensitivity, and specificity of registration of a patient with first skin cancer were 97%, 94%, 97%, and 94%, respectively. The PPV, NPV, sensitivity, and specificity of tumor history (ie, a new primary tumor) were 99%, 100%, 100% and 90%, respectively. The PPV was 96% for localization, 85% for size, and the PPV of the date of treatment was 91%. The PPV of treatment modality registered was 95%. PPV, NPV, sensitivity, and specificity were calculated for the different types of treatment ranging from 50%–100% (Table 6). PPV, NPV, sensitivity, and specificity of complications in relation to treatment was 100%, 98%, 67%, and 100%. PPV, NPV, sensitivity, and specificity of recurrences registered at either 1st or 2nd follow-up was 100%, 98%, 75%, and 100%. However, the number of recurrences was low and the medical record review revealed an incomplete registration of recurrences mainly due to the incomplete registration of follow-up visits. At 1st follow-up visits, two recurrences were registered in the NMSC database. The medical record review revealed one additional recurrence which had been mistakenly registered as no recurrence. At 2nd follow-up visit, one recurrence was registered. The medical record review revealed four additional recurrences which had not been registered due to incomplete registration of follow-up visits, and the sensitivity of registration of recurrence (including the missing registration of follow-up visits) was then 38% (3/8).

Table 5

Validity of variables registered in the nonmelanoma skin cancer database. Verified in medical records

	NMSC database	Medical record		Total	PPV % (95% CI)	NPV % (95% CI)	Sensitivity % (95% CI)	Specificity % (95% CI)
		yes	no
Histological diagnosis overall	Total	78*	0	78	78/78 = 100 (95–100)
Basal cell carcinoma	Yes	72	0	72	72/72 = 100 (95.0–100)	6/6 = 100 (54–100)	72/72 = 100 (95–100)	6/6 = 100 (54–100)
Squamous cell carcinoma	Yes	2	0	2	2/2 = 100 (15.8–100)	76/76 = 100 (95–100)	2/2 = 100 (15.8–100)	76/76 = 100 (95–100)
Mb. Bowen	Yes	2	0	2	2/2 = 100 (15.8–100)	76/76 = 100 (95–100)	2/2 = 100 (15.8–100)	76/76 = 100 (95–100)
Keratoacanthoma	Yes	2	0	2	2/2 = 100 (15.8–100)	76/76 = 100 (95–100)	2/2 = 100 (15.8–100)	76/76 = 100 (95–100)

Notes: Histological diagnosis confirmed in the medical record.

Abbreviations: CI, confidence interval; PPV, positive predictive value. NPV, negative predictive value.

Table 6

Validity of variables registered in the nonmelanoma skin cancer database. Verified in medical records

	NMSC database:	Medical record		Total	Missing	PPV % (95% CI)	NPV % (95% CI)	Sensitivity % (95% CI)	Specificity % (95% CI)
		Yes	No
First skin cancer ever (incident case)	Yes	31	1	32		31/32 = 97 (85–100)		31/33 = 97 (85–100)
	No	2	32	34			32/34 = 94 (80–99)		32/34 = 94 (80–99)
	Unknown		1
	Total	33	34
Tumor registered is a new primary tumor	Yes	67	1	68	1	67/68 = 99 (92–100)		67/67 = 100 (95–100)
	No	0	9	9			9/9 = 100 (66–100)		9/10 = 90 (55–100)
	Total	67	10
Localization of tumor		75*	3*	78		75/78 = 96 (89–99)
Size		29*	5*	34	44	29/34 = 85 (69–95)
Treatment date		71*	7*	78		71/78 = 91 (82–96)
Treatment modality overall	Total	73*	4*	77		73/77 = 95 (87–99)
	missing				1
Curettage ± cautery	Yes	62	3	65		62/65 = 95 (87–99)		62/63 = 98 (91–100)
	No	1	11	12			11/12 = 92 (62–100)		11/14 = 79 (49–95)
	Total	63	14
Other treatments incl. excision, photodynamic therapy, imiquimod creme	Yes	2	1	3		2/3 = 67 (9–99)		2/4 = 50 (7–93)
	No	2	72	74			72/74 = 97 (91–100)		72/73 = 99 (93–100)
	Total	4	73
Referred to hospital treatment	Yes	9	0	9		9/9 = 100 (66–100)		9/9 = 100 (66–100)
	No	0	68	68			68/68 = 100 (95–100)		68/68 = 100 (95–100)
	Total	9	68
Recurrence at 1st or 2nd follow-up visit	Yes	3	0	3		3/3 = 100 (29–100)		3/4 = 75 (19–99)
	No	1	43	44			43/44 = 98 (88–99)		43/43 = 100 (92–100)
	Total	4	43		3
	missing	4	11
Complications in relation to treatment	Yes	2	0	2		2/2 = 100 (16–10)		2/3 = 67 (9–99)
	No	1	51	52			51/52 = 98 (90–100)		51/51 = 100 (93–100)
	Total	3	51
	missing	0	8		3

Notes: Confirmed in medical record (yes/no).

Abbreviations: CI, confidence interval; NPV, Negative predictive value; PPV, Positive predictive value.

Discussion

Our study showed a high PPV, NPV, sensitivity, and specificity for the prognostic and treatment related variables (diagnosis, localization, size, skin cancer history, and treatment). We had a low degree of missing data on these variables in the database, due to its electronic validation of the questionnaire before transfer of information to the DCR. Thus, registrations of prognostic and treatment related variables in the database are of high accuracy.10,14 PPV, NPV, sensitivity, and specificity of the outcome variables (recurrence and complication) ranged from 67%–100%. However, our estimates were imprecise, due to small numbers of outcomes.

The completeness of patient registration in the NMSC database was high in clinic 1 and low in clinic 2, illustrating the challenge of registering data in everyday clinical practice. NMSC is a very common cancer. Complete registration of this cancer is vulnerable to, for example, physician time pressure and manpower devoted to data registration in the NMSC database. Due to the modest overall completeness, the NMSC database cannot yet be used to calculate the true incidence of NMSC among patients treated in dermatology clinics.

The modest completeness of registration of 2nd follow-up visits in both clinics caused missing information on NMSC recurrences among patients registered in the database. Clinic 1 had the highest completeness resulting in the lowest number of missing data. Missing data caused inaccuracy of the registration of complications and recurrences at 2nd follow-up visits. Inaccuracy of registration of outcome related variables is also seen in other databases.15–17 Hence in the future, measures will be taken to obtain higher completeness of registration. Clinics reporting to the database will therefore receive lists every third month on missing registration compared to DPR, as well as a list of missing registration of follow-up visits. This has shown to be effective in other databases.17,18 Additionally we will also regularly contact each clinic by phone in order to solve any registration problems and to act as a reminder to register patients in the database.

Strengths of this validation study include the extensive review of medical records with very detailed data on prognostic factors, treatment, and outcome. This enabled validation of a high number of variables in the NMSC database. The risk of information bias was low since medical records were systematically reviewed using a standard form. Nonetheless, our study had some limitations including a lack of information on cosmetic results in medical records, as well as an incomplete registration of tumor sizes in the medical records. The low number of recurrences and complications registered in the NMSC database also caused imprecise estimates of the PPV. Another limitation is the lack of randomization of the two clinics included in the validation study. However, the clinics were chosen in order to obtain 15 months of follow-up, and patients treated at these clinics appear to be representative of all patients in the database.

The completeness of the NMSC database must be evaluated compared with available NMSC data sources. In Denmark, data on NMSC diagnoses is collected by the DCR. However, only basic information on incident cases is collected, and the completeness of incident BCC cases is estimated at only 50%.1 Data on NMSC is not routinely collected in the major cancer databases.19,20 High quality data on NMSC is therefore sparse.

The NMSC database is unique compared with other data sources because of its detailed data on prognostic factors, treatment, and outcome. We have shown that PPV, NPV, sensitivity, and specificity of prognostic and treatment related variables (diagnosis, localization, size, skin cancer history, and treatment) was high. Due to low numbers of outcome (recurrence and complications) risk estimates were imprecise, and further examination of accuracy of the outcome variables is desirable.

Even though completeness is still not satisfactory, the high values of PPV, NPV, sensitivity, and specificity ensure high quality of data registered. In future, therefore, the database can be used to monitor treatment and to facilitate improvements in treatment of NMSC in everyday clinical practice by providing detailed data on predictors for disease outcome, treatment, and treatment out come.

In conclusion, this validation study shows that the NMSC database has the potential to become a significant resource for epidemiological research of NMSC, and for monitoring and facilitating improvement of treatments of NMSC, Bowen’s disease, and keratoacanthoma in dermatology clinics. However, there is still room for improvement of registration of both patients and their follow-up visits which will be a future aim.