Validation of prostate-specific antigen laboratory values recorded in Surveillance, Epidemiology, and End Results registries.

BACKGROUND: Researchers have used prostate-specific antigen (PSA) values collected by central cancer registries to evaluate tumors for potential aggressive clinical disease. An independent study collecting PSA values suggested a high error rate (18%) related to implied decimal points. To evaluate the error rate in the Surveillance, Epidemiology, and End Results (SEER) program, a comprehensive review of PSA values recorded across all SEER registries was performed.
METHODS: Consolidated PSA values for eligible prostate cancer cases in SEER registries were reviewed and compared with text documentation from abstracted records. Four types of classification errors were identified: implied decimal point errors, abstraction or coding implementation errors, nonsignificant errors, and changes related to "unknown" values.
RESULTS: A total of 50,277 prostate cancer cases diagnosed in 2012 were reviewed. Approximately 94.15% of cases did not have meaningful changes (85.85% correct, 5.58% with a nonsignificant change of <1 ng/mL, and 2.80% with no clinical change). Approximately 5.70% of cases had meaningful changes (1.93% due to implied decimal point errors, 1.54% due to abstract or coding errors, and 2.23% due to errors related to unknown categories). Only 419 of the original 50,277 cases (0.83%) resulted in a change in disease stage due to a corrected PSA value.
CONCLUSIONS: The implied decimal error rate was only 1.93% of all cases in the current validation study, with a meaningful error rate of 5.81%. The reasons for the lower error rate in SEER are likely due to ongoing and rigorous quality control and visual editing processes by the central registries. The SEER program currently is reviewing and correcting PSA values back to 2004 and will re-release these data in the public use research file. Cancer 2017;123:697-703.

INTRODUCTION

Prostate‐specific antigen (PSA) is a serologic biomarker often used in screening for prostate cancer, and there are substantial variations in recommended guidelines for its value in determining potential risk.1, 2, 3 The PSA level at the time of diagnosis is prognostic for outcome,4, 5, 6, 7, 8, 9, 10, 11, 12 and since 2010 the PSA level and Gleason score have been incorporated into the seventh edition of the American Joint Committee on Cancer (AJCC) Tumor‐Lymph Node‐ Metastasis (TNM) staging system for prostate cancer. In conjunction with the anatomic extent of disease, the values of these 2 nonanatomic factors help to distinguish disease stages I, IIA, and IIB.13 The PSA value collected by central cancer registries frequently has been used by researchers to categorize tumors according to their potential for aggressive clinical disease in the absence of curative‐intent treatment. The Surveillance, Epidemiology, and End Results (SEER) program has been collecting the PSA value at the time of diagnosis since 2004. The PSA value is recorded as a 3‐digit field with an implied decimal point between the second and third digits (eg, a PSA of 4.0 ng/mL should be coded as “040”). Currently, any PSA value ≥98 is coded as “980” due to this 3‐digit limitation.

In late 2014, an 18% error rate related to the implied decimal point for the PSA value was reported in a routine quality assessment of cancer staging. This study included > 800 registrars from both hospital and central cancer registries and was based on a random set of 57 prostate cancer cases for which deidentified medical records (source documents in the original format) were obtained for abstraction for quality control studies.14, 15 Decimal‐related errors were likely caused by the implied decimal point in the data collection field, which resulted in the potential for a 10‐fold difference between the actual and recorded PSA value. Initial investigation into SEER data suggested that the error rate likely was lower in the data reported by SEER, possibly due to visual editing at the central registry because data are consolidated across multiple reports from different facilities such as hospitals and pathology laboratories. Because of the SEER program's commitment to high‐data quality standards, PSA data were removed from the SEER research database available from the National Cancer Institute Web site until these data could be reviewed and errors more accurately quantified and corrected. The current study describes the protocol, methods, and results of a comprehensive review of PSA laboratory values recorded across all SEER registries for cases diagnosed in 2012, and the proposed mechanism for reviewing and correcting the PSA laboratory values back to 2004, the first year this data element was collected.

MATERIALS AND METHODS

This data quality review covered all invasive prostate cancer cases diagnosed in 2012 within the SEER central registry regions (Alaska Native, Connecticut, Detroit, Georgia, Greater California, Hawaii, Iowa, Kentucky, Los Angeles, Louisiana, New Jersey, New Mexico, San Francisco/Oakland/San Jose‐Monterey, Seattle‐Puget Sound, and Utah). Cases reported from death certificate, autopsy, or nursing home/hospice only were excluded. All source records available at the central registry were used to evaluate the PSA values coded in the consolidated SEER central registry tumor record, including pathology reports and text documentation abstracted from medical records and recorded in the cancer abstracts. Registry personnel who abstract data from medical records are trained to provide text to support the coded values in the abstract. Because central registry staff generally do not have access to medical records, this process of visually comparing the abstracted text against coded values is a key quality control activity performed by central registries.

For each eligible prostate cancer case in the SEER registry, the consolidated PSA value was reviewed and compared with text by an experienced staff member. The registrars were reminded to apply the most appropriate instructions for PSA abstracting and coding, as available in the User Documentation and Coding Instructions (version 02.05) of the AJCC Collaborative Stage Data Collection System.16 When a discrepancy was identified, the staff updated the case with the corrected PSA value in the SEER registry database. If there was no text supporting the consolidated PSA value in any of the underlying source records, the SEER staff flagged the record and then followed back to the reporting source of the abstract to validate the coded PSA value.

On completion of the review, both original and final coded values for PSA were supplied to the National Cancer Institute along with related data items (eg, TNM stage group). After the data were received from all participating registries, errors were classified into 4 general categories, as follows.

Classification of Errors

Implied decimal point errors were defined as a difference of a factor of 10 between the original recorded and the corrected PSA value. From the previous example, a PSA of 4.0 ng/mL should be coded as “040.” If instead the PSA was coded as “004” or “400,” the case was classified as an implied decimal error. This category also included decimal errors in which one of the codes was “980” (≥98 ng/mL) and the other code was <“980.” The highest individual PSA value allowed in the current data collection system is 97.9 ng/mL. Every PSA above this value is coded as 980. These errors also were due to either not moving the decimal point to translate the PSA units from ng/mL or moving the decimal point in the wrong direction.

Abstraction or coding implementation errors were related to the incorrect application of the coding rules16 or errors in the abstract coding process (ie, coding the wrong value after abstracting the proper value in the text). The User Documentation and Coding Instructions (version 02.05) of the AJCC Collaborative Stage Data Collection System provided instructions for recording the highest PSA value closest to but obtained before the diagnostic biopsy and the initiation of treatment.16 The majority of errors in this category were due to coding the incorrect PSA test when multiple tests appeared in the medical record. Other errors included in this category were transcription errors of >1 ng/mL.

Nonsignificant errors were defined as changes with differences of <1 ng/mL (or a difference of <10 units in the PSA code scale) between the original PSA value recorded and the corrected value. These differences were primarily due to rounding. It is important to point out that although the absolute change in the PSA value for records in this group is small, the changes nevertheless can impact the disease stage group due to the specific PSA cutpoints used in the staging definitions.

Changes related to unknown values comprised the fourth category. Three separate codes could be used when the PSA value was not known, including “997” (test ordered, results not in chart), “998” (test not done [test not ordered and not performed]), and “999” (unknown or no information, not documented in the patient record). The changes involving an unknown values category included changes from one of the unknown codes to another unknown code (not clinically relevant), and relevant changes, which included changes from a known PSA value to an unknown code, or changes from an unknown code to a known PSA value.

RESULTS

Of the 50,277 invasive prostate cancer cases diagnosed in 2012, a total of 43,163 (85.85%) required no change based on the available supporting text (ie, were correct) and 2804 (5.58%) had a nonsignificant change mostly due to rounding (Table 1). These 2 categories represented 91.43% of the prostate cancer cases being identified as “correct” or a “nonsignificant error.” Changes involving unknown PSA codes occurred in approximately 5.09% of cases. Of these, 2.76% were simply changes from one unknown value to another unknown value and were not clinically relevant, thereby increasing the percentage of coded PSA values without meaningful changes to 94.19% of all prostate cases.

Table 1

Types of Errors Identified During PSA Laboratory Value Review by SEER Registrya

	Total	No Change (Correct)		Implied Decimal Point		Abstraction and Implementation of Coding Rules		Nonsignificant Change		Changes Involving Unknown Codes
Registry	No.	No.	%	No.	%	No.	%	No.	%	No.	%
Overall	50,277	43,163	85.85	972	1.93	776	1.54	2804	5.58	2562	5.09
A	3096	2679	86.53	59	1.91	35	1.13	142	4.59	181	5.85
B	826	644	77.97	18	2.18	14	1.69	102	12.35	48	5.81
C	1308	1130	86.39	17	1.30	13	0.99	61	4.66	87	6.65
E	10,649	9271	87.06	192	1.80	162	1.52	597	5.61	427	4.01
G	679	517	76.14	16	2.36	29	4.27	39	5.74	78	11.49
H	4129	3636	88.06	75	1.82	56	1.36	278	6.73	84	2.03
I	2879	2635	91.52	22	0.76	41	1.42	82	2.85	99	3.44
J	3350	2670	79.70	70	2.09	66	1.97	311	9.28	233	6.96
K	6228	5152	82.72	172	2.76	111	1.78	239	3.84	554	8.90
L	2783	2377	85.41	86	3.09	24	0.86	226	8.12	70	2.52
M	4446	3786	85.16	93	2.09	66	1.48	228	5.13	273	6.14
N	5710	5036	88.20	78	1.37	79	1.38	288	5.04	229	4.01
O	2236	1829	81.80	65	2.91	57	2.55	165	7.38	120	5.37
P	1890	1748	92.49	b	0.11	22	1.16	44	2.33	74	3.92

Abbreviations: PSA, prostate‐specific antigen; SEER, Surveillance, Epidemiology, and End Results.

Registry names were omitted to preserve the anonymity of the registries. Data from registries in San Jose‐Monterey and San Francisco were combined to represent the Greater Bay. Atlanta, Rural Georgia, and Greater Georgia also were combined.

Two cases were ineligible for staging by the 7th edition of the American Joint Committee on Cancer TNM staging system.

Implied decimal point errors were defined as a difference of a factor of 10 between the original recorded and the corrected PSA value.

Abstraction or implementation errors were related to the incorrect application of the coding rules or errors in the abstract coding process.

Nonsignificant errors were differences of <1 ng/mL (or a difference of <10 units in the PSA code scale) between the original PSA value recorded and the corrected value.

Changes related to unknown values comprised separate codes used for PSA value, including 997, which indicated test ordered, results not in chart; 998, which indicated test not done (test not ordered and not performed); and 999, which indicated unknown or no information, not documented in patient record.

Registries with < 50 cases were included in the overall calculations but were not included as a by‐registry calculation to protect patient confidentiality.

Categories with < 10 cases were not reported to protect patient confidentiality.

Meaningful changes occurred in 5.7% of reviewed prostate cancer cases. Significant changes related to implied decimal point errors were found in 1.93% of the cases, whereas abstraction and implementation of coding rule errors occurred in only 1.54% of cases. Changes from known to unknown PSA values occurred in 0.68% of cases and from unknown to known values in 1.65% of cases.

Overall, the results varied by registry, with the percentage of cases with no changes ranging from 76.14% to 92.49% and implied decimal errors ranging from 0.11% to 3.09% of cases.

Tables 2 and 3 show the impact on disease staging for the cases in which the PSA value was incorrect due to an implied decimal error (Table 2) or any of the other errors (Table 3), resulting in PSA changes. When focusing on the implied decimal errors (Table 2), 258 of the 972 corrected values (26.54%) resulted in a change of disease stage for that case, with 7.82% (76 cases) shifted from a lower to a higher stage group whereas 18.72% (182 cases) moved from a higher to a lower stage group. For all the other records resulting in PSA changes combined (Table 3), 161 of the 6140 corrected values (2.62%) resulted in a change of disease stage. Corrections of these errors were more likely to move a case to a lower stage group. Overall, only 419 of the original 50,277 cases (0.83%) in the current study had a change in disease stage due to a corrected PSA value. Table 4 shows the impact on staging for all errors combined.

Table 2

Impact on Disease Stage Due To Implied Decimal Error (N=972; 1.93% of the Total Number of Prostate Cancer Cases)

	Corrected
	Stage I		Stage IIA		Stage IIB		Stage III		Stage IV		Unknown Stage
Original	No.	Percent	No.	Percent	No.	Percent	No.	Percent	No.	Percent	No.	Percent
Stage I	42	51.22	26	31.71	14	17.07	0	0	0	0	0	0
Stage IIA	3	2.70	72	64.87	36	32.43	0	0	0	0	0	0
Stage IIB	98	20.37	81	16.84	302	62.79	0	0	0	0	0	0
Stage III	0	0	0	0	0	0	72	100	0	0	0	0
Stage IV	0	0	0	0	0	0	0	0	168	100	0	0
Unknown stage	0	0	0	0	0	0	0	0	0	0	58	100

Table 3

Impact on Disease Stage Due to All Other Errors Except Implied Decimal Error (N=6140; 12.21% of the Total Number of Prostate Cancer Cases)

	Corrected
	Stage I		Stage IIA		Stage IIB		Stage III		Stage IV		Unknown Stage
Original	No.	Percent	No.	Percent	No.	Percent	No.	Percent	No.	Percent	No.	Percent
Stage I	1511	95.75	52	3.30	15	0.95	0	0	0	0	0	0
Stage IIA	27	2.00	1293	95.92	28	2.08	0	0	0	0	0	0
Stage IIB	13	0.75	26	1.50	1691	97.75	0	0	0	0	0	0
Stage III	0	0	0	0	0	0	457	100	0	0	0	0
Stage IV	0	0	0	0	0	0	0	0	391	100	0	0
Unknown stage	0	0	0	0	0		0	0	0	0	636	100

Table 4

Impact on Disease Stage Due to All Errors Combined (N=7112; 14.15% of the Total Number of Prostate Cancer Cases)

	Stage I		Stage IIA		Stage IIB		Stage III		Stage IV		Unknown Stage
Original	No.	Percent	No.	Percent	No.	Percent	No.	Percent	No.	Percent	No.	Percent
Stage I	1553	93.55	78	4.70	29	1.75	0	0	0	0	0	0
Stage IIA	30	2.06	1365	93.56	64	4.38	0	0	0	0	0	0
Stage IIB	111	5.02	107	4.84	1993	90.14	0	0	0	0	0	0
Stage III	0	0	0	0	0	0	529	100	0	0	0	0
Stage IV	0	0	0	0	0	0	0	0	559	100	0	0
Unknown stage	0	0	0	0	0		0	0	0	0	694	100

DISCUSSION

The most significant category of PSA changes was the correction of errors related to the implied decimal point, a type of error that generates a 10‐fold difference from the real value. The implied decimal error rate was only 1.93% of all cases in the current validation study. This was substantially lower than the 18% decimal error rate identified in the original quality assessment study. Reasons for the lower error rate are likely related to processes that occur routinely in the central SEER registry compared with a single registrar assigning the code in a study. Specifically, the central registry routinely performs ongoing and rigorous quality control and visual editing processes. This occurs for single abstracts and also with the consolidation of multiple abstracts and data sources at the central SEER registry. Multiple reviews of individual sources lead to the selection of the best value, in which multiple values are available.

Finally, the SEER registries ensure that the most difficult cases are reviewed by experienced quality control managers and that multiple quality assurance measures are in place to avoid errors. The previous data quality study included registrars from across all types of facilities, including hospitals, central registries, and contracting staff, which may have resulted in a broader variation in the error rate based on differing skill levels of abstraction. These implied decimal errors would be eliminated by modifying the input field in the abstract form by removing the implied decimal. Although this change in formatting needs to be implemented in the North American Association of Central Cancer Registries standards to incorporate into data entry fields, within the SEER program software, the abstractor now will be required to confirm the value that was entered.

In addition to the errors due to the implied decimal point, there were incorrectly reported PSA values related to the incorrect interpretation of coding rules. In this assessment, there was a 5.81% error rate, resulting in meaningful differences in what originally was coded versus the correct PSA value. The collection of this data item began with cases diagnosed in 2004, which preceded the use of PSA as a prognostic factor in the seventh edition of the AJCC TNM cancer staging classification. The implementation of PSA screening combined with the use of PSA in staging (2010), active surveillance, and monitoring of recurrent disease increased the number of PSA assays performed per patient with cancer dramatically, in particular with regard to the number of measurements at the time of the cancer diagnosis. Source abstracts transmitted to the central cancer registry from various data streams frequently include more than a single PSA test result in the text documentation. Registrars have been instructed to select the value closest to and, when available, before biopsy. This process requires visual inspection of all available abstracts at the central registry; the selection of the recommended PSA test out of multiple tests; and the process of applying coding rules, which includes measurement unit transformation, rounding, and the removal of the implied decimal point.

One important consideration related to errors in the recorded PSA value was how they impacted staging of the prostate cancer. As described above, the AJCC rules for staging incorporated PSA in 2010. Before that time, AJCC stage was calculated independent of the PSA value. Because of this potential impact on staging, we examined the effect that these errors would have on disease stage at diagnosis, and found that the impact was minimal, with very few changes in stage of disease found to result from errors in the recorded PSA value. We assessed whether stage of disease was downgraded or upgraded according to the error type. For the errors related to the implied decimal point, corrections resulted in 76 cases (or 0.15% of all prostate cases in 2012) moving to a more advanced disease stage category and 182 cases (0.36%) moving to a less advanced disease stage category. PSA errors impact only stages IA, IIA, and IIB, stages that share similar survival outcomes and treatment recommendations. Other types of errors were found to be even less likely to change the stage of disease. A total of 161 cases (0.32% of all cases reviewed) had their staging changed, with 66 cases having their stage of disease upgraded and 95 having it downgraded.

Given the wide use of PSA as a component of risk group stratification schemas,17, 18 we further analyzed the impact of PSA value corrections on the recurrence risk categorization of patients with prostate cancer. Comparison of PSA values before and after corrections identified 418 patients (0.83% of the study cohort) who potentially could have been placed in a lower risk category before PSA corrections. Similarly, we identified 487 patients (0.97% of the study cohort) who potentially could have been placed in a higher risk category before PSA value corrections. These findings indicate that the impact of PSA corrections on the distribution of patients by risk group categories in either direction was <1%. Accordingly, the magnitude of this potential nondifferential risk group stratification error is unlikely to have resulted in type I errors for studies that used the original PSA value.

The PSA value coding schema allows for the accurate recording and analytic categorization of reasons for an unknown PSA laboratory value at the time of diagnosis. Although the current coding schema is valuable within the context of quality improvement, the current study demonstrated a low level of interclass validation for the code values of 997, 998, and 999. A comparison of PSA values before and after review resulted in an estimated 5.09% error rate involving unknown PSA code values. Meanwhile, the estimate of cases with an unknown PSA value decreased after PSA review, from 2221 (4.42%) to 1727 (3.43%). Investigators often exclude observations with unknown PSA values from prostate cancer analyses.19, 20, 21, 22, 23 Therefore, the difference between observations that would have been excluded before PSA validation review but retained after the review is within the range of 1%, which is significantly less than the 5.09% of cases reported in Table 1 to require corrections in PSA unknown codes.

SEER data from 2012 demonstrated that the meaningful error rate for PSA coding at the time of diagnosis was 5.81% overall. Although this was substantially lower than what was observed in the original quality study, critical errors were identified. SEER is in the process of reviewing all PSA values dating back to 2004. Data from 2010 through 2013 have been reviewed and corrected and were re‐released in April 2016 in the public use file that SEER maintains for research support. To the best of our knowledge, SEER is the only cancer surveillance program that actively reviews and, when necessary, corrects the PSA value in its publicly available data sets. Compared with cancer data sets based solely on hospital registry data, SEER has access to multiple source abstracts and text reports per case, which allows for the development and implementation of new quality control data checks. In addition to correcting the errors for all years of PSA collection, the SEER program, as part of its quality improvement activities, also will further increase the value of PSA at the time of diagnosis by providing the actual PSA value for test results >98.0 ng/mL.

As mentioned above, SEER currently is implementing software modifications to reduce the risk of implied decimal errors. The SEER registries will target education to their registrars to reduce coding interpretation and application errors, which represented a higher percentage of the errors identified. Attention will be directed toward the ongoing review of PSA at the central SEER registry level to ensure that the PSA value is coded correctly in the future. The additional quality assurance steps for PSA laboratory values will include automated quality control checks comparing the consistency of coded values with text documentation provided in the source abstracts, a step previously limited by the labor‐intensive resources required by manual review. When necessary, inconsistencies flagged by automated text review will be adjudicated by registrar review, a quality initiative to ensure that the SEER database will continue to offer the largest, most accurate data set of population‐based PSA laboratory values for the development of research and health policies.

FUNDING SUPPORT

Supported by Surveillance Research Program of the Division of Cancer Control and Population Sciences at the National Cancer Institute/National Institutes of Health contracts with the Surveillance, Epidemiology, and End Results registries. The authors gratefully acknowledge the contributions of the state and regional cancer registry staffs for their work in collecting the data used for this study.

CONFLICT OF INTEREST DISCLOSURES

The authors made no disclosures.

AUTHOR CONTRIBUTIONS

Each author contributed equally to this article.