A Validation Study of Administrative Claims Data to Measure Ovarian Cancer Recurrence and Secondary Debulking Surgery.

OBJECTIVE: Administrative claims data offer an alternative to chart abstraction to assess ovarian cancer recurrence, treatment and outcomes. Such analyses have been hindered by lack of valid recurrence and treatment algorithms. In this study, we sought to develop claims-based algorithms to identify ovarian cancer recurrence and secondary debulking surgery, and to validate them against the gold-standard of chart abstraction.
METHODS: We conducted chart validation studies; 2 recurrence algorithms and 1 secondary surgery among 94 ovarian cancer patients treated at one hospital between 2003-2009. A new recurrence algorithm was based on treatment timing (≥6 months after primary treatment) and a previously validated algorithm was based on secondary malignancy codes. A secondary debulking surgery algorithm was based on surgical billing codes.
RESULTS: The new recurrence algorithm had: sensitivity=100% (95% confidence interval [CI]=87%-=100%), specificity=89% (95%CI=78%-95%), kappa=84% (SE=10%) while the secondary-malignancy-=code recurrence algorithm had: sensitivity=84% (95%CI=66%-94%), specificity=44% (95%CI=31%-=57%), kappa=23% (SE=8%). The secondary surgery algorithm had: sensitivity=77% (95%CI=50%-92%), = specificity= 92% (95%CI=83%-97%), kappa=66% (SE=10%).=.
CONCLUSIONS: A recurrence algorithm based on treatment timing accurately identified ovarian cancer =recurrence. If validated in other populations, such an algorithm can provide a tool to compare effectiveness of recurrent ovarian cancer treatments.

Introduction

First linked in 1992, the Surveillance, Epidemiology, and End Results (SEER)–Medicare registry offers a unique data source to evaluate the comparative effectiveness of cancer treatments.1–3 Despite its availability for more than 20 years, little has been published validating its use to assess treatments of recurrent ovarian cancer.4 Because 75 percent of ovarian cancers are diagnosed at advanced stages5,6 and 60–95 percent of women with advanced stage ovarian cancer experience a recurrence,5,7 identifying the most effective treatments for recurrent ovarian cancer is of critical importance. Typically, recurrent disease is treated with chemotherapy; in more recent years, the National Comprehensive Cancer Network also recommends secondary debulking surgery.8,9 Although definitive results from randomized trials regarding this surgery’s effectiveness are lacking,10 secondary debulking surgeries are increasingly being performed.

The following are challenges for using SEER–Medicare data to evaluate treatments for recurrent ovarian cancer: (1) the lack of a validated algorithm to identify cancer recurrence; and (2) the lack of codes for secondary debulking surgery. In this study, we sought to develop claims-based algorithms to identify ovarian cancer recurrence and secondary debulking surgery, and to internally validate them against the gold standard of chart abstraction. Since there was a previously validated algorithm to identify a solid cancer recurrence using secondary malignancy codes for breast cancer,11 we adapted this algorithm to detect ovarian cancer recurrence and validate it against chart abstraction data.

Materials and Methods

Study Population and Sample Selection

From one academic medical institution in New York City (NYC), new, primary cases of ovarian cancer were identified through the institution’s Data Warehouse (DW), which captures all inpatient and most outpatient hospital discharge and billing data. The DW consists of clinical, operational, and financial data derived from patient care processes at the institution. Detailed inpatient and outpatient data are extracted from transactional systems, transformed, and loaded into the DW at the end of each day. The DW contains data collected since 2003, sourced from 20 transactional systems, for more than 3 million patients. The principal objective of the DW is to make data easily accessible for planning and executing clinical and translational research, and for quality of care and process improvement projects.

All ovarian cancer patients—International Classification of Disease Revision 9 (ICD-9) code 183—diagnosed and treated for ovarian cancer between January 1, 2003 and December 31, 2009 were identified. We defined primary treatment as primary debulking surgery alone (more common among early stage patients), or followed by at least one cycle of chemotherapy (more common among advanced stage patients). The ICD-9 and Current Procedural Terminology (CPT) codes to identify primary debulking surgery are listed in Appendix 1. Because we were interested in identifying recurrence, we merged outpatient claims from the faculty practice billing database with the DW inpatient and outpatient clinical data to capture treatment among women receiving care within the academic medical system. We excluded 29 women who received continuous, primary chemotherapy postsurgery (e.g., ongoing billing codes for chemotherapy) typically used for persistent disease, leaving a cohort of 522 women with primary ovarian cancer from which we randomly selected our sample. Our final sample consisted of 94 cases. The study was approved by the Mount Sinai School of Medicine Institutional Review Board. We obtained a Health Insurance Portability and Accountability Act (HIPAA) waiver of informed consent to access patient medical records.

Algorithms to Identify Ovarian Cancer Recurrence and Secondary Debulking Surgery

The first of two strategies to identify ovarian cancer recurrence was based on timing and utilization of either secondary debulking or secondary chemotherapy. For the cancer to be considered “recurrent,” a 180-day treatment-free window after completion of “primary surgery and chemotherapy” was required, before the patient underwent secondary debulking or secondary chemotherapy. This 180-day treatment-free window was used to distinguish between recurrent and persistent disease. Note, because Stage IV disease is not considered curable, growth of cancer after primary treatment for stage IV disease is commonly described as “progression” rather than “recurrence.” Therefore, our term “recurrence” is meant to indicate recurrence for Stage I–III cases and disease progression for Stage IV cases.

Secondary debulking surgical procedures were identified with ICD-9 procedure and CPT codes outlined in Appendix 2. Because there is no single billing code to reflect secondary debulking surgeries, we asked gynecologic oncology billers from the Midwest, South, Southeast, Northeast, and Mid-Atlantic regions of the country for the codes they use to bill, and we used these codes as our definition of secondary debulking. The second surgery had to occur after a 180-day treatment-free window (i.e., no chemotherapy or surgery) following completion of primary treatment.

Secondary chemotherapy codes included diagnosis (ICD-9) and procedure—CPT/Medicare Healthcare Common Procedure Coding System (HCPCS)—codes for chemotherapy regimens commonly administered for recurrent ovarian cancer, also culled from gynecologic oncology billers (Appendix 3).

As a second strategy to identify recurrence, we adapted a list of secondary malignancy diagnosis codes previously validated in a breast cancer cohort11 to reflect ovarian cancer spread (Appendix 4).

Medical Chart Review (Gold Standard)

An experienced chart abstractor, blinded to the claims-based data, reviewed patients’ medical records and identified dates and types of treatments received for ovarian cancer. The chart abstractor was a physician assistant, trained to review pathology and radiology reports, lab results, physicians’ notes from surgical and follow-up visits, and chemotherapy orders. Ten percent of medical charts were randomly selected for review by a second investigator, for validation of the chart reviews, and there was agreement between reviewers for all cases.

Data Analysis

We calculated the sensitivity and specificity—with 95 percent confidence intervals (CI)—of the recurrence and secondary debulking algorithms compared to gold standard data abstracted from patients’ medical records. Kappa statistics were calculated for each algorithm. Algorithms were considered to be “accurate” if the accuracy—(number of true positives + number of true negatives) / total sample—was 90 percent or greater. All analyses were performed using STATA version 11.2.

Results

Description of Sample

The mean age of the 94 women included in our validation sample was 56 years (range 18–81 years). Seventy-six percent were non-Hispanic White, 6 percent African American, 4 percent Hispanic, and 4 percent Asian; 10 percent were other or unspecified. The majority of women were diagnosed with advanced cancer (77 percent Stage III or IV), 15 percent were diagnosed with Stage I, and 8 percent with Stage II disease (see Table 1).

Table 1.

Characteristics of Patients Included in Chart Abstraction

		N=94
AGE AT DIAGNOSIS
	mean ± SD [range]	56 ± 14 [18–81]
	<50 years	29.8%
	50–64 years	41.5%
	≥65 years	28.7%
RACE/ETHNICITY
	Non-Hispanic White	75.6%
	African American	6.4%
	Hispanic	4.2%
	Asian	4.2%
	Other/Unknown	9.6%
STAGE AT DIAGNOSIS
	Stage IA–IC	15.3%
	Stage IIA–IIC	8.2%
	Stage IIIA–IIIC	69.4%
	Stage IV	7.1%

Performance of Recurrence Algorithms

From medical record review, 34 percent of women in our sample (32/94) experienced a recurrence of their ovarian cancer (95 percent CI = 25–45 percent). The sensitivity of the new recurrence algorithm based on timing and utilization of secondary debulking or secondary chemotherapy was 100 percent (95 percent CI = 87–100 percent; see Table 2); 32/32 recurrences were correctly identified by the algorithm. The specificity was 89 percent (95 percent CI = 78–95 percent); 55/62 of nonrecurrent cases were correctly identified by the algorithm, while 7/62 of the nonrecurrent cases were incorrectly identified as recurrences. Of the 7 patients misclassified by our algorithm as “recurrent,” 5 had undergone secondary surgical procedures more than six months after completion of primary treatment, but charts referred to these procedures as “second look” surgeries (e.g., surgeries performed to check for residual tumor) rather than debulking surgeries. For the remaining 2 misclassified cases, the algorithm calculated a 180-day chemotherapy-free window between “first” and “second” chemotherapy administration, while the medical charts indicated that chemotherapy was in fact administered during that window. The agreement of the algorithm with medical records was 93 percent with a kappa of .84, and the accuracy was 93 percent.

Table 2.

Test characteristics: Hospital Billing and Secondary Malignancy Code Algorithms (N=94)

ALGORITHM	SENSITIVITY	(95% CI)	SPECIFICITY	(95% CI)	PERCENT AGREEMENT	KAPPA (SE)
RECURRENCE
Hospital billing codes	100%	(87%–100%)	89%	(78%–95%)	93%	84% (10%)
Secondary malignancy codes	84%	(66%–94%)	44%	(31%–57%)	58%	23% (8%)
SECONDARY DEBULKING
Hospital billing codes	77%	(50%–92%)	92%	(83%–97%)	89%	66% (10%)

In contrast, the sensitivity of the existing algorithm using secondary malignancy codes to identify recurrence was 84 percent (95 percent CI = 66–94 percent; see Table 2); 27/32 recurrences were correctly identified by the algorithm, while 5/32 recurrent cases were incorrectly identified as nonrecurrences. Specificity was 44 percent (95 percent CI = 31–57 percent); only 27/62 nonrecurrent cases were correctly identified by the algorithm, while 35/62 of the nonrecurrent cases were incorrectly identified as recurrences. The agreement with medical records was 58 percent with a kappa of .23 (see Table 2). The accuracy of this algorithm was 57 percent.

Performance of Secondary Debulking Algorithm

Eighteen percent of women in our sample (17/94) underwent secondary debulking surgery according to medical chart review (95 percent CI = 11–28 percent). The sensitivity of our billing codes to identify these procedures was 77 percent (95 percent CI = 50–92 percent; see Table 2); 13/17 secondary debulking surgeries were correctly identified by the algorithm, while 4/17 true secondary debulking cases were incorrectly identified by the algorithm as not having undergone secondary debulking surgery. Of the 4 true secondary debulking cases missed by our algorithm, 2 had no record of secondary debulking procedures in the DW. For the remaining 2 misclassified cases, the algorithm misclassified the procedures as part of the primary regimen as they occurred before the end of the six-month chemotherapy-free window. The specificity of the algorithm was 92 percent (95 percent CI = 83–97 percent); 71/77 cases who did not undergo secondary debulking procedures were correctly identified as such, while 6/77 were incorrectly identified as having undergone secondary debulking surgery when, as per chart review, they had not. Of the 6 patients misclassified by our algorithm as having undergone secondary debulking, all underwent “second look” surgeries more than six months after completion of primary treatment. The agreement of this algorithm with medical records was 89 percent with a kappa of .66 (see Table 2), and the accuracy was 90 percent.

Discussion

We found that our algorithms based on timing and utilization of select billing codes accurately identify cancer recurrence and secondary debulking procedures in a population of ovarian cancer patients from a single NYC medical institution. The new recurrence algorithm utilizing timing of procedures measured ovarian recurrence with greater accuracy than a previously validated algorithm using secondary malignancy codes.11 Pending randomized trial results comparing surgery to chemotherapy for recurrent ovarian cancer, SEER–Medicare data can be used to compare effectiveness of secondary debulking surgery using these algorithms.

In early work to validate the use of Medicare claims to identify cancer recurrence, Lamont used a secondary malignancy code algorithm among node positive breast cancer patients enrolled in a clinical trial. The algorithm performed well in that setting, with 100 percent sensitivity and 97 percent specificity.11 Unfortunately for our project, when adapted to reflect ovarian cancer spread, this secondary malignancy code algorithm did not perform well.

Other attempts to identify recurrent cancer using claims data include Earle’s algorithm to identify recurrence for patients with acute myelogenous leukemia using inpatient chemotherapy procedure codes with diagnosis codes following a four-month treatment gap indicating remission.2 This approach yielded high specificity (99 percent, 95 percent PPV, 96 percent NPV) and predictive values with moderate sensitivity (86 percent). Specificity and positive predictive value decreased with the addition of outpatient chemotherapy codes to the algorithm.

Recently, Hassett et al. attempted to create a new algorithm to detect recurrence for many different types of cancers,12 utilizing Cancer Care Outcomes Research and Surveillance (CanCORS)/Medicare and HMO Cancer Research Network (CRN) patient populations who underwent primary treatment. The sample included lung, breast, colorectal, and prostate cancer patients, and the purpose was to validate Lamont’s and Earle’s algorithms. Hassett identified recurrence using the presence of ICD-9 codes for secondary malignancy and chemotherapy claims related in time to the date of diagnosis. This algorithm performed well for some cancers but not others. No code reliably detected prostate cancer recurrence. Sensitivities varied from 75 to 85 percent and specificities of 70–88 percent. Hassett suggests, as did Earle, that certain cancers may be more amenable to utilizing algorithms to detect recurrence and such algorithms need to be disease specific.

Our approach differs from Lamont’s and Hassett’s in important ways. Our approach did not set time limits between diagnosis and completion of primary treatment. Rather, we required a treatment-free window following completion of primary therapy that typically signals disease remission. Our algorithm was based on billing codes culled from gynecologic oncology billers across the nation. The algorithm includes claims for chemotherapy and secondary surgery, treatments that extend previous algorithms’ reach. We limited the algorithm to ovarian cancer recurrence, an important distinction as different cancers have different risks of recurrence within different time frames and utilize different treatment modalities.

The challenge of using treatment utilization codes to define a recurrence is the potential to misclassify, as nonrecurrent, untreated patients who do experience a recurrence. Such misclassification is a significant concern when using claims data of older populations who are more likely than younger patients to forego aggressive cancer treatments. We believe this misclassification was low in our population for two reasons. One, our algorithm performed similarly across age groups (<65 years versus ≥65 years, data not presented) indicating that our secondary treatment-based codes have the potential to work well in older, population-based registries to identify women with recurrent ovarian cancer. Second, in a larger SEER–Medicare analysis that applied our recurrence algorithm to women with ovarian cancer, death rates in the “nonrecurrent” group (e.g., excluding those patients with persistent cancer) were significantly lower than death rates among women in the recurrent group,13 suggesting that the nonrecurrent group included women who were truly nonrecurrent, not simply women misclassified as recurrent because they went untreated.

Our study addressed potential limitations to defining recurrence by secondary treatment in key ways. We factored timing of treatment into the algorithm to distinguish persistent from recurrent disease. Patients had to have a significant (six-month) treatment-free window following completion of primary treatment to be classified as recurrent. Further, because we were interested in creating an algorithm that was accurate overall, we oversampled those diagnosed with advanced cancer and, thus, those at greatest risk of a recurrence.

Study Limitations

Several limitations of our study are worth noting. Our sample size was small. In addition, our study included claims for patients from only a single institution, limiting the generalizability of findings. We were not able to externally validate the sensitivity and specificity of our algorithms. As a result, our study can only be considered an internal validation. In addition, we did not have full access to claims for treatment received outside our institution. It is possible that some patients had a recurrence but were seen outside the institution for treatment. Note, we do know that at least 77 percent of cases received follow-up care within the institution, increasing the likelihood of capturing recurrences. A further limitation is that our recurrence algorithm would not have captured recurrences that were not treated with chemotherapy or surgery. However, we believe the number of such cases to be negligible. Finally, we limited our reviewers to a single experienced chart reviewer to maximize reliability.

To ensure abstractions were reliably capturing chart data, the principal investigator of the study reviewed a random sample of 10 percent of the charts, and these abstractions were compared with the primary abstractor. There was agreement for all cases.

Given the limitations outlined, these newly developed algorithms based on the timing and utilization of administrative billing codes must be evaluated in other populations, as our sample size was small and limited to a single institution in which patients received treatment for both primary and recurrent cancer. Future research should evaluate these algorithms with larger samples and in a variety of institutions and settings. Where possible, efforts should be made to ensure that all treatments for patients included in the samples are captured, regardless of where they occurred. If our algorithms can be validated in other populations with larger samples and comprehensive data on follow-up treatment, these administrative claims data offer a promising alternative to expensive chart abstraction to assess comparative effectiveness of different treatments for recurrent ovarian cancer.

Conclusions

Algorithms based on timing and utilization of select administrative billing codes for secondary treatment may be used to identify ovarian cancer recurrence and secondary debulking procedures. Applying these algorithms to existing sources of administrative data can enable comparisons of treatment effects on recurrent ovarian cancer morbidity and mortality that can inform treatment decision-making in the absence of clinical trial results.

ICD-9 PROCEDURE CODES	DESCRIPTION
40.3	Regional lymph node excision
40.5	Radical excision of other lymph nodes
54.3	Excision/destruction abdominal wall lesion
54.4	Omentectomy, excision, destruction peritoneal tissue
65.31	Laparoscopic Unilateral Oophorectomy
65.39	Other Unilateral Oophorectomy
65.41	Laparoscopic Unilateral Salpingo-Oophorectomy
65.49	Other Unilateral Salpingo-Oophorectomy
65.51	Other Removal Both Ovaries
65.52	Other Removal Remaining Ovary
65.53	Laparoscopic Removal Both Ovaries
65.54	Laparoscopic Removal Remaining Ovary
65.61	Other Removal Ovaries/Tubes
65.62	Other Removal Remaining Ovary/Tube
65.63	Laparoscopic Removal Ovaries/Tubes
65.64	Laparoscopic Removal Remaining Ovary/Tube
68.31	Laparoscopic Supra-cervical Hysterectomy
68.39	Other Subtotal Abdominal Hysterectomy
68.4 & 68.41 & 68.49	Total Abdominal Hysterectomy
68.51	Laparoscopic Ast Vaginal Hysterectomy
68.59	Other Vaginal Hysterectomy
68.6	Radical Abdominal Hysterectomy
68.7	Radical vaginal
68.9	Hysterectomy Nec/NOS
70.32	Excision/destruction cul de sac lesion

ICD-9 PROCEDURE CODES	DESCRIPTION
40.30	Regional lymph node excision
40.50	Radical excision of other lymph nodes
54.20	Biopsy of peritoneum
54.30	Excision or destruction of lesion or tissue of abdominal wall or umbilicus; debridement of abdominal wall; omphalectomy
54.40	Excision or destruction of peritoneal tissue

ICD-9 PROCEDURE CODE	DESCRIPTION
99.25	Infusion of therapeutic substance into intraperitoneal cavity

ICD–9-DX CODE	SECONDARY MALIGNANT NEOPLASM OF THE:
196.00	unspecified, lymph nodes
197.00	lung
197.10	mediastinum
197.20	pleura
197.30	other respiratory organs
197.40	small intestine, including duodenum
197.50	large intestine and rectum
197.60	retroperitoneum and peritoneum
197.70	liver
197.80	other digestive organs and spleen
198.00	kidney
198.10	other urinary organs
198.30	brain and spinal cord
198.40	other parts of the nervous system
198.50	bone and bone marrow
198.70	adrenal gland
198.80	other sites
198.82	genital organs