Granular Quality Reporting for Cervical Cytology Testing.

Quality reporting for cervical cancer prevention is focused on patients with normal cervical cytology, and excludes patients with cytological abnormalities that may be at higher risk. The major obstacles for granular reporting are the complexity of surveillance guidelines and free-text data. We performed automated chart review to compare the cytology testing rates for patients with 'atypical squamous cells of undetermined significance' (ASCUS) cytology, with the rates for patients with normal cytology. We modeled the surveillance guidelines, and extracted information from free-text cytology reports, to perform this study on 28101 female patients. Our results show that patients with ASCUS cytology had significantly higher adherence rates (94.9%) than those for patients with normal cytology (90.4%). Overall our study indicates that the quality of care varies significantly between the high and average risk patients. Our study demonstrates the use of health information technology for higher granularity of reporting for cervical cytology testing.

Introduction

Mortality resulting from cervical cancer is attributed to lack of adequate cervical cytology screening as well as lack of appropriate surveillance following an abnormal test result. However, the national quality measure for measuring the quality of care for cervical cancer prevention is limited to screening, and excludes measurement of surveillance. Given the dearth of studies on patients with abnormal findings that require surveillance, we have investigated the cervical cytology testing rates for surveillance cases by performing automated chart reviews.

Background

Preventive care for cervical cancer involves an evaluation of cervical cytology through a liquid-based specimen, commonly referred to as Papanicolaou (PAP) smear. Human Papilloma Virus (HPV) testing may be additionally performed to detect the presence of high-risk strains of HPV that cause of cervical cancer. Preventive care has two components– i) screening patients with normal cytology, and ii) when abnormal findings are found in screening, surveillance is required in the form of follow-up cervical testing at shorter intervals or colposcopy examination. The guidelines for screening and surveillance have been developed by several national organizations. These include the American College of Surgeons (ACS), U.S. Preventive Services Task Force (USPSTF), American College of Obstetricians and Gynecologists (ACOG) and the American Society for Colposcopy and Cervical Pathology (ASCCP). (1–5) The USPSTF guidelines provide recommendations for screening, and the other guidelines relate to surveillance. While the screening guidelines are easier to comprehend, the surveillance guidelines are complicated because they consider several combinations of abnormal findings and hence they are difficult to model. In addition, the abnormal findings are often reported in free-text notes that are not computationally amenable. Consequently, it is difficult to measure quality of surveillance as compared to quality of screening.

Moreover, the surveillance guidelines are challenging for clinicians to recall. Hence, patients with abnormal findings are prone to miss optimal care. Ironically these patients are at higher risk for developing cervical cancer, and provision of optimal care for such patients is critical for the success of the preventive program. It is estimated that 7–9% of women have an abnormal finding that requires surveillance (6). Despite the population size and elevated risk, there is a dearth of studies on patients with abnormal findings, since quality reporting and research on guideline adherence are generally focused on screening.(7–14)

To address this information gap, the current study aims to compare the adherence rates for cytological testing of patients with ‘atypical squamous cells of undetermined significance’ (ASCUS), which is the predominant cytological abnormality, with those of patients with normal cytology. The goal is to provide knowledge about the adherence at a higher level of granularity. Findings of this study will inform studies performing decision analysis for refinement of the guidelines. (15)

Methods

The study population included approximately 60,000 female patients above 21 years of age that received primary care at Mayo Clinic Rochester. The institutional review board at Mayo Clinic Rochester approved this study. We performed an automated chart review of patients receiving primary care at Mayo Clinic Rochester, in order to test the hypothesis that patients with cytological abnormalities have differential cervical cytology testing rates as compared to those with normal findings. Specifically, we compared the cytology testing rates for patients with ASCUS cytology, with the rates for patients with normal cytological findings. This study was carried out using the framework for decision support and natural language processing (NLP) that we have previously developed. (6, 16)

We determined the cytology testing status for the study population on 1st June 2013. We restricted the study population to patients that had at least one cervical cytology in last 8 years (between June 2005 to May 2013). The status of inadequate cytological testing was defined as the absence of cervical cytology three months beyond the optimal testing interval defined in the national guidelines (Table 1).

Table 1.

Guideline logic used to determine the appropriate screening intervals. This is based on the consensus guidelines from ACS/ASCCP/ASCP released in 2012. Cotest indicates that HPV test is always performed along with cytology, and reflex indicates that HPV will be performed only if cytological abnormality is detected.

Cytology	HPV	Screening interval from index PAP
ASCUS	Negative	PAP-HPV cotest at 5 years if last screening was cotest or PAP-HPV reflex at 3 years if last screening was reflex
Negative	Positive	PAP-HPV cotest at 1 year if previous HPV was negative
	Negative	PAP-HPV reflex at 1year, if high risk
		PAP-HPV reflex at 3 years, if age<30 years
		PAP-HPV cotest at 5 years if last screening was cotest or PAP-HPV reflex at 3 years if last screening was reflex

We developed an automated data extraction system, by reusing the framework developed earlier. (6) The system was designed to exclude patients that were ineligible for cytology testing due to hysterectomy and abnormalities warranting colposcopy. We applied the guideline logic as described in Table 1, to measure the adherence of care for two groups of patients; those with normal (negative) cytology and patients with ASCUS. Patients with ASCUS having positive HPV are recommended colposcopy and were excluded from this study. The statistical significance of the adherence rates was determined using test of two sample difference of proportion.

Information extraction: The following variables were extracted from the EHR— gender, age, history of hysterectomy, cervical cytology, HPV test result, risk of cervical cancer, history of Cervical intraepithelial neoplasia (CIN) 2/3. The gender and age variables were obtained from the patient registration system, to include female patients that were above 21 years of age. The history of hysterectomy was determined by searching for mentions of hysterectomy in the clinical notes, problem list, and annual survey information provided by the patient.

The cervical cytology reports are free-text but conform to the 2001 Bethesda system of nomenclature. They were classified using NLP rules into 4 categories: negative (no abnormality detected), unsatisfactory for evaluation, ASCUS and other abnormality greater than ASCUS. HPV testing was either reported in the cervical cytology reports or as a separate laboratory test, and are either positive or negative. Cervical cytology can be ordered in three combinations with HPV testing (17): 1) co-test indicating always perform HPV test along with cytology, 2) reflex indicating that HPV will be performed only if cytological abnormality of ASCUS is detected, or 3) only cytology. In patients less than 30 years old, HPV testing is not performed for screening as a co-test due to the high prevalence of self-limiting HPV infection.

The NLP rules comprised of if-then set of lexer and parser rules. (6) The rules were constructed by identifying patterns of recurring groups of words based on the pathology department’s templates for report generation. The identified word patterns were essentially mapped to concepts that are conveyed in the Pap report, as defined in the 2001 Bethesda system of nomenclature. (18). Before application of the NLP rulebase, the input cytology report was preprocessed, by removing all non-alphanumeric characters, followed by conversion to lower case. The lexer and parser rules used regular expressions to identify the relevant concepts. Hysterectomy was determined from clinical notes by spotting hysterectomy in the past history section of the clinical notes. This information was supplemented by coded information from the problem list and patient survey database.

High risk status is inferred if the following conditions are coded in the problem list: HIV, cervical dysplasia, carcinoma cervix, in utero (di-ethyl silbesterol) DES exposure, Hodgkin’s disease, multiple myeloma, lymphoma, leukemia, organ transplant and stem cell transplant. CIN2/3 is inferred from the problem list as well as comprehensive analysis of all previous cytology reports, by searching for ICD-9 codes for CIN2/3 in the problem list and for searching for CIN2/3 in the cytology reports, respectively.

Validation and Error analysis: We performed manual review for 100 randomly selected cases stratified across the two groups to verify the results of automated extraction. Specifically manual chart review was performed by a clinical expert for 100 cases to ensure that the correct screening/surveillance interval was computed, and that the computation was based on accurate information extracted for computing the interval. The information extraction system flagged cases for errors in data processing. These cases were manually analyzed for error analysis.

Results

Figure 1 and Table 2 summarize the results. A total of 28101 female patients above 21 years of age had a cervical cytology report in the last 8 years. 5.1% of these underwent hysterectomy and hence were not eligible for a cervical cytology. Of the remaining 26660 patients, 19.2 % were excluded due to the following reasons: 1) their cytological findings had high degree of abnormality requiring colposcopy, 2) their samples were unsatisfactory for evaluation, 3) the patients were reported to have had cervical cytology at another institution, 4) their last cytology report indicated other abnormalities like inadequate endocervical transformation zone or positive HPV with negative cytology, 5) they had exited cervical cancer screening as they were past age of 65 years, or 6) they had ASCUS cytology but HPV was not performed. 47 cases were error-flagged by the system.

Figure 1.

Overview of study design and results

Table 2.

Summary of results

	Count	past due by 3 months (% of total)
Normal findings	20951	2018 (9.63)
ASCUS-HPV negative	545	28 (5.14)
Excluded	5117	–
Manual curation	47	–
Total eligible	26660	–

The system identified a total 21496 patients that were indicated for cervical cytology testing. These comprised of 20951 and 545 patients with negative cytology and ASCUS cytology respectively. The adherence rates for the groups were 90.4% and 94.9% respectively. This difference in the rates was statistically significant (p-value <0.01).

Validation and Error analysis: Manual spot checking of 100 cases across the two cohorts, confirmed the validity of the system to 100% accuracy. We manually reviewed the 47 cases that could not be processed by the system. For 13 of the 47 cases, the pathological reports did not relate to cervical cytology, e.g. some were surgical biopsies, neck (cervical) lymph node biopsies. In 7 cases, the specimen for cytology testing was rejected or performed in external laboratory. In 14 cases, the cytology reports could not be processed by the NLP parser, since the reports that did not follow the modelled format. In 5 cases the HPV testing was performed separately from cytology. 8 patients were not eligible for cytology testing as they had completed the recommended routine screening beyond age 65 years, or had hysterectomy. This analysis indicates that the unprocessed cases are very small in number, and there is no systematic bias in the automated system.

Discussion

We performed automated chart review to compare the cytology testing rates for patients with ASCUS cytology with the rates for patients with normal cytological findings. The rates were 94.9% and 90.4%, respectively. Patients with ASCUS cytology were found to have significantly higher adherence rates than patients with normal cytology.

Our approach is in contrast to most other automated efforts on quality reporting for cervical cancer prevention (19, 20), that are generally focused on patients having normal cytology, and exclude patients with abnormal cytology, possibly because the surveillance guidelines are complex and require information embedded in free-text reports. Oddly, the surveillance provided to the patients with abnormal findings has greater benefits, since these patients are at a higher risk for cancer. Our approach enabled us to segregate patients for surveillance and screening, which was not done previously.

The Healthcare Effectiveness Data and Information Set (HEDIS) measure for cervical cancer screening measures “cervical cytology performed every three years in women 21–64 years of age and cervical cytology/HPV co-testing performed every five years in women 30–64 years of age”. (21) The screening measure refers to patients with normal cytology, and seems to assume that the entire population is average risk. However 7–9% of cervical cytology tests are known to have abnormal findings, which indicates higher risk for these patients and the national organizations for cervical cancer testing provide elaborate guidelines to define optimal care standards for surveillance of patients with cytological abnormalities. Hence screening and surveillance are both important components for providing quality care. There is a need to improve quality metrics to better account for diversity of patient population.

The majority of the previous research on measuring adherence has been conducted by manual chart review on small cohorts of patients, or automated review on large populations using discreet data. (19, 22, 23) While manual chart review represents the gold standard due to high accuracy, the latter is likely to be affected by coarse granularity of the coded data. Our attempt has been to increase the granularity of automated chart review by modeling the complex surveillance guidelines and by using NLP to extract information from cytology notes. NLP has been previously successfully applied for improving quality reporting in other domains.

A limitation of our study is that it was carried out at a single institution. The textual patterns for cytology reporting may be highly variable at other institutions, which poses a challenge for reliable data extraction.

Conclusion

Our approach of modeling complex surveillance guidelines in conjunction with NLP, allowed us to segregate patients with abnormal cervical cytology from those with normal cytology. Further we were able to accurately characterize the PAP adherence rates for patients with ASCUS cytology, which is the predominant cytological abnormality. Patients with ASCUS cytology were found to have significantly higher adherence to recommended Pap testing than average risk patients. Overall, this study indicates that there is significant variation in quality of care of high and average risk patients, and demonstrates the use of automated chart review to enhance the granularity of reporting for cervical cancer testing. Similar studies need to be carried out for comparing colposcopy rates for patients with cytological abnormalities and in other domains like colon and prostate cancer surveillance.