Validity of an Automated Algorithm to Identify Cirrhosis Using Electronic Health Records in Patients with Primary Biliary Cholangitis.

BACKGROUND: Biopsy remains the gold standard for determining fibrosis stage in patients with primary biliary cholangitis (PBC), but it is unavailable for most patients. We used data from the 11 US health systems in the FibrOtic Liver Disease Consortium to explore a combination of biochemical markers and electronic health record (EHR)-based diagnosis/procedure codes (DPCs) to identify the presence of cirrhosis in PBC patients.
METHODS: Histological fibrosis staging data were obtained from liver biopsies. Variables considered for the model included demographics (age, gender, race, ethnicity), total bilirubin, alkaline phosphatase, albumin, aspartate aminotransferase (AST) to platelet ratio index (APRI), Fibrosis 4 (FIB4) index, AST to alanine aminotransferase (ALT) ratio, and >100 DPCs associated with cirrhosis/decompensated cirrhosis, categorized into ten clusters. Using least absolute shrinkage and selection operator regression (LASSO), we derived and validated cutoffs for identifying cirrhosis.
RESULTS: Among 4328 PBC patients, 1350 (32%) had biopsy data; 121 (9%) were staged F4 (cirrhosis). DPC clusters (including codes related to cirrhosis and hepatocellular carcinoma diagnoses/procedures), Hispanic ethnicity, ALP, AST/ALT ratio, and total bilirubin were retained in the final model (AUROC=0.86 and 0.83 on learning and testing data, respectively); this model with two cutoffs divided patients into three categories (no cirrhosis, indeterminate, and cirrhosis) with specificities of 81.8% (for no cirrhosis) and 80.3% (for cirrhosis). A model excluding DPCs retained ALP, AST/ALT ratio, total bilirubin, Hispanic ethnicity, and gender (AUROC=0.81 and 0.78 on learning and testing data, respectively).
CONCLUSION: An algorithm using laboratory results and DPCs can categorize a majority of PBC patients as cirrhotic or noncirrhotic with high accuracy (with a small remaining group of patients' cirrhosis status indeterminate). In the absence of biopsy data, this EHR-based model can be used to identify cirrhosis in cohorts of PBC patients for research and/or clinical follow-up.

Introduction

Although biopsy remains the gold standard for determining liver damage, fibrosis, and cirrhosis in patients with primary biliary cholangitis (PBC), it is invasive and performed on a relatively small subset of patients. Transient elastography has shown promise for use in PBC patients1,2 but has not been universally implemented in health care systems that are not supported by specialty gastroenterology and hepatology clinics. An efficient system to identify cirrhosis in PBC patients using data from electronic health records (EHR)—such as diagnosis and procedure codes (DPCs), and laboratory results—may inform epidemiologic research and clinical trials, as well as the identification of subgroups of PBC patients who could benefit from earlier intervention.

Biomarkers for liver fibrosis calculated from results of laboratory tests, such as the Aspartate Aminotransferase to Platelet Ratio Index (APRI) and Fibrosis-4 (FIB4), have been well described and validated among patients with viral hepatitis. However, the distinct etiology and natural history of PBC mean that the ability of these biomarkers to identify cirrhosis cannot be assumed, and there are currently no studies developing or validating PBC-specific cutoffs for cirrhosis. Likewise, elevated alkaline phosphatase (ALP), total bilirubin, and the ratio of aspartate aminotransferase to alanine aminotransferase (AST/ALT) are known to be important prognostic markers for PBC progression and response to treatment with ursodeoxycholic acid (UDCA).3,4 It is likely that the inclusion of these variables could increase the utility of any marker for cirrhosis among patients with PBC.

The FibrOtic Liver Disease Consortium (FOLD) comprises a cohort of more than 4000 PBC patients drawn from 11 US health systems. We applied machine learning techniques to develop and validate an automated algorithm combining EHR-based data—including DPCs and routine laboratory results—for the identification of cirrhosis among patients with PBC.

Methods

The FOLD Consortium has been previously described.3,5 Briefly, FOLD comprises 11 geographically diverse health systems, representing four US Census Bureau-defined regions of the US (Northeast, Midwest, Northwest, and South). FOLD follows the guidelines of the US Department of Health and Human Services for the protection of human subjects. The study protocol was approved by the Institutional Review Board of each participating site (see ). All authors had access to the study results and reviewed and approved the final manuscript.

Cirrhosis Cohort Identification

FOLD PBC patient identification methods have been previously described.5 All cases were confirmed with chart abstraction performed by trained medical abstractors. We used EHR data to identify FOLD PBC patients who had undergone liver biopsy. Fibrosis staging from biopsy results was collected by abstraction from pathology reports, and mapped to an F0–F4 equivalency scale:6 F0, no fibrosis; F1, portal fibrosis without septa; F2, portal fibrosis with few septa; F3, numerous septa without cirrhosis; and F4, cirrhosis. FOLD hepatologists provided adjudication of indeterminate cases. If the patient had more than one biopsy, the biopsy with the highest fibrosis stage was considered. The outcome of interest was a biopsy with F4/cirrhosis biopsy staging.

Possible Covariates/Classifiers

Table 1 details covariates considered for the model, including patient demographics (age, gender, race, Hispanic ethnicity); total bilirubin; ALP and albumin (classified in relation to “normal” as defined by the assay used at each site); APRI; FIB4 index; and AST/ALT ratio. We collected laboratory data, liver-related diagnosis and procedure codes (International Classification of Diseases Ninth and Tenth editions [ICD9/10] and Current Procedural Terminology [CPT] codes), all measured within six months before/after biopsy. In cases where more than one laboratory result was available, the one closest to the date of biopsy was used. ICD9/10 and CPT codes were grouped into ten clusters (C1 to C10, detailed in Table 2); these were used as dichotomized (presence/absence) variables in the classification analysis. An “unknown” category was used for all variables that had missing data.

Table 1

Laboratory Data

Test	Collection Criteria/Additional Information
Serum alkaline phosphatase (ALP)	Four categories (normal, 1-<2 times ULN, 2-<3 times ULN, or ≥3 times ULN), as defined by the assay used at each FOLD site
Ratio of aspartate aminotransferase (AST) to alanine aminotransferase (ALT)	Four categories (unknown; <1.1; 1.1-<2.2; ≥2.2) when tests were available within 7 days of one another
Albumin	Normal or abnormal, as defined by the assay used at each FOLD site
Total bilirubin	Seven categories (≤0.4, >0.4–0.5, >0.5–0.7, >0.7–1.0, >1.0–1.5, >1.5–2.0, >2.0 mg/dL)
AST to Platelet Ratio Index (APRI)	Numerical value, when tests were available within 7 days of one another
Fibrosis 4 Index (FIB4)	Numerical value, when tests were available within 7 days of one another
Platelet count	Numerical value

Table 2

ICD-9/10 and CPT Codes Comprising the Ten Cluster Variables (C1–C10)

Conditions	Associated ICD-9/10 and CPT Codes
C1: Liver transplant	V42.7, 996.82, 50.5, 50.51, 50.59, 47135, 47136, T86.40, T86.41, T86.42, 0FY00Z0,0FY00Z1, 0FY00Z2
C2: Liver cancer	155.0, 155.1, 155.2, C22.0, C22.2, C22.7, C22.8, C22.1, C22.9
C3: Hepatorenal syndrome	572.4, K76.7
C4: Hepatic encephalopathy	572.2, K72.09, K72.91
C5: Portal hypertension/portal decompression procedures	572.3, 37140, 37160, 37180, 37181, 37182, 37183, K76.6
C6: Esophageal varices complications (bleeding) and procedures	456.0, 456.20, 42.91, 44.91, 96.06, 43204, 43205, 43243, 43244, 43400, 43401, I85.01, I85.11, 06L30CZ, 06L30DZ, 06L30ZZ, 06L33CZ, 06L33DZ, 06L33ZZ, 06L34CZ, 06L34DZ, 06L34ZZ, 06L20ZZ, 06L23ZZ, 06L24ZZ, 0DL57DZ, 0DL58DZ
C7: Other gastrointestinal hemorrhage	530.7, 530.82, 578.0, 578.1, 578.9, K22.6, K22.8, K92.0, K92.1, K92.2
C8: Ascites/paracentesis procedures	789.5, 789.59, 54.91, 49080, 49081, 49082, 49083, R18.8, 0D9S30Z, 0D9S3ZZ, 0D9S40Z, 0D9S4ZZ, 0D9T30Z, 0D9T3ZZ, 0D9T40Z, 0D9T4ZZ, 0D9V30Z, 0D9V3ZZ, 0D9V40Z, 0D9V4ZZ, 0D9W30Z, 0D9W3ZZ, 0D9W40Z, 0D9W4ZZ, 0W9F30Z, 0W9F3ZZ, 0W9F40Z, 0W9F4ZZ, 0W9G30Z, 0W9G3ZZ, 0W9G40Z, 0W9G4ZZ, 0W9J30Z, 0W9J3ZZ
C9: Other sequelae of chronic liver disease	572.8, K72.10, K72.90
C10: Cirrhosis	571.2, 571.5, K70.30, K74.0, K74.60, K74.69

Statistical Analysis

Data were randomly divided into two sets at a 2:1 ratio; learning data (2/3) were used to build the classification model and testing data (1/3) were used for model validation. We performed analysis using machine learning approaches to build the model, including SPM (Salford Predictive Modeler, version 8.0) Least Absolute Shrinkage and Selection Operator (LASSO)7 and several machine-leaning R packages,8 including Classification and Regression Tree (CART), K-Nearest-Neighbor (KNN), polynomial support vector machines (SVMs), neural networks, random forest models, and eXtreme Gradient Boosting (xgb)Trees.8–10

The model building process started with variable selection for the initial multivariable model. Highly correlated variables (eg, AST/ALT ratio, APRI, and FIB4) were fit into the model one at a time with other covariates to determine which would be selected. The same modeling approach was repeated using laboratory data without DPCs, given that FIB4 (a commonly used laboratory data-based biomarker) has been used to identify cirrhosis among patients with chronic hepatitis.6 The final modeling approach and multivariable model were selected for optimal classification accuracy (defined by the highest area under the receiver operating characteristic curve [AUROC]). Final model selection was based on classification accuracy in the validation set, with estimation of model goodness-of-fit measured by AUROC. Models are considered to have “reasonable” and “good” accuracy when the AUROC is 70–80% and 80–90%, respectively. We also identified an optimal cut-off point to provide clinical utility to correctly classify patients as either cirrhotic or non-cirrhotic.

Results

Among 4328 confirmed PBC patients observed from 2006 to 2016, 1350 (32%) had biopsy data with F0–F4 staging; 121 (9%) were histologically staged F4 (cirrhosis). The median number of biopsies per patient was 1; 25th and 75th percentiles were 1 and 1 with a range of 1 to 7. Table 3 presents the two-group comparison for all covariates of interest.

Table 3

Two-Group Comparison for Covariates of Interest

Variables*	Response**	F0–3	F4	p-value
		(N= 1229)	(N= 121)
Age in years		56.4 ± 11.6	58.7 ± 10.2	0.041
Gender	Women	1066 (87%)	99 (82%)	0.133
	Men	163 (13%)	22 (18%)
Race	Asian/ Native American/ Pacific Islander	98 (8%)	6 (5%)	0.054
	Black/ African American	109 (9%)	15 (12%)
	White	804 (65%)	88 (73%)
	Unknown	218 (18%)	12 (10%)
Hispanic ethnicity	Yes	366 (30%)	10 (8%)	<0.001
	No	780 (63%)	99 (82%)
	Unknown	83 (7%)	12 (10%)
Alkaline phosphatase (x ULN)	Normal	267 (22%)	28 (23%)	0.385
	[1, 2)*ULN	482 (39%)	40 (33%)
	[2, 3)*ULN	206 (17%)	28 (23%)
	≥3*ULN	259 (21%)	23 (19%)
	Unknown	15 (1%)	2 (2%)
Alanine aminotransferase (x ULN)	Normal	561 (46%)	38 (31%)	0.013
	[1, 2)*ULN	370 (30%)	45 (37%)
	[2, 3)*ULN	149 (12%)	14 (12%)
	≥3*ULN	146 (12%)	24 (20%)
	Unknown	3 (0%)	0 (0%)
Aspartate aminotransferase (x ULN)	Normal	429 (35%)	22 (18%)	<0.001
	[1, 2)*ULN	401 (33%)	33 (27%)
	[2, 3)*ULN	162 (13%)	23 (19%)
	≥3*ULN	184 (15%)	42 (35%)
	Unknown	53 (4%)	1 (1%)
Platelet count (x LLN)	Normal	923 (75%)	68 (56%)	<0.001
	(0.5, 1.0)*LLN	99 (8%)	34 (28%)
	(0.33, 0.5]*LLN	11 (1%)	6 (5%)
	(0, 0.33]*LLN	143 (12%)	11 (9%)
	Unknown	53 (4%)	2 (2%)
AST/ALT Ratio	<1.1	839 (68%)	36 (30%)	<0.001
	1.1>2.2	287 (23%)	68 (56%)
	≥2.2	37 (3%)	16 (13%)
	Unknown	66 (5%)	1 (1%)
Total Bilirubin (IU/L)	(0, 0.4)	324 (26%)	10 (8%)	<0.001
	(0.4, 0.5)	139 (11%)	7 (6%)
	(0.5, 0.7)	204 (17%)	17 (14%)
	(0.7, 1.0)	208 (17%)	16 (13%)
	(1.0, 1.5)	118 (10%)	17 (14%)
	(1.5, 2.0)	39 (3%)	12 (10%)
	>2.0	99 (8%)	41 (34%)
	Unknown	98 (8%)	1 (1%)
Albumin (xLLN)	<LLN	268 (22%)	51 (42%)	<0.001
	Normal	720 (59%)	40 (33%)
	Unknown	241 (20%)	30 (25%)
C1	No	1199 (98%)	97 (80%)	<0.001
	Yes	30 (2%)	24 (20%)
C2	No	1211 (99%)	119 (98%)	0.87
	Yes	18 (1%)	2 (2%)
C3	No	1222 (99%)	116 (96%)	<0.001
	Yes	7 (1%)	5 (4%)
C4	No	1206 (98%)	108 (89%)	<0.001
	Yes	23 (2%)	13 (11%)
C5	No	1180 (96%)	100 (83%)	<0.001
	Yes	49 (4%)	21 (17%)
C6	No	1211 (99%)	112 (93%)	<0.001
	Yes	18 (1%)	9 (7%)
C7	No	1182 (96%)	109 (90%)	0.002
	Yes	47 (4%)	12 (10%)
C8	No	1162 (95%)	88 (73%)	<0.001
	Yes	67 (5%)	33 (27%)
C9	No	1193 (97%)	100 (83%)	<0.001
	Yes	36 (3%)	21 (17%)
C10 (≥2 records)	No	1066 (87%)	53 (44%)	<0.001
	Yes	163 (13%)	68 (56%)
APRI		2.0 ± 4.6	4.6 ± 12.6	0.033
Aspartate aminotransferase (mean ±SD)		82.7 ± 180.3	220.0 ± 423.8	<0.001
Alkaline phosphatase (mean ±SD)		278.6 ± 232.6	304.7 ± 359.0	0.438
Platelet count (mean ±SD)		211.6 ± 111.6	158.4 ± 80.0	<0.001
AST/ALT Ratio (mean ±SD)		1.1 ± 1.0	1.4 ± 0.7	<0.001
Fibrosis-4 Index (mean ± SD)		10.8 ± 33.9	12.4 ± 29.6	0.634

Notes: *all data were collected within six months of the biopsy assessment. **determined based on the highest fibrosis stage biopsy results if multiple assessments were involved.

The LASSO approach—using three laboratory variables (ALP, total bilirubin, AST/ALT), gender, and ethnicity—had “good” model classification accuracy; AUROC was 0.81 (learning data) and 0.78 (testing data). The model equation is expressed as: Lscore = −2.10444 - 0.0380115 [if ALP normal] - 0.10366 [if ALP 1-<2*ULN] + 0.0703424 [if ALP 2-<3*ULN] - 0.0859862 [if ALP≥3*ULN] - 0.0961179 [if male] + 0.0552183 [if non-Hispanic ethnicity] - 0.0557998 [if Hispanic ethnicity] - 0.0553804 [if bilirubin ≤0.4] - 0.0701528 [if bilirubin 0.5>0.4] - 0.0239056 [if bilirubin 0.7>0.5] - 0.0881663 [if bilirubin 1.0>0.7] + 0.0658567 [if bilirubin 1.5>1.0] + 0.115739 [if bilirubin 2.0>1.5] + 0.136269 [if bilirubin >2.0] - 0.0865529 [if AST/ALT<1.1] + 0.0898492 [if AST/ALT 1.1-<2.2] + 0.144404 [if AST/ALT ≥2.2]. At the optimal cutoff of 0.1 in this model, sensitivity was 70% and specificity was 72% based on validation results.

A LASSO model with three laboratory variables (ALP, total bilirubin, AST/ALT), two DPC clusters for diagnosis of hepatocellular carcinoma and cirrhosis, and ethnicity (Hispanic yes/no) demonstrated the best performance; this model reached “excellent” classification accuracy, with AUROCs of 0.86 on learning data and 0.83 on testing data. This model combining laboratory and DPC data had significantly better predictive ability (AUROC) compared to the model using laboratory data without DPCs (p=0.001). The equation for this final LASSO model is expressed as : Lscore = −2.80400 + 0.303777 [if ALP 2-<3*ULN] - 1.11856 [if Hispanic ethnicity] - 0.325175 [if bilirubin ≤0.4] + 0.28772 [if bilirubin >0.5-0.7 mg/dL] + 0.512881 [if bilirubin >1.0-1.5 mg/dL] + 0.9406 [if bilirubin >1.5-2.0 mg/dL] + 0.756801 [if bilirubin >2.0 mg/dL] - 0.652222 [if AST/ALT<1.1] + 0.645455 [if AST/ALT 1.1-<2.2] + 0.681193 [if AST/ALT ≥2.2] + 0.707349 [if diagnosis of hepatocellular carcinoma] + 1.3713 [if two diagnoses of cirrhosis]. A single cut-off of 0.08 (derived from the formula Prob = 1.0/(1.0 + exp(-score)) yielded sensitivity of 76% and specificity of 75% based on validation results. Two optimal cut-offs (0.07 and 0.10) divided patients into three groups—non-cirrhotic (≤0.07); indeterminate 0.7≤0.10); and cirrhotic (>0.10)—and yielded improved specificities of 81.8% for absence of cirrhosis and 80.3% for presence of cirrhosis.

Other modelling approaches using the same covariates reached similar or lower model classification accuracy (); performance of the xgbTree model was similar to the LASSO model (AUROC=0.82 on testing data) but required ten variables (age, gender, ethnicity, albumin, ALP, AST/ALT ratio, total bilirubin, platelet count, and DPCs related to hepatocellular carcinoma and cirrhosis), making this model less useful in the “real world” setting.

Discussion

Using data drawn from the FOLD consortium, we applied machine learning methods to EHR-based laboratory results and DPCs to develop and validate a method for identifying cirrhosis among patients with PBC. Our previous work has shown that cirrhosis is an important prognostic marker for poor outcomes among patients with PBC.3–5 However, in these analyses, cirrhosis identification was based on a limited number of patients with biopsy data (32%). Transient elastography has gradually begun to replace biopsy, but has not yet been universally implemented, especially in health systems without specialty hepatology clinics; only 12% of patients in our real-world cohort had elastography data available. Our EHR-based model could help address that gap in the identification of PBC patients with cirrhosis. The classification accuracy of our model using both laboratory data and DPC codes was “good” (AUROC=0.83 on testing data) and was significantly better than an alternate model using laboratory results without DPCs. The combined model with two-optimal cuts (0.07 and 0.10) divided patients into three groups (cirrhotic and non-cirrhotic, with a small group [<7%] as indeterminate); this model yielded 81.8% specificity for the absence of cirrhosis and 80.3% specificity for the presence of cirrhosis.

We believe this is the first validated model for use of EHR-based data for cirrhosis identification among PBC patients. Although previously developed markers for cirrhosis, such as APRI (which combines AST, ALT, and platelet count) and FIB4 (which combines age, AST, ALT, and platelet count), have been validated in populations with viral hepatitis, it is not clear if they are optimized for use in patients with PBC. In a model replacing AST/ALT ratio with FIB4, classification accuracy was moderate (AUROC=0.75 on testing data). Our analysis found that a combination of total bilirubin, ALP, and AST/ALT ratio—rather than APRI, FIB4, or the individual components of those markers—provided better accuracy (AUROC=0.83 on testing data). In light of our recent study showing that total bilirubin, ALP, and AST/ALT ratio were independent risk factors for all-cause mortality in patients with PBC,4 our current findings suggest that these variables may be the most appropriate biomarkers for cirrhosis and poor outcomes.

One limitation of our analysis is that—although the overall model classification accuracy reached 83%—sensitivity and specificity remained only moderate (75–76%) with the use of a single cut-off (0.08). We addressed this issue with the use of two cutoffs (0.07 and 0.10), which improved specificity to >80% for determining the absence of cirrhosis and presence of cirrhosis, and left only 6.8% of patients classified as “indeterminate.” While the use of more extreme cutoffs (eg, 0.05 and 0.16) could yield specificity in the range of 85–88%, it would classify more patients (28%) as indeterminate. Limitations to this method can be further addressed by using a hierarchical approach for cirrhosis identification: 1) cirrhosis determination based on biopsy or transient elastography when available; 2) use of our model with two cutoffs. Analyses can categorize those patients who fall into the “indeterminate” group as “unknown.” We have successfully implemented a similar approach for cirrhosis identification in patients with viral hepatitis.11,12 An additional unavoidable limitation of classification models that they are most accurate when applied to a sample with patient characteristics similar to those used to build the model. Likewise, this model will need to be validated using external data from a similar patient population.

In conclusion, our study showed that a model using EHR-based data can be used to efficiently identify PBC patients with cirrhosis. Using a hierarchical approach that also takes into consideration cirrhosis determination via biopsy/transient elastography data, when such data are available, we expect that this model will be useful for research in patients with PBC, and could serve as a quality improvement tool to ensure the best available care for such patients. Our model may also be useful in the identification of risk factors for decompensation in large observational studies of patients with PBC. There are interventions that mitigate risk of cirrhotic patients' progression from compensated to decompensated cirrhosis, and poor outcomes of decompensation—this tool may help clinicians identify and monitor such patients.