Traditional Risk Indices as Predictors of Future Utilization and Charges in the Context of Population Health for an Uninsured Cohort.

INTRODUCTION: The uninsured population presents unique challenges to the application of an integrated approach to population health. Our objective is to compare and test population risk indices for identifying a cohort of uninsured patients at high-risk for avoidable healthcare utilization and costs.
METHODS: Patients who had a least one visit at a safety-net clinic, had a primary address in Mecklenburg County, were aged 18-74, and had the most recent healthcare visit coded as 'uninsured' were identified in the baseline period. The five risk indices used were: the HHS Hierarchical Conditions Category (HCC), the Charlson Comorbidity Index (CCI), Total Cost Index, Total Inpatient Visits Index, and Total Emergency Department Visits Index. First, agreement across the five indices was analyzed. Then, the accuracy of the five risk indices was tested in predicting future utilization and costs for the subsequent 12-month follow-up period.
RESULTS: Kappa statistics and percent overlap values showed below average to poor agreement between indices when comparing scorers.The strongest predictors of being in the 90th percentile of total cost during the 12 months follow-up period were the Total Cost Index at baseline (C statistic=0.75) and the HCC (C-statistic=0.73). The CCI and Total Inpatient Visit Index's demonstrated the lowest accuracy for predicting an unnecessary ED visit (C-statistic=0.51, for both). DISCUSSION/
CONCLUSION: Prior cost and ED utilization were key in predicting their corresponding 12-month metrics. In contrast, the Total Inpatient Visit Index had the worst predictive performance for future hospitalization rates. Some indices were similarly predictive as compared to insured cohorts but others showed contrasting results.

Introduction

A fundamental transformation of healthcare delivery is underway as health systems seek to deliver high quality, patient-centered, cost-conscious care at the population level [12]. As this paradigm of care and related reimbursements shift from quantity towards quality and from the individual to populations, healthcare systems are grappling with how to most effectively deploy population health strategies [34]. These strategies frequently employ data analytics to stratify populations into the segments deemed most at risk for a given outcome. Identifying smaller, at risk populations should enable health systems to better target the triple aim (i.e.: care, health, and cost) [1] by matching scarce resources with patients deemed most in need [5].

Utilizing various billing and visit data has become a viable and common strategy used by many in the healthcare industry for population segmentation [6]. Patients are identified through methods that typically incorporate elements from historical diagnoses, utilization, and billing data to predict which patients will experience high utilization and charges over a future timeframe. These various methods differ based on the type of elements they incorporate and the weights given for particular characteristics such as diagnosis, type of visit, and age. The proliferation of electronic health records has facilitated the use of these risk indices because the required data can be easily gathered and analyzed. Some of these indices involve a combination of age and diagnosis codes that require calculation of a final score while others may consist just of a count of a particular type of visit [789].

Indices typically are developed to predict specific outcomes and then validated in defined populations. For example, the Charlson Comorbidity Index (CCI) and Hierarchical Condition Categories (HCC) were originally designed to predict mortality and future costs respectively and were validated in mostly insured populations [1011]. Sometimes existing indices like CCI and HCC are later deployed to predict outcomes or segment populations beyond the scope of the original purpose [12]. In the case of CCI and HCC, both have also been shown to identify patients responsive to care management interventions, and CCI has additionally been reported to predict hospital readmissions [13141516]. While there is growing evidence to support new applications of some indices, many have not been rigorously evaluated and most evaluation have only been conducted in majority insured cohorts, thus limiting the generalizability to a vulnerable, uninsured population. In a local context, the indices examined in this study have enjoyed widespread use within the sponsoring healthcare system; however their performance has neither been tested, nor compared in an uninsured population [17].

In general, the uninsured represent a particularly vulnerable population that is frequently the target of outreach interventions due to their disproportionate disease burden and recurrent utilization of acute care settings [18]. While some care management and system-driven interventions such as sponsored free clinics have been shown to be a cost-effective alternative to emergency and inpatient care, little is known about the best approaches to risk stratification in uninsured populations [19]. Uninsured and insured patients have distinctly different utilization patterns and ultimately health outcomes, so it remains unknown how these indices might perform within a largely uninsured population [20]. In this study we aimed to characterize the performance of five commonly used risk indices within an uninsured population to better inform and target future outreach efforts.

Methods

Overview

This is a retrospective cohort study comparing methods to identify risk of utilization and charges in an uninsured population of patients from within a large, vertically integrated healthcare delivery system. The first objective was to determine the level of agreement across five commonly utilized risk indices. The second objective was to test the accuracy of each index from data collected in a 12-month baseline period in predicting utilization and charges occurring in the subsequent 12-month prediction period. The third objective was to compare the results to those from a previously published insured cohort.

Study Setting

Data were obtained on patients who attended one of four safety-net clinics and associated acute care facilities operated by Carolinas Healthcare System (CHS) based within Mecklenburg County, North Carolina. Mecklenburg County is an urban area with a population of 990,977, an uninsured rate of 17.7 percent, and a median household income of $54,278. The city is diverse in terms of race and ethnicity with the populations of approximately 49 percent White, 31 percent African American, 13 percent Hispanic, and 5 percent Asian. About 10 percent of the population is over the age of 65. These figures are based from 2014 estimates [21].

Eligibility

Patients were eligible if, during the 12-month baseline period from September 30, 2013 to September 30, 2014, they had: (1) at least one primary care visit at one of the four safety-net clinics within CHS; (2) a primary address in Mecklenburg County; (3) the last visit coded as ‘uninsured’; (4) aged 18-74; and (5) not deceased. The safety-net clinics are operated by CHS and offer patients a sliding scale fee system making care financially accessible to uninsured patients.

Data Retrieval

Utilization and total charges data were collected from CHS facilities in the 12-month baseline period and 12-month follow-up prediction period. Billing and quantity of visits data was captured from the CHS billing system and all clinical diagnosis data was captured from the Cerner Millennium (Cerner Corporation, Kansas City KS) Electronic Medical Record (EMR).

Diagnosis codes were gathered from the two billing systems and the Cerner EMR as ICD-9 codes. All diagnoses were extracted for any encounter recorded within each timeframe, and these EMR diagnoses were combined with billing diagnoses.

Billing information was based on the total charges that were originally billed to the patient and not what the patient actually paid. Health system costs contain considerable variability in fixed and variable cost assumptions, making such an estimate both difficult to estimate and potentially limiting in system generalizability while charge data reflects what cost would be from a payer perspective.

Demographics and Patient Characteristics

Race, gender and age for each patient were listed in the Cerner EMR. Race and gender were self-reported upon a patient’s first visit within the health system. Race was broken out into African American, White, American Indian or Alaska Native, Other, and Unknown. The “Other” and “Unknown” categories were combined because of the very few patients with an unknown race listed. Ethnicity was broken out into Hispanic, Non-Hispanic, and Unknown. Highly prevalent chronic diseases were chosen to describe the comorbidities of this cohort and defined by diagnosis codes from the billing system and EMR. Six chronic diseases were chosen,as they were the top six most prevalent based on problem-lists throughout the entire EMR for all patients in the system.

All patients remained uninsured throughout the entire baseline period. Insurance status in the prediction period was not used in the analysis, but is reported to provide additional characteristics of the population and insight into the variability of insurance status over time. During the prediction period, we deemed patients insured if their last visit in the timeframe had a billing activity by a third party. If a patient had no visit in the prediction period but had an insurance payment during the prediction period for a service that took place in the baseline period, then he or she would be considered insured. Insurance status was assumed to be uninsured for patients with neither a visit nor an insurance billing activity in the prediction period.

The study was approved by the Carolinas HealthCare System Institutional Review Board.

Index Descriptions

The Charlson Comorbidity Index (CCI) is a measure using diagnosed chronic diseases and age. This index was first developed to predict mortality in patients, and it has recently been used for utilization and charge prediction. Each chronic disease is given a weight of 1, 2, 3, or 6 and incorporated into the calculation with an additional point given for every decade over 40 years of age [10].

The US Department of Health and Human Services (HHS) leverages the Hierarchical Condition Category (HHS-HCC) system to create a risk-adjusting index to compensate plans for differences in the health status of enrollees. Because of this type of risk adjustment, the model was designed to predict future charges for the patient. The model incorporates demographics and diagnosis codes to calculate the score [11].

Because HCC and CCI require historical data for their calculation, the beginning of the baseline collection period for this type of data was extended to January 1st 2012 in order to more accurately capture relevant diagnoses.

The Total Charges Index is the sum of all charges for a specific patient in the CHS billing records within the given timeframe. The billing records are sourced from the inpatient/outpatient billing system and a separate ambulatory services billing system. The Total Emergency Department Visits Index is determined by the number of unique arrival dates for encounters classified as “Emergency”. This classification is based on a combination of the patient class code and emergency acuity level code. Similarly, the Total Inpatient Visits Index is determined by the number of unique arrival/discharge dates for encounters classified as “Inpatient”. Visits classified as “Observation” were not included in the inpatient visit counts unless the visit was upgraded to an inpatient status during the index stay. The three latter indices are basic measures created by and used routinely by the sponsoring healthcare system to identify population segments for care management activities.

Statistical Analysis

To analyze these indices, we set out to compare both the index agreement at baseline and prospective agreement. For baseline agreement, we dichotomized the risk indices at the 90th percentile to allow comparisons of their ability to identify the same patients. We calculated the percent overlap of positive case (90th percentile) patients and K statistics to determine agreement for each of the pairwise comparisons. Percent overlap for each comparison was calculated with the numerator including patients deemed as a positive case by both indices and the denominator including patients deemed as a positive case by one or both of the indices.

To evaluate prospective performance, each index was analyzed for its ability to predict high utilization and charges in the prediction period between October 1, 2014 and October 1, 2015, one year directly following the patient baseline period. We chose four dichotomized outcomes for predicting utilization and charges during the 12-month prediction period: (1) the 90th percentile of charges; (2) any inpatient visit; (3) any Emergency Department (ED) visit; and (4) any unnecessary ED visit. These outcomes were chosen to facilitate the subsequent comparison of the results to those of Haas et al [13].

Unnecessary ED visits were defined as having an Emergency Severity Index [22] triage level of 4 or 5. We used ED triage levels to identify unnecessary or “non-emergent” ED utilization. Triage nurses assign patients one of five triage levels upon ED admission to project priority and resource need. Resources are defined as services such as radiographs, computed tomography scans, administration of intravenous medications, laboratory tests, and simple procedures. Triage Levels 1 and 2 are assigned to patients with life threatening conditions and Level 3 to patients with abnormal vital signs, while Levels 4 and 5 are reserved for patients expected to use one or zero resources, respectively [2324].

Predictive accuracy was determined using logistic regression with each of the five risk indices included separately as continuous measures. The area under the receiver operating curve and 95 percent confidence intervals were used to estimate the predictive accuracy of each measure. This method was used to allow for a predictable comparison between continuous indices and binary outcomes. Observed rates of the outcomes at the lower and upper deciles are also reported for comparison.

All analyses were conducted using SAS Enterprise Guide version 6.1. For purposes of interpretation, C-statistics only above 0.70 were considered acceptable with a 95 percent confidence interval [25].

To understand how the predictive performance of these indices may differ when used with an insured cohort, we compared our findings with Haas et al.’s similar study (Table 4) [13]. Both our study and Haas et al. use HCC and CCI as indices and include the outcomes of 90th percentile of total charges, having any inpatient visits, and having any unnecessary ED visits.

Table 4

Comparison of C-statistics (95% CI) between Uninsured and Insured Cohort (Haas et al) [11]

RISK INDEX	90TH PERCENTILE OF TOTAL CHARGES	ANY INPATIENT VISIT	ANY UNNECESSARY ED VISIT
HCC
Uninsured	0.73	0.71	0.53
Insured (Haas et al)	0.70	0.67	0.58
CCI
Uninsured	0.67	0.62	0.51
Insured (Haas et al)	0.70	0.68	0.59

Results

We identified 4,715 uninsured patients meeting the inclusion criteria. Over 60 percent were female, 64.5 percent African-American, 65 percent between 45-64 years of age, and 82 percent non-Hispanic (Table 1). The most prevalent condition was diabetes (31 percent of the cohort). During the prediction period, nearly 30 percent became insured at some point. In the baseline period, 48 percent of the cohort had at least one ED visit, 16 percent of the cohort had at least one inpatient visit, and the mean charges per person were $21,051.56 with a median of $7,666.64. The rates of utilization and charges slightly decreased during the prediction period, and approximately 12 percent of the cohort had no visits after the baseline year. A small portion (1.5 percent) of patients in the prediction period had expenditures despite having no CHS visit. This may have been due to a variety of reasons. Patients may have received a service late in the baseline period and then were subsequently billed early in the prediction period. Also, it is possible that a patient had charges from a non-visit encounter such as pharmacy or labs ordered on previously acquired specimens.

Table 1

Cohort Description: Demographics, Utilization, and Charges

CHARACTERISTICS	BASELINE PERIOD FOR UNINSURED COHORT 9/30/13 – 9/30/14 (n=4715)	PREDICTION PERIOD 10/1/14 – 10/1/15 (n=4715)
AGE, n (%)
18–44	1558 (33.0)	–
45–64	3065 (65.0)	–
65–75	92 (2.0)	–
Mean age	47.8	–
Female, n (%)	2877 (61.0)	–
Male, n (%)	1838 (39.0)	–
RACE
Black/African American	3042 (64.5)	–
White	597 (12.7)	–
American Indian or Alaska Native	73 (1.5)	–
Asian	49 (1.0)	–
Other/Unknown	954 (20.2)	–
ETHNICITY
Hispanic	771 (16.4)	–
Non-Hispanic	3868 (82.0)	–
Unknown	76 (1.6)	–
INSURANCE STATUS, n (%)
No Insurance	4715 (100)	2875 (61.0)
Unknown	0	438 (9.3)
Medicaid	0	539 (11.4)
Medicare	0	136 (2.9)
Private	0	273 (5.8)
Managed Care	0	342 (7.3)
Correctional Facility	0	34 (0.7)
Workers Compensation	0	2 (<0.1)
Other form of coverage*	0	76 (1.6)
MEDICAL CONDITIONS, n (%)
Diabetes	1474 (31.3)	–
CAD/MI/CHF	338 (7.2)	–
Stroke	61 (1.3)	–
COPD	420 (8.9)	–
Cancer	182 (3.9)	–
Dementia	21 (0.4)	–
Any ED visits, n (%)	2275 (48.3)	2018 (42.8)
Number of ED visits, mean (SD)	1.2 (2.4)	1.1 (2.3)
Median (IQR)	0 (0, 2)	0 (0, 1)
Any Inpatient visits, n (%)	764 (16.2)	542 (11.5)
Number of Inpatient visits, mean (SD)	0.2 (0.6)	0.2 (0.7)
Median (IQR)	0 (0, 0)	0 (0, 0)
Total charges, mean (SD) per person	$21,051.56($43,362.55)	$19,070.16($45,870.18)
Median	$7,666.64	$5,275.55
No expenditures, n (%)	0 (0)	491 (10.4)
No CHS visit, n (%)	0 (0)	559 (11.9)

*Includes small PPO plans, local plans, and Medicaid waivers

Baseline Agreement between Indices

Each of the pairwise comparisons showed below average to poor agreement with the largest kappa statistic being only 0.51 for prior total charges and prior inpatient visits (Table 2). The percent overlap of patients within indices was also very small, with no two indices having more than 40 percent overlap in 90th percentile scorers. The comparison between CCI and Total ED Visits Index had the worst agreement with a percent overlap of 8.9 percent and a kappa statistic of 0.05 (95 percent CI, 0.02–0.08)

Table 2

Tests for Baseline Agreement amongst Indices 90th Percentile Scorers (n=4715)

INDICES (90TH PERCENTILE)	PERCENT OVERLAP (%)	KAPPA (95% CI)
CCI (6)
HCC (15.53)	25.1	0.33 (0.29, 0.38)
Total ED Visits (3)	8.9	0.05 (0.02, 0.08)
Total Inpatient Visits (1)	13.7	0.13 (0.10, 0.17)
Total Charges ($50,369.37)	17.5	0.22 (0.18, 0.26)
HCC
Total ED Visits	11.2	0.09 (0.06, 0.13)
Total Inpatient Visits	24.2	0.30 (0.27, 0.34)
Total Charges	29.7	0.40 (0.35, 0.44)
TOTAL ED VISITS
Total Inpatient Visits	15.8	0.14 (0.11, 0.17)
Total Charges	17.4	0.20 (0.16, 0.24)
TOTAL INPATIENT VISITS
Total Charges	39.8	0.51 (0.47, 0.54)

Predicting Future Utilization and Charges

The five indices varied in their ability to predict outcomes (Table 3). The strongest index predictor of future high total charges was the Total Charges Index with a C-statistic of 0.75 (95 percent CI, 0.72-0.77). One-third of patients in the highest decile of the Total Charges Index at baseline were a positive case for the 90th percentile of total charges outcome compared to 1.5 percent of those from the lowest decile. Total Inpatient Visits and Total ED Visits Indices had the worst predictive ability for total charges both with C-statistics of 0.63 (95 percent CI, 0.60–0.65 and 0.61–0.66, respectively)

Table 3

Prediction Performance of Risk Indices During Follow-up Prediction Period (n=4715)

BASELINE RISK INDEX PERFORMANCE	90TH PERCENTILE OF TOTAL CHARGES IN PREDICTIVE PERIOD	ANY INPATIENT VISIT IN PREDICTIVE PERIOD	ANY ED VISIT IN PREDICTIVE PERIOD	ANY 4 OR 5 TRIAGE LEVEL ED VISIT IN PREDICTIVE PERIOD
CCI
Observed rate percentage (lowest decile, highest decile)	(5.1, 26.0)	(10.0, 25.0)	(40.6, 51.7)	(27.1, 26.5)
C statistic (95% CI)	0.67 (0.65,0.70)	0.62 (0.60, 0.65)	0.53 (0.51, 0.55)	0.51 (0.49, 0.53)
HCC
Observed rate percentage (lowest decile, highest decile)	(2.6, 32.6)	(4.3, 33.9)	(40.2, 52.8)	(25.8, 26.9)
C statistic (95% CI)	0.73 (0.71, 0.76)	0.71 (0.68, 0.73)	0.57 (0.55, 0.58)	0.53 (0.51, 0.55)
Total Charges
Observed rate percentage (lowest decile, highest decile)	(1.5, 32.4)	(2.8, 29.9)	(24.2, 55.1)	(12.3, 30.7)
C statistic (95% CI)	0.75 (0.72, 0.77)	0.69 (0.66, 0.71)	0.64 (0.62, 0.65)	0.60 (0.58, 0.62)
Total Inpatient Visits
Observed rate percentage (lowest decile, highest decile)	(7.4, 23.3)	(8.9, 24.7)	(41.5, 49.6)	(25.4, 28.0)
C statistic (95% CI)	0.63 (0.60, 0.65)	0.61 (0.59, 0.63)	0.52 (0.51, 0.53)	0.51 (0.50, 0.52)
Total ED Visits
Observed rate (lowest decile, highest decile)	(7.0, 23.2)	(8.1, 23.1)	(27.6, 80.2)	(13.7, 59.1)
C statistic (95% CI)	0.63 (0.61, 0.66)	0.62 (0.59, 0.64)	0.70 (0.69, 0.71)	0.71 (0.69, 0.73)

The strongest predictors of having any inpatient visit were the HCC Index (C-statistic=0.71, 95 percent CI 0.68–0.73) and The Total Charges Index (C-statistic=0.69, 95 percent CI 0.66–0.71). The CCI, Inpatient Visits, and ED Visits indices were equally poor predictors of this outcome.

The Total ED Visits Index outperformed all other indices when predicting the two outcomes of having any ED visit and having any unnecessary ED visit (C-statistics=0.70 and 0.71, 95 percent CI 0.69–0.71 and 0.69, 0.73). The Total Charges Index performed almost at a reasonable level with these two outcomes with C-statistics of 0.64 and 0.60 (95 percent CI, 0.62–0.65 and 0.58–0.62), respectively while the remaining indices all had C-statistics well below 0.60.

The worst overall predictive performances were the CCI and Total Inpatient Visits Index’s prediction of having any unnecessary ED visit, both having C-statistics of 0.51 (95 percent CI, 0.49–0.53 and 0.50–0.52, respectively). To further illustrate this, 27.1 percent of patients scoring in the lowest decile of the CCI Index had an unnecessary ED visit, and 26.5 percent of patients scoring in the highest decile of the CCI Index had an unnecessary ED visit. Thus, the upper and lower deciles of the CCI index were nearly indistinguishable in predicting this outcome.

When compared to Haas et al.’s results, the HCC Index performed slightly better in the uninsured cohort when predicting total charges with a C-statistic of 0.73 (95 percent CI, 0.71–0.76) in our study compared to a 0.70 C-statistic (95 percent CI, 0.70–0.71) in Haas et al.13 The HCC Index performance for inpatient visits was similar, with a 0.71 and 0.67 C-statistic (95 percent CI, 0.68–0.73 and 0.67–0.68) in uninsured and insured cohorts respectively. The HCC Index was an equally poor performer for both cohorts when predicting unnecessary ED visits with C-statistics below 0.60. Conversely, the CCI was consistently stronger in the insured cohort for predicting each of the three similar outcomes.

Discussion

Our study demonstrates that there is significant heterogeneity amongst the evaluated indices within an uninsured population. The pairwise comparisons show average to poor agreement and low percent overlap in 90th percentile of scorers across all indices. Even indices in which one might expect homogeneity simply based on an assumed relationship, like inpatient visits and total charges, demonstrated weak kappa values and percent overlap (40 percent). These results indicate that, when incorporating risk indices into population health outreach strategies, there is considerable variability in the population that might be targeted depending on the index that is employed. Because each identified high-risk group is very different, further research is needed to characterize groups and understand which are most likely to benefit from specific outreach interventions.

The indices also demonstrated considerable variability in their comparative predictive performance, with additional variability seen for the individual indices depending on whether charges or utilization was used as the outcome. A given index might be a very strong performer for one outcome and a very poor performer for another. Even within an uninsured cohort, prior history of the outcome of interest appears to serve as the best predictor For example, the best predictor of being in the 90th percentile of total charges was the Total Charges Index. The same was true for the two ED visit outcomes (regular or unnecessary) with the Total ED Visits Index being the best predictor for both. While one would expect indices that define risk using a historical variable to perform well using that same variable as the predicted outcome (for example, prior ED utilization predicts future ED utilization), such an assumption is not always true and must be tested. For example, this relationship did not exist for prediction of inpatient visits with the Total Inpatient Visits Index having a C-statistic of 0.61 compared to the strongest predictor the HCC Index with a C-statistic of 0.71 (95 percent CI, 0.59–0.63 and 0.68–0.73, respectively).

Our findings suggest that the most reliable indices to stratify an uninsured population are using prior charges to predict future charges and HCC to predict future charges and inpatient visits. Unfortunately with the exception of the Total ED Visit Index, the other evaluated indices proved to be ineffective predictors for ED utilization, leaving a very important question unanswered as the emergency department remains a highly utilized form of care for the uninsured population [26]. In the comparison of indices from Haas et al. and this study, there were some notable differences in measures and methodology [13]. Haas et al. used the Center for Medicare and Medicaid Services (CMS) version of HCC, while we used HHS. These two versions of HCC (HHS and CMS) differ based on the type of population used and timeframe of collection. Specifically, the HHS version typically uses current year data to predict current year charges while the CMS version uses prior year data for prediction. The newer HHS version was also adapted to account for a non-Medicare commercial population by focusing more on salient conditions such as pregnancy or neonatal complications, which is more appropriate for our cohort. Lastly the HHS version also differs by predicting both drug and medical spending whereas the CMS version only accounts for medical spending [27]. This study also differed from Haas et al.’s methodology in its definition of ED outcomes. Haas et al. defined an ED visit as unnecessary if the visit did not result in a hospitalization as opposed to using triage levels. In another study of an insured cohort, frequent outpatient visit utilization and various comorbidity classifications were found to be associated with frequent ED use [7]. Specifically the authors found that an association exists between having ten or more outpatient visits in one year and having four or more ED visits in one year (Odds Ratio 11.4 with 95 percent CI). The analogous association with the Total ED Visit Index predicting future ED utilization appears to be true in the uninsured population. These authors also found that patients with coronary artery disease and asthma are associated with having four or more ED visits (Odds Ratio 1.61 and 1.58, respectively with 95 percent CI). This conflicted with findings from our study where CCI and HCC (comorbidity-related indices) performed poorly in predicting ED outcomes. These areas of discordance between insured and uninsured cohorts suggest that additional factors like social determinants of health may need to be incorporated into predictive modeling within vulnerable populations.

Limitations

This study has several limitations. Because of the retrospective nature, data availability and quality are limited to what is available within the EMR and billing records. For example, historical records likely underestimate actual disease burden, which is a key component of many indices. Also, determining whether the patient was uninsured in the baseline period was limited to the coded status at the most recent visit in that period; a patient could be insured for part of the period but still qualify for the study as an uninsured patient if he or she lost coverage at the time of the most recent visit. Our data also only reflect contact within the sponsoring healthcare system’s facilities. While patients may have obtained care at other facilities for which we do not have data, we expect this impact to be small because this healthcare system provides the majority of care to the uninsured in Mecklenburg County To further control for this limitation and to make it more likely that care would be sought within the same system, we included the eligibility requirement that patients had to be seen at a system primary care clinic. This inclusion criterion selected a population for which we have more complete data, which does limit the generalizability of our findings. However from a practical standpoint, health care systems may be more likely to target populations who are already integrated into established care processes like safety-net clinics [19].

Perhaps as a reflection of the selection bias inherent to this process and the urban setting, the demographics in this study differ slightly from national averages for the uninsured. This study’s population was largely older (33 percent between ages 18 and 44 in the study cohort compared to 65 percent national average), African-American (65 percent in the study cohort versus 15 percent national average), and female (61 percent in the study cohort versus 45 percent national average) [2829].

We also observed a high rate of patients that had no follow-up data during the prediction period (11.9 percent). The potential outcomes for these patients span from not needing care, to seeking care elsewhere, to death. While the analysis could have included only patients for whom we had data in the predictive period, this approach would less accurately depict the real-world predictive value of the indices. Similarly we chose not to exclude those who obtained insurance during the predictive period (29.7 percent). This rate was much higher than expected, and it could have influenced outcomes. However, those employing risk indices to guide population health outreach have to use risk indices based on data that is currently available even in a vulnerable population, which might be expected to be both transient in insurance status and variable in utilization patterns.

As previously mentioned, the direct comparison to the CMS-HCC index used by Haas et al. is also limited by the small differences between the two versions, HHS and CMS. Also, the contrasting definitions of an unnecessary ED visit used in both studies limit comparisons. Lastly, our comparison between performance in insured and uninsured populations is limited by comparing results across two geographically and culturally distinct settings. A number of confounders in addition to insurance status may alternatively explain the variation in predictive values observed.

The comparison to an insured cohort would gain more strength if it were to be compared with a local insured cohort collected within the same timeframe. This study was limited in that it lacked this direct comparison, and future research may attempt to incorporate such a cohort to strengthen the analysis.

Finally the application of our results to practice is also limited by the need to have deeper understanding of the population segments and their underlying characteristics. For example, it is unknown whether the small overlapping groups within each identified cohort represent a small, reoccurring group of patients or unique patients. Additionally further characterization comparing and contrasting demographics, including social determinants of health are needed.

Conclusion

Our results have implications for healthcare delivery systems implementing programs designed to impact population health for uninsured community members. First, deploying a single risk index, while trying to impact different utilization and charges outcomes, may not identify the population most at risk for the individual outcomes. Second, the index used to identify high-risk patients matters considerably and the groups of patients that will be identified by each index are unique. Third, the predictive index performance varies based on the insurance status of the target population.

Until further research is performed, health systems looking to gain practical insights from this study should consider that for this uninsured population, HCC and prior charges proved to be reliable at predicting future inpatient visits and total charges, respectively. Further, the only successful predictor of ED utilization was prior ED utilization. Systems should also note that prior inpatient visits surprisingly proved to be unreliable in predicting future inpatient visits. When designing both population health interventions and evaluation metrics that are based on poorer performing indices, health systems should employ caution and consider the inherent limitations.

To fully capture the potential of predictive analytics for guiding population health, particularly in vulnerable populations, future research should:

Improve the understanding of characteristics that drive health seeking behaviors in high-risk populations. To effectively make use of such indices for an intervention, organizations must gain a deeper understanding of how and why these patients seek care.

Better define characteristics that show which members within the high-risk groups are most likely to be impacted by interventions. Such efforts will require further exploration into the underlying social determinants of health, qualitative studies, and prospective outcomes evaluations.

Explore new methods that incorporate social determinants of health data into predictive indices to better inform outreach to vulnerable populations.