Socially situated risk: challenges and strategies for implementing algorithmic risk scoring for care management.

OBJECTIVE: To characterize challenges and strategies related to algorithmic risk scoring for care management eligibility determinations.
MATERIALS AND METHODS: Interviews with 19 administrators from 13 physician organizations representing over 2200 physician offices and 8800 physicians in Michigan. Post-implementation interviews were coded using thematic analysis.
RESULTS: Utility of algorithmic risk scores was limited due to outdated claims or incomplete information about patients' socially situated risks (eg, caregiver turnover, social isolation). Resulting challenges included lack of physician engagement and inefficient use of staff time reviewing eligibility determinations. To address these challenges, risk scores were supplemented with physician knowledge and clinical data. DISCUSSION AND
CONCLUSION: Current approaches to risk scoring based on claims data for payer-led programs struggle to gain physician acceptance and support because of data limitations. To respond to these limitations, physician input regarding socially situated risk and utilization of more timely data may improve eligibility determinations.

LAY SUMMARY

Many payers have developed care management programs to support patients beyond what is provided in traditional primary care. Some of these programs use algorithmic risk scores to determine which patients are eligible for care management. In order to understand the challenges and strategies related to algorithmic risk scoring in this context, we interviewed physician organizations responsible for implementing payer care management programs. We found that algorithmic eligibility determinations were limited due to outdated claims and incomplete data on patients’ social contexts. To address these challenges, risk scores were supplemented with physician knowledge and clinical data. Due to limitations of claims-based risk scoring, physician input regarding socially situated risk and utilization of more timely data may improve eligibility determinations.

BACKGROUND AND SIGNIFICANCE

The use of algorithmic risk scoring continues to expand in the US healthcare system with the goal of optimizing resource allocation by identifying high-risk patients and directing appropriate support to them. Among many applications, risk scores have accurately identified risk of hospitalization, ICU admission, and heart failure. They have also been used to improve care coordination through more effective utilization prediction. However, challenges persist. Patient risk changes over time, with some high-need patients moving in and out of high-risk categories within months., Bias in algorithmic risk scores also remains a major challenge and concern, given existing health disparities reflected in available data and the unequal social contexts in which the algorithms are applied.

Data for risk scoring are also often limited to claims data, which fail to comprehensively capture some patient needs., Although data collection has expanded in many ways to include some aspects of social determinants of health, prior work has identified challenges. In particular, socially situated risks complicate the utilization of risk scores. These include aspects of patient environments or circumstances that impact health but are not consistently or meaningfully captured in patient records, such as degree of social support and socioeconomic status.

Because care management is often designed to respond to socially situated patient needs, it is not clear the extent to which risk scoring alone is ideal for eligibility determinations. Prior work has identified discrepancies between algorithmic risk scores and provider risk assessment based on providers’ first-hand knowledge of patients’ social and behavioral needs. This raises questions about the implementation of claims-based algorithms for care management, especially when those algorithms do not incorporate provider input or allow for physician referral. Claims-based algorithms may miss critical aspects of risk that are directly relevant to care management. Thus, until these issues can be addressed, algorithmic risk scores may need to be supplemented by provider input.,

OBJECTIVE

To explore the ways physician organizations (POs) and their affiliated practices perceived the utility of claims-based risk-scoring algorithms to support care management enrollment, including identification of challenges and improvement strategies.

MATERIALS AND METHODS

Setting

Many payers have developed and implemented care management programs to provide support for patients beyond what is provided in traditional primary care., These programs include combinations of interventions and tools, including comprehensive health assessments, medication reconciliation, patient care plans, care transition management and palliative care, care coordination, home visits, and closing gaps in care. These program components respond to patients’ healthcare needs and socially situated risks that impact their health.

POs, as groups that represent individual physicians and practices, assist with program implementation and compliance. Our study was motivated by one such program: Blue Cross Blue Shield of Michigan implemented an intensive care management program with POs for high-need Medicare Advantage (MA) beneficiaries called the High-Intensity Care Model (HICM). HICM was adapted from the GRACE Team Care model from Indiana University, combining care management services with primary care provider coordination in a team-based approach. HICM specifically targets complex MA patients with multiple chronic conditions whose eligibility for the program is determined by a claims-based algorithmic risk score. The algorithm was centrally created and administered, with POs receiving eligible patient lists from the payer.

Sample

HICM was launched in 2014 with 6 participating POs across the state of Michigan. The program initially enrolled about 1500 patients. We recruited all 6 POs who implemented HICM and 7 other POs participating in a variety of other payer programs to capture a range of experiences with payer-led care management. Together, these 13 POs represented over 2200 physician offices and 8800 individual physicians in Michigan. We interviewed a total of 19 administrators across the 13 POs. Respondents included quality and clinical operations directors, executive directors, and population health managers. POs in the study represented between 53 and 191 provider practices. The number of physicians represented by a single PO ranged from 166 to 2001.

Data collection

We created a semi-structured interview guide focused on experiences with payer-led, high-need care management programs (including but not limited to HICM). This interview guide covered challenges and strategies to address those challenges, one of which was the claims-based risk scoring tools used for patient eligibility determination. We did not capture payer-specific approaches in order to compare their differences. Rather, we sought to understand perceived utility of HICM and similar care management models, the challenges experienced, and the corresponding strategies employed. Interviews were conducted via phone by both authors from December 14, 2017, to May 1, 2018. Each PO interview lasted approximately 1 h. This study was approved by the University of Michigan IRB (HUM00136965). All interviews were transcribed by the research team and coded using MaxQDA software.

Analysis

We used a thematic analysis approach to qualitative coding. In this approach, the initial codebook was structured by interview topics and the researchers inductively created sub-codes. This allowed us to identify broad themes across interview transcripts related to various program components, challenges, strategies, and perceived program impact. Sub-codes included patient eligibility determinations, the role of information technologies, and physician engagement. Under the theme of patient eligibility, risk-scoring algorithms emerged as a key challenge. Once iterative coding was completed, the researchers constructed and populated analytic matrices with coded content according to topic. This allowed us to capture respondent quotes relevant to the identified themes.

RESULTS

Approach to use of claims-based risk scores

Among the payer-led care management programs described by participants, a claims-based risk scoring algorithm was typically used to determine patient eligibility. Lists of patients designated high need by a payer algorithm were provided to POs and, subsequently, the care management staff or provider in the practice. Practices would then reach out to patients to enroll them in the program. POs generally supported practices with program management and implementation.

Theme 1: Poor utility from perceived “false negatives” and “false positives” from the algorithmic risk scores

Participants described low utility from risk-scoring algorithms due to misalignment with their expectations. The algorithms failed to identify patients who would be well-suited for supplemental care management and often included those who were not perceived to need it, based on PO and provider assessment of needs. The latter was perceived to be driven by outdated data as risk scores relied on lagged quarterly claims data. For example, a high-cost hospitalization for an acute problem that had since been managed and resolved could result in a risk score that qualified a patient for care management months later. Failure to identify those who would benefit most from additional care management was attributed to incomplete information about social context in claims data. Because risk scores could not fully account for the socially situated nature of certain risks, they were missing key information that would indicate lower risk or decreased need for intensive care management, such as being physically active or having strong family support. Similarly, risk factors that would increase risk, like loss of a caregiver or lack of access to transportation, were not included in the risk score. This type of information was possessed by physicians, such that it could inform decisions about who to enroll. As a result of the poor concordance between PO or provider expectation and risk scores, PO and practice staff time was required to review, edit, verify, and pass the list on to the staff responsible for patient enrollment. This multi-step process was identified as inefficient and cumbersome.

“The patients we felt were high-need, [weren’t] eligible for their program. So, based on whatever algorithm they were using or whatever approach they were using, our populations didn’t overlap and that was sort of frustrating to our physicians. They looked at the list of [approved] patients and said, ‘these people aren’t eligible for what I would say is high-need managed care, but these other people are’and then we couldn’t get them covered.”—PO5

“My complaint about the list is not about the predictive ability of any algorithm. My complaint is [that it] comes quarterly… so we’re always late and disjointed in talking to physicians about which patients we want to engage.”—PO12 (speaker 1)

“The algorithm is not a well-oiled machine. Physician knowledge just surpasses all of the algorithms that we’ve ever used… It’s because they know the patients so well and they can understand. It’s so hard for an algorithm to take into account patient engagement and support structures. So, without knowing those things, you could have a [patient labeled as] high risk because they're spending a lot of money but they're actually undercontrol”—PO3

“When you call a patient and they're on the golf course you see that they might not need [intensive care management]. Someone with one chronic condition and one medication, they might need services, but they don’t qualify even if they’re almost 100 years old.”—PO7

Theme 2: Physician acceptance and participation

For practices and POs, participation in care management programs required investment of time and financial resources. Practices changed their documentation processes, sometimes hired or reallocated staff, and built workflows for patient outreach. The success of care management programs hinged on physician buy-in, which was often tenuous because physicians already faced multiple demands on their time. POs described significant efforts to support and engage physicians because of their importance for the success of the program. The challenges to physician buy-in were compounded by algorithmic risk scores that contradicted their patient assessments, undermining confidence in the program. This was especially true when the highest need patients were ineligible, leaving physicians feeling limited in their ability to connect patients to important resources. This led to frustration and concerns about physicians’ willingness to participate.

“You don’t just turn care management on…There’s a lot of work to it and a lot of redesigning processes internally in the practice.” —PO9

“The physicians see any piece that’s added on…as a distinct new thing they have to do, and they feel like all they keep getting is new stuff. So, we tried to present it as… this care management is a new tool in your toolbox.”—PO6

“When the program first started, physicians weren’t able to say…‘I have Mrs. Smith over here who’s not on your list but could really use care management’.It’s hard to work with a practice when…the health plan isn’t making sense.”—PO12 (speaker 2)

Theme 3: Combining clinical and claims data offers a direct way to improve risk scoring

Some POs created supplementary eligibility criteria they could layer on top of claims-based scores. These included PO-designed or EHR-based risk scores. Because these scores utilized clinical data and information from admissions, discharges, and transfers (ADT) feeds, the results were considered more relevant than scores based on claims. Supplementary scores were used to improve patient eligibility determinations and minimize staff time spent revising claims-based patient lists.

“Often a big share of that [risk stratification algorithm] is based on claims experience, so the care managers were finding that when they called a lot of the patients, whatever episode created that high cost the previous year was no longer a problem…So the ADT gives current information and it’s been really helpful.”—PO4

Theme 4: Provider input on risk scoring to determine final enrollment targets as a workaround

Since many POs were unable to contribute their own risk scores, allowing for provider input on patient eligibility was identified as a workaround for ensuring appropriate enrollment in care management programs. This strategy also had the benefit of engaging physician in a way that supported their buy-in:

“Most of them have built in that first tier of risk stratification, which is an algorithm of some sort that can be applied across the whole population. Then there is a second part of risk stratification, which is the clinician’s perception. The patient may be risk stratified as low, but maybe the physician or care manager knows that the patient just lost their spouse or that their family support system is ill so they can override that number-type algorithm and say‘no, they're no longer low. They're a high-risk person’.”—PO6

DISCUSSION

While the limitations of claims data are well-known, they continue to be widely used for risk scoring. Although this may be sufficient for some aspects of care, our study reveals that it continues to impede the effective, efficient identification of patients for payer-led care management. We did not identify resistance to the concept of risk-scoring algorithms. On the contrary, most POs and practices used at least one form of algorithmic risk scoring for patient identification. Primary challenges were the dynamic and social nature of many relevant patient needs that are not captured in claims.

In order to improve patient eligibility determinations, 2 key issues require attention. The first is the social nature of many patient needs that are not effectively captured in claims, but necessarily inform the need for care management. Our findings on the need to combine physician input and claims-based risk stratification to improve patient eligibility determination build on previous analysis demonstrating providers’ increased confidence in risk stratification that incorporates human review.,, Because providers often incorporate psychosocial risks in their assessments of complexity, they provide key insight into patient needs that is not available in claims data or the patient record. Furthermore, missing data on patient needs introduces bias, despite growing efforts to incorporate social determinants of health in the EHR and risk assessments. While a comprehensive and coordinated effort to standardize collection of social determinant data has been suggested, this data will remain limited because socially situated risks evolve in ways that are evident to physicians but missing from the EHR. Supplementing algorithmic risk scores with physician input is feasible and can increase confidence in eligibility determinations. It can also improve the efficiency of the patient enrollment process by reducing staff time spent on eligibility revisions. If provider assessments are not explicitly incorporated in the risk scores themselves, direct physician referral into a care management program may also preserve physician buy-in and effectively account for socially situated risk.

The second key issue is related to data timeliness and relevance. In order to respond to the issue of timeliness, incorporation of current ADT data or other updated information would also be responsive to the patient eligibility challenges we identified. In conjunction with physician input, the use of clinical data in determining patient eligibility would likely streamline the process. This approach could minimize inefficiencies in use of staff time and bolster physician support for care management.

Limitations

Although this study incorporates the perspectives of a diverse group of POs representing a large number of physicians, our findings may not generalize to all POs or all approaches to care management eligibility. Some states do not have the PO infrastructure that is present in Michigan. Additionally, these data were collected prior to increased general public awareness of algorithmic bias. Future qualitative analysis of provider concerns about algorithmic bias would be an appropriate expansion of this work.

CONCLUSION

As predictive analytics and risk scoring continue to proliferate, optimized patient eligibility determinations are critical. We identify particular challenges in algorithmic risk scoring of eligibility for intensive care management programs including incomplete inclusion of socially situated risks and outdated patient information. These issues lead to inefficiencies in use of staff time and reduce physician buy-in. We also identify 2 potential strategies for improvement; physician input and supplementary risk scores based on current clinical data.