Brian C Ricci1, Jonathan Sachs2, Konrad Dobbertin3, Faiza Khan4, David A Dorr5,6. 1. Department of Internal Medicine, Division of General Internal Medicine and Geriatrics, Oregon Health & Science University, School of Medicine, 3181 SW Sam Jackson Park Rd. Mail Code: L-475, Portland, OR, 97239, USA. riccib@ohsu.edu. 2. Department of Clinical Informatics and Clinical Epidemiology, Oregon Health & Science University, School of Medicine, Portland, USA. 3. Office of Advanced Analytics, Information Technology Group, Oregon Health & Science University, School of Medicine, Portland, USA. 4. Division of Business Intelligence and Advanced Analytics, Information Technology Group, Oregon Health & Science University, School of Medicine, Portland, USA. 5. Department of Internal Medicine, Division of General Internal Medicine and Geriatrics, Oregon Health & Science University, School of Medicine, 3181 SW Sam Jackson Park Rd. Mail Code: L-475, Portland, OR, 97239, USA. 6. Department of Medical Informatics and Clinical Epidemiology, Oregon Health & Science University, School of Medicine, Portland, USA.
Abstract
BACKGROUND: In primary care risk stratification, automated algorithms do not consider the same factors as providers. The process of adjudication, in which providers review and adjust algorithm-derived risk scores, may improve the prediction of adverse outcomes. OBJECTIVE: We assessed the patient factors that influenced provider adjudication behavior and evaluated the performance of an adjudicated risk model against a commercial algorithm. DESIGN: (1) Structured interviews with primary care providers (PCP) and multivariable regression analysis and (2) receiver operating characteristic curves (ROC) with sensitivity analyses. PARTICIPANTS: Primary care patients aged 18 years and older with an adjudicated risk score. APPROACH AND MAIN MEASURES: (1) Themes from structured interviews and discrete variables associated with provider adjudication behavior; (2) comparison of concordance statistics and sensitivities between risk models. KEY RESULTS: 47,940 patients were adjudicated by PCPs in 2018. Interviews revealed that, in adjudication, providers consider disease severity, presence of self-management skills, behavioral health, and whether a risk score is actionable. Provider up-scoring from the algorithmic risk score was significantly associated with patient male sex (OR 1.24, CI 1.15-1.34), age > 65 (OR 2.55, CI 2.24-2.91), Black race (1.26, CI 1.02-1.55), polypharmacy >10 medications (OR 4.87, CI 4.27-5.56), a positive depression screen (OR 1.57, CI 1.43-1.72), and hemoglobin A1c >9 (OR 1.89, CI 1.52-2.33). Overall, the adjudicated risk model performed better than the commercial algorithm for all outcomes: ED visits (c-statistic 0.689 vs. 0.684, p < 0.01), hospital admissions (c-statistic 0.663 vs. 0.649, p < 0.01), and death (c-statistic 0.753 vs. 0.721, p < 0.01). When limited to males or seniors, the adjudicated models displayed either improved or non-inferior performance compared to the commercial model. CONCLUSIONS: Provider adjudication of risk stratification improves model performance because providers have a personal understanding of their patients and are able to apply their training to clinical decision-making.
BACKGROUND: In primary care risk stratification, automated algorithms do not consider the same factors as providers. The process of adjudication, in which providers review and adjust algorithm-derived risk scores, may improve the prediction of adverse outcomes. OBJECTIVE: We assessed the patient factors that influenced provider adjudication behavior and evaluated the performance of an adjudicated risk model against a commercial algorithm. DESIGN: (1) Structured interviews with primary care providers (PCP) and multivariable regression analysis and (2) receiver operating characteristic curves (ROC) with sensitivity analyses. PARTICIPANTS: Primary care patients aged 18 years and older with an adjudicated risk score. APPROACH AND MAIN MEASURES: (1) Themes from structured interviews and discrete variables associated with provider adjudication behavior; (2) comparison of concordance statistics and sensitivities between risk models. KEY RESULTS: 47,940 patients were adjudicated by PCPs in 2018. Interviews revealed that, in adjudication, providers consider disease severity, presence of self-management skills, behavioral health, and whether a risk score is actionable. Provider up-scoring from the algorithmic risk score was significantly associated with patient male sex (OR 1.24, CI 1.15-1.34), age > 65 (OR 2.55, CI 2.24-2.91), Black race (1.26, CI 1.02-1.55), polypharmacy >10 medications (OR 4.87, CI 4.27-5.56), a positive depression screen (OR 1.57, CI 1.43-1.72), and hemoglobin A1c >9 (OR 1.89, CI 1.52-2.33). Overall, the adjudicated risk model performed better than the commercial algorithm for all outcomes: ED visits (c-statistic 0.689 vs. 0.684, p < 0.01), hospital admissions (c-statistic 0.663 vs. 0.649, p < 0.01), and death (c-statistic 0.753 vs. 0.721, p < 0.01). When limited to males or seniors, the adjudicated models displayed either improved or non-inferior performance compared to the commercial model. CONCLUSIONS: Provider adjudication of risk stratification improves model performance because providers have a personal understanding of their patients and are able to apply their training to clinical decision-making.
Keywords:
healthcare utilization; mortality; patient care management; population health; primary health care; racism; risk assessment; value-based care
Authors: Clemens S Hong; Steven J Atlas; Jeffrey M Ashburner; Yuchiao Chang; Wei He; Timothy G Ferris; Richard W Grant Journal: J Gen Intern Med Date: 2015-12 Impact factor: 5.128
Authors: Richard W Grant; Jeffrey M Ashburner; Clemens S Hong; Clemens C Hong; Yuchiao Chang; Michael J Barry; Steve J Atlas Journal: Ann Intern Med Date: 2011-12-20 Impact factor: 25.391
Authors: Mary E Charlson; Robert E Charlson; Janey C Peterson; Spyridon S Marinopoulos; William M Briggs; James P Hollenberg Journal: J Clin Epidemiol Date: 2008-07-10 Impact factor: 6.437
Authors: Lindsey R Haas; Paul Y Takahashi; Nilay D Shah; Robert J Stroebel; Matthew E Bernard; Dawn M Finnie; James M Naessens Journal: Am J Manag Care Date: 2013-09 Impact factor: 2.229
Authors: John Hsu; Mary Price; Christine Vogeli; Richard Brand; Michael E Chernew; Sreekanth K Chaguturu; Eric Weil; Timothy G Ferris Journal: Health Aff (Millwood) Date: 2017-05-01 Impact factor: 6.301
Authors: A Matthew Prina; Theodore D Cosco; Tom Dening; Aartjan Beekman; Carol Brayne; Martijn Huisman Journal: J Psychosom Res Date: 2014-11-08 Impact factor: 3.006