Does Preexisting Practice Modify How Video Games Recalibrate Physician Heuristics in Trauma Triage?

BACKGROUND: A majority of severely injured patients fail to receive care at trauma centers (undertriage), in part, because of physician judgment. We previously developed two educational video games that reduced physicians' undertriage compared with control in two clinical trials. In this secondary analysis, we investigated heterogeneity of treatment effect of the interventions by assessing physicians' preexisting practice patterns in claims data. We hypothesized that physicians with high preexisting undertriage would benefit most from game-based training.
METHODS: Using Medicare claims records from 2010 to 2015, we measured physicians' preexisting triage practices before their participation in one of two trials conducted in 2016 and 2017. We categorized physicians as having received game-based training versus control and noted their postintervention simulation triage performance in the trials. We used multivariable linear regression models to assess the heterogeneity of game-based training effect among physicians with high and low preexisting undertriage.
RESULTS: Of the 394 eligible physicians from our trials, we identified 275 (70%) with claims for Medicare fee-for-service beneficiaries suffering severe injury between 2010 and 2015. On average, the physicians were 44 y old (SD 8.4) with 12 y (SD 8.2) of experience. We found significant interaction between preexisting practice and intervention efficacy (P = 0.04). Physicians with high undertriage before enrollment improved significantly with game-based training compared with the control (46% versus 63%, P < 0.001). Those with low preexisting undertriage did not (58% versus 56%, P = 0.76).
CONCLUSIONS: Using claims-based data, we found heterogeneity of treatment effect of interventions designed to recalibrate physician heuristics. Physicians with high preexisting undertriage benefited most from game-based training.

Introduction

Severely injured patients experience reduced mortality when cared for at trauma centers.[1] The National Academy of Medicine advocates for the regionalization of trauma services to ensure that patients receive the right level of care and to reduce preventable deaths after injury.[2] Despite wide dissemination of triage guidelines, up to 70% of severely injured patients who presentinitially to nontrauma centers do not receive care at trauma centers (undertriage).[1,3] Patient factors such as age, injury mechanism, and insurance status as well as institutional factors such as profit-status and distance to trauma center contribute to undertriage.[4-6] Current quality improvement efforts targeting these variables have made little headway in reversing this phenomenon.[7,8] Our prior research suggests that physician judgment plays a key role in undertriage and may be a critical locus for intervention.[4,9]

Physicians often rely on heuristics—mental shortcuts or pattern recognition—to make clinical judgments, especially under conditions of time pressure and uncertainty.[10,11] When heuristics are well calibrated to practice guidelines, they can lead to efficient and “accurate” decision-making. When they are poorly calibrated, they can lead to predictable deviations from those guidelines.[11] We previously developed two video game interventions to recalibrate heuristics for triage decisions and found that game-based training improved physician decision-making compared with exposure to traditional text-based training or nothing at all.[12-14] However, little is known about whether individual differences exist in how physicians respond to behavioral or educational interventions. Better understanding of those differences would ensure the most effective dissemination of training programs.

We sought to assess if any heterogeneity of treatment effect of video game—based training existed among physicians with different preexisting triage practices. We estimated physician triage in clinical practice using Medicare claims data, and we hypothesized that physicians with high preexisting undertriage would show greater response to the interventions than those with low preexisting undertriage.

Methods

Overview of prior studies and present analysis

We have shown that physicians often rely on the representativeness heuristic (pattern recognition) when determining the disposition of injured patients.[9,15] Physicians using this heuristic are more likely to transfer representative trauma cases (gunshot wound to the torso) than nonrepresentative trauma cases (frail patient with a fall and at a high risk for complications) although both patients may be deemed severely injured and require transfer according to the American College of Surgeons’ Committee on Trauma guidelines.[3,9] We designed two video game interventions to recalibrate this heuristic for triage decisions. The first, Night Shift, used narrative engagement with captivating stories in an adventure video game. The second, Shift: The Next Generation, used analogical reasoning delivered via a puzzle video game.[13,14] Both games included cases of representative and nonrepresentative patients and incorporated mechanisms to provide feedback based on triage decisions.

In 2016 and 2017, we conducted two trials using a validated virtual simulation to study physician decision-making in trauma triage.[12] In the first trial, we tested the effect of exposure to the first video game on physician undertriage on the simulation. We randomized participants to play the game for 1 h or to complete a text-based educational training program modeled on theAdvanced Trauma Life Support course.[13] In the second trial, we compared the effect of exposure to either one of the two video games, a text-based educational training program, or nothing at all.[14] In both trials, all physicians were required to complete the same simulation within 4 wk of completing their designated intervention. The main results from these studies have been previously reported.[13,14]

In the current effort, we performed a post hoc secondary analysis by linking physician data from our previous trials to claims records acquired from the Centers for Medicare and Medicaid Services (CMS). The elderly experience high undertriage and mortality after injury, allowing insight into the determinants of variation in triage practices.[7] Medicare claims allow for longitudinal case tracking and physician identification. For these pragmatic reasons, we decided Medicare claims data would be best for achieving our objectives. Given the lag with which CMS releases Medicare claims, we do not have access to the billing data for the epoch after the trials. For this reason, we used a validated simulation to measure postintervention undertriage in our clinical trials. Our objective in this study was, therefore, to investigate the heterogeneity of treatment effect of game-based training by using Medicare data to assess the degree to which preexisting practice patterns might mediate the efficacy of the interventions. The University of Pittsburgh Institutional Review Board approved this study and permitted a waiver of consent, given the retrospective nature and minimal risk to participants. We performed all analyses using Stata 15 (StataCorp, TX) with alpha set at 0.05.

Subjects

We initially included all board-certified or board-eligible emergency physicians recruited for the two previous studies. Enrollment occurred at two separate meetings of the American College of Emergency Physicians in 2016 (Study 1) and 2017 (Study 2).[13,14] We obtained information about physicians including age, sex, race, years of experience, and work environment. Subjects received a wage-based incentive ($100/ h) for participating.[16]

For this analysis, we excluded trial participants who had less than 1 y of board certification by the end of 2015 because they lacked sufficient Medicare case volume to allow the calculation of preexisting triage patterns. Using the National Plan and Provider Enumeration System database, we identified each of these remaining physicians’ unique National Provider Identifier (NPI). We further excluded physicians for whom we could not definitively identify a unique NPI. Finally, we excluded physicians who did not treat any severely injured patients (Injury Severity Score [ISS] > 15). We then used NPI numbers to match physicians’ preexisting undertriage percentage in Medicare with their simulation undertriage percentage.

Physician undertriage in preexisting practice

We requested all inpatient, outpatient, and professional claims from CMS filed between January 1, 2010 and October 1, 2015 that included an International Classification of Diseases, version 9, Clinical Modification diagnostic code associated with an Abbreviated Injury Scale ≥ 3 (i.e., those codes that indicated the presence of moderate to severe injury). These files contain patient demographics, clinical information, and NPI numbers for billing physicians.

We describe the detailed analytic strategy for estimating triage practices from Medicare claims in the Appendix. Briefly, we began by identifying patient visits that occurred within 1 d of each other in the inpatient and outpatient analytic files and linking them to acute care hospitals. We built episodes of care for each patient who presented to the emergency room and spent at least 1 d in the hospital, organizing them based on the date of presentation and hospital identifier. We excluded those patients who initially presented to a hospital with a level I/II trauma center designation (identified by linking CMS records with data from the Trauma Information Exchange Program), as those hospitals represent definitive care centers. Using a validated algorithm for transforming diagnostic codes into Abbreviated Injury Scale, we calculated ISS for each patient and excluded those with ISS < 15.[17] For the remaining episodes, we identified the treating emergency department (ED) clinician using NPI.

Using these NPI numbers, we grouped episodes by physician. Because physicians may work at multiple hospitals, we further subgrouped episodes by presenting hospital. For each physician-hospital dyad, we tabulated the total number of severely injured patients seen and their dispositions. We defined undertriage as those episodes in which the patient was not transferred to a level I/II trauma center directly from the ED or within 1 d of admission.

Primary outcome: physician undertriage on the simulation

Physicians in our prior studies had completed a validated virtual simulation at the conclusion of the study protocol.[12] The virtual simulation consisted of ten patient cases presented over 42 min, meant to simulate a busy 8-hour ED shift at a nontrauma center. Four patients had severe injuries, two had minor injuries, and four had nontraumatic complaints. Each case included a two-dimensional image of the patient, a chief complaint, vital signs that updated regularly, and a physical examination. Physicians evaluated and managed patients using a set of 250 studies, medications, and procedures. Patients’ clinical status changed according to diagnostic or management decisions. The cases ended when physicians made a disposition decision (admit, discharge, or transfer) or the patient died. The simulation outcome was the percentage of severely injured trauma patients who the physician undertriaged (admitted to the nontrauma hospital or discharged).

Statistical analysis

We summarized physician characteristics including experience, primary hospital trauma center designation, and median number of severely injured patients seen before study enrollment using means (standard deviation [SD]) or medians (interquartile range) for continuous data and proportions (%) for categorical data as appropriate.

We estimated preexisting undertriage for each physician as the percentage of severely injured patients not transferred to a level I/II trauma center within 24 h of presentation. We first computed undertriage at each hospital at which they worked and then averaged these percentages (weighting by hospital case volume) to calculate an aggregated preexisting undertriage percentage.

To assess physicians’ performance on the simulation, we reviewed their disposition decisions (transfer, admit, discharge) for each severely injured patient case. We categorized patients who died before a disposition decision as “transferred,” because we could not predict what the physician would have done given a successful resuscitation and wanted to give him/her the benefit of the doubt. We then calculated each physician’s undertriage percentage on the simulation, which we defined as the number of patients not transferred divided by the total number of severely injured patients.

To determine whether some physicians might benefit more from exposure to the interventions than others, we tested the interaction between preexisting triage practices and the effect of the interventions on simulation practice. We collapsed physicians into two groups: physicians exposed to game-based training versus physicians in the control arm (active [text-based training] or passive [nothing]). We began by graphically exploring the relationship between preexisting undertriage and intervention group on simulation performance. We dichotomized preexisting triage practice at three different cut points (median, 45%, 40%) based on our graphical assessment of the relationship between preexisting undertriage and intervention group on simulation performance. We fit multiple linear regression models with the dependent variable being physician simulation undertriage and independent variables being intervention group, preexisting triage practice (parameterized either continuously or dichotomized at previously stated cut points), and their interaction term. We used Akaike Information Criterion to identify the parameterization that yielded the best fitting model. Our final linear regression analysis included intervention group, preexisting performance dichotomized into two groups (≤40% undertriage, > 40% undertriage), and the interaction between these two independent variables.

Results

Among the 524 physicians from these two studies, 394 were in practice long enough to make estimation of preexisting triage practices feasible. Of these, we excluded seven physicians because we were not able to verify a unique NPI number in the National Plan and Provider Enumeration System database. We further excluded 87 physicians with unique NPIs because we could not identify claims filed by them for a Medicare fee-for-service beneficiary from 2010 to 2015. This left 300 (76%) physicians who had filed a claim for a Medicare fee-for-service beneficiary from 2010 to 2015. Of these, 25 physicians were excluded because they did not treat any severely injured patients (ISS > 15). This left 275 physicians in the final cohort. (see Fig. 1).

Fig. 1-

Cohort flow diagram.

Overall, the cohort had a mean age of 44 y (SD 8.4) with a mean of 12 y of experience (SD 8.2). Most physicians were Caucasian (68%) and male (67%). Table 1 shows more detailed cohort characterization. There were no significant differences between these and characteristics of physicians who were excluded. Physicians managed a median of four severely injured Medicare patients (interquartile range 2–8) over the 5 y for preexisting triage practice determination.

Table 1.

Cohort summary statistics (n=275).

Variable	Value
Age, mean (SD)	44 (8.4)
Sex, n (%)
Male	186 (67)
Female	82 (30)
Unspecified	7 (3)
Race, n (%)
Caucasian	188 (68)
African-American	11 (4)
Asian	32 (12)
Hispanic	18 (7)
American-Indian	4 (1)
Other/unspecified	22 (8)
Years of experience, mean (SD)	12 (8.2)
Work at a trauma center, n (%)
Yes	30 (11)
No	245 (89)

Physicians undertriaged an overall mean of 74% (SD 30%) of patients in preexisting practice, and both groups were similar (mean 74% control versus 73% game-based training). Graphical analysis showed observable differences in intervention effect between the groups at various strata of preexisting undertriage (see Fig. 2). In general, physicians with higher preexisting undertriage showed improved simulation performance compared with control, whereas physicians with lower preexisting undertriage did not. When adjusting for preexisting practice as a continuous variable, we found that physicians who completed game-based training undertriaged 14% (95% CI−6% to −21%, P < 0.001) fewer patients in the simulation compared with the control group. The interaction between intervention group and preexisting performance in this model was not statistically significant (P = 0.25).

Fig. 2-

Heterogeneity of game-based training effect by preexisting triage.

Our analysis revealed the best-fitting model to be the one with the cut point for preexisting undertriage at 40%. After dichotomizing preexisting practice around this value, we found the interaction between intervention group and preexisting practice to be significant (P = 0.04). Among physicians who undertriaged fewer patients (i.e., had better preexisting performance), game-based training conferred no improvement in simulation undertriage compared with control (2%, 95% CI –14% to 20%, P = 0.76). However, among those who undertriaged more patients before participation in the trial (i.e., had worse preexisting performance), game-based training significantly improved simulation undertriage compared with control (−17%, 95% CI −9% to −24%, P = <0.001). Absolute undertriage percentages for high and low preexisting performers in the control and game-based training arms are shown in Table 2.

Table 2.

Percent under-triage on the simulation among physicians with different pre-existing triage practice.

Pre-existing triagepractice	Control(n = 132)	Game-based training (n = 143)		Interactionterm
	% Under-triage (95% CI)	% Under-triage (95% CI)	P Value	P Value
Low under-triage	56 (43–68)	58 (47–70)	0.76	0.04
High under-triage	63 (57–68)	46 (41–51)	<0.001

Discussion

In this secondary analysis of data collected from two behavioral trials, we demonstrated the feasibility of using administrative data to capture physicians’ preexisting practice patterns, offering a novel and pragmatic method of comparing real-world practice patterns with outcomes from trials targeting provider behavior. We found that preexisting triage patterns modified the efficacy of video game interventions on physicians’ simulation triage performance.

Efforts to improve quality of care by reducing diagnostic errors, standardizing practice patterns, and expediting the dissemination of therapeutic advances to everyday practice are broad priorities.[2,18,19] One of the barriers to developing interventions to change provider practice is the difficulty in assessing effectiveness. Most testing relies on process or interim outcome measures, such as whether providers tolerate the intervention or whether exposure improves knowledge of clinical practice guidelines.[20] Even well accepted continuing medical education programs like Advanced Trauma Life Support lack strong evidence demonstrating that certification in these skills changes practice patterns.[21] To combat this problem, we used a novel method of linking physician trial data to claims records. Such innovative approaches are becoming increasingly used in patient trials where they can lower costs, aid recruitment, and facilitate longitudinal outcome tracking.[22-25] We demonstrated that this method is feasible in physician trials, and, therefore, could help to establish baseline performance, track decision-making, and test the efficacy of interventions over time.

We tested the heterogeneity of treatment effect of our novel interventions to recalibrate physician heuristics in trauma triage. In the context of patient-centered outcomes research, understanding heterogeneity of treatment effect can identify patient subgroups who might most benefit from clinical or behavioral therapy.[26] For example, patients exhibit significant differences in their responses to behavioral interventions such as exercise training for nonactive individuals and cognitive training for older adults.[27,28] Little is known about the differential response of physicians to behavioral interventions. Better understanding of treatment heterogeneity among physicians is critical to ensure effective dissemination of these programs. We speculated that preexisting practice patterns might be one potential modifier of responsiveness to game-based training—specifically that those with worse preexisting performance would improve moreda hypothesis that we confirmed. And overall, physicians who completed game-based training showed improved simulation performance compared with control.

We observed that physicians with low preexisting undertriage showed worse absolute undertriage on the simulation than in their prior practice. We surmise a few potential explanations for these effects. First, some could question the simulation’s ability to represent real-world triage decision-making. However, we have previously confirmed the validity of the simulation undertriage estimates in a separate cohort of physicians.[12] Second, the simulation may have detangled physicians from the confounding effects of their work environments. High-performing physicians may undertriage fewer patients in practice because of institutional factors encouraging transfer, and, these effects may be lost when testing physicians on the simulation. Finally, the interventions may have recalibrated high-performing physicians’ heuristics for the worse. Evidence from the video game literature suggests that badly designed intervention strategies can have unintended consequences and deteriorate performance.[29] Our findings underscore the need for further research on determinants of treatment heterogeneity. We have not directly tested the games’ effects over time, but we previously found that triage improvement was sustained for 6 mo.[13] We cannot yet recommend specific training programs using these games, but our findings highlight the importance of a personalized and strategic approach to quality improvement in trauma triage rather than the current “one-size fits all” approach used by most stakeholders.

Limitations

Our study has several limitations. First, we recruited convenience samples of emergency physicians at national meetings. These physicians may have been disproportionately eager to learn and capable of improving with training. At minimum, our cohort demographics mirrored those of the national emergency medicine workforce.[30] Second, we relied on Medicare claims data to establish preexisting practice patterns, with associated concerns about how we adjusted for injury severity. We minimized bias by using a validated algorithm to estimate injury severity. This yielded case volume and undertriage percentages in line with previously published findings. Patients enrolled in Medicare differ from the general population. However, the elderly make up an increasing proportion of the trauma population, are more likely to be undertriaged, and more likely to suffer adverse consequences after injury.[7,8,31] Evidence of efficacy for this cohort would justify dissemination of the interventions. Third, we had to exclude many trial participants because of our inability to match them with filed Medicare claims. This difficulty is consistent with the discordance between the number of provider and facility claims filed for ED patients.[32] Fourth, preexisting practice may modify the effect of each of the video games differently. We collapsed physicians who played either game into one group because we saw similar efficacy in our trials and because we sought to preserve statistical power for our heterogeneity of treatment effect analysis. Finally, we were underpowered to detect small to moderate effect size differences in our primary model. We retained power to detect large differences between high and low performing physicians, and ultimately, this bears more significance for future efforts to disseminate training.

Conclusions

We demonstrated the feasibility of using claims data to measure preexisting practice and to test the heterogeneity of treatment effect of behavioral interventions designed to recalibrate physician heuristics. Video game-based training significantly improved undertriage. Low-performing physicians in preexisting practice derived the greatest benefit from game-based training. This highlights the need for better understanding of treatment heterogeneity and the value of targeted training in trauma triage.