Learning optimal opioid prescribing and monitoring: a simulation study of medical residents.

OBJECTIVE: Hospitalized patients often receive opioids. There is a lack of consensus regarding evidence-based guidelines or training programs for effective management of pain in the hospital. We investigated the viability of using an Internet-based opioid dosing simulator to teach residents appropriate use of opioids to treat and manage acute pain.
MATERIALS AND METHODS: We used a prospective, longitudinal design to evaluate the effects of simulator training. In face-to-face didactic sessions, we taught 120 (108 internal medicine and 12 family medicine) residents principles of pain management and how to use the simulator. Each trainee completed 10 training and, subsequently, 5 testing trials on the simulator. For each trial, we collected medications, doses, routes and times of administration, pain scores, and a summary score. We used mixed-effects regression models to assess the impact of simulation training on simulation performance scores, variability in pain score trajectories, appropriate use of short- and long-acting opioids, and use of naloxone.
RESULTS: Trainees completed 1582 simulation trials (M = 13.2, SD = 6.8), with sustained improvements in their simulated pain management practices. Over time, trainees improved their overall simulated pain management scores (b = 0.05, P < .01), generated lower pain score trajectories with less variability (b = -0.02, P < .01), switched more rapidly from short-acting to long-acting agents (b = -0.50, P < .01), and used naloxone less often (b = -0.10, P < .01). DISCUSSION AND
CONCLUSIONS: Trainees translated their understanding of didactically presented principles of pain management to their performance on simulated patient cases. Simulation-based training presents an opportunity for improving opioid-based inpatient acute pain management.

BACKGROUND AND SIGNIFICANCE

Pain is common among hospitalized patients and is often undertreated. Inappropriate pain management is associated with adverse events, readmissions, higher levels of anxiety, and depression. In a study of non-surgical patients admitted in 286 U.S. hospitals, Herzig et al. found that more than 50% of patients were treated with opioids. In spite of higher rates of opioid use for hospitalized patients, nearly 50% of patients with significant pain reported that their pain was not satisfactorily controlled., Comparisons across U.S. hospitals found that only 71% of hospitalized patients stated that their pain was well controlled at all times. Additionally, higher rates of opioid exposure were associated with adverse events, with approximately 0.6% of opioid exposed patients experiencing severe opioid-related adverse events.

Managing acute pain requires striking a balance among patient-centered pain relief, opioid-related adverse effects, and the risk of long-term dependence. Inpatient opioid prescribing behaviors are often guided by previous negative experiences with opioid orders (eg, inadvertent overdose, and patient misuse) and institutional culture (eg, patient satisfaction, avoiding re-admission for an adverse opioid event). The lack of universally agreed-upon evidence-based guidelines on pain management and the lack of opioid-related training have resulted in persistent variation in opioid prescriptions and pain relief.,

Traditionally, pain management training is only a small part of medical school or residency curricula. Training, when offered, tends to focus on chronic pain management in outpatient settings, emphasizing cognitive and behavioral techniques to manage the functional and emotional well-being of patients.

For acute pain management, there is limited training, and there are few objective guidelines for safe practice. Attempting to fill this gap and reflecting on experience in clinical quality improvement in large inpatient settings, we formulated 2 conceptual principles for management of acute exacerbations of pain (see Conceptual Principles section). We incorporated these principles into an interactive, Internet-based opioid dosing simulator that teaches prescribers how to safely dose and monitor hospitalized patients in acute pain.

As a part of a larger study on opioid management, this study was motivated by the fact that pain management in acute care settings has been largely ineffective,, and that traditional mechanisms of pain care training have been not been successful. The opioid dosing simulator affords an opportunity to practice opioid management skills without risk of patient harm, and, therefore, has appeal for reasons of patient safety, educational quality, cost-effectiveness, and medical ethics. In this exploratory study, we investigate the viability of using the simulator as a mechanism for teaching conceptual principles of pain management and monitoring.

Toward this end, using the simulator, we explored the following research questions related to trainee performance on simulated patient cases: (1) Does training on the simulator lead to improvements in simulated pain management? (2) How effectively do trainees learn to combine short- and long-acting opioids to achieve smooth pain score trajectories?, (3) Does simulator training reduce the use of naloxone? (4) Do trainees learn to monitor pain scores at intervals concordant with the pharmacokinetic properties of different drugs? (5) Do trainees learn to transition from short- to long-acting opioids?

OPIOID DOSING SIMULATOR

In this section, we describe the conceptual principles underlying the simulator, its design, how to use the simulator, and the mechanisms for evaluating trainee performance.

Conceptual principles

Based on a series of empirical studies, we developed 2 conceptual principles regarding safe and effective pain management. First, it is desirable to avoid saw-tooth type pain trajectories. Often, a patient’s pain trajectory follows a saw-tooth-shaped pain trajectory due to overreliance on short-acting agents. To mitigate the peaks and valleys in pain, prescribers should use appropriately dosed long-acting opioids, with the long-acting dose derived from a patient’s response to short-acting opioids. Appropriately timed short- and long-acting opioid doses can potentially lead to smoother declines in pain score response.

Second, clinicians should assess pain scores at intervals concordant with a given opioid’s pharmacokinetics. The peak effect of a short-acting opioid is about 1 h after a dose; for long-acting opioids, it is approximately 4 h after an 8-h dose. As much as workflow permits, pain scores should be monitored at times corresponding to the administered agent’s peak effect. Learning to control pain with opioids also requires an understanding of how to titrate a dose based on a patient’s observed response to a given dose and route of a particular opioid formulation.

Simulator design

Guided by these principles, we developed a device-agnostic, Internet-based simulator, with input from a team of patient safety researchers, physicians, nurses, pharmacists, and software developers. Development took place via an iterative series of studies on dose-response of opioids used at a local academic medical center including: (1) collecting empiric dose-response curves, (2) reconciling literature-based response curves with empiric curves, (3) reconciliation of empiric curves with pharmacy-derived response curves, and, finally, (4) pilot testing and calibration against actual pain score responses to opioids.

Internally, the simulator consists of a set of default curves that, for each of the 13 possible drug products, relate time to reduction in pain score in 10-min intervals for a standard dose (ie, morphine IV equivalent dose of 1 mg). The drugs and routes in the simulator are morphine (PO, IV bolus, and drip), morphine sustained release (SR), hydromorphone (PO, IV bolus, and drip), oxycodone, oxycodone SR, fentanyl (IV bolus), hydrocodone, methadone (oral), and codeine. The simulator also included the antidote, naloxone (injection), for use in case of an overdose.

Using the simulator

Training on the simulator involves working through assigned patient cases. For each case (see Figure 1A), trainees select one or more opioid agents from the list of available “Meds” (see Figure 1B) and “signs the order.” Trainees choose when to measure (and re-measure) pain scores by selecting simulated time increments from 10 min to 8 h after a dose (see Figure 1C). The simulator displays a graphical pain score trajectory once the trainee selects an observation interval. The timing, dose, type, and route of administration of all the opioids and naloxone administered during a simulated care episode determines the shape of the resulting pain score trajectory.

Figure 1.

The various interfaces that are part of the opioid dosing simulator. Clockwise (A–F). (A) Presentation of the clinical case. (B) Medication ordering interface. (C) Selecting the “simulation time”. (D) Pain trajectory after administration. (E) 48-h pain score trajectory (ie, at the end of the simulation). (F) Log of medications ordered for a patient.

For each case, trainees perform 48 simulated hours of pain management. At any point, trainees can view the list of doses administered, current pain score trajectory, or use a dosage calculator (see Figure 1D–F). The dose calculator, incorporated within the simulator, helps in dose conversions between opioid products and displays pharmacokinetic curves for each agent.

Evaluating trainee performance

The 48-h pain score trajectory is the basis for trainee evaluation. We defined overdose as a pain score less than zero or any naloxone administration. Either of these situations result in a failing grade for that case. For all trajectories without an overdose, we calculated the performance score using an adjusted area under the curve (AUC) metric. A standard AUC does not account for the distress to patients caused by a saw-tooth trajectory. Hence, the scoring system penalized variation, and the penalty depended on the severity of the pain. For example, the penalty for variation between pain Scores 7 and 9 was greater than that for variation between pain Scores 1 and 3. To permit comparison across cases with different initial pain scores, we normalized the adjusted AUC to account for the initial pain score in each case. Based on the normalized AUC score, the use of naloxone, and whether the pain score ever fell below zero, each case was assigned a letter Grade A, B, C, or F. The simulator displayed a letter grade to the trainee as immediate feedback. For all trainees with non-failing grades a corresponding numerical score was also logged in the database for subsequent analysis.

METHOD

Setting and participants

We conducted the study at the University of Illinois at Chicago, an urban academic medical center. Between March 2015 and December 2016, we recruited and trained 120 residents (108 internal medicine, 12 family medicine). We compensated residents for their participation. The institutional review board of the university approved the study, and we obtained written consent from all participants.

Design and procedure

We used a prospective, longitudinal design to investigate the effect of simulation on performance on simulated cases. We trained all consenting participants during an hour-long face-to-face session conducted by 1 of 3 physicians (co-authors R.M., S.F., or W.L.G.). Training sessions included a description of the conceptual principles for effective pain management, walk-through of practice cases, how to perform various tasks on the simulator, and instructions on how best to achieve a smooth pain score trajectory. We instructed trainees to aim for a stable pain score in the range of 2–3, and to avoid a pain score of zero. We provided trainees a link to a video version of the tutorial for future reference.

After the didactic session, we emailed all participants instructions on how to access the simulator and instructed them to complete a set of 10 training cases. After the completion of the training cases, we emailed instructions to complete 5 additional cases.

Patient cases

We developed and incorporated 25 cases (20 training and 5 testing cases) of acute or acute on chronic pain exacerbation into the simulator. Of the 25 cases, 11 were cancer-related, 9 general medical, 3 sickle cell disease, 1 surgical, and 1 trauma. By varying the case sensitivity to opioids randomly within predetermined ranges (low, medium, and high), we configured the simulator such that no two patient cases would ever have identical pain responses. This was useful in the training phase where we wanted to prevent trainees from being able to memorize cases. During the testing phase, we set sensitivity to a constant such that all trainees encountered precisely the same cases. A set of 5 example cases and their corresponding sensitivities are provided in Table 1.

Table 1.

Case description and associated case sensitivity for 5 patient cases that were used in the simulator

Case description	Case sensitivity
A 58-year-old woman 5 y post-mastectomy for Stage 2 breast cancer presented to her outside physician with pain in her ribs and chest. A bone scan revealed metastatic disease. She has been taking 20 mg of IV Morphine Sulfate daily in a drip but has no venous access at present. She is mildly obese but in normal health otherwise. Her pain score is 10 and she has had no pain medicine since transfer to your service.	Low
A 45-year-old man has pancreatic cancer involving his stomach and liver. He is writhing in pain with a pain score of 10. He has been on pain medicine but cannot recall the dose. He had no medicine for the last 8 h. His health is otherwise normal.	Medium
A 70-year-old woman, slips on ice and fractures her radius and femur. She called 911 was picked up and brought to the hospital. She is alert and oriented and reports a pain score of 9. She has a history of Hypertension and coronary artery disease.	High
A 55-year-old woman with Human Immunodeficiency Virus, HTN, and Degenerative Joint Disease status post-hip replacement 2 y ago, has severe hip pain, is thought to have a septic joint. She has normal renal function and weighs 80 kg. Her pain score is 9 of 10.	High
A 43-year-old woman with sickle cell disease (Hgb SS) complicated by Avascular Necrosis of the hip, HTN, and a stroke presents with right shoulder pain and bilateral leg pain for 2 d. She was taking acetaminophen/Codeine at home with little help. Her pain score is 10 and she feels that this is a typical vaso-occlusive crisis.	Medium

Data collection

We collected the following information for 15 simulation trials for each participant (we refer to each completed case as a “trial” of the simulator): medication identities, doses, and routes of administration, administration times, time-stamped pain scores, and the numerical performance score.

Data analysis

Trials 1–10 were training trials, and 11–15 were testing. We assessed trainee performance based on the numerical score assigned to each trial. As described previously (see Evaluating Trainee Performance section), the numerical score could be: (1) a continuously-valued numerical score for the AUC of the pain score trajectory, adjusted for variability in pain response; (2) −1 for a simulated pain score below 0 (ie, overdose); or (3) −2, if naloxone was used (both −1 and −2 were considered failing scores).

In order to represent trainee performance on the same continuum, we partitioned the numerical scores into quintiles of performance. Higher AUC scores represented more pain, thus worse performance. We grouped numerical scores into quartiles by computing the 25th, 50th, and 75th percentile cutoffs. We then reverse-coded the scores as follows: (1) failing scores (−1 or −2), to the first quintile (ie, worst performance), (2) scores greater than the 75th percentile, to the second quintile, (3) scores in the 50th to 75th percentile to the third quintile, (4) scores in the 25th to 50th percentile to the fourth quintile, and (5) scores in the 0–25th percentile to the fifth quintile (ie, best performance). We used a series of regression analyses to perform the following comparisons:

To assess the effect of training on performance, we estimated a mixed-effects ordinal regression with performance quintile as the dependent variable and number of completed trials as the independent variable.

To assess the effect of training on the variability in pain scores, we examined the association between number of completed trials and the standard deviation of pain scores for each trial (ie, whether there were peaks and valleys in pain scores). We also compared the mean pain trajectories for training and testing trials respectively.

To investigate whether participants appropriately timed their pain assessment, we determined the mean time to pain measurement after short-acting and long-acting opioid agents respectively.

To investigate whether participants initiated long-acting opioids appropriately, we examined the association between number of completed trials and the mean time to initiate long-acting drug therapy.

To investigate whether participants internalized the goal of rapidly transitioning to long-acting opioid therapy and reducing use of short-acting therapy over time, we plotted the frequency of use of morphine IV, morphine SR, and naloxone over 48 simulated hours across all participants.

Finally, to characterize the effect of training on naloxone use, we used a mixed-effects logistic regression model with naloxone use as the dependent variable, and number of completed trials as the independent variable.

Alpha was set to 0.05 for all tests of statistical significance, and regression estimates were computed using the Supermix software package.

RESULTS

One hundred twenty resident trainees participated in this study over a period of 21 months (March 2015 through December 2018), and completed a total of 1582 simulation trials (M = 13.2, SD = 6.8, range = 1–29). Of these, 81 (67.5%) completed all 15 trials. On average, each trial simulated 47.6 h of patient care (SD = 3.6 h). Participants completed the trials over a period of 20.4 days (SD = 43 days).

Of the 1582 trials that were started, 1566 (99%) lasted the full simulated 48 h. Thirty-eight percent of the trials were completed in under 2 min (n = 601), 55% (n = 870) in under 3 min, and 90% (n = 1424) in under 9 min. After excluding 21 trials that lasted longer than 60 min, when a trainee likely exited the simulator for an extended period of time, mean trial duration was 3.6 min (SD = 4.5 min). Trainee performance, as measured by the mean quintile score, during training trials was 1.43 (SD = 1.45, n = 1014 trials); during the testing trials, the mean quintile score was 1.81 (SD = 1.58, n = 416 trials).

Based on the mixed-effects regression, we found a significant effect of the number of training trials on performance (b = 0.05, s.e. = 0.007, z = 7.96, P < .01) (see Figure 2A). We also found the mean pain score trajectory was lower during the testing trials (b = −0.34, s.e. = 0.08, z = −4.02, P < .01) (see Figure 2B). Additionally, variability in the pain score trajectory, as measured by its standard deviation, decreased with increasing training (b = −0.02, s.e. = 0.00484, z = −4.19, P < .01) (Figure 3A).

Figure 2.

(A) Training performance over time based on the overall score for each trial. Over time, the overall performance improved, as shown by the increasing average quantile score. (B) The overall mean pain trajectory for the training and testing trails.

Figure 3.

(A) The variation in the standard deviation of the pain scores, with a decreasing slope. (B) Effect of training on mean time to initiation of long-acting therapy.

The mean time between medication administration and pain score measurement was 1.78 h for short-acting opioids (SD = 1.8 h, median = 1 h, mode = 1 h), and 4.49 h for long-acting opioids (SD = 3.4 h, median = 4 h, mode = 8 h; t = −58.74, P < .01). The mean time from the start of a care episode to initiation of long-acting therapy decreased significantly as training progressed, starting at about 16 simulated hours on trial 1, and decreasing to 10 simulated hours by trial 15 (b = −0.50, s.e. = 0.09, P < .01) (See Figure 3B). Similarly, there was a transition from morphine IV to long-acting morphine SR with increasing simulated time (at simulated hour 0 [morphine IV, morphine SR, naloxone]: 44.7%, 14.3%, 0%; hour 24: 7.9%, 51.4%, 1.7%; hour 48: 0.56%, 61.4%, 0.28%) (See Figure 4A). Finally, the probability of using naloxone for opioid overdose declined significantly as the number of completed trials increased (b = −0.10, s.e. = 0.02, z = −5.80, P < .01) (See Figure 4B).

Figure 4.

(A) Proportion of trainees using short-acting (morphine IV), long-acting (morphine sustained), and naloxone (antidote) over the course of 48 simulated hours. (B) Proportion of trainees using naloxone as a function of number of simulation trials completed.

DISCUSSION

In a prospective study, we found that trainees showed sustained improvements in their simulated pain management practices. Over time, trainees generated lower mean pain score trajectories with less variability, used naloxone less often, and switched more rapidly from short- to long-acting opioid agents.

As previously described, the purpose of this exploratory study was to evaluate whether trainees translated didactically presented pain management principles onto simulated cases. Taken together, the results show that trainees successfully translated conceptual principles of pain control to their management of simulated cases.

To the best of our knowledge, this is the first simulator for training pain management using opioids. This is noteworthy given that a recent survey by the Association of American Medical Colleges found that approximately 90% of respondents used simulators during their first 3 years of training. In a recently published meta-analysis of 609 technology-enhanced simulation programs, Cook et al. found that these programs were associated with gains in knowledge, process (eg, efficiency), and skills (eg, procedural dexterity). However, there were only moderate effects of such training on patient-related outcomes. None of simulation programs were designed for pain management or opioid administration.

Given the current focus on opioid related issues, and the lack of training opportunities for medical students and residents, simulator-based training provides trainees an experiential approach to learning about safe opioid use without the risk of harming patients. Further research is underway to determine whether, and to what extent, improvement on simulated cases translates to clinical practice. We are currently measuring inpatient pain management practices (pain scores, opioid orders) of trainees who participated in this study to evaluate whether they translated their understanding on the simulator to real-world practice. Past research on the transfer of cognitive skill has shown that knowledge gained in one setting can transfer to new settings., We hypothesize that participants performing better on the simulator are also likely to utilize their learned pain management skills in real-world settings.

The flexible, web-based design of the simulator enables easy customization for its use for training in other institutions or settings. For example, one could easily create and evaluate pain-related patient cases that reflect different practice settings (eg, post-surgical care) or with different clinical groups (eg, nurses or pharmacists). It would also be possible to incorporate different conceptual principles (eg, related to the preventing opioid dependence) for its use in other settings or situations (eg, in primary care). Given the flexibility of the simulator’s system architecture, it is also possible to incorporate interactive cognitive support during the simulator use such as providing content-based feedback (eg, alerts for switching to a long-acting opioid), timing of such feedback at appropriate intervals (eg, for timing a pain score measurement), and remediation strategies (eg, suggestions for dosage).

From an informatics perspective, the simulator addresses a previously unaddressed technological gap—a potential mechanism for training medical students, residents, fellows, and nurses on pain and opioid management. Although preliminary, results from this study highlight the viability of trainee learning on opioid management using the simulator. The transfer of such learning to real-world settings for pain care can have significant impact on residency training, safe use of opioids, and patient safety.

Limitations

Our study had several limitations. The study included only 2 groups of trainees (family and internal medicine residents) from 1 academic medical center. We only measured performance within the simulator, not in real patient care situations. The effect of simulator training on real-world pain management practice is currently on-going. The study design lacked a control group, and comparisons were based on participants acting as their own control. The cases used for this simulation were artificial but were similar to previously encountered patient cases. The choice of the number of training and testing trials were based on convenience. We also did not account for the number of years of experience of the residents. Approximately 32% of the participants did not complete all the 15 assigned trials.

Finally, we based the design of the simulator on principles of inpatient pain management derived from preliminary research studies and from insights of clinical experts on our research team. There is limited consensus regarding the best practices for inpatient pain management, and we did not design this study to assess the merits of our particular model of inpatient pain care. Rather, the study shows how medical educators can use simulation training to inculcate any guidelines they choose, as long as they can translate those guidelines into an evaluation metric suitable for trainee feedback regarding their opioid prescribing behaviors.

CONCLUSIONS

We designed and developed an internet-based opioid dosing simulator based on 2 principles of safe pain care. Based on a longitudinal, prospective study, we found that simulation training led to improved performance within the simulator. As learners completed more trials on the simulator, simulated pain scores were lower, trajectories were less variable, trainees started long-acting agents more rapidly and used antidotes less frequently.

We are currently investigating how such simulator-based training improvements translate to clinical practice. Given the lack of evidence-based guidelines or specific pain management training, the simulator, a first of its kind for pain management, affords opportunities for potentially improving opioid-based inpatient acute pain management. The flexible platform used for the simulator development also allows its use for training in other clinical settings (eg, surgical care), and providing cognitive support and feedback during training.