An Investigation of Radiation Treatment Learning Opportunities in Relation to the Radiation Oncology Electronic Medical Record: A Single Institution Experience.

PURPOSE: A modern radiation oncology electronic medical record (RO-EMR) system represents a sophisticated human-computer interface with the potential to reduce human driven errors and improve patient safety. As the RO-EMR becomes an integral part of clinical processes, it may be advantageous to analyze learning opportunities (LO) based on their relationship with the RO-EMR. This work reviews one institution's documented LO to: (1) study their relationship with the RO-EMR workflow, (2) identify best opportunities to improve RO-EMR workflow design, and (3) identify current RO-EMR workflow challenges. METHODS AND MATERIALS: Internal LO reports for an 11-year contiguous period were categorized by their relationship to the RO-EMR. We also identify the specific components of the RO-EMR used or involved in each LO. Additionally, contributing factor categories from the ASTRO/AAPM sponsored Radiation Oncology Incident Learning System's (RO-ILS) nomenclature was used to characterize LO directly linked to the RO-EMR.
RESULTS: A total of 163 LO from the 11-year period were reviewed and analyzed. Most (77.2%) LO involved the RO-EMR in some way. The majority of the LO were the results of human/manual operations. The most common RO-EMR components involved in the studied LO were documentation related to patient setup, treatment session schedule functionality, RO-EMR used as a communication/note-delivery tool, and issues with treatment accessories. Most of the LO had staff lack of attention and policy not followed as 2 of the highest occurring contributing factors.
CONCLUSIONS: We found that the majority of LO were related to RO-EMR workflow processes. The high-risk areas were related to manual data entry or manual treatment execution. An evaluation of LO as a function of their relationship with the RO-EMR allowed for opportunities for improvement. In addition to regular radiation oncology quality improvement review and policy update, automated functions in RO-EMR remain highly desirable.

Introduction

A record and verify system (RVS) can be defined as computer software used in the radiation oncology environment to store and verify treatment plan parameters against delivery parameters, as well as to record a history of treatment delivery parameters used for each treatment. Some of the earliest literature on the RVS for radiation treatments appeared in the late 70s and early 80s1, 2, 3, 4 and described the initial design and progress of the RVS. The recent and significant advancements of technology in radiation oncology have given us the ability to deliver more sophisticated treatments. One such example is volumetric modulated arc therapy, where all of the required motions of the gantry, collimator and multileaf collimator are stored and managed by modern versions of the RVS. Subsequently, the modern RVS has evolved further, to entail a comprehensive radiation oncology electronic medical record (RO-EMR) system, which contains patients’ medical and treatment records relevant to radiation therapy, including imaging data from daily treatment setup. In today's modern radiation oncology department most, if not all, of the treatment preparation and management, including planning and treatment workflow steps, are performed in and through the RO-EMR. Due to the broad role the modern RO-EMR now plays in the delivery and management of patient care, the entire radiation therapy care team—radiation oncology physicians, nurses, dosimetrists, medical physicists, and therapists—have come to rely heavily on the RO-EMR. As such, the RO-EMR has become one of the most frequently used tools for team-based communication in the radiation oncology department.

It is a well-established fact that deviations from standardized processes in medicine are unavoidable due to the so-called human factor and such deviations are, perhaps, even more likely to happen in more complex (modern) radiation oncology environments., The modern RO-EMR represents a sophisticated human-computer interface that has the potential to reduce human driven errors and improve patient safety by way of enhanced and improved communication and by automation of processes that may be more vulnerable to human error. However, if not optimized for accuracy and efficiency of workflow, or if simply misused, the RO-EMR can also affect clinical workflow adversely and, thus, inadvertently contribute to deviations from intended processes that could affect safety and quality of patient treatment.,9, 10, 11, 12, 13 As routinely used, the RO-EMR is a central repository for information and facilitates essential communication between health care workers, with both data input and output functionality. One of the mechanisms for data input to the EMR is a human-computer interface where information is manually entered into the RO-EMR. On the output side, there is another human-computer interface used to deliver information to either a human who must receive, interpret, and execute the output of information, or another computer-computer interface for treatment execution.

As it has been well described in To Err Is Human: Building a Safer Health System, “Not all errors result in harm … Errors that do not result in harm also represent an important opportunity to identify system improvements having the potential to prevent adverse events. Building safety into processes of care is a more effective way to reduce errors than blaming individuals.” In our institution, we have identified the importance of documenting incidents to enable the opportunities for future improvements and established an internal incident learning system (ILS) more than one decade ago. Additionally, to encourage all level of staff members to identify any unsafe conditions (with or without harms) to promote the culture of safety,, we have started to identify these cases as “learning opportunities” (LO), instead of using terms such as errors, events, or incidents. We believe that recording of circumstances that lead to unintended deviations from standard process can allow for learning through root cause analysis, and potentially lead to effective mitigation strategies, which can ultimately improve quality of care and patient safety. Lastly, evaluation of “families” of related LO from an ILS, based on their relationship to the RO-EMR, may help identify more meaningful workflow deficiencies compared with common root cause analysis practices.

In this study, we are interested in the LO that flow through the RO-EMR and potentially increase the risk of errors second to the RO-EMR's design or use. We analyzed our single institution data to characterize the role that the RO-EMR plays based on their relationship to the LO at various stages of the workflow (input or output), and to identify potential opportunities for improvement in RO-EMR workflow design, or use. Our hypothesis is that many of the process deviations we observe in radiation oncology can be traced back to having “flowed through” the RO-EMR workflow design and that having knowledge of this relationship will allow easier identification of effective mitigation strategies. In this report, we investigate the frequency of LO that in some way involved the RO-EMR. We also used the Radiation Oncology Incident Learning System's (RO-ILS) contributing factor nomenclatures to study their correlations with various RO-EMR involved LO. Lastly, we endeavored to identify patterns in LO related to the workflows or clinical process (whether human-related or due to limitations of the RO-EMR) to inform future mitigation strategies.

Methods and Materials

Database of learning opportunities

Our institution is a university-based hospital that treated approximately 110 patients per day on average in 4 to 6 vaults during the study period, from 2007 to 2018, resulting in roughly 27,000 patient treatments annually, or approximately 300,000 total patient treatments. In our institution, the LO reporting system has evolved continuously over the past decade, growing in sophistication and detail. Similar to many institutions, as our understanding and appreciation of the notion of a “culture of safety” has grown, the detail and frequency of LO recording and study has also grown. In this study, an LO was defined as whenever one or more technical parameters during the radiation oncology process deviated from what was planned, or whenever workflows deviated in any meaningful way from defined policies and procedures. The process deviation did not need to ever “reach the patient” to be counted as an LO, and this is logical in that rigorous QA processes have been established to catch deviations before they reach the patient, and a process deviation does not need to directly affect a patient to be of value for learning.

RO-EMR-based LO categorization

Basic analyses of LO were performed to determine: whether it involved human error, the time point when the LO occurred (eg, simulation), and when it was identified. To determine the relationship between the LO reported and our department RO-EMR, MOSAIQ (Elekta Inc, Stockholm, Sweden), every LO was assigned to 1 of 5 main categories: (I) RO-EMR related, input side, (II) RO-EMR related, output side, (III) RO-EMR unrelated, workflow specific, (IV) RO-EMR unrelated, network, linear accelerator (linac), or computer hardware issues, and (V) RO-EMR unrelated, other patient safety. The definition of each category is listed in Table 1. The category assigned to each LO was determined by 3 experienced medical physicists in our institution based on the decision tree as shown in Fig. 1. Within the RO-EMR related categories (1 and 2), we further analyzed specific components of the RO-EMR that may be related to the LO (Fig. 1). Some of the components of the RO-EMR at the time of the LO did not exist or were not used in our institution. However, based on the LO description and our projected workflow in the current environment, we were able to deem the LO relevant to the RO-EMR. In this case, the assignments of the relevant components of the RO-EMR were based on the current functionality. For example, the utilization of treatment session schedule function (treatment calendar) was launched in 2011 in our institution. However, we assigned previous LO that could be related to treatment calendar before 2011 to the treatment calendar category, assuming its current function and usage.

Table 1

Definitions of RO-EMR based LO categories

Event categories	Definition	Example
(I) RO-EMR related, input side	Unitended information entered in RO-EMR; LO occurred on the input side of RO-EMR.	Treatment field scheduling information entered into treatment calendar not as intended.
(II) RO-EMR related, output side	Information entered in RO-EMR was correct but was misinterpreted or executed differently than intended.	Schedule for treatment field delivery misinterpreted.
(III) RO-EMR unrelated, workflow specific	LO resulted from workflow outside of RO-EMR. Existing policy incomplete, unclear, or not followed by the operator.	Treatment couch was not zeroed according to defined processes.
(IV) RO-EMR unrelated, network, linac, or computer hardware issues	LO caused by non RO-EMR related network or hardware issues.	Linac couch hardware issue caused couch drifting with changes in couch lateral positional readout.
(V) RO-EMR unrelated, other patient safety	Safety related LO not related to any of the previous categories; including nonexisting policy.	Patient rolled off the table during treatment.

Abbreviations: LO = learning opportunities; RO-EMR = radiation oncology electronic medical record; Linac = linear accelerator.

Figure 1

Radiation oncology electronic medical record (RO-EMR) related learning opportunities (LO) categorization decision tree.

RO-ILS contributing factors analysis

To further understand the potential causes for each LO, RO-ILS contributing factor nomenclatures were used. We assigned various contributing factors to each LO to examine possible common contributing factors within our operational processes. Depending on the LO pathway, each LO might be assigned several contributing factors. Additionally, 9 common categories for the contributing factors were also used to find their occurrence rates. The relationships between the 5 RO-EMR based LO categorization and the contributing factors were studied.

Results

During the 11-year study period, a total of 163 LO were recorded. Within the 5 main categories, most of the LO were related to RO-EMR, from (II) output (N = 70, 42.9%) or (I) input side (N = 55, 33.7%). For the RO-EMR unrelated categories, the highest LO count was from (III) workflow specific (N = 18, 11.0%), followed by (IV) other patient safety (N = 16, 9.8%), and (V) network, linac, computer hardware glitch (N = 4, 2.5%). The majority of the LO were related to human errors (N = 155, 95%); of those 11.7% were related to training, and 19.0% related to inadequate staff coverage or time pressure. Most of the LO occurred at the treatment machine (N = 111, 68.1%), followed by during treatment planning (N = 28, 17.2%). The majority of the LO were discovered at the machine either during verification simulation (v-sim) or after treatment started (N = 149, 92.0%). Within all LO, 18.4% (N = 30) resulted from wrong shifts (either initial setup shift to treatment isocenter or image guided shift), and 17.8% (N = 29) from using the wrong treatment related devices (block, bolus, etc). It is worth noting that 73.6% (N = 120) of reported LO were associated with a policy not being followed, 8.6% (N = 14) with situation in which there was no existing policy, and 17.8% (N = 29) of LO were not related to policy.

RO-EMR related LO analysis

The RO-EMR was involved in 76.7% (N = 125) of the events. Among them, 44.0% (N = 55) were related to RO-EMR input side and 56.0% (N = 70) were related to output side. All but 3 LO (2 input and one output side) within all of the RO-EMR related LO were human-error related. Figure 2 shows the specific RO-EMR components contributing to either input side LO (Fig. 2a) or output side LO (Fig. 2b). Regardless of RO-EMR input or output side LO, 4 commonly involved RO-EMR components were patient treatment position steup related documentation (23.3%, N = 38), treatment session schedule functionality (16.0%, N = 26), RO-EMR used as communication tools (14.7%, N = 24), and issues with treatment accessories (12.9%, N = 21).

Figure 2

Results of specific radiation oncology electronic medical record (RO-EMR) components contributing to learning opportunities (LO) from either (a) RO-EMR input side or (b) RO-EMR output side.

Contributing factor analysis

The counts and occurrence rates based on 9 main RO-ILS contributing factor categories are listed in Table 2. Some of the highest-ranking contributing factors in our institution were staff lack of attention (N = 143, 87.7%), policy not followed (N = 120, 73.6%), and treatment timeout (verification of approvals and documents) not done (N = 96, 58.9%). Within all of the RO-EMR related LO, either on the input side or output side, contributing factors were further analyzed. Most of the RO-EMR input LO (N = 55) were related to documentation (N = 36, 65.5%). For the RO-EMR output LO (N = 70), treatment timeout verification (not done or not done properly) was the most common contributing factor (N = 63, 90.0%). The resulting correlation summary between various dominant contributing factors and the RO-EMR based LO categories are shown in Table 3.

Table 2

List of contributing factors and their categories

Contributing factor category	Counts (occurrence rate)	Contributing factor
Communication	62 (38.3%)	Inadequate communicationIncorrect communicationMisunderstood prescriptionMisunderstood techniquePhysician orders misunderstoodProblem using RO-EMR for communicationScheduling issue
Documentation	49 (30.2%)	Verification of approvals and documents not done*Unclear or incomplete setup documentation
Equipment	15 (9.3%)	Human-computer interface problem
Patient	4 (2.5%)	Ineffective communicationPatient id not verified
Policy and procedure	132 (81.5%)	Policy nonexistentPolicy not followed
Treatment	98 (60.5%)	Timeout-verification of approvals and documents not done†Timeout-patient identification verification not completed
Staff	144 (88.9%)	Comprehension issuesInadequate human resourcesNegligencePoor judgmentStaff rushedStaffing inadequate
Training	10 (6.2%)	Inadequate knowledge or skillsStaff not adequately trained
Work environment	158 (97.5%)	Increased workloadNot condusive to safetyPhysical environment inadequateStaff distractionsStaff interruptionsStaff lack of attentionStaff lack of information

Abbreviation: RO-EMR = radiation oncology electronic medical record.

Verification during pretreatment check (by either a therapist or a physicist) not performed according to defined processes.

Verification right before treatment delivery not accroding to defined processes.The results of contributing factors counts by categories are also shown.

Table 3

Summarized correlations between various contributing factors and the RO-EMR based LO categories

		RO-EMR input side	RO-EMR output side	Workflow specific	Network, linac, computer hardware issues	Other patient safety
Contributing factor category and specific items
Documentation	Verification of approvals and documents not done	X
	Unclear or incomplete setup documentation	X
Equipment	Human-computer interface problem				X
Policy	Policy not followed		X
Treatment	Timeout-verification of approvals and documents not done	X	XXX
Staff	Poor judgment					XXX
Work environment	Not conducive to safety		XX

Abbreviations: LO = learning opportunities; RO-EMR = radiation oncology electronic medical record; X = somewhat related; XX = mostly related; XXX = highly related.

Discussion

Study uniqueness

The process of radiation therapy is complex particularly owing to the number of steps and staff involved, as each hand-off or data transfer presents an opportunity as the beginning of the LO pathway. Many studies have investigated radiation therapy safety, analyzing events and risk factors to identify potential weak links in the process.,18, 19, 20, 21, 22, 23, 24 Even though many of the conclusions from these studies still apply to current practice, it is important to consider the changes in the radiation oncology field with modern technologies and how they are affecting our daily workflow to treat patients. Although a previous study by Patton et al investigated the types of events related to RVS in 2003, no other recent report has looked into the relationship specifically between modern RO-EMR and the LO in RO. Since 2010, ILS started drawing attention in the RO community as a mechanism to promote patient safety and quality of care, as an ILS can be used to identify vulnerabilities in the processes and provide opportunities to minimize potential future LO. In our study, we demonstrate that an ILS-based investigation that includes events’ relationship to the RO-EMR can help us identify more specific and meaningful workflow deficiencies. Although our concepts or results may not be novel, we hope and believe sharing our experiences could help reiterate the potential weak links in the daily RO clinical workflow involving RO-EMR and bring continuous awareness to the field.

RO-ILS event reporting differences and promoting safety culture

Similar to other early ILS findings, when the internal event report system was established in our department initially, only the LO cases that occurred at the machine or when the treatment course had started were reported and well documented. Therefore, most of the LO in this study occurred at the machines during v-sim or treatments as opposed to the RO-ILS report where most events occurred during treatment planning steps. In comparison, our institution observed less LO related to treatment planning or other processes before patients’ arrival at the machines. The majority of the LO recorded during the study period were entered and documented by the therapist group, while no dosimetrist or physician reports were recorded. Since late 2018, our institution has joined RO-ILS and established a workflow where everyone can submit LO from any workstation to encourage more LO reporting in every stage of the process hoping to remove barriers to incident reporting. With the change of LO reporting mechanism, we are expecting to see more LO reported upstream before they reach the patients.

Aside from creating easy access for LO report entry, we also worked to create transparent and nonpunitive environments to encourage incident reporting. For example, we established a “great catch” program to promote the safety culture in the department. Looking forward, adding a Safety_II perspective29, 30, 31 could provide additional opportunities to enhance the safety culture initiatives.

Best opportunities for LO prevention

We have found that information flow through the RO-EMR interface whether on the input or output side contributed to the majority of the LO in this study. The accuracy of the information flow through the RO-EMR is critical as the RO-EMR serves as the central information repository in RO; and information either entered (input side) or executed (output side) in an unintended manner creates LO.

With the current RO-EMR setting and functionality in our clinic, we found the best mitigation strategy to prevent input LO is to establish an additional layer in the chart checking process, therapist chart check, as a required step before starting any new treatment. The therapist chart check step reviews the patient's treatment delivery plans from a different point of view than the physicist plan check/review. Although the physicist plan check/review focuses on the accuracy and quality of the plan itself, the therapist chart check emphasizes more on the completeness and readiness of the plan of treatment. For example, in the therapist chart check checklist, the readiness of treatment related accessories in the treatment room is verified. Having the additional layer of therapist chart check in the processes provides opportunities for us to ensure all required documents and equipment are ready before the patient arrives at the machine. Consequently, we minimize the risk for the machine therapists having to spend time fixing the issues under time pressure, which could increase LO possibilities with the contributing factors listed under the staff or work environment categories (Table 2). A sample therapist chart check checklist is shown in Table 4. We believe the therapist chart check step has the highest potential to eliminate most of the input events as many of the check items are designed to catch the high frequency input errors.

Table 4

Therapist chart check detail checklist items for new patient treatment preparation

Therapist chart check: New patient treatment checklist
Consent signed by the patient
Prescription approved by the attending
Treatment plan document approved by the attending
Physicist plan check completed
Face photo in RO-EMR*
Treatment related accessories ready in treatment room*
Simulation photo for treatment position setup in RO-EMR*
Patient treatment position steup related documentation clear and understandable*
Check existance of previous treatment tatoos*
Treatment devlivery parameter accurate
Digital reconstructed radiographs display and associated to the correct fields
Skin rendering available when appropriate*
Special instruction entered accordingly when necessary*
Image guidance instruction appropriate and approved
Additional action needed for special physics consult
IMRT QA performed and approved
Isocenter coordinate correct for image guidanded modalities (eg, CBCT)
Dry run performed when appropriate
Treatment delivery related third party software information/entry ready (eg, surface imaging modalities, ultrasound imaging guided systems, etc)*
Is this an inpatient?
Is plan ready 3 hours before scheduled time?
Will the patient receive chemotherapy as well?
Does the patient have transportation issues?
Does the patient need an interpreter?
Insurance preauthorization treatment technique and number of treatment fractionations verified

This is a required step before any patient treatment course can be started in our institution. Items denoted with an asterisk (*) are related to some of the high frequncy input LO found in this study.

Abbreviations: CBCT = cone-beam computed tomography; LO = learning opportunities; RO-EMR = radiation oncology electronic medical record; IMRT = intensity modulated radiation therapy; QA = quality assurance.

On the output LO prevention side, we believe that a detailed and specific therapist time out procedure at the machine is critical to help eliminate many output related LO found in this study. In the proposed procedure (Table 5), the therapists are required to perform different time out items at different patient encounter time points: pretreatment time out, patient time out, and treatment delivery time out; and record the time out action as part of the daily treatment record in the RO-EMR. The pretreatment time out is carried out before the patient is brought back to the console. This time out gives the therapists the opportunity to get an overview of the current treatment status of the patient. The special instructions functionality is used in our institution as a mechanism to communicate any changes requiring special attention by any of the team members during a treatment course, which is why it is specifically listed out in the pretreatment time out. This is the last time point the therapists could make sure no changes have been made and/or are needed for the current treatment fraction. Examples of special instructions are (1) removing bolus for treatment from a certain fraction and onward, (2) patient specific dosimetry measurement requests, and (3) physician to see the lesion in the treatment position. The patient time out is the routine patient identification verification as per United State Nuclear Regulatory Commissioning regulation to ensure the correct patient has been brought to the treatment area. The treatment delivery time out is the time out to verify the treatment field parameters listed in the approved plan in the RO-EMR match the linac parameters including the treatment accessories correct placements.

Table 5

Detail therapist time out items at various patient encounter time point

Therapist patient encounter time out
Pretreatment time out: Before bringing the patient back to the treatment console
Patient name
Dose site summary: Prescribed number of treatment and fraction number/total fractions
Special instructions related to current treatment course: current and new items*
Imaging guidance instruction*
Patient time out: Verbally confirm the patient's name and date of birth before bringing the patient into the treatment procedure room
Treatment delivery time out: Before treatment begins on each treatment field
Treatment site and field number
Energy to be delivered
Monitor units to be delivered for the field(s)
Start and stop gantry angles when applicable
Treatment accessory or device to be used on the field (bolus, wedge, etc)*
Detail field parameters (gantry angle, collimator angles, x-/y- jaw setting, etc) for SRS treatments
Additional time out checklist will be in the RO-EMR chart for special procedures such as TBI and TSE

The therapists are required to perform all steps for all patient treatments, and document in the RO-EMR that all time outs are done according to defined procedure. Items denoted with an asterisk (*) are related to some of the high frequncy RO-EMR output LO found in this study.

Abbreviations: LO = learning opportunities; RO-EMR = radiation oncology electronic medical record; TBI = total body irradiation; TSE = total skin electron beam therapy; SRS = stereotactic radiosurgery.

Lastly, although not directly linked to the RO-EMR output LO, continuous optimization of the document layout within the RO-EMR also provides opportunities for LO prevention. For instance, if we simplify and standardize a one-page display in the RO-EMR for pretreatment time out, we could eliminate the number of various locations within the RO-EMR the therapists have to navigate manually to perform the time out task, therefore reducing the risk of RO-EMR output LO. As suggested elsewhere, this practice will minimize the load on problem solving and reduces reliance on memory, and therefore reduce the likelihood of errors. An example of the Pre-Treatment Time Out display in one view from our RO-EMR is shown in Fig. 3.

Figure 3

An example of single page display for pretreatment time out. This example contains the patient name, date of birth, prescription/dose site overview, special instruction, and imaging guidance instruction for the treatment site.

In summary, the best opportunities to prevent RO-EMR related LO based on this study were (1) building in the therapist chart check as a required step before starting any patient treatment, (2) designing a rigorous time out procedure at different patient encounter time point, and (3) continuously working to achieve effective RO-EMR document display.

Current RO-EMR products deficiencies

RO-EMR usage and number of interactions between the RO-EMR and the user have increased drastically in recent years. In our present RO-EMR workflow, most, if not all, of the input and output steps are performed manually, which can be improved easily if appropriate automation solutions exist. Other RO studies have shown that automatic functions from the linac and the RO-EMR help prevent LO from occurring.,32, 33, 34, 35 In our study, 95% of the events were human-error related. In our present clinical practice, a lot of treatment related data are manually entered by the therapists during v-sim and this data becomes the baseline for the subsequent treatment sessions. An earlier study by Barthelemy-Brichan el al also demonstrated the LO caused by manual entry and the importance of first treatment entry as this is the beginning of possible LO pathway. Similar to our findings, other studies also showed the importance of accurate manual processes either at the treatment plan preparation stage or execution by the therapists at the machine., Among the RO-EMR related LO we studied, 63% were from output LO, which meant the setup or documentation in the RO-EMR were correct but not executed accordingly by the therapists. It is worth mentioning that many LO in this study were caused by failure to follow policy. In our present RO-EMR environment and workflow design, there are only a handful of functions in the RO-EMR can be used as a forcing function to follow our policy, which meant the LO caused by failure to follow policy most likely were executed as a manual step. Therefore, as a department, we should look for ways to implement automatic mechanisms to stop the process step, based on policy defined. Based on our results, 2 of the best opportunities to eliminate LO using automated functions are (1) an automated mechanism in the RO-EMR that ensures the treatment related accessories/devices (eg, bolus, immobilization device) are used during treatment based on the plan and recorded back accordingly in the RO-EMR; and (2) an automated RO-EMR solution that understands the patient geometry and its relationship to the machine isocenter that can replace manual patient setup based on the manually entered setup documentation, photos, or shifts. Although no single product or software can resolve these shortcomings of manual processes, possible solutions do exist in the market, mostly as third party solutions. Currently, a limited number of products is available in the market that has the potential to help eliminate most of the treatment position setup and accessory related errors, as concluded in another study. However, the integration of the third party product with the RO-EMR remains a challenge. In some cases, to apply a third party product with current RO-EMR workflows may require more manual steps, which creates other risks in the processes or reduced efficiency. The radiation oncology department should consider these factors when looking into possible solutions to overcome the manual processes. Furthermore, when considering automatic solutions to implement in the clinic, it is imperative to understand that the increasing reliance of automation could decrease staff awareness., A careful evaluation, staff training, redesign of process, and implementation of new policies are warranted steps before any new automatic hardware or software solution is deployed.

Recognizing some of the deficiencies in the RO-EMR workflow and functionality, we also urge the RO-EMR manufacturers to identify high-risk areas for product improvement consideration. As pointed out also by Horsky, it is important for vendors to adapt their design practice to include formal user-centered, interactive design processes. The resulting product should derive from clinical environment evidence, for example this study's findings, or insights gained by observing clinicians’ daily practice in the real clinical environment. As a community, we can only achieve high quality patient care by continuous quality improvement projects by the RO department and collaboration with the manufactures for more efficient, safer, and innovative RO-EMR products.

Conclusions

Based on the LO documented in the internal ILS, most of the LO that occurred in our institution were related to the RO-EMR. The high-risk areas were identified to be most related with the manual steps either in data entry (input) or treatment execution (output). The RO-EMR input LO can be prevented by the pretreatment therapist chart check, and the RO-EMR output LO can be prevented by a rigorous therapist timeout, along with effective RO-EMR document display. The RO department should perform regular quality improvement reviews and policy updates as technology improves or workflow changes. Last but not least, automated functions in RO-EMR remain highly desirable.