A Clinical Decision Support System for Increasing Compliance with Protocols in Chemotherapy of Children with Acute Lymphoblastic Leukemia.

Objective: In this survey, a protocol-based Chemotherapy Prescription Decision Support System (CPDSS) was designed and evaluated to reduce medication errors in the chemotherapy process of children with ALL.
Methods: The CPDSS algorithm was extracted by the software development team based on the protocol used by doctors to treat children with ALL. The ASP.Net MVC and SQL Server 2016 programming languages were used to develop the system. A 3-step evaluation (technical, retrospective, and user satisfaction) was performed on CPDSS designed at 2 children's hospitals in Tehran. The data were analyzed using descriptive statistics. At the technical evaluation step, users provided recommendations included in the system.
Results: In the retrospective CPDSS evaluation step, 1281 prescribed doses of the drugs related to 30 patients were entered into the system. CPDSS detected 735 cases of protocol deviations and 57 (95%, CI = 1.25-2.55) errors in prescribed chemotherapy for children with ALL. In the user satisfaction evaluation, the users approved two dimensions of the user interface and functionality of the system. Conclusions: With the provision of alerts, the CPDSS can help increase compliance with chemotherapy protocols and decrease the chemotherapy prescribing errors that can improve patient safety.

Background

Leukemia is a type of malignancy that begins in hematopoietic tissues. This type of cancer accounts for about 8% of all human cancers and is recognized as the fifth most common cancer in the world. Acute Lymphoblastic Leukemia (ALL) is the most common type of cancer among children aged 1 to 16 years, and its peak incidence is in children aged 3 to 7 years.[1-5] The treatment of pediatric ALL is one of the most outstanding achievements achieved in cancer science.[6,7] Progress in the understanding and treatment of the disease in the 1990s led to approximately 90% improvement in the survival rate of pediatric ALL patients.[5,8] The primary treatment for children with ALL is chemotherapy that lasts 2 to 2.5 years. They receive chemotherapy based on complex chemotherapy protocols. The complexity of the chemotherapy protocols in children makes it vulnerable to errors. The complexity of the protocol, coupled with the potential for interrupting the implementation of the protocols (due to unstable physiological conditions), increases the probability and effect of errors and generates more critical complications in a vulnerable population (ie, Children).[10,11]

Chemotherapy errors are a significant concern for healthcare systems. Chemotherapy, as a specialized medication process, includes the risk of errors in the calculation of the drug dose in children, the risks associated with narrow therapeutic indexes, and the high potential for acute and cumulative toxicity of chemotherapy agents.[10,12,13] These errors can occur at any stage of the chemotherapy process (prescription, preparation, dispensing, and administration), especially in the prescribing stage.[14-17] In one comprehensive pediatric oncology medication error article, 74% prescription errors, 13.5% administration errors, 9.4% preparation errors, and 3.1% transcription errors were reported. More significantly, the risk of errors with the potential for harm is 3 times higher for children, especially those receiving chemotherapy. Institute of Medicine (IOM) has suggested using evidence to guide the best practice and minimize variation to achieve the goal of providing safe care for pediatric patients. Part of this evidence is protocols and therapeutic guidelines. Although chemotherapy protocols are the best evidence for cancer treatment, compliance with these protocols is low. One of the reasons for this low compliance with protocols is that oncologists need to keep track of many multi-dimensional and complex paper-based protocols (with limited user-friendliness) in inherently long-term chemotherapy process and to adjust them to each specific patient who appears to be difficult for them and leads to chemotherapy errors.[22,23]

It has been progressively recognized that some clinical decision support systems (CDSSs) have the potential to overcome the problems of paper-based protocols, improve compliance with chemotherapy protocols and decrease chemotherapy prescription errors.[21,24-27] As Collins et al pointed out: “Oncology is a very data-rich and protocol-driven specialty, making it perfect for the use of a CDSS.” Such CDSSs are coupled with CPOE (Computerized Provider Order Entry) can provide special alerts and recommendations based on standardized chemotherapy protocols for the patient at the time of the prescription and increase patient safety,[28,29] particularly while considering user interest in the early stage of the CDSS design process. Basically, the poor outline of a framework interface can promptly back off Healthcare specialists as they may invest more energy in looking for relevant data than in investigating suggestions.[24,30]

Specifications of chemotherapy prescription decision support system (CPDSS) such as dose/drug/order/regimen recommendation according to the respective patient, performing a complex calculation of the dose and adjusting the dosage based on the various parameters (such as cumulative dose, kidney, and liver function tests, calculations based on age, height, weight and etc.) And drug interaction, allergy, abnormal weight, height, BMI, and body surface area (BSA) alerts were described in a recent review study.

The purpose of the present study was to design a CPDSS that was evaluated using a 3-step retrospective evaluation study to assess its effect on compliance with protocols and user satisfaction of CPDSS.

Methods

The method includes software development, 3-step retrospective evaluation, and data analysis description.

Development

A multidisciplinary team consisting of 2 oncologists and 3 information system specialists experienced in CDSS developed the CPDSS algorithm based on the IC-BFM 2002 protocol. The rules defined in this protocol can be used as “IF-Then” in decision support systems. These rules are translated into the computer-readable format as the algorithms of CPDSS.

The CPDSS checks the orders with the standard protocol agreed by the departmental experts and warns if they do not comply. The doctor can correct this deviation or ignore the alert. If the user determines that deviation from the protocol is necessary, the system displays a message and asks the user to justify the reason for deviating from the protocol according to the patient’s particular circumstances. However, all the alert drug/order characteristics, such as the type of alert, the prescriptive response to the alert, reasons of rejected alert, dose, frequency, or other modified profile of the drug, are recorded in the system’s database.

The design and architecture of CPDSS were intended to check and verify the parameters listed in Table 1 and provide the necessary alerts.

Table 1.

CPDSS alerts.

Row	Parameters	CPDSS alerts
1	Drug	Drug deviation from the chemotherapy protocol occurs when:– The drug is not in compliance with the protocol– According to the protocol, the drug should not be prescribed because of the abnormality of some patient parameters such as white blood cell count (WBC), platelets, absolute granulocyte count, hemoglobin (Hb), CNS status, shortening fraction (SF), ejection fraction (EF), Creatinine/CLCr, infection, ALT/AST, and bilirubin
2	Dose	The dose of the drug is not in compliance with the protocol
3	The days of giving the chemotherapy drug	Administration days of the drug are not in compliance with the protocol
4	Unit	The unit of the drug is not in compliance with the protocol
5	Frequency	The frequency of the drug is not in compliance with the protocol
6	Route	The route of the drug is not in compliance with the protocol
7	Solvent	The solvent of the drug is not in compliance with the protocol
8	Rate	The rate of the drug is not in compliance with the protocol
9	Drug allergy	The patient had allergic reactions to the chemotherapy drug

The system was developed in ASP.NET MVC with SQL Server as a database management system.

Evaluation

This study was evaluated at 2 teaching children’s hospitals in Tehran with an oncology ward that admitted children up to 18 years of age.

Information required for the technical and the retrospective clinical validation was extracted from the paper records and consisted of all the children with ALL between the ages of 1 and 18. Six physicians treated these patients from January to December 2019. Patients had received chemotherapy according to Protocol I/I’Phase 1, Protocol I/I’ Phase 2, Protocol Mm/M of the IC-BFM 2002 main protocol. The software was carried out at the hospital for 6 months.

A validation strategy developed by Scheepers-Hoeks et al was modified and used for this study. The development and validation strategy developed by Scheepers-Hoeks consisted of 4 steps. The first step was a retrospective technical validation based on which CDSS was investigated to confirm that parameters are correctly linked to other related data. In the second step, an expert team examined all alerts in terms of clinical relevance, feasibility, and usefulness. In step 3, to ensure the accuracy of the alerts, CDSS was used in daily clinical practice. The fourth step was the CDSS improvement in practice, which was carried out by constant maintenance after implementing the system. In this study, the third to fourth steps were left out, and user satisfaction assessment of the CPDSS as another step was added to the evaluation process.

Technical evaluation

The technical evaluation step aimed to determine if the CPDSS algorithm worked technically and correctly. To perform this step, 4 oncologists examined the parameters and alerts in the algorithm.

Retrospective evaluation

At this evaluation step, the ability of the CPDSS to detect protocol deviation was studied using prescriptions of the study population. Initially, patients’ data such as the type of ALL, other diseases, allergies, height, age, weight, and patient tests during the treatment with chemotherapy were collected from paper records and entered into CPDSS. Subsequently, the chemotherapy prescriptions of these patients were entered into CPDSS; their compliance with the chemotherapy protocol was compared, and in case of non-compliance, the system alerted. The specifications of the alerts were stored in the system database.

User Satisfaction evaluation

Data related to user satisfaction were collected by a checklist in the form of a questionnaire in Persian. The questions of this questionnaire were designed based on previous studies and the opinions of a team of experts. The validity of the questionnaire was examined and approved by 5 medical informatics experts. The reliability of the questionnaire based on Cronbach’s alpha was estimated to be .90.

The questionnaire consisted of 2 parts (user interface, functionality) and 15 questions. User satisfaction evaluation of CPDSS was conducted on 2 aspects of user interface (8 yes/no questions) and functionality (7 yes/no questions). This questionnaire was distributed among 10 oncologists who had experience working with the CPDSS.

Data analysis

At first, the characteristics of patients were examined. Subsequently, the number of deviations from the protocol was determined for the parameters of the drug, dose, unit, frequency, route, solvent, rate, and date of drug delivery to the patient based on the alerts stored in the CPDSS database. In addition, the number of drug allergies alerts was examined. Two oncologists (who were different from prescribing oncologists) reviewed these alerts. The number of cases of protocol deviation and chemotherapy errors was determined, followed by an evaluation of users’ satisfaction with this system.

Ethical considerations

The medical center ethics committee approved the study and waived the need for specific patient consent due to its retrospective nature.

Results

Results of technical evaluation

In the technical evaluation of the software, three of the alert messages were corrected, changes were made to the user interface, and the warning was suggested if the doctor prescribed 10% more or less than the recommended dosage of the medication protocol.

Results of retrospective evaluation

In the retrospective evaluation step, 1281 doses prescribed for 30 patients (15 patients in each hospital) were entered into the system. The study was conducted on children aged 1 to 18 years old, of which the sample age was between 1 and 14. The number of drug doses in each protocol is summarized in Table 2.

Table 2.

Number of drug doses per protocol for all patients.

	Protocol I/I’ Phase 1	Protocol I/I’ Phase 2	Protocol Mm/m	Sum
Hospital A	255	198	195	648
Hospital B	240	198	195	633
Sum	495	396	390	1281

Table 3 presents the frequencies and percentages of alerts related to drug, dose, and day parameters in each of the treatment protocols for 30 patients. This Table shows that hospital A has 5 protocol deviations of the drug parameter from the Protocol I/I’ Phase 2, which is 0.076% of all prescriptions for this hospital in this Phase. Hospital B has 1 protocol deviation of the drug parameter from the Protocol I/I’ Phase 1 and one from the Protocol I/I’ Phase 2, which is 0.013% to 0.015% of its total prescriptions in each Phase, respectively. Also, CPDSS found 8 cases of the protocol deviation of the dose parameter in the Phase 1, 63 cases in the Phase 2, and 24 cases in the Phase 3, constituting 0.094%, 0.318%, 0.123% of the total dose in each Phase regarding the hospital A. For hospital B, 8 cases of the protocol deviation of the dose parameter were found in the first Phase, 24 cases in the Phase 2, and 30 cases in the Phase 3, constituting 0.1%, 0.121%, 0.154% of the total dose in each Phase. Furthermore, CPDSS found 126 cases of protocol deviation for day parameter in the Phase 1, 132 in the Phase 2, and 36 in the Phase 3, constituting 0.494%, 0.667%, 0.185% of total prescriptions of hospital A, respectively. Hospital B had 99, 108, 24 protocol deviation of day parameter from the first, second, third Phase, constituting 0.123%, 0.545%, 0.413% of the total prescriptions of this hospital prescriptions in each protocol respectively.

Table 3.

Frequency of the stored alerts in CPDSS database.

		Protocol I/I’ Phase 1		Protocol I/I’ Phase 2		Protocol Mm/m		Sum
		Number	%	Number	%	Number	%	Number	%
Hospital A	Drug	0	0	15	0.076	0	0	15	0.023
Hospital B		3	0.013	3	0.015	0	0	6	0.009
Hospital A	Dose	24	0.094	63	0.318	24	0.123	111	0.171
Hospital B		24	0.100	24	0.121	30	0.154	78	0.123
Hospital A	Day	126	0.494	132	0.667	36	0.185	294	0.453
Hospital B		99	0.413	108	0.545	24	0.123	231	0.364

With the implementation of CPDSS, oncologists prescribed unit, frequency, route, solvent, and rate of the drugs in compliance with the protocol. No patient allergy to the drug was mentioned in the patients’ files, and the system did not record any alert about this parameter.

Evaluation results of user satisfaction

The results of the user satisfaction assessment provided by 10 oncologists are shown in Tables 4 and 5. The CPDSS user interface was rated 0.86, and CPDSS functionality was 0.96, which were both acceptable. At this point, users offered suggestions like showing a preview of the selected section of the protocol on the CPDSS page and creating links to more resources in the alerts messages to increase user satisfaction.

Table 4.

The results of evaluating the user satisfaction of the CPDSS-User Interface.

Row	User interfaces	Yes	No	No reply	Total
1	Are the terminologies used for the icons and menus intelligible and understandable?	9	1	0	10
2	Are the information elements on the page well suited?	8	2	0	10
3	Is the display of alerts and reminders in the practical location of the page?	8	2	0	10
4	Do alerts in the system take place at the right time and place?	10	0	0	10
5	Are messages related to alerts and reminders intelligible and understandable?	7	3	0	10
6	Is it easy to enter data associated with the name of the drug, dose, unit, frequency, route, rate, solvent, and days of giving the medication?	9	1	0	10
7	Is there an option to automatically insert time and date in the system?	10	0	0	10
8	Do you think the system is generally user-friendly in terms of user interface?	8	2	0	10
	Sum	69	11	0	80
	Average	0.86	0.14	0	1

Table 5.

The results of evaluating the user satisfaction of the CPDSS-Functionality.

	Functionality	Yes	No	No reply	Total
1	Does the use of the system, given that it is protocol-based (guideline-based), facilitate decision-making for you?	10	0	0	10
2	Has the system been useful in your awareness of the treatment protocol?	9	1	0	10
3	Do the alerts given by the system notify you of the compliance process provided to the patient with the existing therapeutic protocol?	10	0	0	10
4	Is the system capable of reducing the errors associated with protocol deviation?	10	0	0	10
5	Is the system capable of reducing the errors associated with the name of the drug?	10	0	0	10
6	Given the fact that the system is a protocol-based chemotherapy prescription system, does it increase your confidence and subsequently improve patient safety?	10	0	0	10
7	Do you think the system is generally user-friendly in terms of functionality?	10	0	0	10
	Sum	69	1	0	70
	Average	0.98	0.02	0	1

Results of data analysis

In the data analysis step, during the meetings, all the alerts recorded in the CPDSS database by 2 oncologists were analyzed. Experts have appraised all the cases of deviation from the protocol found by the CPDSS. Besides, based on the patient’s situation and paper records, they determined which of the protocol deviations were errors. Table 6 demonstrates the number of dose errors broken down by protocol for each hospital.

Table 6.

Frequency of prescription dose errors.

	Protocol I/I’ Phase 1		Protocol I/I’ Phase 2		Protocol Mm/m		Sum		CI (α = .05)
	Number	%	Number	%	Number	%	Number	%
Hospital A	24	0.094	18	0.091	0	0	42	0.065	0.92-2.88
Hospital B	6	0.025	9	0.045	0	0	15	0.024	1.33-2.47
Sum	30	0.061	27	0.682	0	0	57	0.044	1.25-2.55

Abbreviation: CI, confidence interval.

According to experts, the CPDSS correctly found 42 medication errors occurred in hospital A, which accounted for 0.065% of the total doses prescribed by this oncologist (95%, CI = 0.92-2.88), as well as 15 drug dosing errors occurred in hospital B, which accounted for 0.024% of all doses prescribed in that hospital (95%, CI = 1.33-2.47). Based on the reasons given in Table 7, none of the deviations from the protocol in the drug parameter and drug day and 132 cases of the dosing parameter deviations that were detected by the CPDSS were documented as errors. CPDSS did not record deviations from the protocol for other parameters that led to an error. These findings showed that only the dose parameter in prescriptions for the 30 patients was erroneous; there was no error in other parameters of the chemotherapeutic agents.

Table 7.

List of reasons provided by oncologists.

1. The cumulative dose was more important than the single dose.2. The patient had weakness and lethargy.3. The dose has been reduced based on the general condition of the patient (height, weight, race, etc.)4. The patient had low WBC, but it was not registered in his/her paper record.5. The patient had allergic reactions to a high dose of the drug but did not register in his/her paper record.6. Drug poisoning occurred in a therapeutic dose.7. The patient had an infection.8. The patient had a fever.9. Because both drugs are hepatotoxic, the physician considers a 1-day interval between the two drugs (IC-BFM 2002).10. Patients missed appointments.11. From the doctor’s point of view, starting the drug from the First day of the week was more slightly.12. Starting the drug from that day leads to more compliance with the protocol (because the duration between doses was more important than delay)13. The outpatient clinic did not provide any service to patients on weekends.

Discussion

In this paper, a prescription chemotherapy CDSS for children with ALL was designed and evaluated. In the retrospective evaluation, 1281 doses of chemotherapy administered to 30 patients were prescribed with the CPDSS. Most of the 735 alerts recorded in the CPDSS database belonged to day, dose, and drug parameters with 525 (71%), 189 (26%), and 21 (3%), respectively. There was no alert for other parameters (including unit, frequency, route, solvent, rate, and allergy). As other studies had considered using the CDSSs to increase compliance with the treatment protocol,[21,28,33,34] the CPDSS accurately identified 100% of the deviations from the treatment protocol. The results of this study demonstrated the ability of the CPDSS to find cases of deviation from the ALL chemotherapy protocols in drug, dose, unit, frequency, route, solvent, rate, and day parameters for children.

Several studies have pointed to the role of various CDSSs in reducing drug errors.[10,24,35-38] Especially, CPOE coupled with CDSS has been able to significantly reduce therapeutic medication errors based on the rules derived from the therapeutic protocols.[26,38-41] The results of this study on alert cases deviating from the chemotherapy protocol by CPDSS showed that this system could help find chemotherapy drug errors. All errors in prescription chemotherapy drugs occurred in the dose parameter (57 cases [30%] of the 189 cases deviated from the protocol [95%, CI = 1.25-2.55]). This is also supported by other studies[14,42,43] in which dose errors were the most reported medication errors in the pediatric population, particularly at the stage of drug prescription.

The evaluation results related to user satisfaction from CPDSS also indicated that users were satisfied with the system based on 2 criteria of the user interface and system functionality. Even though in the evaluation of the previous step, many deviations from the chemotherapy protocol were not known as errors, especially in the case of the field of the drug and day, in assessing user satisfaction, from the perspective of a CPDSS user, the alert in these fields was useful. The users also confirmed the ability of the system to find errors in the prescription of chemotherapy drugs, especially regarding dose errors. They acknowledged that the system had played an active role in the decision-making process, and it has been instrumental in ensuring and subsequently improving patient safety. At this point, users provided suggestions to increase user satisfaction, such as showing a preview of the selected section of the protocol on the CPDSS page and creating links to more resources in alerting messages. As noted in other studies, by demonstrating scientific documentation for chemotherapy protocols, alerts can play a significant role in learning the treatment process with evidence-based medicine and, users accepted the role of CPDSS in teaching and decision-making, especially for less experienced physicians.

Limitations of this research include lack of control of some factors by CPDSS, such as the use of other drugs, the simultaneous implementation of different interventions, and the cumulative dose of chemotherapy drugs. The CPDSS only makes good decisions about patients who are not in exceptional circumstances and, if the patient is in a particular condition, CPDSS will be able to decide while instructing the physicians.

The shortage of resources in the hospital, such as minimum hardware needed for such programs (eg, tablets that can be carried by physicians easily), and a comprehensive hospital information system which provides complete information on the treatment of patients, were also the limitations of this research. Since these preliminary results are encouraging, further investigation with providing complete information for CDSS and a larger sample is recommended to generalize these results.

Conclusion

The overall outcome indicated the ability of the CPDSS to find cases of deviation from the chemotherapy protocol on drug, dose, unit, frequency, route, solvent, rate, and day of drug administered to the patients, which led to the discovery of prescription errors in the chemotherapy regimen. The system can contribute to increasing the matching of chemotherapy protocols by providing alerts that lead to an increase in patients’ safety.

The results of the user satisfaction evaluation showed that the CPDSS was acceptable in terms of user interface features. Additionally, the system’s function in contributing to decision-making, awareness of the protocol, reducing the errors in prescribing chemotherapy drugs, reducing drug name errors, and ultimately, users’ confidence and improved patient safety were accepted. The results show that the system can reduce the prescribing errors of chemotherapy.