A new approach of assessing patient safety aspects in routine practice using the example of "doctors handwritten prescriptions".

AIMS AND
OBJECTIVES: To test the method of self- and external assessment as a feedback system to decrease illegibility and incorrectness of handwritten prescriptions and to reduce additional workload for nursing staff.
BACKGROUND: Illegibility and incorrectness of handwritten prescriptions occur very often and are the most crucial factors affecting patient safety.
DESIGN: Self- versus external assessment using a 15 items checklist.
METHODS: Nurses randomly selected five fever charts of their wards. Each fever chart was self- as well as externally assessed. Nurses and doctors took part in the self-assessment, and the external assessment was performed by external experts. According to a monitor suspension system, assessment results were considered "green," "yellow" or "red." After the first assessment and issuing feedback of the results "red" scored wards by the external assessment, additional trainings were performed. Thereafter, a second assessment was performed to rate eligibility and completeness of prescriptions. The research and reporting methodology followed squire 2.0.
RESULTS: In total, 580 fever charts were self- as well as externally assessed (290 in each of the two assessment periods). Out of the 58 participating wards, 31 were surgical and 27 were non-surgical wards. Averaging over all checklist items, surgical and non-surgical wards improved only slightly over time. Linear regression models for ward means showed that there were significant improvements over time for non-surgical wards.
CONCLUSIONS: This method directly involves those who commit errors and stimulate learning from errors. The approach of self- and external assessment was a useful instrument to detect inadequate prescriptions and to monitor improvements. RELEVANCE TO CLINICAL PRACTICE: Significant improvements were achieved regarding correctness and legibility of handwritten prescription and helped to decrease additional workload for nursing staff and thereby enhanced patient safety.

What does this paper contribute to the wider global clinical community?

Incorrect prescription forces nursing staff to check with doctors more frequently incorrect prescriptions and thus lead to an additional workload of both professional groups.

To stimulate quality improvements, the method of self‐ and external assessment can have great impact for practice.

Nursing staff randomly selected five fever charts and self‐assessed together with doctors all prescriptions against a checklist. An external assessment was used to assess the accuracy of the self‐assessment and to provide objective feedback.

Significant improvements were achieved regarding correctness and legibility of handwritten prescription and helped to decrease additional workload for nursing staff and thereby enhanced patient safety.

INTRODUCTION

Handwritten medication prescriptions still represent a common practice in hospitals worldwide. Prescription may be illegible as well as incomplete in terms of missing prescriber initials, missing or wrong dosage, missing or wrong frequency, and missing or wrong route of administration (Brits et al., 2017; Gommans, McIntosh, Bee, & Allan, 2008; Naik, 2016; Thirumagal, Ahamedbari, Samaranayake, & Wanigatunge, 2017). It is generally known that illegible handwriting among doctors is prevalent. It is also one of the most important factors affecting patient safety and results in adverse clinical outcomes (Brits et al., 2017; Bruner & Kasdan, n.d.; Michaelson et al., 2017; Naik, 2016; Roy, Bhunia, Das, Dhar, & Pal, 2017; Scanlin, 2013; Sokol & Hettige, 2006). Annually, around 7,000 mortalities have been reported due to medication errors, and it is reasonable to assume that this is only the tip of the iceberg (Sheikh, Mateti, Kabekkodu, & Sanal, 2017). Medication errors occur due to a lack of knowledge, a poor performance, and/or psychological lapses (Sheikh et al., 2017). Poor performance is associated with poor legibility and incompleteness of handwritten prescriptions (Jadhav et al., 2017).

BACKGROUND

Besides doctors who know what is written, nurses or pharmacists for example have to subsequently deal with illegibility and incompleteness of prescriptions when dispensing medication (Brits et al., 2017). Illegibility and incompleteness are preventable, either by a change of habit on the part of prescribing doctors or through the digitization of the prescription process (Boehringer, Rylander, Dizon, & Peterson, 2007; Vaughn & Linder, 2018). Electronic prescription is often considered the superior alternative to avoid illegible and incomplete handwritten prescriptions. While the use of an electronic medication prescription system demonstrably led to fewer errors, it has been shown that medication errors were more severe in these cases (Hinojosa‐Amaya et al., 2016).

Previously, the most frequently used methods to detect handwritten prescription errors were chart reviews of pharmacists, direct observation during medication dispensation by a nurse or pharmacist, incident reporting, or auditing (Akoria & Isah, 2009; Al‐Khani, Moharram, & Aljadhey, 2014; Dean, Schachter, Vincent, & Barber, 2002; Dean Franklin, Reynolds, Atef Shebl, Burnett, & Jacklin, 2011; Naik, 2016). However, these methods did not directly involve the prescribing doctor; rather, they focused on the medication process thereafter. Doctors were thus not confronted with their illegible and incomplete prescriptions and had no opportunity to learn about, and from, their prescribing errors.

The aim of this study was to test the method of self‐ and external assessment sketched above as a feedback system for doctors, using the example of handwritten prescriptions. A second objective was to determine the self‐assessment ability of doctors through external assessment. Finally, we investigated whether the method of repeated self‐assessment and external assessment of illegible and incomplete prescription had any effect on the quality of handwritten prescriptions in fever charts over time.

METHODS

Checklist for self‐ and external assessment

Applicable legislation defines that a prescription has to be completed by a doctor; however, legislation does not define which items are necessary for a correct and complete prescription (Bundesgesetz, 1998). Therefore, an in‐house standard operating procedure (SOP) for correct prescription on fever curves was prepared by experts and deliberates known causes of prescribing errors (Boehringer et al., 2007; Dean et al., 2002). The self‐ and the external assessments of handwritten prescription on fever charts were performed according to a checklist with 15 items to determine the legibility and completeness of prescriptions.

Are prescription legible written on the fever chart?

Are prescription written with a waterproofed pen?

Is the available generic medication prescribed on the fever chart?

Are medications prescribed with the dosing including the unit? (true: 15 mg; false: 1 vial)

Are medications prescribed including the concentration? (true: 500 mg in 100 ml NaCl 0.9%; false: ad short infusion)

Is the time interval of administration prescribed correctly? (true: 1–0–1–0; false: 2 × 1)

Are medications written out? (true: Neodolpasse; false: NDP)

Are prescriptions signed by doctors?

Are changes of a prescription signed by doctors?

Are prescriptions crossed out in case of changes so that they remain legible?

Are medications only prescribed in the right column?

In one line there is only one medication prescribed?

Are medications prescribed for the next day?

For pro re nata medication the reason is defined (e.g., upon pain, upon vomiting)?

For pro re nata medication the daily maximum dose is defined (e.g., up to three times per day, every 8 hr)?

In order to secure reliability, the checklist was pre‐tested with doctors and nurses. To ensure validity, all items were cross‐checked against the in‐house SOP.

Each checklist item was rated on a 4‐point Likert scale, with categories “fulfilled,” “partially fulfilled,” “to some extend fulfilled,” and “not fulfilled” coded as 1–4. A rating of “1” for a particular checklist item implied that for all medications on the fever chart the respective checklist item was fulfilled, whereas “2” to “4” implied that the checklist item was not fulfilled for exactly one, exactly two, or more than two medications on the fever chart, respectively. In cases where it was not possible to evaluate a checklist item, there was also the possibility to tick “not applicable.”

Self‐assessment in a team‐based process

For increased attention of the prescriber, self‐assessment of doctors was implemented in our university hospital. However, in the past, it has also been demonstrated that doctors have problems with adequate self‐assessment (Davis et al., 2006; Glisson, Morton, Bond, & Griswold, 2011). To increase the accuracy of a self‐assessment tool, it therefore seemed necessary that another healthcare professional, for example a nurse, joined the assessment. The self‐assessment of prescriptions was thus designed as a team‐based process, with a nurse joining the doctor to complete the self‐assessment in dialogue.

In 2017, each head of a department/division was obliged by the management to enforce “legible and complete prescriptions.” As doctors are responsible for the prescription on the fever chart, all doctors were required to complete a provable training of the in‐house SOP defining the correct prescription of medication. After doctors’ training, the managing ward nurse of each of the wards randomly selected fever charts of the respective department/division which included the prescription of the doctor. As an average ward has 25 beds within the hospital, a sample size of 5 (~20%) fever charts was assumed as a representative number resulting in 290 fever charts for each of the assessment periods. The medical department/division head of each respective department nominated a doctor for the self‐assessment. These five fever charts of each ward were then self‐assessed by the nominated doctor and the managing ward nurse using the checklist. For each fever chart, one checklist was to be filled out and the ID‐number of the fever chart as well as the name of the assessing doctor and nurse was to be recorded. Upon completion, a copy of the five fever charts together with the filled‐out self‐assessment checklists was forwarded to the Executive Department for Quality and Risk Management (Supporting Information Appendix S1).

External assessment

To address the poor self‐assessment ability of doctors, an external assessment by experts was also put in place. Through this external assessment, the accuracy of the self‐assessment could be tested, as the illegibility and incompleteness scores of prescriptions were objectively determined and reported back. Each fever chart was assessed by two independent experts with a nursing background who were affiliated with the Executive Department for Quality and Risk Management. As in the self‐assessment, each checklist item was rated according to the 4‐point Likert scale. Results of the self‐assessment and the external assessment of all fever charts were then transferred into an electronic database (EvaSys version 6.0, Healthcare Survey Automation Suite; Electric Paper Evaluationssysteme GmbH, Lüneburg, Germany). Automated reports with results for each ward were generated, which were forwarded to the respective head and divisions/departments as well as to the self‐assessment team. The report included mean values for the 15 checklist items (Supporting Information Appendix S2).

According to a monitor suspension system, self‐ and external assessment results were considered “green” if the mean value for all 15 checklist items was between 1.0–1.4, “yellow” if the mean value was above 1.4 but below 2.0, and “red” if the mean value was at least 2.0. Each medical department/division head as well as the lead nurse received an Email containing the results according to the monitor suspension system (Table 1). Doctors in departments/divisions that were rated “red” by the external assessment team subsequently received additional training by the Executive Department for Quality and Risk Management together with the Legal Department. If the rating was “green” or “yellow,” no further training was offered.

Table 1

Monitor suspension system [Colour table can be viewed at http://www.wileyonlinelibrary.com]

Re‐evaluation process

The procedure described above was repeated 6 months after the first assessment period to detect any improvements (Figure 1). The time period of 6 months between the first and the second assessment was influenced due to the number (a) of fever charts which had to be assessed externally and (b) of trainings for departments/division that were rated “red.” If a department/division was still considered “red” after the second external assessment, the respective department/division was obliged to further work on their prescription habits in the following year using the same procedure (Figure 1). In 2018, 10 of 31 surgical wards still have to perform self‐ and external assessments.

Figure 1

Timeline of interventions

Data analysis

To quantify the agreement between the self‐assessment and external assessment on the 4‐point Likert scale, the per cent agreement and the concordance coefficient according to Kendall, which can take into account the ordinal nature of the data, were calculated at each assessment period for each checklist item separately. Dependencies among the observations, due to the fact that several fever charts come from the same department/ward and were thus self‐assessed by the same team, were ignored in this part of the analysis.

In order to determine whether there are differences in adherence to the 15 criteria for legible and complete prescriptions between surgical and non‐surgical wards as well as whether the additional training led to improvements, we considered linear models for each checklist item separately. Since it is assumed that the external assessments by the Executive Department for Quality and Risk Management were objective, we used the mean external ratings for the respective assessment period in each ward as dependent variables in the models. We considered type of ward (surgical or non‐surgical) and assessment period (initial assessment or re‐assessment) as independent predictors and checked for interaction between these two parameters. In case it was significant, the results (estimates along with their 95% confidence interval) are presented not only for the two main effects but also for their interaction. To allow for easier interpretation of the results, the data are also graphically summarised via boxplots.

For the monitor suspension system, the ratings within a ward and assessment period were obtained by averaging over all 15 checklist items of the five fever charts. The resulting score was then categorised into (using mathematical bracket notation; square ones include the stated value, whereas round ones exclude it) “(1, 1.4)” (i.e., between 1–1.4), “(1.4, 2)” (i.e., larger than 1.4 but less than 2) and “(2, 4)” (i.e., between 2–4), and the number of wards in the respective category was descriptively summarised.

We considered “not applicable” as missing and a p‐value of <0.05 (i.e., the respective 95% confidence interval excluding 0) statistically significant due to the explorative nature of the study. All analyses were performed using r version 3.4.4.

Ethical considerations

The study was approved by the Ethics Committee of Medical University of Graz (vote#: 30‐217 ex17/18). The research and reporting methodology followed squire 2.0 (Supporting Information Appendix S3).

RESULTS

In total, 580 fever charts were self‐ as well as externally assessed (290 in each of the two assessment periods). Out of the 58 participating wards, 31 were surgical and 27 were non‐surgical wards.

Self‐ versus external assessment

As shown in Figure 2, differences emerged between self‐ and external assessment in both assessment periods. According to the monitor suspension system, overestimations of self‐assessors were seen at both assessment periods. Eleven (35.5%) surgical wards rated themselves between 1.0–1.4 at the first assessment (vs. 5 [16.1%] according to external evaluation) and 16 [51.6%] at the second assessment (vs. 8 [25.8%] according to external evaluation). Similar results were seen for non‐surgical wards (14 [51.9%] vs. 4 [14.8%] at the first assessment and 20 [74.1%] vs. 11 [40.7%] at the second). Accordingly, “red” ratings (mean rating between 2–4) were more frequent among external assessors, except for non‐surgical wards at the re‐assessment where there were no such ratings. Averaging over all checklist items, both surgical and non‐surgical wards improved only slightly over time (mean [SD]; surgical: 1.91 [0.42] to 1.76 [0.46], non‐surgical: 1.77 [0.37] to 1.50 [0.22]).

Figure 2

Ward ratings according to the monitor suspension system (n, %) of self‐ and external assessment for the first and second assessment period for surgical and non‐surgical wards [Colour figure can be viewed at http://www.wileyonlinelibrary.com]

However, it can also be seen that some improvements were achieved at the re‐assessment (Figure 3). According to linear models for each checklist item, both surgical and non‐surgical wards improved significantly from the first to the second assessment for checklist items 1, 6 and 11–13. Regarding items 6, 7, 13 and 14, we also found significant differences between surgical and non‐surgical wards, with the latter always being rated more favourably by the external assessors. Furthermore, non‐surgical wards showed significant improvements for item 2, leading to a significant difference between the ward types at the re‐assessment (Table 2).

Figure 3

Self‐ versus external assessment in surgical and non‐surgical wards for the first and the second assessment period for each of the 15 checklist items

Table 2

Linear models for each checklist item according to mean external assessment results per ward to determine differences in adherence to the 15 criteria for legible and complete prescriptions between different ward types as well as change over time

Checklist Item	Difference between ward typea	p‐value	Change over timeb	p‐value
Are prescription legible written on the fever chart?	−0.14 [−0.35, 0.06]	0.171	−0.32 [−0.53, −0.12]	0.002
Are prescription written with a waterproofed pen?c	0.52 [0.18, 0.87]	0.003	−0.08 [−0.41, 0.25]	0.634
Is the available generic medication prescribed on the fever chart?	−0.15 [−0.36, 0.07]	0.176	−0.16 [−0.38, 0.05]	0.129
Are medications prescribed with the dosing including the unit?	−0.21 [−0.49, 0.06]	0.127	−0.18 [−0.45, 0.10]	0.207
Are medications prescribed including the concentration?	−0.27 [−0.60, 0.06]	0.113	−0.18 [−0.51, 0.15]	0.270
Is the time interval of administration prescribed correctly?	−0.49 [−0.84, −0.14]	0.006	−0.41 [−0.76, −0.07]	0.020
Are medications written out?	−0.13 [−0.23, −0.03]	0.014	−0.03 [−0.13, 0.07]	0.603
Are prescriptions signed by doctors?	−0.18 [−0.57, 0.21]	0.357	−0.23 [−0.62, 0.15]	0.234
Are changes of a prescription signed by doctors?	0.00 [−0.35, 0.35]	0.990	−0.26 [−0.61, 0.09]	0.141
Are prescriptions crossed out in case of changes so that they remain legible?	−0.02 [−0.28, 0.24]	0.871	−0.12 [−0.38, 0.14]	0.359
Are medications only prescribed in the right column?	−0.05 [−0.28, 0.17]	0.631	−0.23 [−0.46, −0.01]	0.039
In one line there is only one medication prescribed?	−0.09 [−0.22, 0.03]	0.132	−0.14 [−0.26, −0.01]	0.031
Are medications prescribed for the next day?	−0.53 [−0.82, −0.23]	<0.001	−0.43 [−0.72, −0.13]	0.005
For pro re nata medication the reason is defined?	−0.77 [−1.00, −0.54]	<0.001	−0.09 [−0.32, 0.14]	0.419
For pro re nata medication the daily maximum dose is defined?	−0.13 [−0.35, 0.08]	0.212	0.04 [−0.17, 0.25]	0.731

Results are presented as “effect [95% confidence interval].”

Effect for non‐surgical wards when compared to surgical wards;

Effect for re‐assessment when compared to initial assessment;

With significant interaction (−0.72 [−1.21, −0.24]; p = 0.004), otherwise model was reduced to additive only.

Agreement between self‐ and external assessors

The degree of agreement between the self‐ and external assessors in both assessment periods can be seen in Table 3. Overall, the percentage of agreement increased but concordance did not. This means that with the general improvements in the re‐assessment more agreement would have been expected to occur by chance.

Table 3

Agreement between self‐ and external assessment of each fever chart (agreement [%] and concordance coefficient W) for each of the two assessment periods

Checklist items	Initial assessment period			Re‐assessment period
	N of charts	% agreement	W	N of charts	% agreement	W
Are prescription legible written on the fever chart?	290	52.8	0.457	287	67.6	0.340
Are prescription written with a waterproofed pen?	246	69.9	0.357	220	85.9	0.190
Is the available generic medication prescribed on the fever chart?	64	73.4	0.269	62	83.9	0.102
Are medications prescribed with the dosing including the unit?	286	57.3	0.549	284	61.3	0.474
Are medications prescribed including the concentration?	175	53.1	0.703	157	54.1	0.657
Is the time interval of administration prescribed correctly?	275	54.9	0.672	276	57.6	0.629
Are medications written out?	286	78.3	0.302	284	86.6	0.272
Are prescriptions signed by doctors?	280	52.5	0.614	289	64.0	0.645
Are changes of a prescription signed by doctors?	198	41.4	0.552	183	41.0	0.480
Are prescriptions crossed out in case of changes so that they remain legible?	100	41.0	0.462	92	42.4	0.401
Are medications only prescribed in the right column?	283	78.1	0.432	285	89.8	0.358
In one line there is only one medication prescribed?	289	84.8	0.220	283	91.5	0.123
Are medications prescribed for the next day?	226	63.7	0.485	227	78.4	0.366
For pro re nata medication the reason is defined?	219	72.1	0.591	224	76.8	0.529
For pro re nata medication the daily maximum dose is defined?	218	69.7	0.527	220	67.3	0.574

DISCUSSION

This is the first study to use an approach combining self‐ with external assessment to investigate specific patient safety relevant aspects in the clinical practice of healthcare experts, especially doctors. The combined approach was used to evaluate handwritten prescriptions using 15 checklist items, the fulfilment of which is a prerequisite for legible and complete handwritten prescription. Prescriptions were assessed from two perspectives, an internal and an external expert evaluation. Assuming that the external assessment by nursing experts of the Executive Department for Quality and Risk Management represented the objective, true result, overestimations were revealed among self‐assessors. Legibility and completeness of prescriptions increased over the evaluation period for both ward types, with non‐surgical wards in particular showing a significant learning effect for six items. Improvements regarding adherence to the SOPs from the first to the second assessment period were likely achieved due to training in prescription guidelines for doctors, the involvement of doctors in the self‐assessment and additional in‐house training by experts.

Proper prescriptions require legibility, completeness and traceability, and it is known that handwritten prescriptions are a major source of errors (Brits et al., 2017; Thirumagal et al., 2017). In the past, assessment of prescription errors focused predominantly on the dispensing process of medications, identification of error‐producing conditions or strengthening staff ability to speak up (Akoria & Isah, 2009; Dean et al., 2002; Pfeiffer, Gut, & Schwappach, 2018; Schwappach & Gehring, 2014). Therefore, in this study, we focused on doctors’ participation in assessing their handwritten prescriptions to raise their awareness of the important patient safety aspects of medication errors following illegible and incomplete prescriptions. The combined approach of self‐ and external assessment and its repeating format was the key determinant of this study. Doctors were first trained in correct prescribing habits according to an in‐house SOP and given the opportunity to reflect on prescription quality aspects together with a nurse of the ward. Self‐assessment in a team‐based approach (doctor and nurse) was chosen as doctors have been shown to have limited accuracy in self‐assessment (Davis et al., 2006). Although self‐assessment was performed in a team, our results indicate the possibility that nurses were overruled as overestimation compared to the external evaluation was frequently observed. Informal conversations of nurses confirmed the assumption that a doctor disregarded illegible handwriting as well as incompleteness of prescriptions in cases where a prescription was particularly inadequate.

In Austria, medication management in terms of dispensing is in the purview of the nurse. Therefore, nurses in particular face difficulties in dispensing medication, as too often information crucial for correct dispensation is missing or illegible. Therefore, our safety improvement approach required nurses of the respective ward to pick out five fever charts for each of the two assessment periods. We assumed that nurses would pick out badly written fever charts as they likely hoped for improvements. Our results revealed variations among wards and specializations, with surgical wards showing higher levels of illegibility and incompleteness than others (compare “red” ratings). However, legibility and completeness increased over the evaluation period for all ward types, with a higher learning effect of doctors in non‐surgical wards according to the monitor suspension system (“green” ratings increased from 16.1%–25.8% for surgical wards and from 14.8%–40.7% for non‐surgical wards). Moreover, all non‐surgical wards improved such that they complied with the in‐house limit of acceptability, defined as a rating below 2.0. According to the linear models, non‐surgical wards achieved better scores than surgical wards for checklist items 6, 7, 13 and 14, and there were improvements over time for checklist items 1, 2, 6 and 11–13. It seems that training and confronting doctors with their illegible and incomplete prescriptions was effective in changing habitual behaviour; however, illegibility still remains a surviving dinosaur, especially for surgical disciplines.

The used methodology directly involved those who commit errors and supports stimulation of learning from errors. It is important not to only tell healthcare experts how to improve; direct involvement of healthcare experts in patient safety is needed and is relevant concerning transformation into practice. The generated report for each assessment period gave a simple overview where improvements are needed. In order to handle the well‐known topic of limited accuracy in self‐assessment by doctors, nurses took also part in these self‐assessments. Though, it was likely that nurses were often overruled by doctors when there was a disagreement, the used team‐based approach facilitates patient safety as results of the re‐assessment showed improvements.

LIMITATIONS

A major limitation is that we cannot rule out any bias in the selection of fever charts by the nurses, which may have influenced the findings in both assessment periods.

CONCLUSIONS

Our approach combining self‐assessment with external assessment was a useful instrument to detect inadequate prescription practices. By way of an external assessment, healthcare experts also received objective feedback about their daily practice. Combined with targeted interventions such as training and further reflection on assessment results together with external experts, doctors’ awareness about such patient safety relevant aspects can be strengthened through this safety improvement approach and support as a direct outcome nursing processes.

RELEVANCE TO CLINICAL PRACTICE

The approach combining self‐ and external assessment was a useful instrument to detect inadequate prescriptions and to monitor improvements over a certain time period. Significant improvements were achieved regarding the completeness and legibility of handwritten prescription. Though improvements were obvious, it seems that illegibility and incompleteness remains a surviving dinosaur, especially for surgical disciplines. Results suggest the impact of repeated training and direct involvement to effect a change in habitual behaviour. Improvements also helped to decrease additional workload for nursing staff and thereby enhanced patient safety. Furthermore, the instrument of self‐ and external assessment can be easily transferred to other patient safety relevant topics and processes where it is possible to create checklists. Using the method of self‐ and external assessment has a great implication for practice as it involves those who commit errors in order to stimulate learning from errors. In terms of legibility and completeness of prescriptions, self‐ and external assessments will be repeated for those who were still rated “red” according to the monitor suspension system at the re‐assessment. For all others, samples will be drawn in order to control adherence to the SOP, which defines prescription of medication in our university hospital.

CONFLICT OF INTEREST

The authors declare that they no conflict of interest.

AUTHOR CONTRIBUTIONS

Study design and participation: GS, VG, KL, PT, CR and MH; data interpretation and contribution to discussion: LJ, GS, LPK and GB; statistical analysis: GP; and project supervision: GB.

Click here for additional data file.

Click here for additional data file.

Click here for additional data file.