Validation of a method to assess the severity of medication administration errors in Brazil: a study protocol.

BACKGROUND: Medication errors are frequent and have a high economic and social impact and is critical to know their severity. A variety of tools exist to measure and classify the harms associated with medication errors, but few are internationally validated. DESIGN AND METHODS: It was decided to validate a method proposed by Dean and Barber for assessment of the potential severity of medication administration errors. A number of thirty health care professionals (doctors, nurses and pharmacists) from Brazil will receive an invitation to take part by scoring 50 cases of medication errors gathered from an original UK study regarding their potential harm to the patient on scale 0 to 10. Sixteen cases with known actual harm outcomes will be used to assess the validity of their scoring. By looking at 10 errors (out of the 50 cases) scored twice, reliability shall be assessed; and potential sources of variability in scoring will be evaluated depending on the severity of each of error case, the occasion when the scores were given, the scorer, their profession, and interactions among these variables. Generalizability theory will be used for analysing data. Expected impact of the study for public health: This study was submitted to the evaluation of the Research Ethics Committee of the Complexo Hospitalar Universitário Professor Edgard Santos and approved under no. 3.102.570/2019. This is the first validation of this method for use in Brazil, and will allow researchers to conduct more standardised evaluations of interventions to reduce the impact of medication errors.

Introduction

Several studies conducted in hospitals have shown that medication errors are frequent and have high economic and social impact.[1-7] Recently, the World Health Organization launched the global “medication without harm” challenge with the goal of reducing medication-related harm by a half by 2022. For institutions to achieve this goal, it is critical not only to know the frequency and nature of errors but also their severity.[8]

Among the steps in the medication process in hospitals, the administration of medications is considered a critical step, subject to a high occurrence of errors and the highest probability of patient harm because it is the last step before the error reaches the patient. This is due, in part, to the complexity in medication administration processes and the absence of many of the barriers that could prevent errors from occurring.[9] Assessing the severity of medication errors is a crucial point in improving patient safety during medication use. This assessment makes it possible to differentiate errors in relation to their severity, and thus to establish risk minimization strategies targeting those errors with the greatest potential to harm patients.[10] The term “error severity” refers to the extent of the potential or actual impact of medication errors. However, in many studies it is not clear when reporting the prevalence and severity of medication errors whether what is being assessed is actual or potential harm to the patient.[11] This distinction between actual and potential harm is crucial because an actual harm has an obvious severity when compared to a potential harm. Medication errors that actually cause harm represent a small fraction of errors, and many are intercepted before reaching the patient.[11] The assessment of potential harm and actual harm are different processes, each one involving two steps: 1) identifying the potential or actual harm to the patient related to a medication error; 2) rating the degree or severity of that harm.[12] A variety of tools exist to measure and classify the harms associated with medication errors. A systematic review on harm related to prescription errors identified over 40 harm classification tools used prior to 2013.[13] The authors sought to identify acceptable interexaminer reliability and validity through reviewer judgment of potential harm compared to actual harm in situations where actual harm was known. Only two of these tools met the criteria of interreviewer reliability and validity: the National Coordinating Council for Medication Error Reporting and Prevention (NCC MERP)[14] for classifying actual harm and the Dean and Barber 10-point scale for classifying potential harm.[15,1].

The NCC MERP and DEAN and BARBER methods are the only methods that have been validated internationally.[13] The NCC MERP method classifies an error according to the severity of the outcome. It considers factors such as whether the error reached the patient, if the patient was harmed, and to what degree. This method classifies errors into 9 categories (A to I) where A means no error and I, error causing death. The Dean and Barber scale assesses the potential severity of medication administration errors by calculating the mean subjective score of four different healthcare professionals (including pharmacists, nurses and physicians). This method has already been used to assess the potential clinical significance of medication administration errors identified in studies conducted in the UK[15] and Germany[17] and has been shown to be valid and reliable in the contexts in which these studies were conducted.

A recent systematic review on medication administration errors detected by the direct observation method in Latin American hospitals identified 10 studies that estimated the rate of medication administration errors (MAEs); however, none of them assessed the severity of these errors (personal communication). As far as we are aware, this is the the first scientific work on the validity of a scale to assess the severity of MAEs in South America and particularly in Brazil. In this context, we have decided to use the method developed by Dean and Barber[15] for assessing the potential severity of MAEs because it is a more appropriate method for research when compared to the NCC MERP, it has been validated in studies conducted in other countries (UK and Germany) for this purpose, and it may later be used to assess the potential severity of medication errors, which do not have a known outcome, in Brazilian hospitals. Considering the differences between Brazil and countries such as Germany and the United Kingdom regarding health systems, professional training and performance, and cultural context, it is necessary to validate the method within the Brazilian context.

Thus, this study aims to validate the existing Dean and Barber method for assessing the potential clinical significance of medication errors developed in the UK for use in Brazil, using the same procedures involved in developing and testing the method in the UK. For due that our specifics objectives will be: i) To determine the minimum number of judges required to produce a reliable mean severity score in the Brazilian context; ii) to determine whether the judge’s profession has an effect on the score; iii) to determine if the repeated assessment has an effect on the score; and iv) to explore the validity of the mean severity score.[15]

Design and methods

The existing method

When creating their method Dean and Barber[15] chose 50 medication error cases from the literature in nearly equal numbers showing minor, moderate and severe potential clinical outcomes; in 16 of these cases, the patient outcome was already known. These cases were then sent to 30 different healthcare professionals (ten physicians, ten nurses and ten pharmacists). These judges were asked to score the potential clinical significance on a visual analogue scale ranging from 0 to 10 (with 0 corresponding to “no harm” and 10 corresponding to “patient death”). Specifically, this error severity classification involves: 1) Minor - very unlikely that the patient will develop any adverse event; 2) Moderate - likely to cause an adverse event in the patient or interfere with the therapeutic goal, but very unlikely to cause death or harm lasting more than a week; 3) Serious - error that could lead to permanent harm or death to the patient. A subset of ten cases was evaluated on a second occasion by all judges. The data were analyzed using generalizability theory.[18]

Generalizability theory

Cronbach et al.[18] developed generalizability theory, a method that systematically allows the effect of multiple sources of variance and their interactions on scores to be measured at the same time in a single study, based on the premise that in any assessment procedure, variance in scores can be attributed to different identifiable sources.

Generalizability theory also emphasizes the estimation of variance components. Once the variance attributed to each of these sources can be calculated, the most efficient way to reduce unwanted variation can be determined. The results of this can be used to identify methods for improving the reliability of a test.[19]

The application of generalizability theory takes place in stages. In the first, a generalizability analysis begins with the specification of a universe of admissible observations through the identification of different sources of variation. In the second stage, a generalizability or Gstudy estimate variance components for this universe. This involves creating an appropriate research design, collecting data, and determining the extent to which each of the variables influences the score. Different coefficients of variation can be calculated representing the different situations. For example, a coefficient can be calculated showing the extent to which one can generalize the score assigned to a case by a physician to the score assigned to the same case by a pharmacist. The final step is a decision (or D-study) associated with a prespecified universe of generalization.[18,19] Broadly speaking, D studies emphasize the estimation, use, and interpretation of variance components for decision-making with well-specified measurement procedures. Perhaps the most important D study to consider is the specification of a universe of generalization, where the universe to which a decisionmaker wants to generalize based on the results of a D study using a particular measurement procedure.[20] From the estimated variance, the effect of a change in the number of observations on the generalization coefficient can be explored. For example, the change in the generalization coefficient can be determined by changing the number of judges. This is done by dividing each term (variance) by the number of observations. This step allows exploration of the conditions that can achieve a sufficient level of reliability.

Case selection

The original instrument will be used, keeping the described cases. These cases will be translated into Portuguese, updated (if the drugs are no longer available or not in routine use), and adapted to the Brazilian context (making any necessary adjustments regarding the drugs, doses, concentrations, units of measurement, pharmaceutical forms and available presentations). The maintenance of the cases submitted to evaluation will allow comparison with the previous studies carried out in the UK and Germany.

Translation of the cases to Portuguese

All 50 cases will be translated by the principal investigator and adapted if needed. (Appendix A presents the original cases and the translation into Portuguese). The reason for doing so is because some of the drugs mentioned in the original cases may not be used in Brazil. The translated and adapted version will then be submitted to the evaluation by two experienced hospital pharmacists regarding the pharmaceutical product, drug concentration, route of administration, pharmaceutical dosage to make sure the degree of severity remains unchanged; and, in case there is no consensus, they will be sent to a more experienced pharmacist with expertise in clinical medicine and patient safety.

After this process, the document will be translated back to English; and, to ensure that this process has preserved the essential characteristics of the errors described in the original version, the adapted document will be sent to the authors of English version.

Recruitment of the evaluators

After contacting and receiving the permition of the chief of services in each hospital, thirty health professionals (10 physicians, 10 nurses, 10 pharmacists) with at least three years of clinical practice will be invited and recruited from different public and private hospitals, from all five Brazilian geographic regions. Health professionals from specialized areas such as pediatrics and oncology and with less than three years of clinical practice will not be included in the sample. In each hospital, two physicians, two nurses and two pharmacists will be initially selected. Next, the indicated physicians, nurses, and pharmacists will be contacted via email, a letter will be sent to participants for their consent, plus a document explaining the objectives of the study, the method for assessing the severity of medication administration errors based on the scoring scale, and practical examples of how to perform the scoring.

The professionals who agree to participate in the evaluation will be grouped according to profession, degree of training and the country’s region, and a stratified random sample of thirty professionals (ten physicians, ten nurses, ten pharmacists) will be selected using SPSS software. Those not randomly selected will be informed through a thank you letter for agreeing to participate in the study.

No incentives will be offered to professionals to participate in this study.

Scoring process

The 30 professionals initially selected will receive a file with the descriptions of the 50 cases of MAEs and will be instructed to score the cases in terms of their potential clinical significance, using the scale proposed by Dean and Barber.[15] The scores provided by those professionals will then be analyzed.

Two weeks after the receiving of the severity assessments based on the fifty cases, each respondent will be sent ten of the cases randomly selected, for rescoring. In this way it will be possible to measure whether the occasion on which the cases were scored was an important source of variance of the responses obtained.

Evaluators will be asked to record the time spent assessing all fifty cases and invited to make relevant comments about the scoring process in a specific space of the form and complete a short questionnaire on demographic details, including their occupation and the number of years of work experience (Appendix A).

Reliability analysis

Universe of observations

The analysis in this study will be identical to that of the original study.[15] The sources of variance in the process of assessing the severity of medication administration errors will be considered as inherent in the cases themselves (CASE), the occasion on which they are assessed (OCASION), the evaluator (JUDGE), the professional background of the judge (PROFESSION), and the interactions among these sources. Since each judge is a member of a single profession, the JUDGE factor is considered nested with the PROFESSION factor (JUDGE:PROFESSION).

Since the scores for the fifty cases of errors will be obtained on two occasions in a sample of ten cases, there are two models for conducting the G-study, depending on the data set used:

Model 1: OCASION X CASE X JUDGE (using the ten cases scored twice).

Model 2: CASE X JUDGE:PROFESSION (using all 50 cases).

Model 1 ignores the effect of different professions, while model 2 ignores the effect of occasion. A model that would take into account all sources of variance for the ten cases with repeated scores, OCASION X CASE X JUDGE:PROFESSION, will not be used because the variance per case is anticipated to be too high to perform an analysis of variance.

G Study

The data will be evaluated considering models 1 and 2 in order to determine the contributions of each factor to the variance in scores. First, repeated measures of variance analysis will be performed, using SPSS software (version 26.0, SPSS mc, Chicago, IL, USA).

An analysis of variance will be performed and seven sources of variance will be estimated for model 1, being these: case, occasion, occasion x case, judge, judge x case, occasion x judge, and judge x case x occasion. For model 2, the sources of variance are: profession, judge ‘nested’ in profession, case, case x profession, and a residual variance (case x judge:profession). The equations used to calculate each variance (estimated mean square) will be provided with the results. Equations to calculate the generalizability coefficients are provided in Appendix B. The data will be analyzed using the Statistical Package for the Social Sciences (version 26.0, SPSS mc).

The resulting mean square values will then be used to calculate the attributable variance for each source, using equations for the mean squares based on that described by Streiner and Norman[20] and Cronbach et al.[18]When the estimated variance components are computed as negative, a value of zero will be assumed.[21]An overall generalizability coefficient, coefficients equivalent to inter-examiner reliability and test-retest reliability will be computed.

D Study

In a D study, the effects of different modifications in the evaluation procedure on the generalizability coefficient will be investigated, and the accuracy of the obtained measurement results will be evaluated. Therefore, different scenarios based on the results of study G will be investigated in study D. The same model of study G will be used to calculate the generalizability coefficients for different numbers of judges and different occupations. This will be done to allow identification of how many judges were needed to obtain a reliable average score. Study D will also investigate whether judges need to be of different professions or of the same profession. Generalizability coefficients for different numbers of judges and different numbers of test occasions will be calculated using the formula described by Streiner and Norman.[20] As in previous studies, a generalizability coefficient greater than 0.8 will be considered to represent acceptable reliability.[15]

Validity analysis

A sample of 16 medication administration errors with known outcomes will be included among the cases that will undergo evaluation by the judges. The premise is that if the scoring method is valid, the scores assigned to cases with known outcomes should reflect the relative severity of those outcomes. In this way, it will be possible to test the validity of the method by comparing the scores assigned by the 30 raters to the 16 MAEs with previously established scores.

The researchers in the original study grouped the 16 cases that had a known severity into cases with a ‘minor’ outcome, meaning the errors resulted in no adverse effects, ‘moderate’, meaning the errors resulted in some adverse event with no lasting impairment, and cases with a ‘severe’ result, meaning the errors resulted in death or lasting impairment. The cases with known severity were distributed as follows: 5 cases with minor severity, 5 cases with moderate severity, and 6 cases considered severe.[15] The average scores assigned to these 16 cases by the raters will be compared to the known outcomes described for the same 16 cases in the original study.

Ethics and disclosure

This study is part of a larger study on MAEs in a university hospital: incidence, severity, and associated factors, which was submitted to the evaluation of the Research Ethics Committee of the Complexo Hospitalar Universitário Professor Edgard Santos and approved under opinion number 3.102.570/2019.

We believe that the results of this study will be particularly important for an audience of professors, researchers, and health professionals from health institutions in Brazil. These results will be published in international peer-reviewed journals, as well as disseminated through scientific congresses focused on patient safety and quality of healthcare services.

The validation of this scale in Brazil will allow the expansion of research in the area of patient safety with the aim of measuring the potential harm related to medication errors, particularly medication administration errors in hospitals and health care institutions.

Patients and public involvement

Patients and the public were not involved in co production of this protocol.