Development and Psychometric Evaluation of the Pre-hospital Medical Emergencies Early Warning Scale.

INTRODUCTION: The number of requests for emergency medical services (EMSs) has increased during the past decade. However, most of the transports are not essential. Therefore, it seems crucial to develop an instrument to help EMS staff accurately identify patients who need pre-hospital care and transportation. The aim of this study was to develop and evaluate the psychometric properties of the Pre-hospital Medical Emergencies Early Warning Scale (Pre-MEWS).
MATERIALS AND METHODS: This mixed-method study was conducted in two phases. In the first phase, a qualitative content analysis study was conducted to identify the predictors of medical patients' need for pre-hospital EMS and transportation. In the second phase, the face and the content validity as well as the internal consistency of the scale were evaluated. Finally, the items of the scale were scored and scoring system was presented.
RESULTS: The final version of the scale contained 22 items and its total score ranged from 0 to 54.
CONCLUSIONS: Pre-MEWS helps EMS staffs properly understand medical patients' conditions in pre-hospital environments and accurately identify their need for EMS and transportation.

INTRODUCTION

The number of requests for emergency medical services (EMSs) and transportation to medical facilities has increased drastically during the past decade[12] due to technological advances, increase in population age, and people's greater awareness of and sensitivity toward their own health.[3] However, studies have shown that most of the transport are not essential. For instance, the results of a study conducted in England showed that from 215 patients who had been considered by EMS staffs to be eligible for transportation, only 139 patients (65%) were admitted to hospital.[1] Studies conducted in our country, Iran, also reported that 29%–50% of medical patients transported by EMS did not really need emergency transportation.[45]

One strategy for preventing unessential transportations by EMS is using credible instruments for quick and comprehensive pre-hospital patient assessment.[6] Nonetheless, current patient assessment instruments have been developed for in-hospital assessments[17] and hence cannot be used for pre-hospital emergency situations.[6] One of the emergency patient assessment instruments is the early warning score (EWS).[7] The EWS includes items on heart rate, respiratory rate, blood pressure, body temperature, and level of consciousness and is usually used for monitoring severely ill hospitalized patients.[78] Currently, the modified version of the EWS[7] and the pandemic medical EWS[1] are used for pre-hospital assessment of patients with medical problems. However, Ebrahimian et al. and Ebrahimian et al. reported that these instruments were useful and sensitive only for identifying severely ill patients and had low specificity for identifying patients who are not severely ill. Moreover, they found that even patients who had obtained a zero score had been transported by EMS and hospitalized in different hospital wards.[46] Accordingly, these instruments are not useful for accurate pre-hospital triage and, hence, developing a credible instrument for accurately identifying patients who really need EMS services seems essential. The challenging question here is, “for what point of an emergency mission should this instrument be developed to both maintain the quality of pre-hospital EMS and minimize unessential EMS transportations?”

In general, there are two major points in an EMS mission for deciding upon the need for transportation. One point is in emergency medical dispatch (EMD) centers where incoming calls-for-help are assessed, categorized, and prioritized and, subsequently, decisions are made about whether or not to send an ambulance.[9] However, despite using different methods and strategies for predicting patients' conditions,[101112] these centers cannot accurately identify patients who really need transportation even after doing overtriage.[1314] Moreover, as EMD staffs cannot directly see and assess patients who are in emergency situations, they usually have problems in identifying patients' need for transportation. The studies have shown that in one case, EMD staffs had even mistakenly diagnosed a cardiac arrest as a seizure.[1516] Accordingly, deciding upon whether or not to send an ambulance based on the data received in an EMD can potentially endanger patients' lives. Another point for determining the need for transportation is at calling patient's bedside.[17] As they attend emergency situations, EMS staffs are at a unique position for assessing patients and deciding upon the best treatments to administer, patients' need for transportation, and the best transportation destination.[1718] However, studies did not support the soundness of all decisions made by EMS staffs about transportation.[171819] Fullerton et al. reported that pre-hospital EMS staffs had different levels of expertise and made transportation-related decisions mainly based on mental processes and personal experience rather than scientific evidence.[17] Consequently, it seems crucial to develop an instrument for helping EMS staffs accurately identify patients who need pre-hospital EMS and transportation. The aim of this study was to develop and evaluate the psychometric properties of the Pre-hospital Medical Emergencies Early Warning Scale (Pre-MEWS).

MATERIALS AND METHODS

This mixed-method study was conducted in two phases. In the first phase, a qualitative content analysis study was conducted to identify the predictors of medical patients' need for pre-hospital EMS and transportation in pre-hospital environments. The results of this phase were used for developing the Pre-MEWS. In the second phase, the face and the content validity as well as the internal consistency of the scale were evaluated. Finally, the items of the scale were scored and the final Pre-MEWS and its scoring system were presented. These phases are explained below.

The first phase: Item generation

In this phase, a qualitative content analysis was conducted for generating the items of the Pre-MEWS. The study population included EMS specialists affiliated to Sina, Rasoul Akram, Imam Khomeini, and Shohaday-e Haftom-e Tir Hospitals as well as the staffs of Tehran EMS and EMD, Tehran, Iran. The inclusion criterion was at least 1-year work experience in medical emergency departments. The study participants were recruited by employing the purposive sampling technique. The semi-structured, face-to-face interviews were conducted for data collection. Primarily, we made appointments with the participants and then referred to their workplace for conducting the interviews. Three distinct interview guides were used for interviewing the EMD staffs, EMS staffs, and EMS specialists. Interview guides included broad, open-ended questions about the predictors of patients' need for medical EMS and transportation from the perspectives of EMD staffs, EMS staffs, and EMS specialists. The three opening questions which were asked from EMD staffs, EMS staffs, and EMS specialists were respectively as follows,

How do you make decisions about whether to send ambulance for transporting patients with medical problems?

How do you determine whether a patient with medical problems needs pre-hospital EMS and transportation?

What factors do you value most when managing patients with medical problems?

All interviews were digitally recorded and immediately transcribed verbatim. We obtained no new information from the data after conducting 26 interviews. However, we conducted two more interviews for ensuring data saturation.

Study data were analyzed using the summative content analysis approach in the following seven steps:[20]

Data preparation: We transcribed the interviews verbatim and read the transcripts for several times to understand what was happening in the data;

Determining the level of analysis: A whole-interview transcript was considered as the unit of analysis. Moreover, we decided to code all words, expressions, and sentences which conveyed a similar meaning as a single code;

Deciding upon the coding of repetitive meaning units: The frequency of each concept was important to the analysis. In other words, the more frequent appeared concepts were considered as the more important predictors of medical patients' need for pre-hospital EMS and transportation in pre-hospital environments

Deciding upon how to differentiate concepts: Words and expression which conveyed a similar meaning – either directly or indirectly – were coded under a similar code;

Establishing the rules of coding: Initially, we established several coding rules, documented them, and used them during the process of coding. Moreover, we established new rules during the coding process and added them to the list of original rules. Occasionally, some old rules were reformulated and analyses were performed by using the new reformulated rules. For instance, one of the coding rules was, “Any abdominal pain is considered as a pain related to problems in the digestive system.”

Coding the text: This step was taken after reading each transcript for several times. The MAXQDA v. 10 (VERBI Software based in Berlin, Germany) was used for managing the data. This software helped us manage the data, code the transcripts, and categorize the generated codes. The soundness of our analytical activities was monitored and confirmed through inviting a qualitative researcher to code the first interview transcript using the established rules of coding. Then, our generated codes were compared with the codes generated by him. Conflicts and discrepancies in coding were discussed until reaching agreement. An extensive list of codes was assembled during the process of coding

Analyzing the results: During the process of coding, we constantly compared the generated codes with each other and also with the data and categorized the codes accordingly. The importance of each concept was determined based on its “frequency.” As mentioned earlier, codes with higher frequency were considered as the more important predictors of medical patients' need for pre-hospital EMS and transportation. Moreover, we used our clinical expertise to prioritize the codes with similar frequencies. Finally, a comprehensive list of categories, subcategories, and predictors of medical patients' need for pre-hospital EMS and transportation was created. The list contained 102 predictors.

The second phase: Scoring and psychometric evaluation

This was a quantitative phase and included four steps as follows.

The first step: Item reduction

The first step of the second phase of the study was a descriptive–analytic study conducted from August to November 2013. All patients being 15-year-old or more with medical problems who had called Semnan EMS, Semnan, Iran, constituted the study population. Multiple logistic regression was used for selection of important variables, as lower important, latent and collinear variables were excluded. Selection criteria for key variables was amount of P value that acquired by logistic regression analysis. In this manner, variables with P < 0.1 are considered as key agents [Table 1]. These variables were used to develop scoring scale of the necessity of sending medical patients from pre-hospital settings. As we used logistic regression analysis in this step, the study sample size was determined to be 5–10 times more than the number of items.[21] The number of variables, i.e., the predictors, was equal to 102; therefore, 1020 patients were needed. The study instrument was the checklist of predictors created in the first phase of the study. This checklist contained the most characteristic signs and symptoms of medical problems, items on patients' socioeconomic status, and items on patients' immediate environment. Items were answered simply on a dichotomous yes/no scale. Accordingly, the checklist was easily completed by EMS staffs without interfering with their practice. Before starting sampling, twenty EMS staffs affiliated to Semnan EMS, Semnan, Iran, were educated about how to complete the checklist. Then, they were asked to use the checklist during their EMS missions for assessing patients with medical problems. When patients had been transported to hospitals, attending EMS specialists assessed them and requested blood testing, radiography studies, and other diagnostic evaluations. Three medical emergency physicians participated in this study. In each work shift, there was one physician in emergency ward. They were requested laboratory, X-ray graphics, and other paraclinical tests according to patients' status. After comprehensive patient assessment, the attending EMS specialists were asked to determine whether patients had really needed transportation or not.

Table 1

Multiple logistic regression analysis

The SPSS version 16.0 (IBM Corporation) was used for data management and analysis. Logistic regression analyses were performed for identifying the items which significantly contributed to patients' need for transportation. Such items were used for developing the Pre-MEWS. The level of significance for regression analysis was set at below 0.1.

The second step: Validity assessment

In this step, the face and the content validity of the Pre-MEWS was assessed. The content validity of the scale was evaluated by calculating the content validity ration (CVR) and the content validity index (CVI). We used the Lawshe's technique for calculating CVR.[22] Accordingly, a panel of fourteen EMS specialists were invited to rate the items of the Pre-MEWS on a three-point scale – essential, useful but not essential, not necessary. Then, the CVR of each item was calculated using the following formula, CVR = (ne – [N/2])/(N/2). Finally, we referred to the table of CVR critical values[22] and excluded items with a CVR of <0.52. Consequently, 22 items remained in the Pre-MEWS.

The CVIs of the items were calculated by adopting the Waltz and Bausell technique.[23] The 22-item Pre-MEWS were provided to another expert panel of fourteen EMS specialists. They were asked to rate the relevance, clarity, and simplicity of the Pre-MEWS items on a four-point scale – from 1 to 4. The CVI of each item was calculated by dividing the number of experts which had rated that item as 3 or 4 by the total number of the experts. Finally, the CVIs of all items were summed and divided by 22 to determine the total CVI of the scale.

On the other hand, we assessed the face validity of the scale by inviting ten EMS staffs to comment on the clarity/ambiguity, wording, terminology, and arrangement of the items. Finally, an expert panel of two EMS specialists, two methodologists, and one EMS staff made the necessary amendments recommended by aforementioned 28 EMS specialists and ten EMS staffs.

The third step: Reliability assessment and the scoring of the Pre-hospital Medical Emergencies Early Warning Scale

In this step, a cross-sectional study was conducted in 2014 to assess the internal consistency of the scale and also to determine its scoring. This step was taken in the same way as the first step of the second phase. The only difference between these two steps was that in the first step, we had used the 102-item Pre-MEWS while in the third step; the scale contained only 22 items. Accordingly, 447 patients were assessed using the scale and the Cronbach's alpha was calculated for determining its internal consistency. Moreover, the simple logistic regression analysis technique was employed for calculating odds ratio (OR) for each item and determining the scoring of the scale[2425] OR values of 4–7.49, 7.5–10.49, 10.5–13.49, 13.5–16.49, and 16.4–20 were scored 1, 2, 3, 4, and 5, respectively. The output of the third step was the final Pre-MEWS and its scoring system.

The fourth step: Determine of sensitivity, specificity

The aim of designing and performing of this step was to determine of sensitivity, specificity, and cutoff point of scale. The data gathering scale in this step were final derived and scored scale that acquired from third steps of second stage. In this step, 474 patients were studied. The method in this step was similar to the first step, with the difference that in this step, a 22-item scored scale was used. At first, the questionnaire data were saved in SPSS software, 16 edition version. Then two-third of data were selected randomly. These data were formed learning group. The acquired data from learning group were used to estimate of necessity parameters. In this group, receiver operating characteristic (ROC) curve analysis performed and appropriate cutoff point for instrument was detected. The remaining data (one-third) formed the testing group. The accreditation indices (amount of sensitivity and specificity) were calculated according to detected cutoff point and the data of testing group.

RESULTS

Ten EMS specialists, nine EMD staffs, and nine EMS staffs participated in the qualitative phase of the study. The means of their ages and work experience were 31.43 ± 6.86 and 6.29 ± 4.92 years, respectively. The findings of this phase fell into two main categories, five subcategories, 22 concepts, and 102 codes. We used these 102 codes and developed a 102 scale which was completed for 1020 patients who had requested EMS. Eighty-five (8.3%) scales had been filled out incompletely and hence were excluded. Consequently, 935 completely filled scales – completed for 490 male and 445 female patients – were included in the final analysis (a response rate of 91.7%). The mean of these 935 patients' ages was 49.10 ± 21.32 years. The EMS specialists affiliated to the study setting had reported that only 656 patients (70.2%) were eligible for and really needed transportation. Multiple logistic regression analysis revealed that 35 items were significantly correlated to patients' need for transportation [Table 1].

These 35 items remained in the primary version of the Pre-MEWS and other items were excluded.

Content validity assessment also revealed that thirteen items had a CVR of <0.52. These items were excluded and, hence, 22 items with a CVR of >0.51 remained in the final version of the scale [Table 2]. Then, a CVI was calculated for each Pre-MEWS item. The CVIs of all items of the Pre-MEWS were >0.86 and the total CVI of the scale was 0.971.

Table 2

Means and content validity ration values for acceptation or rejection results

The Cronbach's alpha of the Pre-MEWS was 0.759. Table 3 shows the scoring of each Pre-MEWS item. As mentioned above, this scoring system was developed by conducting logistic regression analysis and calculating OR values. Consequently, the final version of the scale contained 22 items and its total score ranged from 0 to 54 [Figure 1]. The ROC curve area for developed instrument was calculated to be 0.915 [Figure 2]. The cutoff point for final instrument was detected to be 10.5. The sensitivity and specificity of designed instrument were 0.814 and 0.850, respectively.

Table 3

Calculating odds ratio for each item and determining the scoring of the scale

Figure 1

Pre-hospital Medical Emergencies Early Warning Scale

Figure 2

Receiver operating characteristic curve for Pre-hospital Medical Emergencies Early Warning Scale as predictor of need for emergency department care

DISCUSSION

The aim of this study was to develop and evaluate the psychometric properties of the Pre-MEWS. The item of “special cases/conditions” obtained a score of 5. About 90.7% patients who had been considered by EMS staffs to be special cases or to have special conditions really needed transportation. Consequently, although the item of “special cases/conditions” is a subjective item which can be responded variously, it is exceptionally valuable for determining patients' need for transportation. This item highlights the big difference between pre-hospital and clinical patient care.[26] Moreover, the item of “chest pain” scored 5 while the items of “abnormal perspiration” and “dyspnea” obtained a score of 4. These three items are among the manifestations of acute medical conditions. A patient who has two or three of these manifestations may have a serious health problem such as myocardial infarction, pulmonary emboli, or acute respiratory failure[27] and, hence, need immediate transportation. The findings of the quantitative phase of the study also indicated that 89.6% of patients with chest pain, 86.3% of patients with dyspnea, and 85.6% of patients with abnormal perspiration really needed transportation.

Items of “altered level of consciousnesses calculated using the voice-pain-unconsciousness (VPU) scale – and 'availability of irregular pulse'” were also scored 4. The results of the quantitative phase of the study revealed that 91.6% of patients with cardiac arrhythmias and 95.7% of patients with decreased level of consciousness really needed transportation. Accordingly, these items were valuable enough for determining medical patients' need for transportation and, hence, remained in the final version of the Pre-MEWS. Any alterations in level of consciousness or any abnormal changes in cardiac rhythm may be an early sign of critical conditions. However, patients and their families usually do not call for ambulance and EMS in case of temporary alterations in level of consciousness and already-diagnosed cardiac arrhythmias. The score of the complete consciousness item is equal to 0. Moreover, the scores of other VPU parameters in the Pre-MEWS are equal to 4. A total score of 4 was allocated to these three items of consciousness because, in pre-hospital environments, any changes in level of consciousness should be considered as potentially serious. Seymour et al. also allocated scores 1 and 2 to Glasgow coma scale values of 8–14 and <8, respectively.[28]

Moreover, logistic regression analyses revealed a score of 3 for items such as “previous history of asthma,” “pain severer than 6,” “sudden changes in skin color,” and “previous history of heart disease.” Study findings also revealed that 76.6% of patients with previous history of asthma, 84.5% of patients with pain severer than 6, 74% of patients experiencing sudden changes in skin color, and 88.4% of patients with previous history of heart disease really needed transportation. Although pain perception is affected by different cultural and environmental factors, pain severer than 5 reflects an acute pathologic condition needing immediate attention. Accordingly, patients with such severe pain would be a candidate for transportation provided that he/she is also experiencing other accompanying pathologic findings. The score of severe pain in the Simplified Diagnostic Predictors Scoring System for assessing peptic ulcer perforation is 4.5.[29] Moreover, the parameter of pain is a significant predictor in the system of National Early Warning Scores[30] On the other hand, skin manifestation is common in a lot of pathological conditions. Coldness, pallor, erythema, bruising, urticaria, purpura, rash, and many other integumentory manifestations reflect problems in internal organs.[2731] Accordingly, when happening suddenly, these changes in the skin can be the early signs of an acute condition. The previous history of heart disease(s) can also help determine the acuteness of patients' conditions. Heart is a vital bodily organ whose impaired function can significantly affect other organs. Consequently, the item of previous history of heart disease was also considered as a valuable middle-rank predictor of medical patients' need for transportation.

Finally, the items which obtained a score of 2 were “active bleeding,” “abnormal blood pressure,” “inability to maintain body balance during movements,” and “pupil size disturbances” while the items of “an age of 60 years or greater,” “abnormal blood glucose values,” “abnormal heart rate,” “abnormal oxygen saturation values,” “ailment,” “recurrent signs and symptoms,” “rebound tenderness,” and “self-care deficit” were scored 1. Seymour et al. also reported a score of 1 for items of “a blood pressure of <90 mmHg,” “a heart rate of 120 or faster,” and “an oxygen saturation of <87%.”[28] The reason behind the low score of these items can be the existence of other items in the scale which exert the same effects as these items. For instance, active and uncontrollable bleeding can affect blood pressure, heart rate, oxygen saturation, skin color, perspiration, and vitality. Another example would be a patient with rebound tenderness who also has severe pain, recurrent signs and symptoms, and increased heart rate. Consequently, as expected from coherent instruments, the Pre-MEWS covers a wide spectrum of medical emergency conditions which require EMS and transportation. Moreover, the scoring of the scale follows a consistent and logical pattern.

The total score of the 22-item Pre-MEWS ranges from 0 to 54. Given the number and the type of their items, other scales for identifying medical emergencies have different total scores. For instance, the total scores of the six-item Rapid Emergency Medicine Score, the Ottawa Heart Failure Risk Scale, and the VitalPAC Early Warning Score are 0–26,[32] 0–15,[33] and 0–21, respectively[34] and the maximum possible score of the Resuscitation Management score (THERM) is 37. In the THERM scale, scores of 30 or less, 30.1–35, and 25.1–37 are interpreted as low, medium, and high risk, respectively.[35] The sensitivity and specificity of designed instrument were 0.814 and 0.850, respectively. In scoring system for risk diagnosis in heart failure patients, the sensitivity and specificity in higher cutoff point 2 were reported 0.619 and 0.507, respectively.[33] In addition, the sensitivity and specificity in Worthing physiological scoring system in cutoff point of 3 were 0.63 and 0.72, respectively.[36] Therefore, in comparison to other scoring system that developed for medical disease diagnosis, the sensitivity and specificity of developed instrument in this study are desirable.

CONCLUSIONS

Study findings indicate that factors affecting pre-hospital EMS staffs' decisions about patients' need for EMS and transportation are different from factors contributing to risk assessment in other healthcare settings. Besides patients' physical problems, EMS staffs participating in this study also valued other factors such as environmental and contextual ones. Consequently, both factors related to patients' physical conditions as well as environmental and contextual factors were included in the Pre-MEWS. The total score of the Pre-MEWS ranges from 0 to 54 – the higher the score, the greater is the need for EMS and transportation. The Pre-MEWS helps EMS staffs properly understand medical patients' conditions in pre-hospital environments and accurately identify their need for EMS and transportation. Like many other instruments, the Pre-MEWS is not fully sensitive nor specific; however, it would be helpful for assessing medical patients who call for EMS and also for identifying patients who really need EMS and transportation. The Pre-MEWS is a simple, quite precise, cost-effective, and user-friendly scale and little education is needed for learning how to use it. Therefore, we recommend using it in daily EMS missions as well as in epidemics of medical conditions.

Financial support and sponsorship

This study was supported by Iran University of Medical Sciences (grant No: IUMS/SHMIS_ 2013.4.7/7).

Conflicts of interest

There are no conflicts of interest.