Designing Iranian Pre-hospital Stroke Scale.

BACKGROUND: Various studies have shown that stroke morbidity and mortality could be decreased if patients receive early diagnosis and treatment. Iranian Pre-hospital Stroke Scale (IPSS) is designed based on other prehospital stroke scales available across the world as well as experiences of emergency medicine specialists and pre-hospital emergency technicians to improve the diagnostic accuracy of the stroke scale in Iran.
METHODS: Using a mixed method, the study was carried out in two main phases. In the first phase, concept elicitation was conducted based on the review of the literature. Related stroke diagnosis and usual instruments in pre-hospital emergency stroke centers were documented. The IPSS was designed based on semi- structured interviews with 35 neurologists, emergency medicine practitioners and physicians working in hospitals and emergency technicians in the pre-hospital field. In the second phase, the face and content validity, and reliability were checked.
RESULTS: According to results from the first phase of this study (items generation), three domains were introduced as the most important factors in detection of early signs and symptoms of stroke. In the second phase (items reduction), the face validity of the IPSS was checked based on the comments from participants (the experts and EMS technicians). The content validity was calculated based on Lawshe index. The IPSS scale content validity index (S-CVI/Ave) was calculated as 89%. To determine the criterion validity of the instrument, the IPSS scores were compared with the final diagnosis based on results from brain CT scan in hospital.
CONCLUSION: During this study we developed IPSS to be used by emergency technicians in pre-hospital field with a dichotomous items and simple and easy administration. It is recommended for future studies to apply this tool to emergency dispatch units as well as triage procedures in hospitals.

Introduction

Stroke is the most common neurological disease and an important cause of death and disability worldwide (1). The incidence of stroke is 15 million worldwide annually; with over 5.7 million mortalities (2, 3). The cost of the disease was estimated as $73.7 billion in 2010 (4). According to released reports, stroke is increasing across the Asian continent (5,6). In Iran, studies have reported incidence of stroke among all age groups as 372 cases per 100 thousand, and in people older than 45 years 500 cases per 100 thousand people (7, 8). The incidence of stroke in Western countries is estimated as 100 to 300 cases per 100 thousand people (8). Today, by improving the care and treatment for stroke patients, the stroke mortality rate is decreased in developed countries (3). About 87% of mortalities of stroke patients is occurred in middle and low income countries (9). Stroke complications impacts on patients’ roles and daily activities as well as quality of life of their caregivers (1,4).

In Iran, the mortality rate due to stroke is approximately between 15.3 -31.5%, which is far higher than the global rate of about 12.6% for all stroke types (2, 10, 11). Multiple methods for assessment of signs and diagnosis of stroke patients at pre-hospital field by emergency technicians has caused increasing delay in transferring patients to medical centers and also increasing misdiagnosis of stroke in the pre-hospital phase (12).

In Iran, the assessment of stroke patients in the pre-hospital phase is in two ways: 1. In some emergency centers neurological examinations are conducted by emergency technicians and the information will be recorded for the final decision by a general practitioner attended in that center. 2. In some pre-hospital emergency centers without a doctor, the neurological examinations and the final decision are based on emergency medical technicians’ actions and diagnoses.

Medical technicians’ applied methods of diagnosis are often experience-based and thus the administered treatment may not be the same by different technicians (13). Therefore, the American Heart Association, American Stroke Association and other organizations involved in prevention and treatment of stroke, recommend use of diagnostic tools for more accurate diagnosis at the pre-hospital phase by emergency medical technicians (14). These tools can help pre-hospital emergency technicians to perform accurate neurologic examinations resulting better care for optimum results. At present, there is no exclusive tool recommended for stroke diagnosis in Iran.

An effective tool for diagnosis of stroke is one that is easy to learn about who it works and used by pre-hospital emergency technicians with proper reliability and validity (15). The aim of this study was to design and assess a suitable psychometric tool for diagnosis of stroke by pre-hospital emergency technicians.

Methods

This study used mixed methods and was carried out in two main phases, as follows.

Generating of Items

In this first phase, concept elicitation was conducted using a review of the literature identification of usual instruments in pre-hospital emergency stroke centers including NIHSS, LAPSS, CPSS, FAST, OPSS, KPSS. A series of semi- structured individual interviews were performed with 35 neurologists, emergency medicine practitioners, physicians and emergency technicians. Data was analyzed and categorized in order to identify those concepts relevant to participants’ experience, and to explore variability of symptoms. Qualitative analysis of transcripts was used to support item generation process and resulted in a preliminary draft of the measure.

Finally, the most important elements for pre-hospital diagnosis of stroke in a panel of experts were identified. After that the primary tool comprising 17 items was developed. Afterwards, weighting and ranking of items of the instrument were performed using the Delphi technique.

In end of first phase, we had a pool of items which were drafted for each concept based on qualitative data from experts’ input and support from the literature.

Item Reduction

At this second phase we determined the psychometric properties and the cutoff point of the instrument. Initially, validity of instrument was tested using three methods including; face validity, content validity and criterion validity.

Face validity: To this step, 10 emergency technicians were interviewed by face to face technique and the level of difficulty (difficulty of understanding words and phrases), the fit (fit and proper communication with the statements of the questionnaire) and uncertainty (probability of false impressions despite the failure of the meanings of words or phrases) were studied, also fifteen experts reviewed the items. (16).

Content Validity: Quantitative determination of content validity was performed for scale utilization. Fifteen faculty members from University of Social Welfare and Rehabilitation Sciences, Tehran University of Medical Sciences, Arak University of Medical Sciences and University of Khorasan Razavi Medical Sciences, offered their opinion about the necessity, fitness, clarity and relevance of items to assess content validity ratio (CVR) and content validity index (CVI) (16).

At the end of this phase, we calculated S-CVI/Ave. S-CVI/Ave definite Average of the I-CVIs (Content Validity of individual items) for all items on the scale.

Criterion validity: The primary scale was used for 96 suspected stroke patients and results compared with discharge diagnosis by brain CT scan, (as standard clinical criteria for stroke diagnosis) (17).

Internal consistency: To determine internal consistency coefficient of IPSS, Kuder-Richardson method was used to determine the objectivity of instrument.

In this stage, for determining the rate of reliability, the tool was given to emergency pre-hospital technicians and based on tool guidelines they applied it for the patients who had the inclusion criteria. Finally about 140 questionnaires were returned. However, 20 of them due to uncompleted information and 24 of them due to no registered name of patient by emergency pre-hospital technicians and not having a final diagnosis were excluded. It should be noted that the final diagnosis of stroke or transient ischemic attack (TIA) was proved by the neurologist based on the gold standard of brain CT scan.

In this study the elimination of one non-harmonic question was eliminated for improvement of the overall reliability of the instrument.

Inter-rater Reliability: In this study, due to the nature of the tool, the response to some items depended on its users. Moreover, there was no possibility to implement a unique test because of the changing in patients’ conditions over time; therefore, the between observers’ reliability was used. For this, three questionnaires were completed for 10 patients, simultaneously, by different persons. As there was only one technician for each urgency mission, one questionnaire was completed by emergency technician and two other by nurses at the hospital triage units. Then, scores of these three questionnaires using interclass correlation coefficient (ICC) were compared.

Determination of the cutoff point: After entering the data of 96 suspected stroke patients into the SPSS v.16, using receiver operation characteristic curve (ROC), the best cut-off point, the sensitivity, specificity, and area under the curve were calculated.

Regarding ethical considerations, the study was performed under formal permissions from University of Social Welfare and Rehabilitation Sciences, Arak University of Medical Sciences, University of Khorasan Razavi Medical Sciences. Anonymity of the participants was secured and informed consent was obtained for all participants. Names of all patients who met the inclusion criteria removed and replaced with numeric codes.

Results

According to results from the first phase of this study, three categories were introduced with the most important factors for detection of early signs and symptoms of stroke: the signs and symptoms of stroke, the distinguished factors between acute stroke and mimics, and the main risk factors of stroke (Table 1). Following this phase, IPSS with 17 primary dichotomous (yes/no) items was identified.

Table 1

The results extracted from the first phase of first stage

Signs and symptoms of stroke	Common risk factors	differentiate the conditions mimicking stroke
o Unilateral weakness or paralysis	o TIA	o Having history of seizures or loss
o Speech disorders	o MI	of consciousness
o Face paralysis	o CVA	o altered level of consciousr
o Sudden onset	o DM	motor symptoms compare
o Decline in strong gripping	o HLP	Previuos status
o Imbalance	o HTN	o glucose 60 to 400 mg
Impaired vision	o Age greater than 45 years	o
Headache	o Patient's blood pressure is higer
Loss of consciousness	than usual, or 200/120 mmHg
Urinary incontinence
Nausea and vomiting

Finally, the items were weighted and ranked by experts; the first 4 items were scored 3, score 2 was assigned to the next 5 and other items received scored 1.

In the second phase, for the face validity and clarity of items some changes were made based on the comments received from participants (the experts and EMS technicians).

For the content validity, two items with less than 0.49 (minimum of acceptable level for 15 experts based on Lawshe table) were excluded (18). Also, according to the results of ICV, some items were corrected. The S-CVI/Ave for IPSS was calculated as 89%.

To determine the criterion validity of the instrument, IPSS scores were compared with the final diagnosis based on brain CT scan result in hospital (Table 2). As in Table 2, 14 patients had other diagnosis (false positive) and 6 patients had lower scores of cutoff point index despite having a stroke; the tool was unable to find them as a suspected stroke (false negative).

Table 2

Comparison of the results of tool pre-hospital diagnosis of stroke patients in the hospital with a cut to the final diagnosis results in a total of 11 patients having criteria

Results Patient Tool	The final diagnosis in hospital
	Patients Number (%)	Healthy Number (%)
Above cut-off (patient)	61(81.3%)	14(18.8%)
Below cut-off (non-patient)	6(28.6%)	15(71.4%)

Also, the comparison of IPSS scores with a final diagnosis of stroke patients by discharge records in hospital, made it possible to estimate the positive predictive value (PPV), negative predictive value (NPV), positive likelihood ratio (PLR), and negative likelihood ratio (NLR)(Table 3).

Table 3

Numerical values ​​of the validity criterion of pre-hospital diagnosis tool in Iran

The criterion validity of utility values	The presented value	Confidence interval 95%
Sensitivity	92%	81.51 to 96.62%
Specificity	51%	32.54 to 70.54%
Positive Predictive Value (PPV)	81%	70.67 to 89.39%
Negative predictive value (NPV)	71%	47.83 to 88.65%
Positive Likelihood Ratio (PLR)	1.89	1.28 to 2.77
Negative Likelihood Ratio (NLR)	0.17	0.07 to 0.40

The positive predictive value of 81% was in compliance with the final diagnosis in the hospital; when patients are detected as stroke or TIA, with 81% probability of IPSS indicating suspected stroke or TIA patients. The negative predictive value of 71% was in compliance with the final diagnosis in the hospital, when a patient is not diagnosed with stroke or TIA. The IPSS confirmed the probability of 71% for stroke or TIA in patients. PLR and NLR also computed as 1.89 and 0.17, respectively.

For the remaining 15 items of instrument the internal consistency reliability, the coefficient Kuder-Richardson was 0.68.

After this step, one item due to low coefficient correlation internal consistency of the instrument was removed and finally, the coefficient Kuder-Richardson for IPSS (total of 14 items) was calculated as 0.7.

To determine the inter-rater reliability, the coefficient correlation was calculated for scores assigned to patients by three different observers and the ICC of total scores was calculated 0.96, that was significant (p<0.001). Then, based on the result of ROC curve, the cutoff point for sensitivity (92%) and specificity (51%) was calculated (10.5) respectively ( Fig. 1). That means that in coordination with the final diagnosis, patients with score less than 11 were considered as healthy (no stroke) and those with score 11 and upper, suspected as stroke or TIA.

Fig.1

ROC curve

Accuracy of the results is based on the area under the curve ROC 0.79 (CI 95%, 0.68-0.90) and was calculated (p<0.001), indicating that about 79% of cases were diagnosed correctly.

Discussion

A fast and accurate method for assessment of stroke patients by emergency medical technicians in the pre-hospital or hospital triage can reduce the delay in the early treatment or post assessment by a physician. Now, different tools are used for identifying stroke and avoid consequences of stroke in different settings. In order to minimize any delay in the start of treatment, designing and implementing a pre-hospital stroke tool for fast recognition is a current trend in the pre-hospital phase. However, there is no definitive diagnostic tool for the pre-hospital phase of stroke in whole the world. Researchers have performed several studies about designing a perfect and appropriate tool for detecting stroke in the pre-hospital stage.

Consistent with the changes in the treatment of stroke, the design of IPSS, for the first time in Iran, was established based on psychometric process and experts’ opinions.

A unique innovation in this study was the benefit of clinical experience of pre-hospital emergency technicians and other doctors associated with the diagnosis and treatment of stroke patients.

Researchers reviewed the literature to learn about other research experiences of designing similar tools and also for common signs of stroke patients.

Stroke demonstrates a wide range of neurological signs and symptoms this are similar to other central nervous system diseases and disorders such as epilepsy, diabetes, dementia and brain tumors and abscesses (19). Therefore, taking history and performing clinical examination (by technicians in the pre-hospital emergency and not too time consuming) can help to distinguish stroke from other diseases.

Brott et al, in 1989 studied previous stroke scales and common brain neurological symptoms in stroke. This research has led to the design of NIHSS (19). The NIHSS was then used as a basis for designing other pre-hospital stroke recognition tools such as ETSS, CPSS and KPSS (15, 20, 21)

This scale has 15 items with 3 to 4 parts for response range, and requires the user to spend a considerable time in pre-hospital emergency care (22).

CPSS and LAPSS are the most popular diagnostic tools in the pre-hospital setting that are recommended by the American Heart Association (AHA), American Stroke Association (ASA) and The European Stroke Organization (ESO) (23).

Despite this, none of them cover vision problems and impaired balance in the posterior circulation problems (24).

Hence, we designed the IPSS based on clinical experience of personnel involved in the care and treatment of stroke patients, a broad overview of relevant resources and previous tools. Our tool consists of 14 dichotomous (yes/no) items. At all stages of the study, we tried to design a simple and brief scale. Short and simple tools help avoid fatigue in technicians and prevent delay in patient transfer. Furthermore, we tried the scale cover common signs and symptoms base on the brain areas involved.

Clinical tools for stroke diagnosis in the acute care setting should be simple, valid and reliable. However, use of some of these tools is not widespread due to problems such as the lack of clarity in reliability and validity (25).

In this study, CVI of IPSS was calculated as 89%. Review of previous studies did not yield the content validity of previous instruments (26-28).

In 2013, Studnek et al reported sensitivity and specificity values ​​of 79% and 24% respectively for the CPSS and 74 and 33 percent respectively in Med PACS tool (29). These values are ​​less than 92 and 51 percent of our tools for sensitivity and specificity respectively. Although the specificity of IPSS is less than ROSIER tool (83%), but ROSIER sensitivity of 90% is below our tool (30). Since IPSS is a screening tool, the higher sensitivity is more important (31).

In this study the internal consistency reliability was indicated with coefficient Kuder-Richardson as 0.7. This result showed that domains have positive correlation and are significant with each other, as well as with total scores. It demonstrated that 30% of variances of total scores of the instrument depend on probability error. The internal consistency value (α>0.9) was good and when (α>0.8) is at low levels may be indicative of the notion that there is more than one good internal consistency value in the α hardware tools. High values ​​indicate good internal consistency α, but must be repeated. Repeated expressions cause unnecessary prolongation of the tools and due to measure of the same concepts, it indicates a good internal consistency (17).

We need different items, which cover stroke symptoms in different parts of the brain. This seems to be one of the reasons of the low coefficient Kuder Richardson in this study. For example, items such as headaches, despite the high frequency in patients with acute cerebral stroke, particularly hemorrhagic stroke, lead to a decrease in internal consistency of the instrument; though we maintain it due to its importance in stroke diagnosis.

Check reliability of the NIHSS has been by test-retest and inter-rater reliability. Its reported correlation in test-retest is 0.66-0.77 and agreement between observer, 0.69 (19).

Inter-rater reliability in our study was 0.96. This outcome suggests the present scale has excellent agreement among the participants, including nurses and pre-hospital emergency technicians.

In this study, one reason for the high level of agreement between observers can be objective responses and the other the dichotomous items.

Besides, there is no training course for pre-hospital emergency technicians in such studies. Only research purposes are described and they are asked to use the tools via the attached tool guide, for completing a scale. We believe one reason for deficit in sensitivity and specificity ratio in some scales is lack of pre-hospital emergency medical technicians’ knowledge about importance of rapid transport and treatment of stroke patients. It is recommended that in future studies; pre-hospital emergency technicians’ take an appropriate training course about the importance of immediate action in the treatment of stroke and emphasizing the role of pre-hospital emergency care in the treatment of these patients.

Gur et al. used ROC curve in order to determine their scale cutoff point (32).

This study reported accuracy of the instrument as 79% in which according to table 4 is set at a good level. Accuracy of the instruments is determined by the area under the curve (AUC) (33).

Table 4

Relationship between the area under the ROC curve and diagnostic accuracy.

Diagnostic accuracy	Area
Excellent	0.9 –1.0
Very Good	0.8 -0.9
Good	0.7 -0.8
Sufficient	0.6 -0.7
Bad	0.5 -0.6
Test not useful	< 0.5

Our research limitations can be lack of access to national studies, and lack of evaluation by other diagnostic and treatment professional groups (for example emergency medical dispatch, triage nurses etc.). Positive points of this tool are preserving the simplicity, eloquence, brevity and logical sequence of items.

During this study we attempted to create a tool for emergency medical technicians based on a good pre-hospital stroke tool definition. After the abovementioned procedures, the IPSS consisted of 14 items; with dichotomous responses (yes/no), with cut-off point of 11 and psychometric properties, was designed and now is ready for application.

Conclusion

IPSS is a pre-hospital stroke scale with simple dichotomous response options for emergency technicians in Iran. The IPSS could be recommended and employed by emergency dispatch units and be used in the triage procedure in the hospital.