Stepwise stroke recognition through clinical information, vital signs, and initial labs (CIVIL): Electronic health record-based observational cohort study.

BACKGROUND: Stroke recognition systems have been developed to reduce time delays, however, a comprehensive triaging score identifying stroke subtypes is needed to guide appropriate management. We aimed to develop a prehospital scoring system for rapid stroke recognition and identify stroke subtype simultaneously. METHODS AND
FINDINGS: In prospective database of regional emergency and stroke center, Clinical Information, Vital signs, and Initial Labs (CIVIL) of 1,599 patients suspected of acute stroke was analyzed from an automatically-stored electronic health record. Final confirmation was performed with neuroimaging. Using multiple regression analyses, we determined independent predictors of tier 1 (true-stroke or not), tier 2 (hemorrhagic stroke or not), and tier 3 (emergent large vessel occlusion [ELVO] or not). The diagnostic performance of the stepwise CIVIL scoring system was investigated using internal validation. A new scoring system characterized by a stepwise clinical assessment has been developed in three tiers. Tier 1: Seven CIVIL-AS3A2P items (total score from -7 to +6) were deduced for true stroke as Age (≥ 60 years); Stroke risks without Seizure or psychiatric disease, extreme Sugar; "any Asymmetry", "not Ambulating"; abnormal blood Pressure at a cut-off point ≥ 1 with diagnostic sensitivity of 82.1%, specificity of 56.4%. Tier 2: Four items for hemorrhagic stroke were identified as the CIVIL-MAPS indicating Mental change, Age below 60 years, high blood Pressure, no Stroke risks with cut-point ≥ 2 (sensitivity 47.5%, specificity 85.4%). Tier 3: For ELVO diagnosis: we applied with CIVIL-GFAST items (Gaze, Face, Arm, Speech) with cut-point ≥ 3 (sensitivity 66.5%, specificity 79.8%). The main limitation of this study is its retrospective nature and require a prospective validation of the CIVIL scoring system.
CONCLUSIONS: The CIVIL score is a comprehensive and versatile system that recognizes strokes and identifies the stroke subtype simultaneously.

Introduction

Stroke remains with a high burden in societies, and improving the recognition of a stroke can help reduce this burden [1]. “Time-is-brain” is a crucial concept in ischemic stroke management [2,3], the importance of early recanalization has been further addressed after the success of endovascular recanalization therapies [4]. In the case of hemorrhagic stroke, early surgical interventions can be beneficial in selected patients [5,6]. Candidates for urgent interventions should be transported to an appropriately-equipped hospital, however, treatment can be delayed due to various reasons [7].

Stroke recognition systems have been developed to reduce time delay in community and hospital settings [8,9]. Various scales and scoring systems have been developed, but there is still no consensus on which scale perform better. [10,11]. Previous systems have issues related to false positives and false negatives, making it difficult for stroke specialists to handle a considerable number of patients [12,13]. In addition, most research has covered only one aspect of stroke, ‘true stroke or not’ or ‘selection of emergent large vessel occlusion (ELVO),’ and clinical parameters can be subjective even after thorough training. Finally, previously published systems have a greater focus on reducing pre-hospital delay [14,15]. Therefore, a comprehensive triaging system for considering next-step treatments should address to reduce the workload at stroke centers with acceptable sensitivity and specificity.

In this context, we have developed a new scoring system using Clinical Information, baseline Vital sign, and Initial Labs (CIVIL). Here, we aimed to evaluate the feasibility of the CIVIL system and compare with previous screening systems in suspicious acute stroke patients.

Methods

Study population

This electronic health record-based observational cohort study was performed in a tertiary referral hospital from January 2012 to December 2015. Care in the tertiary stroke center fulfilled the Brain Attack Coalition’s standardized criteria, and the stroke unit had also obtained certification from the Korean Stroke Association [16]. The regional emergency medical center serves the southern part of Gyeonggi Province of South Korea with a population of approximately four million, and it’s emergency room (ER) has approximately 89,000 patients annually [17]. Previously, we developed the stroke recognition system ‘Cubic S model’ (S1 Fig), which is based on the Electronic Medical Record (EMR) system for suspected stroke patients in the ER [18]. It is based on common signs and symptoms and contains three domains: time, body-part involvement, and symptomatic presentations. When suspicious stroke patients visit the ER, ER physicians check the presence of three domains: sudden onset, one-sided involvement of face/arm/leg, and 6 representative symptoms of stroke.

Each dataset was automatically stored in the database from a prospectively registered critical pathway system for rapid thrombolysis in suspicious stroke patient. The data has been used to improve the quality of registered data through monthly reports. Inclusion criteria for this study were (a) acute neurologic manifestations within 6 hours, (b) acute thrombolysis code activating cases who meets all three domains of EMR-based Cubic S system, and (c) final confirmation with clinical and imaging findings by stroke neurologists. Exclusion criteria were: (a) incorrect activation of an acute thrombolysis code, (b) onset-to-door time > 6 hours, and (c) uncertain final diagnosis due to incomplete study. The study protocol was approved by the Institutional Review Board of Ajou University Hospital (AJIRB-MED-MDB-16-407). Informed consent was waived because of the study’s retrospective nature.

Processing after critical pathway activation

Initial assessment and activation of acute thrombolysis code was performed by ER physicians. Education of ER physicians and nurses was routinely performed every 6 to 12 months by stroke neurologists. Immediately after the activation of acute thrombolysis code, stroke neurologists assessed the patients. All patients except those with contraindications to contrast use underwent computed tomography (CT) scan with angiography. Simultaneously, neurologists meticulously investigated clinical information, baseline vital signs, initial laboratory findings, and stroke images. In cases where recanalization treatments were needed, intravenous recombinant tissue plasminogen activator and/or endovascular therapies were implemented according to critical pathway in our institute. Clinical information was recorded including age, sex, prior medical histories (hypertension, diabetes mellitus, previous stroke occurrence, seizure or syncope, and psychiatric history) from the patient, care-givers, or paramedics. Initial neurological manifestations were assessed with the Cubic S model. Baseline vital signs were comprised of blood pressure, pulse rate, and body temperature, and initial laboratory findings (glucose level and oxygen saturation) were also included.

Confirmation of final diagnosis

The adjudication meeting comprised of stroke specialists was held weekly for final diagnoses of all patients according to the three tiers; as stroke mimic vs. true stroke, ischemic vs. hemorrhagic stroke, non-ELVO vs. ELVO. Final diagnosis was determined after review of ER chart, imaging and laboratory studies for differential diagnoses. True stroke was diagnosed when the neurologic exam was compatible with supportive imaging evidence of CT and/or magnetic resonance imaging including diffusion weighted image. Transient cerebral ischemic attack was classified into the true stroke and ischemic stroke group. Stroke mimics were designated when the clinical details were compatible with non-vascular etiologies. Initial CT angiography confirmed hemorrhagic stroke and large artery occlusion. ELVO was designated as occlusion of the internal carotid artery, M1 or M2 segment of the middle cerebral artery, or basilar artery [19,20].

Development of new scoring system

We intended to develop a new scoring system that is characterized by a “stepwise clinical assessment system which enables rapid discrimination of patients suspected of acute stroke,” potential variables were assessed including Clinical Information, Vital signs, and Initial Labs (CIVIL) used in ER and prehospital settings. There were 23 clinical findings, four vital signs, and two laboratory findings. We analyzed these variables by the three tiers; stroke mimic vs. true stroke, ischemic vs. hemorrhagic stroke, non-ELVO vs. ELVO. At the first tier (stroke mimic vs. true stroke), we assigned +1 to positive variables for true stroke (odds ratio [OR] > 1.0) and -1 to negative variables (OR <1.0). In the second tier, items suggestive of hemorrhagic stroke were derived (only positive scores). Finally, to discriminate ELVO from non-ELVO, we applied the gaze to face-arm-speech-time (GFAST) scoring system [21]. For evaluation of performance, the CIVIL system was compared with three previous recognition systems for acute stroke: Cincinnati Prehospital Stroke Scale (CPSS) [14], Los Angeles Prehospital Stroke Screen (LAPSS) [15], and Recognition Of Stroke In the Emergency Room (ROSIER) system [8].

Statistical analysis

Differences between the two groups at each steps were analyzed using χ or Student t-test for categorical and continuous variables, respectively. Significant variables from univariate analyses (p<0.05) were assessed with multivariate logistic regression models for deduction of scoring items (enter method). Associations were presented as odds ratios (OR) with corresponding 95% confidence interval (CI). Internal validation using receiver operating characteristic (ROC) curve analysis was performed to determine the optimal cut off point for each steps. Diagnostic performance including sensitivity, specificity, positive predictive value, negative predictive value, and Youden index were assessed for each cut-off point. We performed all analyses using SPSS 25.0 for Windows (SPSS Inc., Chicago, Ill).

Results

Patient assessment

The flow chart of study population is shown in S2 Fig. A total of 1,621 patients were screened by acute thrombolysis code activation. Sixty-two patients were excluded, and the remaining 1,559 suspected stroke patients were enrolled, of these, true stroke was confirmed in 1,153 (74.0%). Causes of stroke mimicking symptoms were metabolic disease (18.0%), drug intoxication (15.0%), peripheral neuropathy (14.3%), psychogenic disorder (14.3%), seizure (13.8%), infectious disease (7.4%), syncope (6.9%), and tumorous condition (3.4%). True stroke patients comprised of ischemic stroke (n = 894, 77.5%) and hemorrhagic stroke (n = 259, 22.5%), and the number of ischemic stroke patient requiring recanalization therapy was 291 (32.6%).

Clinical information, baseline vital signs, and initial labs (CIVIL)

Tables 1 and 2 summarizes detailed findings according to the final diagnosis. In the first tier (stroke mimic vs. true stroke), true stroke patients were older and male-dominant. History of stroke risk factor were more frequent in the true stroke group, whereas history of seizure or psychiatric disease were less common. From clinical manifestations, “after awakening”, lateralizing symptoms, “not ambulating”, and “not able to grasp” were more prevalent in the true stroke group, while “mental change” was more frequent in the stroke mimic group. From vital-sign and initial laboratory findings, systolic and diastolic BP were higher in the true stroke group. In contrast, the stroke mimic group included more patients with low systolic BP (≤90mmHg) and extreme glucose level (initial glucose <80 or ≥400 mg/dl).

Table 1

Clinical information, vital signs, and initial labs (CIVIL).

	Stroke- mimic (n = 406)	True stroke (n = 1,153)	pa	Ischemic stroke (n = 894)	Hemorrhagic stroke (n = 259)	pb
Clinical information
Age, years	62.2 ± 15.8	64.5 ± 14.2	0.012	65.6 ± 14.0	60.5 ± 14.5	<0.001
Age ≥ 60 years, n (%)	230 (56.7)	743 (64.4)	0.005	613 (68.6)	130 (50.2)	<0.001
Age ≤ 40 years, n (%)	38 (9.4)	57 (4.9)	0.001	40 (4.5)	17 (6.6)	0.172
Male, n (%)	200 (49.3)	699 (60.6)	<0.001	561 (62.8)	138 (53.3)	0.004
Onset-to-door time (minute)	172.6 ± 209.9	182.9 ± 205.2	0.118	197.9 ± 210.0	131.2 ± 178.7	<0.001
Onset-to-door ≥ 90 min, n (%)	173 (42.6)	466 (40.4)	0.439	314 (35.1)	152 (58.7)	<0.001
Prior history, n (%)
Hypertension	170 (41.9)	606 (52.6)	<0.001	473 (52.9)	133 (51.4)	0.355
Diabetes	100 (24.6)	250 (21.7)	0.419	214 (23.9)	36 (13.9)	<0.001
Cardiac diseases	69 (17.0)	263 (22.8)	<0.001	244 (27.3)	19 (7.3)	<0.001
Previous stroke	78 (19.2)	242 (21.0)	0.120	199 (22.3)	43 (16.6)	0.028
Seizure or psychiatric history	89 (21.9)	21 (1.8)	<0.001	19 (2.1)	2 (0.8)	0.192
Clinical manifestations
Time
“Sudden”, n (%)	365 (89.9)	1038 (90.0)	0.172	786 (87.9)	252 (97.3)	<0.001
“After awakening”, n (%)	32 (7.9)	114 (9.9)	0.021	105 (11.7)	9 (3.5)	<0.001
“As unusual”, n (%)	13 (3.2)	34 (2.9)	0.476	31 (3.5)	3 (1.2)	0.034
Body-spatial
“one-side arm”, n (%)	144 (35.5)	821 (71.2)	<0.001	642 (71.8)	179 (69.1)	0.221
“one-side leg”, n (%)	120 (29.6)	720 (62.4)	<0.001	557 (62.3)	163 (62.9)	0.457
“one-side face”, n (%)	72 (17.7)	401 (34.8)	<0.001	306 (34.2)	95 (36.7)	0.255
“any asymmetry’, n (%)	205 (50.5)	958 (83.1)	<0.001	763 (85.3)	195 (75.3)	<0.001
Symptoms
“not ambulating”, n (%)	79 (19.5)	508 (44.1)	<0.001	388 (43.4)	120 (46.3)	0.222
“not able to speak”, n (%)	185 (45.6)	576 (50.0)	0.077	447 (50.0)	129 (49.8)	0.506
“not able to grasp”, n (%)	37 (9.1)	206 (17.9)	<0.001	164 (18.3)	42 (16.2)	0.245
“mental change” *, n (%)	156 (38.4)	193 (16.7)	<0.001	92 (10.3)	101 (39.0)	<0.001
“abnormal sensation”, n (%)	56 (13.8)	162 (14.1)	0.247	138 (15.4)	24 (9.3)	0.006
“visual disturbance”, n (%)	4 (1.0)	18 (1.6)	0.165	17 (1.9)	1 (0.4)	0.062
Baseline vital signs
SBP, mmHg	137.7 ± 28.9	151.1 ± 28.6	<0.001	147.7 ± 25.4	163.0 ± 34.9	<0.001
SBP ≥ 160mmHg, n (%)	100 (24.6)	473 (41.0)	<0.001	327 (36.6)	146 (56.4)	<0.001
SBP ≥ 140mmHg, n (%)	202 (49.8)	801 (69.5)	<0.001	598 (66.9)	203 (78.4)	<0.001
SBP ≤ 90mmHg, n (%)	12 (3.0)	3 (0.3)	<0.001	1 (0.1)	2 (0.8)	0.128
DBP, mmHg	81.9 ± 34.7	86.8 ± 16.4	0.018	85.6 ± 15.5	90.6 ± 18.7	0.005
Pulse rate, bpm	84.2 ± 19.1	82.9 ± 15.5	0.454	82.8 ± 15.9	83.2 ± 13.9	1.000
Body temperature, °C	36.5 ± 0.8	36.5 ± 0.5	1.000	36.5 ± 0.4	36.4 ± 0.6	0.173
Initial laboratory findings
Glucose, mg/dl	150.0 ± 101.6	146.8 ± 60.1	0.510	144.0 ± 60.5	156.4 ± 57.8	0.051
Extreme glucose level†, n (%)	24 (5.9)	14 (1.2)	<0.001	11 (1.2)	3 (1.2)	1.000
Oxygen saturation, %	99.3 ± 3.2	99.5 ± 2.5	0.062	99.6 ± 2.2	99.2 ± 3.2	0.082

SBP means systolic blood pressure, and DBP indicates diastolic blood pressure. *”mental change” was defined when a decrease in the level of consciousness below drowsiness was observed on initial neurological examination. †initial blood glucose level ≤ 80 or ≥ 400 mg/dl, pa = Stroke mimic vs. true stroke, pb = Ischemic stroke vs. hemorrhagic stroke

Table 2

Clinical information, vital signs, and initial labs (CIVIL) between ELVO and non-ELVO patients.

	ELVO stroke (n = 291)	Non-ELVO stroke (n = 603)	p
Clinical information
Age, years	68.1 ± 13.6	64.4 ± 14.0	<0.001
Male, n (%)	171 (58.8)	390 (64.7)	0.087
Onset-to-door time (minute)	191.8 ± 208.8	200.9 ± 210.7	0.045
Prior history, n (%)
Hypertension	160 (55.0)	313 (51.9)	0.388
Diabetes	59 (20.3)	155 (25.7)	0.075
Cardiac problems	115 (39.5)	129 (21.4)	<0.001
Previous stroke	57 (19.6)	142 (23.5)	0.182
Manifestation
Time domain
“Sudden”, n (%)	260 (89.3)	526 (87.2)	0.363
“After awakening”, n (%)	30 (10.3)	75 (12.4)	0.354
“As unusual”, n (%)	10 (3.4)	21 (3.5)	0.972
Body-spatial domain
“one-side arm”, n (%)	241 (82.8)	401 (66.5)	<0.001
“one-side leg”, n (%)	217 (74.6)	340 (56.4)	<0.001
“one-side face”, n (%)	112 (38.5)	194 (32.2)	0.062
“any asymmetry’, n (%)	257 (88.3)	506 (83.9)	0.081
Symptom domain
“not ambulating”, n (%)	159 (54.6)	229 (38.0)	<0.001
“not able to speak”, n (%)	188 (64.6)	259 (43.0)	<0.001
“not able to grasp”, n (%)	53 (18.2)	111 (18.4)	0.944
“mental change” *, n (%)	59 (20.3)	33 (5.5)	<0.001
“abnormal sensation”, n (%)	14 (4.8)	124 (20.6)	<0.001
“visual disturbance”, n (%)	9 (3.1)	8 (1.3)	0.070
“gaze deviation”, n (%)	197 (67.7)	48 (8.0)	<0.001
Baseline vital signs
SBP, mmHg	143.1 ± 26.4	149.9 ± 24.7	<0.001
DBP, mmHg	83.5 ± 16.0	86.7 ± 15.1	0.003
Pulse rate, bpm	84.2 ± 18.1	82.1 ± 14.7	0.064
Body temperature, °C	36.4 ± 0.4	36.5 ± 0.4	0.004
Initial laboratory findings
Glucose, mg/dl	142.4 ± 49.3	144.8 ± 65.2	0.571
Oxygen saturation, %	99.5 ± 1.7	99.7 ± 2.4	0.202
Stroke characteristics
NIHSS, median (IQR)	16 (12–20)	3 (1–6)	<0.001
TOAST classification, n (%)			<0.001
Large artery disease	69 (23.7)	105 (17.4)
Cardioembolism	157 (54.0)	106 (17.6)
Small artery disease	0 (0.0)	134 (22.2)
Others	65 (22.3)	258 (42.8)
Vessel occlusion, n (%)
ICA	90 (30.9)	-
M1	102 (35.1)	-
M2	7 (16.2)	-
BA	33 (11.3)	-
Others	19 (6.5)	-
tPA use, n (%)	155 (53.3)	83 (13.8)	<0.001
Endovascular treatment, n (%)	150 (51.5)	0 (0.0)	<0.001

*”mental change” was defined when a decrease in the level of consciousness below drowsiness was observed on initial neurological examination. SBP = systolic blood pressure, DBP = diastolic blood pressure, NIHSS = National Institute of Health Stroke Scale, ICA = internal carotid artery, BA = basilar artery, tPA = tissue plasminogen activator

In the second tier (ischemic vs. hemorrhagic stroke), the ischemic stroke group was older and had a higher proportion of males than the hemorrhagic stroke group. The patients with hemorrhagic stroke had shorter onset-to-door time. History of stroke risk factor was more prevalent in the ischemic stroke group. “Sudden onset” was more frequent in the hemorrhagic stroke, while “after awakening” and “as unusual” were more common in the ischemic stroke. The ischemic stroke group showed more common “any asymmetry. “Mental change” was more prevalent in the hemorrhagic stroke group, while “abnormal sensation” was more frequent in the ischemic stroke group. Systolic and diastolic BP were higher in the hemorrhagic stroke group.

Tier 1: Stroke mimic vs. true stroke: CIVIL-

We determined independently significant factors in the CIVIL system using multiple regression analysis (Fig 1A). ge (≥60 years), troke risks (history of cardiac disease), “any symmetry”, and “not mbulating” were positive discriminatory items for diagnosis of true stroke, while younger ge (≤40 years), history of eizure or psychiatric disease were negative discriminatory items. In vital signs and laboratory data, high B (systolic BP ≥140mmHg) was a positive discriminatory item, and low B (systolic BP ≤90mmHg) and extreme ugar level (≤80 or ≥400 mg/dL) were included as negative discriminatory items. Asymmetric leg weakness, non-lateralizing symptoms, mental change, and initial oxygen saturation were indiscriminate. The CIVIL- score was finally determined as overall 7 items (Fig 2): 5 clinical items, 1 vital sign, and 1 initial laboratory finding. The total score ranged from -5 to +6. Retrospective validation on 1,559 suspected stroke patients determined an optimal cut-off point for stroke diagnosis as ≥ +1. At this cut-off point, the diagnostic performance of CIVIL- score was as follows: sensitivity 82.1%, specificity 56.4%, positive predictive value (PPV) 84.3%, and negative predictive value (NPV) 52.6% (Youden’s index 0.385). We compared the performance of the CIVIL-ASAP score to the CPSS, LAPSS, and ROSIER scales in our data set. The sensitivity and specificity of these established recognition systems were 90.4% and 29.1% in CPSS, 69.7% and 67.7% in LAPSS, 93.8% and 34.0% in ROSIER. In ROC curve analysis, CIVIL- score had a superior diagnostic performance than the other three systems per area under the curve (S3 Fig, 0.767 in CIVIL-ASAP vs. 0.751 in ROSIER vs. 0.687 in LAPSS vs. 0.597 in CPSS). Comparisons among early stroke recognition scales were described in Fig 3.

Fig 1

Results of multiple regression analysis and distribution of patients according to the CIVIL scores.

(A) Tier 1: CIVIL- score consisted of ge (≤40 years or ≥60 years), troke risk (cardiac disease history) without eizure or psychiatric history, extreme ugar level (≤80 or ≥400mg/dl), any symmetry, not mbulating, and ressure (SBP≤90 mmHg or ≥140mmHg). (B) Tier 2: CIVIL- included ental change, ge (≤60 years), ressure (SBP≥160mmHg), and no troke risks (history of diabetes mellitus or cardiac disease). (C) Tier 3: To identify emergent large vessel occlusion patients, GFAST score was incorporated into the CIVIL scoring system (aze, ace asymmetry, rm asymmetry, peech disturbance).

Fig 2

Summary of items and scoring in the CIVIL system.

In tier 1 (CIVIL-), 7 items which included clinical information (white), vital signs (grey), and initial labs (dark grey) were used. Stroke-preferred items were assigned positive points and stroke mimic preferred items were negative points (ranged from ―5 to +6). Tier 2 (CIVIL-) allocated 4 items with clinical information and vital signs, and the GFAST system was applied in tier 3 (CIVIL-) for the selection of ELVO patients.

Fig 3

Descriptive comparison of various early stroke recognition scales.

CIVIL = Clinical Information, Vital signs, and Initial Labs, CPSS = Cincinnati Prehospital Stroke Scale, LAPSS = Los Angeles Prehospital Stroke Screen, ROSIER = Recognition Of Stroke In the Emergency Room.

Tier 2: Ischemic vs. hemorrhagic stroke: CIVIL-

To differentiate between ischemic and hemorrhagic stroke, second tier analysis using multiple regression was conducted (Fig 1B). : ental change, ge below 60 years, high blood ressure (systolic BP ≥160mmHg), no troke risk (without history of diabetes or cardiac disease) were positive discriminatory items for diagnosis of hemorrhagic stroke (Fig 2). The CIVIL- score consisted of 3 clinical items and 1 vital sign, and the total score ranged from 0 to +4. Retrospective validation on the 1,153 true stroke patients determined an optimal cut-off point for hemorrhagic stroke diagnosis as ≥ 2. At this cut-off point, the diagnostic performance of CIVIL- score was as follows: sensitivity of 47.5%, specificity of 85.4%, PPV of 50.6%, NPV of 83.8% (Youden’s index 0.329).

Tier 3: Non-ELVO vs. ELVO: CIVIL-

In the final tier, we applied the GFAST score to select ELVO patients (Fig 1C). The score was calculated as the sum of positive symptoms: aze deviation, ace asymmetry, rm asymmetry, and peech disturbance (Fig 2). Retrospective validation on 894 ischemic stroke patients determined the optimal cut-off point for ELVO diagnosis as ≥ 3. At this cut-off point, the diagnostic performance of CIVIL- score was as follows: sensitivity of 66.5%, specificity of 79.8%, PPV of 54.6%, NPV of 86.7% (Youden’s index 0.463). The CIVIL scoring system is summarized in Fig 2.

Discussion

Our data support that the CIVIL scoring system is feasible for identifying suspicious acute stroke patients in a stepwise fashion: true stroke or not, hemorrhagic stroke or not, and ELVO or not. In addition, step-by-step acronyms (ASAP, MAPS, and GFAST) can be used in a wide range of fields of prehospital and ER-based situations to serve as triaging tools for patients to be easily remembered.

The CIVIL scoring system can help us to differentiate different types of stroke at the same time. Acute stroke is an urgent condition that requires rapid evaluation and proper management because the longer a stroke goes untreated, the greater the brain damage (time is brain) [22]. Efficient triaging is important for acute stroke patients to guide proper disposition and early interventions, which may be entirely decisive in some cases [23,24]. Due to limited time window for thrombolytic therapy, numerous prehospital scoring systems for early recognition of ischemic stroke have been developed [8,14,15]. Recently, endovascular recanalization therapy in ELVO patients has been proven as the standard treatment, consequently, several scoring systems to recognize ELVO have been addressed [19-21]. However, current scoring systems have focused only on one aspect of stroke identification: stroke versus stroke mimic, or ELVO discrimination [25]. In addition, little has been elucidated to distinguish hemorrhagic stroke from ischemic stroke especially in situations with limited imaging facilities [26]. To the best of our knowledge, there has been no definitive scoring system that integrates various aspects of stroke diagnosis.

This scoring system features a stepwise approach to triage stroke suspicious patients. When paramedics in emergency medical service (EMS) or ER physicians proceed step by step, they will be able to properly classify stroke patients who require rapid treatment. CIVIL- initially differentiates patients with true stroke from stroke mimics. In this first tier, it is important not to exclude potential patients who need. In this context, as we expected, CIVIL- showed relatively high sensitivity and low specificity for including all possible candidates. Second (CIVIL-) and third tiers (CIVIL-) showed low sensitivity and high specificity so that patients in need of urgent treatments (thrombolysis and/or endovascular therapy) could be selected effectively. The CIVIL scoring system enables rapid identification of patients delivered to the ER with high sensitivity to identify the actual stroke, and also enables the recognition of hemorrhagic stroke and ELVO with high specificity.

The CIVIL scoring system included objective vital signs and laboratory findings as well as clinical information. Previous scoring systems consisting of clinical manifestations may be affected by the examiner’s experience and special training is needed to reduce inter-observer variability [27]. Some validation studies on early recognition scoring systems reported high variability in inter-observer reliability ranging from 69% to 90% [14,28]. To compensate for these variations, ROSIER [8] and LAPSS [15] included laboratory finding such as blood glucose levels. For this reason, the CIVIL scoring system contained vital signs in addition to laboratory findings designed to apply more objective parameters. Moreover, the extreme values of vital signs and laboratory findings-blood glucose level ≤ 80 or ≥ 400mg/dl and systolic BP ≥140 mmHg or ≤ 90 mmHg help to discriminate stroke mimic conditions such as sepsis, shock, or syncope. Therefore, our new scoring system could overcome some potential limitations of other previous scoring systems by including objective and quantitative items.

In this study, the CIVIL scoring system was developed for use in both ER and prehospital settings. The selection of acute stroke patients in the prehospital and emergent setting continue to be the subject of research due to the time-dependent nature of stroke [27]. Various early recognition systems have been used in the EMS to properly transport stroke patients to more appropriate centers. However, there have been several limitations including inconvenience, imperfect accuracy, and time-consuming training [19,20]. Moreover, an increase in items adds complexity to the system for rapid evaluation [29]. The CIVIL scoring system applies an intuitive and easy-to-remember acronym for EMS and other medical professionals to be easily used. We applied simple and familiar GFAST to improve accessibility in the third tier for the identification of ELVO patients.

Our study has some limitations. First, it is an observational study with retrospective nature; however, all information has been automatically stored in a prospectively-collecting database at a large regional emergency and stroke center. Second, data with time windows over 6 hours were not covered in the current study. From recent trials, mechanical thrombectomy is indicated up to 24 hours after stroke onset. Nevertheless, most patients with onset to treatment time less than 6 hours need more urgent treatment regardless of core-penumbra mismatch, so that our recognition system can more properly apply to those patients. Third, there are limits in the conclusions that can be drawn regarding the performance of the CIVIL system in patients with posterior circulation acute ischemic stroke. This scoring system was designed to focus on patients with anterior circulation which is supported by the current guideline for endovascular treatment. Finally, the sensitivity of tier 2 and 3 are less than 80%, the results should be interpreted with caution. In the future, prospective validation of the CIVIL scoring system should include a systematic education program for paramedics to improve performance.

In conclusion, the CIVIL scoring system can be used as a comprehensive and versatile tool to recognize true stroke and identify stroke subtypes simultaneously.

(DOC)

Click here for additional data file.

Deidentified raw dataset.

(XLSX)

Click here for additional data file.

The Korean version of EMR based matrix for stroke suspicious patients.

(Cubic S model)

(PDF)

Click here for additional data file.

Flow diagram of 1,621 suspicious stroke patients.

(PDF)

Click here for additional data file.

Receiver-Operating Characteristic (ROC) curve and corresponding area under the curve (AUC) statistics of the CIVIL scoring system.

(PDF)

Click here for additional data file.

Transfer Alert

This paper was transferred from another journal. As a result, its full editorial history (including decision letters, peer reviews and author responses) may not be present.

20 Jan 2020

PONE-D-19-32649

Stepwise stroke recognition through Clinical Information, Vital signs, and Initial Labs (CIVIL): Electronic health record-based observational cohort study

PLOS ONE

Dear Dr. Hong,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

We would appreciate receiving your revised manuscript by Mar 05 2020 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Juan Manuel Marquez-Romero, M.D., M.Sc.

Academic Editor

PLOS ONE

Journal requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

http://www.journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and http://www.journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. We note that you have indicated that data from this study are available upon request. PLOS only allows data to be available upon request if there are legal or ethical restrictions on sharing data publicly. For information on unacceptable data access restrictions, please see http://journals.plos.org/plosone/s/data-availability#loc-unacceptable-data-access-restrictions.

In your revised cover letter, please address the following prompts:

a) If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially identifying or sensitive patient information) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent.

b) If there are no restrictions, please upload the minimal anonymized data set necessary to replicate your study findings as either Supporting Information files or to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. Please see http://www.bmj.com/content/340/bmj.c181.long for guidelines on how to de-identify and prepare clinical data for publication. For a list of acceptable repositories, please see http://journals.plos.org/plosone/s/data-availability#loc-recommended-repositories.

We will update your Data Availability statement on your behalf to reflect the information you provide.

Reviewers' comments:

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: No

Reviewer #2: Yes

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: No

Reviewer #2: Yes

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This paper is interesting, but the sensitivity results are less than 80% for tier 2 and 3. The stroke is a serious disease with potential treatment therefore requires better levels of sensitivity. I recommend mentioning this finding as a weakness of the study.

Reviewer #2: I had the pleasure to read and review the manuscript: "Stepwise stroke recognition through Clinical Information, Vital signs, and Initial Labs (CIVIL): Electronic health record-based observational cohort study", which is a very interesting article addressing the main objective of develop a scoring system for three different scenarios: ischemic stroke, hemorrhagic stroke and LVO, in a pre-hospital stage. Before considering this manuscript for publication, I have some comments to add:

1) Abstract:

Nothing to add

2) Introduction:

One of the main aspects that should be mentioned in this manuscript, is the current availability of scales and scoring systems in the pre-hospital management of possible stroke cases. Two recent articles (Stroke. 2019;50:e285–e286 and Stroke. 2015 Jun;46(6):1513-7) already intended to analyze the complexity of the different systems that are currently part of many emergency and pre-hospital departments around the world. The justification of doing your research, seems that optimal and more valid scoring systems are needed, but I can't see this rationale the way you are presenting the manuscript.

2) Methods:

- Data availability (repository or by request) should be clarified.

- I understand that information to develop this scoring system is based on information from the Emergency Department, mainly from data that already is being recruited from the Cubic S model, but all this information is captured AT the emergency department, and was transferred to the pre-hospital scenario, which seems logic, but also is a different area of work and the possibility of losing information from the real pre-hospital work is plausible.

- Statistical analysis: which variables were included at the multivariate analysis? Were they pre-defined? If so, which cut-off point was decided to include the variable in the multivariate model.

- Youden's Index together with the scoring system performance should be included.

- OR and CI 95% are included in your analysis, but this also should be mentioned at the "statistics" section

- As you are doing only internal validation of your scoring system, did you consider a bootstraping to evaluate a more accurate performance in your population?

3) Results

- The decision on each variable included in the CIVIL ASAP tool, was done based on current, recent and the highest level of bibliography, or only extracted from the dataset.

- How do you define "mental change"? In terms of the MAPS scoring system?

- I can't see any of the results referring to the multivariate model... Do all the OR are un-adjusted or adjusted? If you adjusted, which variables were included at the model?

- Did you perform a ROC-AUC analysis to evaluate the performance of your scoring system compared to the other systems (you mention that at the methods section)? Could you provide a figure of the comparison of each curve according to the pre-hospital score used to recognize a "confirmed stroke case"

4) Discussion:

- Only one comment: current scoring systems are very easy to use; its performance vary, and seems that the CIVIL has a very good opportunity to prove your rationale, but I think that the applicability of the scoring point is very difficult, so, you should try to convince the readers that they should use this system.

**********

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

19 Feb 2020

Reviewer #1:

Comment 1. This paper is interesting, but the sensitivity results are less than 80% for tier 2 and 3. The stroke is a serious disease with potential treatment therefore requires better levels of sensitivity. I recommend mentioning this finding as a weakness of the study.

Response 1. Thank you for the reviewer’s valuable comment. We totally agree with the reviewer’s comment. As the reviewers mentioned, the sensitivity of CIVIL score is limited, and the results should be interpreted with caution. The CIVIL scoring system had a relative high sensitivity in tier 1 (to identify all possible candidates), and a relative high specificity in tier 2 and 3 (to recognize the need for urgent treatments, such as intravenous thrombolysis or endovascular thrombectomy). Therefore, we added limitation of the sensitivity results in discussion section of the revised manuscript (p20-1). Please check this point.

Reviewer #2: I had the pleasure to read and review the manuscript: "Stepwise stroke recognition through Clinical Information, Vital signs, and Initial Labs (CIVIL): Electronic health record-based observational cohort study", which is a very interesting article addressing the main objective of develop a scoring system for three different scenarios: ischemic stroke, hemorrhagic stroke and LVO, in a pre-hospital stage. Before considering this manuscript for publication, I have some comments to add:

1) Abstract:

Nothing to add

2) Introduction:

Comment 1. One of the main aspects that should be mentioned in this manuscript, is the current availability of scales and scoring systems in the pre-hospital management of possible stroke cases. Two recent articles (Stroke. 2019;50:e285–e286 and Stroke. 2015 Jun;46(6):1513-7) already intended to analyze the complexity of the different systems that are currently part of many emergency and pre-hospital departments around the world. The justification of doing your research, seems that optimal and more valid scoring systems are needed, but I can't see this rationale the way you are presenting the manuscript.

Response 1. Thank you for the reviewer’s valuable comment. Various scales and scoring systems have been developed and analyzed, but there is still no consensus on which scale work better, especially in large vessel occlusion (LVO).1,2 In addition, the accuracy of EMS stroke recognition is not enough due to issues related to false positives and false negatives.3 As noted by the reviewer, the need to develop a more optimized and valid scoring system should have been more convincing. Therefore, we added this rationale in the introduction section of the revised manuscript (p5). Please consider this point.

1. Walker GB, Zhelev Z, Henschke N, et al. Prehospital Stroke Scales as Screening Tools for Early Identification of Stroke and Transient Ischemic Attack. Stroke. 2019;50(10):e285-e286.

2. Zhelev Z, Walker G, Henschke N, et al. Prehospital stroke scales as screening tools for early identification of stroke and transient ischemic attack. Cochrane Database Syst Rev. 2019;4:CD011427.

3. Oostema JA, Konen J, Chassee T, et al. Clinical predictors of accurate prehospital stroke recognition. Stroke. 2015;46(6):1513-7.

2) Methods:

Comment 2. Data availability (repository or by request) should be clarified.

Response 2. In accordance with journal requirements, we added an unidentified data set as supporting information. Please check this point.

Comment 3. I understand that information to develop this scoring system is based on information from the Emergency Department, mainly from data that already is being recruited from the Cubic S model, but all this information is captured at the emergency department, and was transferred to the pre-hospital scenario, which seems logic, but also is a different area of work and the possibility of losing information from the real pre-hospital work is plausible.

Response 3. We fully understand the reviewer’s concern about the applicability of the CIVIL scoring system to real pre-hospital work. As the reviewer mentioned, our data were collected in the emergency department by ER physicians. In detail, the items consist of clinical information, vital signs, and initial labs. The selected items were age, past history, key clinical manifestations, vital sign, and blood glucose level. In actual prehospital situations, clinical manifestations (asymmetry, not ambulating, gaze deviation, speech disturbance) are familiar and have often been used in other well-known scoring systems (CPSS, LAPSS, ROSIER, etc.). Also, the vital signs and blood sugar levels are always checked in ambulances. Therefore, we believe that the CIVIL scoring system can be easily applied to actual pre-hospital work after systematically educating EMS paramedics. In the discussion section of the revised manuscript (p21), we described that EMS paramedics will require systematic training and verification. Please check this point.

Comment 4. Statistical analysis: which variables were included at the multivariate analysis? Were they pre-defined? If so, which cut-off point was decided to include the variable in the multivariate model.

Response 4. We entered variables with p value < 0.05 in univariate analyses into a multivariate logistic regression model. For confounding factors, more significant or intuitive variables were selected. Please check this point in the revised manuscript (p10).

Comment 5. Youden's Index together with the scoring system performance should be included.

Response 5. Thank you for the reviewer’s comment. Youden’s index was 0.385 at a cut-off point ≥ 1 in tier 1, 0.329 at a cut-off point ≥ 2 in tier 2, and 0.463 at a cut-off point ≥ 3 in tier 3, respectively. We have provided these values in the results section of the revised manuscript (p16-7). Please check this point.

Comment 6. OR and CI 95% are included in your analysis, but this also should be mentioned at the "statistics" section

Response 6. We added OR and CI 95% analysis in the statistics section of the revised manuscript (p10). Please check this point.

Comment 7. As you are doing only internal validation of your scoring system, did you consider a bootstraping to evaluate a more accurate performance in your population?

Response 7. Thank you for the reviewer’s valuable comment. As the reviewer’s comment, we performed additional analysis using bootstrap to evaluate more accurate performance. Bootstrap results showed that the parameter estimates closely agreed with the corresponding values in the final model (Theses results were based on 1,000 bootstrap sample.). Please check the tables below.

Tier 1

Model estimates Bootstrap results

Variables β (SE%) OR with 95% CI P β (SE%) OR with 95% CI P

Age ≥ 60 years 0.32 (15.4) 1.38 (1.02-1.86) 0.04 0.32 (15.1) 1.38 (1.03-0.87) 0.03

Age ≤ 40 years -0.58 (26.6) 0.56 (0.33-0.94) 0.03 -0.58 (28.8) 0.56 (0.32-1.00) 0.04

Hypertension history 0.16 (14.3) 1.18 (0.89-1.56) 0.25 0.16 (14.2) 1.18 (0.89--1.54) 0.24

Cardiac disease history 0.40 (18.0) 1.49 (1.05-2.12) 0.03 0.40 (18.2) 1.49 (1.05-2.13) 0.03

Seizure or psychiatric history -2.79 (27.8) 0.06 (0.04-0.11) <0.01 -2.79 (29.3) 0.06 (0.03-0.10) 0.00

After awakening 0.05 (24.6) 1.05 (0.65-1.70) 0.84 0.05 (25.5) 1.05 (0.63-1.76) 0.84

Any asymmetry 1.20 (16.5) 3.32 (2.40-4.58) <0.01 1.20 (16.9) 3.32 (2.40-4.68) 0.00

Not ambulating 0.76 (16.4) 2.14 (1.55-2.95) <0.01 0.76 (16.6) 2.14 (1.54-2.98) 0.00

No grasping 0.11 (21.7) 1.12 (0.73-1.72) 0.60 0.11 (23.0) 1.12 (0.74-1.83) 0.62

Mental change -0.23 (17.0) 0.79 (0.57-1.11) 0.17 -0.23 (16.8) 0.79 (0.57-1.12) 0.16

SBP ≥ 140 mmgHg 0.66 (14.0) 1.93 (1.47-2.53) <0.01 0.66 (14.6) 1.93 (1.46-2.55) 0.00

SBP ≤ 90 mmHg -1.60 (70.2) 0.20 (0.05-0.80) 0.02 -1.60 (455.3) 0.20 (0.00-0.84) 0.03

Extreme glucose level -1.97 (38.3) 0.14 (0.07-0.30) <0.01 -1.97 (42.7) 0.14 (0.06-0.30) 0.00

Tier 2

Model estimates Bootstrap results

Variables β (SE%) OR with 95% CI P β (SE%) OR with 95% CI P

Age ≤ 60 years 0.68 (16.6) 1.96 (1.42-2.72) <0.01 0.68 (16.9) 1.96 (1.43-2.77) 0.00

Diabetes mellitus history -0.45 (21.7) 0.64 (0.42-0.97) 0.04 -0.45 (22.0) 0.64 (0.41-0.95) 0.03

Cardiac disease history -1.49 (27.0) 0.23 (0.13-0.38) <0.01 -1.49 (27.6) 0.23 (0.11-0.36) 0.00

Sudden 1.00 (60.8) 2.71 (0.82-8.94) 0.10 1.00 (65.6) 2.71 (0.80-9.84) 0.08

After awakening -0.39 (53.5) 0.68 (0.24-1.93) 0.46 -0.39 (57.0) 0.68 (0.18-1.71) 0.45

As unusual -0.66 (70.6) 0.52 (0.13-2.06) 0.35 -0.66 (436.7) 0.52 (0.00-2.30) 0.40

Any asymmetry 0.04 (22.1) 1.04 (0.67-1.60) 0.86 0.04 (22.3) 1.04 (0.69-1.68) 0.85

Mental change 1.84 (20.8) 6.28 (4.18-9.44) <0.01 1.84 (22.1) 6.28 (4.25-10.08) 0.00

Abnormal sensation -0.35 (25.4) 0.71 (0.43-1.16) 0.17 -0.35 (27.2) 0.71 (0.39-1.14) 0.20

SBP ≥ 160 mmgHg 0.76 (15.9) 2.13 (1.56-2.91) <0.01 0.76 (16.6) 2.13 (1.56-3.00) 0.00

Tier 3

Model estimates Bootstrap results

Variables β (SE%) OR with 95% CI P β (SE%) OR with 95% CI P

Gaze deviation 3.08 (20.1) 21.77 (14.68-32.27) <0.01 3.08 (21.4) 21.76 (15.04-34.99) 0.00

Face asymmetry 0.04 (20.0) 1.04 (0.70-1.54) 0.85 0.04 (20.3) 1.04 (0.69-1.55) 0.85

Arm asymmetry 0.78 (22.1) 2.19 (1.42-3.37) <0.01 0.78 (22.2) 2.19 (1.47-3.43) 0.00

Speech disturbance 0.88 (19.6) 2.421 (1.65-3.55) <0.01 0.88 (19.5) 2.421 (1.66-3.59) 0.00

3) Results:

Comment 8. The decision on each variable included in the CIVIL ASAP tool, was done based on current, recent and the highest level of bibliography, or only extracted from the dataset.

Response 8. The decision to include variables is important and critical in developing a scoring system. Various aspects had to be considered, such as previous systems, statistical results, and availability in prehospital and emergency room. In tier 1 (CIVIL-ASAP tool), multivariate analysis indicated that nine variables are independent prognostic indicators: age (≥ 60 years, ≤ 40 years), cardiac disease history, seizure or psychiatric history, any asymmetry, not ambulating, systolic BP (≥ 140 mmHg, ≤ 90 mmHg), and extreme glucose level (≤ 80 or ≥ 400 mg/dl). In tier 2 (CIVIL-MAPS), five parameters were independent variables (age ≤ 60 years, diabetes mellitus, cardiac disease, mental change, SBP ≥ 160 mmHg). We developed CIVIL scoring system based on statistical results. These variables are also well-recognized because they are frequently used in previous scoring systems. Please consider this point.

Comment 9. How do you define "mental change"? In terms of the MAPS scoring system?

Response 9. As the reviewer mentioned, the definition of “mental change” needs to be clarified. After the training of ER physicians, a decrease in the level of consciousness below drowsiness during the initial neurological examination was assigned to “mental change”. It is equivalent to a NIHSS level of consciousness score (1a) ≥ 1. We clarified the definition of mental change in the Table 1 and 2 of the revised manuscript (p14-5). Please check this point.

Comment 10. I can't see any of the results referring to the multivariate model... Do all the OR are un-adjusted or adjusted? If you adjusted, which variables were included at the model?

Response 10. Thank you for the reviewer’s comment. Due to spatial limitations, the results of the multivariate model are shown only in Figure 1. Multivariate logistic regression analyze was performed after adjusting significant variables in univariate analyses (P<0.05). We clarified this point in Figure 1 (adjusted OR). Please check this point.

Comment 11. Did you perform a ROC-AUC analysis to evaluate the performance of your scoring system compared to the other systems (you mention that at the methods section)? Could you provide a figure of the comparison of each curve according to the pre-hospital score used to recognize a "confirmed stroke case"

Response 11. The results of ROC-AUC analysis were shown in the supplementary figure (S3) of supporting information. In Tier 1, CIVIL-ASAP score was compared with CPSS, LAPSS, and ROSIER scores. Please consider this point.

4) Discussion:

Comment 12. Only one comment: current scoring systems are very easy to use; its performance vary, and seems that the CIVIL has a very good opportunity to prove your rationale, but I think that the applicability of the scoring point is very difficult, so, you should try to convince the readers that they should use this system.

Response 12. As the reviewer have mentioned, the CIVIL scoring system appears to be more complex than current scoring systems. Usability is also important, but more detailed items may be required to improve the accuracy of the scoring system. Although the CIVIL scoring system included a significant number of items and had a stepwise approach, included items were frequently used in other well-known scoring systems (Figure 2). We also used an intuitive and easy-to-remember acronyms, and existing GFAST score was applied to Tier 3. These efforts can improve the accessibility of the scoring system. The complexity of the CIVIL scoring system can be overcome with systematic education and practical tools (such as mobile applications, checklist and automatic calculation system). We added need for further investigation to increase applicability of the CIVIL scoring system in the discussion section of the revised manuscript (p21). Please consider this point.

Submitted filename: CIVIL_PLOS ONE_response letter.docx

Click here for additional data file.

17 Mar 2020

Stepwise stroke recognition through Clinical Information, Vital signs, and Initial Labs (CIVIL): Electronic health record-based observational cohort study

PONE-D-19-32649R1

Dear Dr. Hong,

We are pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it complies with all outstanding technical requirements.

Within one week, you will receive an e-mail containing information on the amendments required prior to publication. When all required modifications have been addressed, you will receive a formal acceptance letter and your manuscript will proceed to our production department and be scheduled for publication.

Shortly after the formal acceptance letter is sent, an invoice for payment will follow. To ensure an efficient production and billing process, please log into Editorial Manager at https://www.editorialmanager.com/pone/, click the "Update My Information" link at the top of the page, and update your user information. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, you must inform our press team as soon as possible and no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

With kind regards,

Juan Manuel Marquez-Romero, M.D., M.Sc.

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #2: No comments, all the suggestions were addressed therefore I consider it is suitable for publication

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #2: Yes: Miguel A. Barboza

25 Mar 2020

PONE-D-19-32649R1

Stepwise stroke recognition through Clinical Information, Vital signs, and Initial Labs (CIVIL): Electronic health record-based observational cohort study

Dear Dr. Hong:

I am pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please notify them about your upcoming paper at this point, to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

For any other questions or concerns, please email plosone@plos.org.

Thank you for submitting your work to PLOS ONE.

With kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Juan Manuel Marquez-Romero

Academic Editor

PLOS ONE