Validity of cerebrovascular ICD-9-CM codes in healthcare administrative databases. The Umbria Data-Value Project.

BACKGROUND: Validation of administrative databases for cerebrovascular diseases is crucial for epidemiological, outcome, and health services research. The aim of this study was to validate ICD-9 codes for hemorrhagic or ischemic stroke in administrative databases, to use them for a comprehensive assessment of the burden of disease in terms of major outcomes, such as mortality, hospital readmissions, and use of healthcare resources.
METHODS: We considered the hospital discharge abstract database of the Umbria Region (890,000 residents). Source population was represented by patients aged >18 discharged from hospital with a diagnosis of hemorrhagic or ischemic stroke between 2012 and 2014 using ICD-9-CM codes in primary position. We randomly selected and reviewed medical charts of cases and non-cases from hospitals. For case ascertainment we considered symptoms and instrumental tests reported in the medical charts. Diagnostic accuracy measures were computed using 2x2 tables.
RESULTS: We reviewed 767 medical charts for cases and 78 charts for non-cases. Diagnostic accuracy measures were: subarachnoid hemorrhage: sensitivity (SE) 100% (95% CI: 97%-100%), specificity (SP) 96% (90-99), positive predictive value (PPV) 98% (93-100), negative predictive value (NPV) 100% (95-100); intracerebral hemorrhage: SE 100% (97-100), SP 98% (91-100), PPV 98% (94-100), NPV 100% (95-100); other and unspecified intracranial hemorrhage: SE 100% (97-100), SP 96% (90-99), PPV 98% (93-100), NPV 100% (95-100); ischemic stroke due to occlusion and stenosis of precerebral arteries: SE 99% (94-100), SP 66 (57-75), PPV 70% (61-77), NPV 99% (93-100); occlusion of cerebral arteries: SE 100% (97-100), SP 87% (78-93), PPV 91% (84-95), NPV 100% (95-100); acute, but ill-defined, cerebrovascular disease: SE 100% (97-100), SP 78% (69-86), PPV % 83 (75-89), NPV 100% (95-100).
CONCLUSIONS: Case ascertainment for both ischemic and hemorrhagic stroke showed good or high levels of accuracy within the regional healthcare databases in Umbria. This database can confidently be employed for epidemiological, outcome, and health services research related to any type of stroke.

Introduction

In Italy, stroke is one of the leading cause of mortality, and the first cause of long-term disability. The overall Italian population is 60,656,000 and the incidence estimate is 73,116 strokes/year, the prevalence estimate is 351,820 strokes, with 75,252 deaths/year due to stroke [1]. Stroke represents a major social and economic problem: the healthcare estimated cost amounts to € 3,195.9 million, € 53 per capita. In Europe, we expect an increase of about 32% of incidence in the next twenty years, mainly due to the aging of the population [2]. Epidemiological data of stroke in Italy are in line with those of other high-income countries [3].

Trends in epidemiology and survival rates of stroke can be assessed using administrative healthcare databases. Administrative databases have the advantage that they can link different sources of information (such as discharge data, prescription and laboratory data) providing a comprehensive understanding of the burden of stroke in terms of important outcomes such as mortality, disability, hospital readmissions, and use of healthcare resources. However, these databases need to be adequately validated by comparing their main content, that is, the diagnosis represented by the International Classification of Diseases (ICD), with another source of information such as the clinical chart [4, 5]. In addition, such databases can aid in monitoring adherence to drug therapies, including the use of evidence-based therapies. For these reasons, the validation of available administrative databases for cerebrovascular diseases becomes crucial, and it represents the first step for the conduction of subsequent epidemiological, outcome, and health services research.

The objective of the present study was to evaluate the accuracy of the ICD-9-CM codes in identifying patients with hemorrhagic or ischemic stroke in the administrative database of the Regional Health Authority of Umbria.

Materials and methods

Setting and data source

Administrative database

The regional healthcare administrative database of Umbria gathers information regarding all hospital admission medical records on all 890,000 residents, including personal demographics, hospital admission and discharge dates, vital status, hospital department, primary and secondary diagnoses, and surgical or diagnostic procedures. In addition, the database records all drug prescriptions listed in the National Drug Formulary, and it allows identification of the prescriber. The regional administrative database has already been used for pharmacoepidemiology and drug-related outcome research [6-9]. Each resident has a unique identification code within the database, to which the various types of information corresponding to each individual are linked. In Italy, healthcare is provided almost entirely by the Italian National Health System (NHS), therefore most residents’ significant healthcare information can be retrieved within the healthcare databases.

Source population

Source population was represented by permanent Umbria Region residents aged 18 or above. Any resident that had been discharged (with exclusion of voluntary discharge and inter-hospital transfer of patients) from a hospital with a diagnosis of hemorrhagic or ischemic stroke was considered. Residents that have been hospitalised outside the regional territory of Umbria were excluded from analysis.

Case selection and sampling method

The research protocol for this study has been previously published [10]. From the discharge abstract database of Umbria we identified by a simple randomization method using SAS 9.4 procedures six cohorts of “cases”, that is patients with a diagnosis of hemorrhagic or ischemic stroke, between 2012 and 2014 having the ICD-9 codes located in primary position of subarachnoid hemorrhage (ICD-9 code 430), intracerebral haemorrhage (code 431), other and unspecified intracranial hemorrhage (codes 432.x), occlusion and stenosis of paracerebral arteries (codes 433.x1), occlusion of cerebral arteries (codes 434.x1), and acute but ill-defined cerebrovascular disease (code 436).

We excluded prevalent cases, i.e. patients discharged from hospitals with any of the diagnostic codes investigated with the same diagnosis in the five years before the index date. From each cohort, we extracted a random sample of 130 cases (see Statistical Methods for details). At the same time, we identified a cohort of “non-cases”, corresponding to patients who had been discharged in the same period with a diagnosis of cerebrovascular disease (ICD-9 codes 390–459), including transient ischemic attack, but without hemorrhagic or ischemic stroke. From this cohort of non-cases, we extracted a random sample of 80 patients. This sample of non-cases was used as control for each of the six conditions.

Chart abstraction and case ascertainment

We examined medical charts of cases and non-cases from hospitals archives for case ascertainment.

We collected the following information from the medical charts: unique identification patient code, date of birth, gender, dates of hospital admission and discharge, any diagnostic procedure and treatment that contributed to the diagnosis of the disease.

The chart reviewers were physicians appropriately trained in data extraction. A pilot phase was performed in which they independently examined 30 medical charts. The level of agreement between the two reviewers was very high (k> 0.85). To ensure a higher level of agreement the results of the pilot phase were discussed by the working group. Disagreements were resolved through the involvement of a third reviewer (GA). Each reviewer independently completed the data extraction using standardized forms.

The detection of symptoms and diagnostic tests have been considered for the case ascertainment, as described below.

Validation criteria

As defined in the study protocol [10], we used the criteria indicated in the published international guidelines regarding hemorrhagic and ischemic stroke to validate the related ICD-9-CM codes. To validate hemorrhagic stroke ICD-9 codes (430, 431, 432.x), we also considered the criteria defined by the American Heart Association/American Stroke Association (AHA/ASA)[11, 12], while to validate ischemic stroke ICD-9 codes (433.x1, 434.x1, 436) the criteria defined by the AHA/ASA[13] and European Stroke Organization (ESO) were used[14].

According to the above-mentioned guidelines, for the validation of hemorrhagic and ischemic stroke we considered the corresponding ICD-9 codes valid when both of the following conditions were present: (1) detection of focal lesions by neurological examination; (2) imaging test (CT, MRI, or angiography).

Neuroimaging was the main discriminator between the two types of stroke: the presence of hemorrhagic lesion classified the case as hemorrhagic stroke, while negative imaging for hemorrhage classified the case as ischemic stroke.

Statistical analysis

As previously published, for each ICD-9 code we calculated a sample of 121 cases in order to obtain an expected sensitivity of 80%, with a half-width of the 95% CI equal to 8% [10]. For specificity, we calculated a sample of 73 non-cases (cerebrovascular disease patients without the diseases of interest) to obtain an expected specificity of 90%, with a half-width of the 95% CI equal to 8%, according to binomial exact calculation [15]. We considered published systematic reviews to derive the expected accuracy measures [4, 5]. In order to provide for potential missing medical charts, we decided to review a higher number of medical charts than anticipated.

For each ICD-9-CM code we calculated sensitivity and specificity, with their corresponding 95% CI. Sensitivity expressed the proportion of ‘true positives’ (i.e., patients with subarachnoid hemorrhage classified as positive by the administrative database and medical record review) relative to all cases deemed positive by the medical chart review. Specificity expressed the proportion of ‘true negatives’ (i.e., subarachnoid hemorrhage identified as negative by the administrative database and medical record review) relative to all cases deemed negative by the medical chart review. Positive and negative predicting values were also calculated, along with their 95% CI.

Reporting

We ensured quality of reporting following the criteria of Standards for Reporting Diagnostic Accuracy (STARD) 2015 [16] ().

Ethics statement

Ethics approval has been obtained from the Regional Ethics Committee of Umbria (CEAS), registry No 2695/15 of 16/12/2015.

Results

We randomly selected a sample of 130 medical charts for each cohort of cases, and 80 medical charts from the cohort of non-cases.

Overall, we retrieved and analysed 767 medical charts for cases: 129 for subarachnoid hemorrhage (ICD-9 430), 125 for intracerebral hemorrhage (ICD-9 431), 129 for other and unspecified intracranial hemorrhage (ICD-9 432.x), 128 for occlusion and stenosis of precerebral arteries (ICD-9 433.x1), 129 for occlusion of cerebral arteries (ICD-9 434.x1), and 127 for acute, but ill-defined, cerebrovascular disease (ICD-9 436) (Table 1).

Table 1

Characteristics of the overall study sample.

ICD-9 codes	PatientsN	Sex		Age			Instrumental examinations				ProceduresN	Deaths
		M	F	< 60	60–79	≥ 80	CT	MRI	CA	US
430	129	54	75	35	50	44	124	13	64	0	51	26
431	125	61	64	17	48	60	124	14	12	0	12	39
432	129	77	52	12	54	63	127	7	6	0	76	15
433.x1	128	83	45	6	83	39	91	20	35	0	77	4
434.x1	129	69	60	18	55	56	127	47	0	77	12	11
436	127	50	77	5	45	77	118	14	0	43	0	28
Overall, N (%)	767	394 (51)	373(49)	93 (12)	335 (44)	339 (44)	711	115	117	120	228	123 (16)

Notes: CT, Computed Tomography; MRI, Magnetic Resonance Imaging; CA, Cerebral Angiography; US, Ultrasound.

We retrieved and analysed 78 medical charts for non-cases. Tables 2–7 show the characteristics of the patients for each disease.

Table 2

Characteristics of patients with subarachnoid hemorrhage who were identified in the Regional Administrative Database of Umbria.

Subarachnoid hemorrhage
Incident cases (N medical charts reviewed)	129
ICD-9 code, N (%)
430 Subarachnoid hemorrhage	129 (100)
Sex, N (%)
Male	54 (42)
Female	75 (58)
Age, N (%)
< 60	35 (27)
60–79	50 (39)
≥ 80	44 (34)
Instrumental examinations, N (%)
Computed tomography (CT) of head	124 (96)
Arteriography of cerebral arteries	64 (50)
Magnetic resonance imaging (MRI) of brain	13 (10)
Surgical procedures, N (%)
Endovascular repair or occlusion of head and neck vessels	32 (25)
Other surgical occlusion of intracranial vessels	19 (15)
Deaths, N (%)
Patients deceased during hospital admission	26 (20)

Table 7

Characteristics of patients with acute, but ill-defined, cerebrovascular disease who were identified in the Regional Administrative Database of Umbria.

Acute, but ill-defined, cerebrovascular disease
Incident cases (N medical charts reviewed)	127
ICD-9 code, N (%)
436 Acute, but ill-defined, cerebrovascular disease	127 (100)
Sex
Male	50 (39)
Female	77 (61)
Age, N (%)
< 60	5 (4)
60–79	45 (35)
≥ 80	77 (61)
Instrumental examinations, N (%)
Computed tomography (CT) of head	118 (93)
Diagnostic ultrasound of head and neck	43 (34)
Magnetic resonance imaging (MRI) of brain	14 (11)
Deaths, N (%)
Patients deceased during hospital admission	28 (22)

As an additional support information file, a minimal anonymized dataset is provided ().

Diagnostic accuracy measures are reported in Table 8.

Table 8

Cross tabulation of the index test (ICD-9-CM code) for the results of the reference standard (medical chart).

	True Positive	False Positive	True Negative	False Negative
430 Subarachnoid hemorrhage	126	3	78	0
431 Intracerebral hemorrhage	123	2	78	0
432.x Other and unspecified intracranial hemorrhage	126	3	78	0
433.x1 Occlusion and stenosis of precerebral arteries	89	39	77	1
434.x1 Occlusion of cerebral arteries	117	12	78	0
436 Acute, but ill-defined, cerebrovascular disease	105	22	78	0

Hemorrhagic stroke

Subarachnoid hemorrhage

We identified a cohort of 294 patients with subarachnoid hemorrhage (ICD-9 430), from which we extracted a sample of 130 cases, of these 129 clinical charts were analysed (one clinical chart was not available).

shows the basic characteristics of the patients with subarachnoid hemorrhage. The majority of patients were females (58%). Most patients (73%) were > 60 years.

Instrumental examinations performed for the diagnosis included computed tomography (CT) of the head for most patients, followed by arteriography of cerebral arteries, and magnetic resonance (MRI). Forty percent of patients underwent intracranial or neck vessels surgical procedures. Twenty percent of patients died during hospital stay.

shows the diagnostic accuracy measures for subarachnoid hemorrhage: sensitivity 100% (95% CI: 97% - 100%), specificity 96% (95% CI: 90% - 99%), positive predictive value (PPV) 98% (95% CI: 93% - 100%), and negative predictive value (NPV) 100% (95% CI: 95% - 100%). Misclassification of cases and non-cases is described in .

False positives were due to lack of neuroimaging documentation in the medical chart.

Intracerebral hemorrhage

We identified a cohort of 1,259 patients with intracerebral hemorrhage (ICD-9 431), from which we extracted a sample of 130 cases, of these 125 clinical charts were analysed (five clinical charts were not available).

shows the basic characteristics of patients with intracerebral hemorrhage. Fifty-one percent of patients were females. Most patients (86%) were >60 years. Almost all patients underwent CT of the head, while about 10% underwent MRI of the brain or arteriography of cerebral arteries (Table 2). Few patients (10%) underwent neurosurgery procedures. Almost one third of patients died during hospitalization.

The diagnostic accuracy measures for intracerebral hemorrhage were: sensitivity 100% (95% CI: 97% - 100%), specificity 98% (95% CI: 91% - 100%), PPV 98% (95% CI: 94% - 100%), and NPV 100% (95% CI: 95% - 100%) (.

Two patients were considered false positives: one patient had negative instrumental (CT or MRI) diagnosis for intracerebral haemorrhage; in a second patient, imaging confirmation of intracerebral hemorrhage (CT or MRI) could not be found in the medical chart ().

Other and unspecified intracranial hemorrhage

We identified a cohort of 419 patients with “other and unspecified intracranial hemorrhage" (ICD-9 432.x), from which we extracted a sample of 130 cases, of these 129 clinical charts were analysed (one clinical chart was not available).

shows the basic characteristics of patients with other and unspecified intracranial hemorrhage. The most common ICD-9 subgroup was the code 432.1 (subdural hemorrhage) (94%). Sixty percent of patients were males. Most patients (91%) were 60 years and older. The most frequent diagnostic instrumental examination was CT of the head. Almost half of the patients underwent incision of brain and cerebral meninges.

The diagnostic accuracy measures for other and unspecified intracranial hemorrhage were: sensitivity 100% (95% CI: 97% - 100%), specificity 96% (95% CI: 90% - 99%), PPV 98% (95% CI: 93% - 100%), and NPV 100% (95% CI: 95% - 100%) ().

Three patients were considered false positives: two patients were misclassified because they had a diagnosis of subarachnoid hemorrhage instead of subdural, while one patient had no symptoms or signs in past history ().

Ischemic stroke

Precerebral arteries

We identified a cohort of 468 patients with occlusion and stenosis of precerebral arteries (ICD-9 433.x1), from which we extracted a sample of 130 cases, of these 128 clinical charts were analysed (two clinical charts were not available).

shows the basic characteristics of patients with occlusion and stenosis of precerebral arteries. The most common ICD-9 subgroup was the code 433.11 (carotid artery) (93%). Sixty-five percent of patients were males. Most patients (65%) were in the age class 60–79 years. The most frequent diagnostic instrumental examination was CT of the head. Almost half of the patients (45%) underwent endarterectomy of other vessels of head and neck.

The diagnostic accuracy measures for occlusion and stenosis of precerebral arteries were: sensitivity 99% (95% CI: 94% - 100%), specificity 66% (95% CI: 57% - 75%), PPV 70% (95% CI: 61% - 77%), and NPV 99% (95% CI: 93% - 100%) ().

Thirty-nine patients were considered false positives. Twenty-nine patients who had been discharged with a stroke diagnosis were asymptomatic, admitted for planned carotid endarterectomy, and for most of them (26/29) no instrumental confirmation of the disease was found in the medical chart. Seven patients had no diagnostic report of carotid artery ultrasound in the medical chart. Three patients had carotid stenosis <50% ().

A less conservative algorithm that considered true positives patients with a planned hospitalization, lead to an increase in specificity from 66% to 86%, and PPV from 70% to 90%.

One patient in the non-case group was considered a false negative because he had carotid stenosis but with instrumental examination missing in the medical chart that can confirm the absence of cerebral infarction.

Occlusion of cerebral arteries

We identified a cohort of 4,152 patients with occlusion of cerebral arteries (ICD-9 434.x1), from which we extracted a sample of 130 cases, of these 129 clinical charts were analysed (one clinical chart was not available).

shows the basic characteristics of patients with occlusion of cerebral arteries. The most common ICD-9 subgroup was the code 434.01 (Cerebral thrombosis) (67%). Fifty-three percent of patients were males. Most patients (86%) were 60 years and older. Almost all patients (98%) underwent CT of the head. Few patients (9%) underwent pharmacological procedures.

The diagnostic accuracy measures for occlusion of cerebral arteries were: sensitivity 100% (95% CI: 97% - 100%), specificity 87% (95% CI: 78% - 93%), PPV 91% (95% CI: 84% - 95%), and NPV 100% (95% CI: 95% - 100%) ().

Twelve patients were considered false positives: eight of them had negative instrumental diagnosis for cerebral thrombosis or embolism or occlusion of cerebral arteries with cerebral infarction; three patients were misclassified because they had a diagnosis of acute, but ill-defined, cerebrovascular disease; one patient had no instrumental examinations in the clinical chart ().

Acute, but ill-defined, cerebrovascular disease

We identified a cohort of 278 patients with “acute, ill-defined, cerebrovascular disease” (ICD-9 436), from which we extracted a sample of 130 cases, of these 127 clinical charts were analysed (three clinical charts were not available).

shows the basic characteristics of patients with acute, but ill-defined, cerebrovascular disease. Sixty-one percent of patients were females. Most of patients (61%) were 80 years and older. The most frequent diagnostic instrumental examination was CT of the head (93%). One fifth of the patients died during hospitalization.

The diagnostic accuracy measures for acute, but ill-defined, cerebrovascular disease were: sensitivity 100% (95% CI: 97% - 100%), specificity 78% (95% CI: 69% - 86%), PPV 83% (95% CI: 75% - 89%), and NPV 100% (95% CI: 95% - 100%) ().

Twenty-two patients were considered false positives. Of these, ten were symptomatic patients with negative instrumental diagnosis; seven were symptomatic patients with instrumental confirmation not found in the medical chart; three were misclassified because they had diagnosis of thrombosis with cerebral infarction; lastly, two were asymptomatic patients with instrumental confirmation of chronic ischemia ().

Discussion

In administrative databases the diagnosis of a given disease is associated with a specific ICD code. Despite some limitations, the ICD code is an important tool designed to map health conditions to corresponding general disease categories, along with specific variations. These codes have the advantage of being widely available and require much lower effort and cost than consulting medical charts [17].

The present study is the first to validate the main ICD-9 codes related to ischemic and hemorrhagic stroke using the Umbria healthcare database. Through the study an excellent level of accuracy for hemorrhagic stroke and acceptable values of specificity and PPV for ischemic stroke was found. This is the second study in Italy to validate ICD-9 codes at regional level. Spolaore et al[18] assessed the accuracy of discharge diagnoses related to stroke criteria and found that the codes 430, 431, 434, and 436 in primary diagnoses had the highest PPVs ranging from 61% to 78%. The authors used the MONICA criteria[19] and obtained similar results to ours in terms of PPVs. Other studies that measured the validity of stroke related ICD-9 codes in Italy were limited to a hospital level[20] [21], or limited to the validity of ischemic stroke only [20] but the results were substantially comparable to those of our study. Potential discrepancies in terms of accuracy might exist between different healthcare databases and these can be explained by the validation criteria used for case ascertainment or inaccuracies in coding the diagnosis. However, validation of administrative databases is context-specific and the results of the present validation study can be generalizable only to the setting of the Umbria population. Hence, the Umbria healthcare database can be used to perform epidemiological and clinical research on health services, in terms of assessment of efficacy, safety and appropriateness of drugs prescription and use of medical devices, through cross-linkage of several databases such as hospital discharge records and prescription databases.

The diagnostic accuracy results of our study are in line with those described in the literature of validation studies using analysis of medical records as the gold standard.

A systematic review published in 2012 [4] identified 35 studies that across 1990–2010 evaluated the validity of algorithms for identifying ischemic and hemorrhagic strokes (intracranial hemorrhage and subarachnoid haemorrhage). All studies included validated administrative coding data in primary and secondary position through abstraction of medical charts. The source population was heterogeneous (i.e. inpatient, outpatient, incident and prevalent cases).

In addition, while two studies validated stroke diagnosis in a paediatric population, most studies conducted the validation process using hospitalization databases, though 16 studies evaluated stroke as a cause-of-death on death certificates and one study reported on outpatient data.

Criteria for the confirmation of stroke varied widely across the studies. More than half of the studies used a specific set of diagnostic criteria based mostly on the WHO criteria (clinical criteria for stroke with CT negative for lesion) to evaluate the stroke diagnosis.

In terms of accuracy, for most studies evaluating codes 430, 431, or 434.x separately, the reported PPVs were 80% or higher. Most of the studies that validated the 436 code had a PPV ≥ 70%. In contrast, most studies reported low PPV values for the 433.x code, with the exception of a study [22] that evaluated the code 433.x1 separately from the code 433.x0 with a much higher value of PPV (71% compared to 13%). Moreover they compared algorithms using the primary discharge diagnosis with those using diagnoses in any position (primary and secondary diagnoses) and found less than 10% higher PPVs for algorithms using the primary discharge diagnosis alone.

A more recent systematic review published in 2015 [5] identified 77 studies that across 1976–2015 evaluated the validity of ICD-9 or ICD-10 codes related to any cerebrovascular disease and found that in more than half of the studies sensitivity was > 82%, specificity > 95% and PPV > 81%. Hospital discharge positions (primary and secondary diagnosis) and type of administrative data (i.e. inpatient, outpatient) were abstracted from the included studies. Most of the studies used chart review as a gold standard. About half of the included studies used chart reviews sometimes in conjunction with unspecific diagnostic criteria as a gold standard, while about 35% used chart review with a specific set of diagnostic criteria, most often the WHO criteria. Finally, about 15% of the studies used regional stroke registers and clinical databases. The review also separately analysed the three codes related to hemorrhagic stroke, while regarding ischemic stroke, in most studies the code ICD-9 436 or ICD-10 I64 was analysed, paired with the ICD-9 434 or ICD-10 I63.

Regarding subarachnoid hemorrhage codes (ICD-9 430 or ICD-10 I60) the PPV was ≥ 86% in 16 of the 26 studies where this was reported. The sensitivity of these codes, reported by 4 studies, ranged from 35% to 95%. Twenty-six studies evaluated the intracerebral hemorrhage codes (ICD-9 431 or ICD-10 I61) and the PPV was ≥ 87% in 16 of the 25 studies reporting on PPV. The sensitivity of these codes, as reported by 3 studies, ranged from 57% to 95%. The codes ICD-9 432 or ICD-10 I62 were less used and evaluated in 15 studies, in which the PPV was ≤ 67% in all but two.

Regarding ischemic stroke the ICD-9 code 433 was evaluated in 19 papers, and in 14 of the 19 the PPV was ≤ 71%. Two studies reported very low measures of sensitivity for ischaemic stroke (ICD-9 433). The code for occlusion of cerebral arteries (ICD-9 434 or ICD-10 I63) had a PPV ≥ 82% in 20 of the 27 studies. The sensitivity, available from 6 papers, ranged from 2% to 80%.

Finally, 16 studies examined the validity of ICD-9 436 and 434 together, and the PPV was ≥ 82% in 10 of the 16.

Strengths and limitations

A strength of our study is that we used medical charts as reference standards to adjudicate cases of hemorrhagic or ischemic strokes. In addition, we relied on a pre-published protocol with no deviation during the study development. To ensure the quality of reporting we followed the STARD 2015 criteria [16] for diagnostic accuracy studies. Lastly, we used detailed and explicit criteria for case ascertainment, and duplicate and independent processes for medical chart review, data abstraction and analysis. We acknowledge that a potential limitation of our study is that we did not evaluate the accuracy of ICD-9 codes located in secondary position. Another limitation of the present study concerns the generalizability of our results in other populations with different demographic characteristics and disease prevalence. Nonetheless, the study methodology could be replicated in other settings to identify possible differences in diagnostic accuracy results.

Conclusion

The Regional Health Authority of Umbria has started a research activity in the last years regarding case definitions of several diseases [23-26], and has validated ICD-9 codes for several oncological diseases [27-30]. In the present study, we validated the ICD-9 diagnostic codes for important cerebrovascular diseases using the Regional Health database of Umbria. Findings from the present study suggest that the assessed ICD-9 codes are highly predictive of hemorrhagic stroke while they may be considered acceptable for ischemic stroke. In conclusion, our results showed that the Umbrian healthcare administrative database, validated for these diseases, can be confidently used for epidemiological, outcome, and health services research.

STARD-2015-Checklist.

(DOCX)

Click here for additional data file.

Minimal anonymized dataset.

(PDF)

Click here for additional data file.

6 Nov 2019

PONE-D-19-24032

Validity of cerebrovascular ICD-9-CM codes in healthcare administrative databases. The Umbria Data-Value Project

PLOS ONE

Dear Dr. Abraha,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

This study is well conducted. The authors should highlight what is their incremental contribution in terms of methodology to previous studies.

We would appreciate receiving your revised manuscript by Dec 21 2019 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Gianni Virgili

Academic Editor

PLOS ONE

Journal Requirements:

1. When submitting your revision, we need you to address these additional requirements.

Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

http://www.journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and http://www.journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. During our internal evaluation of the manuscript, we found significant text overlap between your submission and the following previously published works, on which you are an author:

https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0218919

We recognize that the publication cited above was written by you and/or your co-authors. However, please note that re-use of text from a previous publication is unacceptable according to PLOS ONE’s editorial policy on text overlap and re-use (http://journals.plos.org/plosone/s/ethical-publishing-practice#loc-plagiarism). Although this previously published article is Open Access, we still ask that you acknowledge the reuse of any text or data, citing your previous article, so as to properly attribute the original published source. We thank you for your attention to our editorial policies.

Additional Editor Comments (as further peer-review):

The manuscript is very clearly written. The study is well designed with an appropriate choice of controls and adequate validation methods.

The Discussion presents details on previous large systematic reviews on this topic, which encompass a larger time span and include studies using different methodologies. One of these reviews also provides subgroups analyses, which are also presented. The authors should highlight what is the incremental value of their methodology with respect to previous research that could be used in future research and in practice.

As a minor comment, avoid presenting numbers (accuracy estimates) in the Abstract conclusions.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This research article focused on validation of ICD-9 codes for several cerebrovascular conditions in the Umbria regional administrative database in order to use them for future epidemiological studies.

The following MAJOR ISSUES should be carefully considered by the authors:

1) The Authors created several group of diagnoses and demographic and clinical characteristics of patients were described. I suggest adding a new table with all cases in order to describe characteristics of general cohort.

2) In the discussion section authors reported what is already available in the literature. However, I miss what this specific study adds to the topic. I suggest to the authors to stress the key-message of the study also in the discussion section.

3) The authors declared that they focus on ICD9 diagnosis in primary position. What about the same diagnosis in the second position? Are difference in the number of cases and validity of codes?

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

9 Dec 2019

Da: PLOS ONE

A: Iosief Abraha

Inviato: mercoledì 6 novembre 2019, 10:59:46 CET

Oggetto: PLOS ONE Decision: Revision required [PONE-D-19-24032] - [EMID:1be9306ff293a376]

PONE-D-19-24032

Validity of cerebrovascular ICD-9-CM codes in healthcare administrative databases. The Umbria Data-Value Project

PLOS ONE

Dear Dr. Abraha,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

This study is well conducted. The authors should highlight what is their incremental contribution in terms of methodology to previous studies.

We would appreciate receiving your revised manuscript by Dec 21 2019 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Gianni Virgili

Academic Editor

PLOS ONE

Journal Requirements:

1. When submitting your revision, we need you to address these additional requirements.

Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

http://www.journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and http://www.journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. During our internal evaluation of the manuscript, we found significant text overlap between your submission and the following previously published works, on which you are an author:

https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0218919

We recognize that the publication cited above was written by you and/or your co-authors. However, please note that re-use of text from a previous publication is unacceptable according to PLOS ONE’s editorial policy on text overlap and re-use (http://journals.plos.org/plosone/s/ethical-publishing-practice#loc-plagiarism). Although this previously published article is Open Access, we still ask that you acknowledge the reuse of any text or data, citing your previous article, so as to properly attribute the original published source. We thank you for your attention to our editorial policies.

Authors’ reply: We modified most of the text overlapping with our previous publication.

Additional Editor Comments (as further peer-review):

The manuscript is very clearly written. The study is well designed with an appropriate choice of controls and adequate validation methods.

Authors’ reply: We thank the Editor for this comment.

The Discussion presents details on previous large systematic reviews on this topic, which encompass a larger time span and include studies using different methodologies. One of these reviews also provides subgroups analyses, which are also presented. The authors should highlight what is the incremental value of their methodology with respect to previous research that could be used in future research and in practice.

Authors’ reply: We added a paragraph in the Discussion section describing what our study adds to the existing literature in the field, with a focus on a comparison with other Italian validation studies.

As a minor comment, avoid presenting numbers (accuracy estimates) in the Abstract conclusions.

Authors’ reply: Thanks for your suggestion. We have deleted the accuracy estimates from the Abstract conclusions.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: This research article focused on validation of ICD-9 codes for several cerebrovascular conditions in the Umbria regional administrative database in order to use them for future epidemiological studies.

The following MAJOR ISSUES should be carefully considered by the authors:

1) The Authors created several group of diagnoses and demographic and clinical characteristics of patients were described. I suggest adding a new table with all cases in order to describe characteristics of general cohort.

Authors’ reply: We added a new table (Table 1) with a summary of the characteristics of the overall sample.

2) In the discussion section authors reported what is already available in the literature. However, I miss what this specific study adds to the topic. I suggest to the authors to stress the key-message of the study also in the discussion section.

Authors’ reply: We added a paragraph in the Discussion section describing what our study adds to the existing literature in the field, with a focus on a comparison with other Italian validation studies.

3) The authors declared that they focus on ICD9 diagnosis in primary position. What about the same diagnosis in the second position? Are difference in the number of cases and validity of codes?

Authors’ reply: We decided to limit our validation study to only consider the codes in primary position because, according to the Italian legislation, the primary diagnosis constitutes the main cause of the need for treatment and/or diagnostic tests, and is mainly responsible for the use of resources.

We acknowledged this in the study limitations: “We acknowledge that a potential limitation of our study is that we did not evaluate the accuracy of ICD-9 codes located in secondary position”.

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

Submitted filename: Response to Reviewers.docx

Click here for additional data file.

26 Dec 2019

Validity of cerebrovascular ICD-9-CM codes in healthcare administrative databases. The Umbria Data-Value Project

PONE-D-19-24032R1

Dear Dr. Abraha,

We are pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it complies with all outstanding technical requirements.

Within one week, you will receive an e-mail containing information on the amendments required prior to publication. When all required modifications have been addressed, you will receive a formal acceptance letter and your manuscript will proceed to our production department and be scheduled for publication.

Shortly after the formal acceptance letter is sent, an invoice for payment will follow. To ensure an efficient production and billing process, please log into Editorial Manager at https://www.editorialmanager.com/pone/, click the "Update My Information" link at the top of the page, and update your user information. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, you must inform our press team as soon as possible and no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

With kind regards,

Gianni Virgili

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

31 Dec 2019

PONE-D-19-24032R1

Validity of cerebrovascular ICD-9-CM codes in healthcare administrative databases. The Umbria Data-Value Project

Dear Dr. Abraha:

I am pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please notify them about your upcoming paper at this point, to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

For any other questions or concerns, please email plosone@plos.org.

Thank you for submitting your work to PLOS ONE.

With kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Gianni Virgili

Academic Editor

PLOS ONE

Table 3

Characteristics of patients with intracerebral hemorrhage who were identified in the Regional Administrative Database of Umbria.

Intracerebral hemorrhage
Incident cases (N medical charts reviewed)	125
ICD-9 code, N (%)
431 Intracerebral hemorrhage	125 (100)
Sex
Male	61 (49)
Female	64 (51)
Age, N (%)
< 60	17 (14)
60–79	48 (38)
≥ 80	60 (48)
Instrumental examinations, N (%)
Computed tomography (CT) of head	124 (99)
Magnetic resonance imaging (MRI) of brain	14 (11)
Arteriography of cerebral arteries	12 (10)
Procedures, N (%)
Incision and excision of skull, brain, and cerebral meninges	6 (5)
Ventriculostomy or other operations to establish drainage of ventricle	6 (5)
Deaths, N (%)
Patients deceased during hospital admission	39 (31)

Table 4

Characteristics of patients with other and unspecified intracranial hemorrhage who were identified in the Regional Administrative Database of Umbria.

Other and unspecified intracranial hemorrhage
Incident cases (N medical charts reviewed)	129
ICD-9 code, N (%)
432 Other and unspecified intracranial hemorrhage	129 (100)
432.0 Nontraumatic extradural hemorrhage	4 (3)
432.1 Subdural hemorrhage	121 (94)
432.9 Unspecified intracranial hemorrhage	4 (3)
Sex, N (%)
Male	77 (60)
Female	52 (40)
Age, N (%)
< 60	12 (9)
60–79	54 (42)
≥ 80	63 (49)
Instrumental examinations, N (%)
Computed tomography (CT) of head	127 (98)
Magnetic resonance imaging (MRI) of brain	7 (5)
Arteriography of cerebral arteries	6 (5)
Procedures, N (%)
Incision of brain and cerebral meninges	63 (49)
Craniotomy or craniectomy	13 (10)
Deaths, N (%)
Patients deceased during hospital admission	15 (12)

Table 5

Characteristics of patients with occlusion and stenosis of precerebral arteries who were identified in the Regional Administrative Database of Umbria.

Occlusion and stenosis of precerebral arteries
Incident cases (N medical charts reviewed)	128
ICD-9 code, N (%)
433.x1 Occlusion and stenosis of precerebral arteries with cerebral infarction	128 (100)
433.01 Basilar artery	3 (2)
433.11 Carotid artery	119 (93)
433.21 Vertebral artery	2 (2)
433.31 Multiple and bilateral	4 (3)
433.81 Other specified precerebral artery	-
433.91 Unspecified precerebral artery	-
Sex
Male	83 (65)
Female	45 (35)
Age, N (%)
< 60	6 (5)
60–79	83 (65)
≥ 80	39 (30)
Instrumental examinations, N (%)
Computed tomography (CT) of head	91 (71)
Arteriography of cerebral arteries	35 (27)
Magnetic resonance imaging (MRI) of brain	20 (16)
Procedures, N (%)
Endarterectomy of other vessels of head and neck	57 (45)
Percutaneous angioplasty or atherectomy of precerebral (extracranial) vessel(s)	20 (16)
Deaths, N (%)
Patients deceased during hospital admission	4 (3)

Table 6

Characteristics of patients with occlusion of cerebral arteries who were identified in the Regional Administrative Database of Umbria.

Occlusion of cerebral arteries
Incident cases (N medical charts reviewed)	129
ICD-9 code, N (%)
434.x1 Occlusion of cerebral arteries with cerebral infarction	129 (100)
434.01 Cerebral thrombosis	87 (67)
434.11 Cerebral embolism	24 (19)
434.91 Cerebral artery occlusion, unspecified	18 (14)
Sex
Male	69 (53)
Female	60 (47)
Age, N (%)
< 60	18 (14)
60–79	55 (43)
≥ 80	56 (43)
Instrumental examinations, N (%)
Computed tomography (CT) of head	126 (98)
Diagnostic ultrasound of head and neck	77 (60)
Magnetic resonance imaging (MRI) of brain	47 (36)
Procedures, N (%)
Injection or infusion of thrombolytic agent	12 (9)
Deaths, N (%)
Patients deceased during hospital admission	11 (9)

Table 9

Hemorrhagic stroke: reasons for incorrect identification of cases and controls.

	430 Subarachnoid hemorrhage	431 Intracerebral hemorrhage	432.x Other and unspecified intracranial haemorrhage
FALSE POSITIVES	- Instrumental confirmation of subarachnoid hemorrhage (CT or MRI) not found in the medical chart (n. 3).	- Negative instrumental diagnosis for intracerebral hemorrhage (CT or MRI) (n. 1);- Instrumental confirmation of intracerebral hemorrhage (CT or MRI) not found in the medical chart (n. 1).	- Misclassification (diagnosis of subarachnoid hemorrhage instead of subdural) (n. 2);- Symptoms or signs not reported in the medical history (n. 1).
FALSE NEGATIVES	None	None	None

Table 10

Ischemic stroke: reasons for incorrect identification of cases and controls.

	433.x1 Occlusion and stenosis of precerebral arteries	434.x1 Occlusion of cerebral arteries	436 Acute, but ill-defined, cerebrovascular disease
FALSE POSITIVES	- Planned carotid endarterectomy in asymptomatic patients (n.29) with:a) instrumental confirmation not found in the medical chart (26)b) negative instrumental diagnosis for cerebral infarction (2)c) negative instrumental diagnosis paracerebral stenosis (1);- Carotid artery ultrasound examination not found in the medical chart (n.7) with:a) instrumental confirmation for cerebral infarction (3)b) negative instrumental diagnosis for cerebral infarction (4);- No evidence of carotid stenosis or moderate or severe stenosis (<50%) (n.3) with:a) symptomatic patient with negative instrumental diagnosis for cerebral infarction (2);b) asymptomatic patient with negative instrumental diagnosis for cerebral infarction (1).	- Negative instrumental diagnosis for cerebral thrombosis or embolism or occlusion of cerebral arteries with cerebral infarction (n. 8);- Acute, but ill-defined, cerebrovascular disease misclassification (n. 3);- Absence of instrumental examinations in the clinical chart (n. 1).	- Symptomatic patients with negative instrumental diagnosis for acute, but ill-defined, cerebrovascular disease (n. 10);- Symptomatic patients with instrumental confirmation not found in the medical chart (n. 7):a) deceased (n. 3,b) > 90 years (n. 3);c) moved from another hospital ward (n. 1)- Thrombosis with cerebral infarction misclassification (n. 3).- Asymptomatic patients with instrumental confirmation of chronic ischemia (n. 2).
FALSE NEGATIVES	- Patient with carotid stenosis but without instrumental examination in the medical chart that can confirm the absence of cerebral infarction (n. 1).	None	None