Registration of acute stroke: validity in the Danish Stroke Registry and the Danish National Registry of Patients.

BACKGROUND: The validity of the registration of patients in stroke-specific registries has seldom been investigated, nor compared with administrative hospital discharge registries. The objective of this study was to examine the validity of the registration of patients in a stroke-specific registry (The Danish Stroke Registry [DSR]) and a hospital discharge registry (The Danish National Patient Registry [DNRP]).
METHODS: Assuming that all patients with stroke were registered in either the DSR, DNRP or both, we first identified a sample of 75 patients registered with stroke in 2009; 25 patients in the DSR, 25 patients in the DNRP, and 25 patients registered in both data sources. Using the medical record as a gold standard, we then estimated the sensitivity and positive predictive value of a stroke diagnosis in the DSR and the DNRP. Secondly, we reviewed 160 medical records for all potential stroke patients discharged from four major neurologic wards within a 7-day period in 2010, and estimated the sensitivity, specificity, positive predictive value, and negative predictive value of the DSR and the DNRP.
RESULTS: Using the first approach, we found a sensitivity of 97% (worst/best case scenario 92%-99%) in the DSR and 79% (worst/best case scenario 73%-84%) in the DNRP. The positive predictive value was 90% (worst/best case scenario 72%-98%) in the DSR and 79% (worst/best case scenario 62%-88%) in the DNRP. Using the second approach, we found a sensitivity of 91% (95% confidence interval [CI] 81%-96%) and 58% (95% CI 46%-69%) in the DSR and DNRP, respectively. The negative predictive value was 91% (95% CI 83%-96%) in the DSR and 72% (95% CI 62%-80%) in the DNRP. The specificity and positive predictive value did not differ among the registries.
CONCLUSION: Our data suggest a higher sensitivity in the DSR than the DNRP for acute stroke diagnoses, whereas the positive predictive value was comparable in the two data sources.

Introduction

Stroke medicine has developed dramatically in recent years with the emergence of new and effective treatment methods and increasing attention paid to implementation of clinical guidelines for treatment and rehabilitation of stroke patients.1,2 This development has led to the establishment of several stroke-specific registries, which are used for monitoring and improving the quality of stroke care and for research in a wide range of countries, eg, Sweden, UK, Germany, Poland, Austria, US, Australia, China, and Denmark.3–11 Although the value of such registries is entirely dependent on the validity and completeness of data,12 few validation studies of stroke-specific registries have been conducted.13 In addition, studies that have directly compared the validity of the registration of patients in stroke-specific and hospital discharge registries are largely missing. The aim of this study was to examine and compare the validity of the registration of acute stroke events leading to hospitalization in the Danish Stroke Registry (DSR) and the Danish National Registry of Patients (DNRP).

Materials and methods

Danish Stroke Registry

The DSR was established in 2003 with the aim of documenting and improving the early treatment and care of acute stroke.11,14 It is mandatory for all 28 Danish hospitals treating patients with acute stroke to report data on selected evidence-based performance measures, which are monitored by the DSR.

All patients aged 18 years or older who are admitted with an acute stroke to a Danish hospital should be registered in the DSR. Data are reported to a central database after discharge of the patient, and detailed instructions on the reporting of data have been provided by the DSR in order to ensure the data validity and completeness (case ascertainment) of the database. Once a year, a structured audit is conducted where the completeness of the database is assessed (ie, the number of persons in the database with stroke/the number of persons with stroke admitted to hospital). Acute stroke should be the primary condition in case the patient is admitted with multiple diagnoses. Data are prospectively collected using a standardized registration form in a multidisciplinary effort involving the entire staff of health professionals involved in the patients’ care, including physicians, nurses, physiotherapists, and occupational therapists. The data are subsequently entered into a web-based database, usually by a nurse or a medical secretary.

Danish National Registry of Patients

The DNRP was established in 1977 as a national administrative registry containing data on all discharges from Danish non-psychiatric hospitals. Reporting to the registry is mandatory and 99.4% of all discharges are registered therein.15 Data include hospital and department codes, dates of admission and discharge, surgical procedures performed during the hospital stay, and up to 20 diagnosis codes at discharge, defined according to the International Classification of Diseases, 10th edition (ICD-10) since 1994.16 Diagnosis codes are assigned at discharge by a physician involved in the care of the patient and reported to the registry by a medical secretary. In both the DSR and the DNRP, patients are identified by a unique Civil Registration Number, which is assigned to all Danish citizens at birth or at the time of immigration. The Civil Registration Number enables unambiguous identification and linkage of patients from different registries.

Study design and population

Our study included two approaches. The aim of the first approach was to examine the validity of the registration of patients on a national level based on the assumption that all hospitalized patients diagnosed with an acute stroke in 2009 would most likely be registered in the DSR, DNRP, or both. For this purpose, we identified 25 patients who were only registered in the DSR, 25 patients who were only registered in the DNRP, and 25 patients who were registered in both data sources. The sample sizes were not based on a formal power analysis, but pragmatically determined based on a priori estimation of the expected precision associated with different sample sizes. The patients were randomly identified in the data sources using a computer algorithm which specified that patients should be at least 18 years old, inpatients diagnosed at a Danish hospital with an acute stroke as the primary diagnosis (ICD-10 codes I61, intracerebral hemorrhage; I63, cerebral infarction; and I64, unspecified stroke), and discharged in the period January 1, 2009 to December 31, 2009.

The aim of the second approach was to examine the validity of the registration of patients on a local level, based on four major neurologic wards that had agreed to participate in a more comprehensive study on the feasibility of extending the use of administrative data for quality improvement activities. These four wards were all characterized by a high volume of patients with stroke (approximately 350–950 patients with stroke/year) and extensive experience with diagnostics and treatment of stroke. We reviewed the medical summaries for all patients discharged from the four participating neurologic wards in the period January 4, 2010 to January 10, 2010, focusing on patients with a potential stroke diagnosis, including patients registered with discharge diagnoses of cerebrovascular disease (I60–I69), inflammatory diseases of the central nervous system (G00–G09), episodic and paroxysmal disorders (G40–G47), hemiplegia/paraplegia and tetraplegia/other paralytic syndromes (G81–G83), symptoms and signs involving the nervous and musculoskeletal systems (R25–R29), head injuries (S062–S069), and care involving use of rehabilitation procedures (Z501). This initial sorting of medical summaries was performed in order to focus efforts most effectively on the potentially stroke-related medical records. If the medical summary indicated that the patient had potentially been admitted to hospital due to stroke, we obtained the medical record in order to verify the stroke diagnosis. In all, we retrieved 160 medical summaries.

Medical record review

In the first approach, the medical records, including descriptions of diagnostic imaging procedures (magnetic resonance imaging [MRI] and/or computed tomography [CT] scans), were retrieved. All information was reviewed by a physician (SPJ), and in the event of an uncertain diagnosis, the case was assessed independently by two consultants in neurology (HKI and KV), who made a final decision by consensus. We defined stroke in accordance with the criteria proposed by the World Health Organization, ie, a condition with a presumed vascular etiology leading to a rapid progression of symptoms of loss of neurologic function lasting >24 hours or causing death.17 Although fulfilling the World Health Organization criteria, we excluded patients with subarachnoid hemorrhage because these patients are not included in the DSR. Patients with subdural hematoma or epidural hemorrhage, retinal infarct or infarct caused by trauma, infection, or an intracranial malignant process, as well as asymptomatic patients with infarct detected only by CT or MRI scan were not confirmed as stroke cases in our study. Similarly, patients with purely diffuse symptoms (eg, isolated vertigo or headache) did not fulfill the diagnostic criteria. A diagnosis of intracerebral hemorrhage was considered confirmed in the presence of clinical symptoms of stroke combined with verification of hemorrhage by CT or MRI scan and the absence of evidence of a previous stroke in the same area. A diagnosis of ischemic infarction was confirmed in the presence of clinical symptoms of stroke with no visualization of hemorrhage by CT or MRI scan. Brain imaging is not required according to the World Health Organization definition; however, all potential patients in our study did have a CT and/or MRI scan. The original MRI and/or CT scans were not reassessed. We excluded one record (1.3%) due to a missing Civil Registration Number (DSR sample) and two records (2.7%) because the available data did not allow a meaningful validation (joint sample), leaving 72 medical records available for data analysis.

In the second approach, medical summaries for patients discharged from the participating neurologic wards in the study period were retrieved and reviewed (by CW and SPJ). On suspected diagnosis of stroke or insufficient information in the summary, we retrieved the medical record and descriptions of diagnostic imaging procedures, and reviewed the material in accordance with the same diagnostic criteria used in the first part of the study (SPJ). Any uncertain diagnoses were settled by the two consultants in neurology. For two patients (1.3%) there were no medical records available to confirm a suspected diagnosis of stroke, and two patients (1.3%) did not have a Civil Registration Number, leaving 156 medical summaries for data analysis.

Permission to conduct the study was obtained from The Danish Data Protection Agency (journal number 2007-58-0016) and The National Board of Health (journal number 7-604-04-67/1/EHE).

Statistical analysis

In the first approach of the study, we assessed the sensitivity and positive predictive value of the registered diagnosis of stroke in the DSR and the DNRP, respectively, using the medical record as the gold standard. We calculated the positive predictive value of stroke in each sample and then, referring to the total number of patients registered with a diagnosis of stroke in the DSR, the DNRP, or in both data sources in 2009, we extrapolated the positive predictive value to the number of patients with stroke on a national level in 2009 in each registry. Using this approach, we were able to estimate the number of verified patients with stroke, and the total number of patients with stroke by gold standard and registry. The sensitivity of the diagnosis of stroke was calculated as the number of patients with a verified diagnosis registered in either the DSR or DNRP divided by the total number of patients with a verified diagnosis of stroke. The positive predictive value was calculated as the number of patients with a verified diagnosis registered in either the DSR or DNRP divided by the total number of patients registered with a diagnosis of stroke in either the DSR or DNRP. Given that the basis of this approach was the assumption that all relevant patients would be registered in either or both of the registries, we were not able to estimate the specificity or the negative predictive value. Since it is complex and debatable how best to estimate the 95% confidence intervals (CIs) accurately when extrapolating to the total stroke population, we did not estimate 95% CIs when extrapolating. Instead, we examined the robustness of our findings by performing additional analyses (“best” and “worst” case scenarios) using the 95% CIs from the positive predictive values of the samples presented above. The “best case scenario” reflected the highest possible extrapolated sensitivity and positive predictive value, whereas the “worst case scenario” represented the opposite. The data analyses in the first approach are described in further detail in the Appendix.

In the second approach, we assessed the sensitivity, specificity, positive predictive value, and negative predictive value of the registered diagnosis of stroke in the DSR and the DNRP, using the medical record as the gold standard. According to the guidelines of the DSR, a patient should not be registered as discharged if he or she is transferred to another hospital ward for further treatment. Because patients are often transferred to other wards from the neurologic ward, the registered date of discharge in the DSR is not necessarily in agreement with the date of discharge from the neurologic ward. We therefore identified patients registered in the DSR with a date of discharge that was within ±30 days of our study period. Given that we included patients according to whether they were discharged in the study period, both long-term and shorter-term hospitalizations were represented. The sensitivity and positive predictive value were calculated as described for the first approach. The specificity was calculated as the number of patients verified not to have had a stroke divided by the total number of patients without a stroke according to their medical record. The negative predictive value was calculated as the number of patients verified not to have had a stroke divided by the total number of patients not registered in the DSR or DNRP with a diagnosis of stroke. The data were analyzed using SAS software version 9.2 (SAS Institute, Cary, NC, USA).

Results

In our sample of 72 patients registered in the DSR, DNRP, or both registries with a diagnosis of stroke in 2009, we were able to verify the diagnosis for 47 patients. Twenty-five patients had not had a stroke as defined by the inclusion criteria for the DSR, of whom the majority were patients with old non-acute strokes and patients with transient ischemic attacks, whereas the rest were admitted with non-stroke related conditions (including unspecified dizziness, migraine, and meningitis); 20 of these unconfirmed patients originated from the DNRP sample, three from the DSR sample, and two from the joint sample. On the basis of these three samples, the positive predictive value in the DSR was calculated as 21/24=88% (95% CI 69%–96%), in the DNRP as 5/25=20% (95% CI 9%–39%), and in both registries as 21/23=91% (95% CI 73%–98%). When extrapolating the positive predictive value to the number of patients with stroke on a national level in 2009, we referred to the total number of patients registered with stroke in one or both of the registries in 2009 (Figure 1 and Appendix). We thus estimated the number of patients with true stroke to be 0.88 × 2,429=2,138 (95% CI 1,676–2,332) in the DSR, 0.20 × 1,722=344 (95% CI 155–672) in the DNRP, and 0.91 × 8,278=7,533 (95% CI 6,043–8,112) in both registries (Figure 2).

Figure 1

Number of patients registered with stroke in the Danish Stroke Registry, the Danish National Registry of Patients, and in both data sources in 2009.

Figure 2

Number of patients with true stroke in the Danish Stroke Registry, the Danish National Registry of Patients, and in both data sources in 2009, extrapolated from review of 75 medical records.

Table 1 shows that based on these samples, the validity of the registration of patients was higher in the DSR than in the DNRP. The sensitivity of the DSR was 97% in contrast to a sensitivity of 79% for the DNRP when using the entire population included in the DSR and DNRP. The positive predictive value of the DSR was 90% compared with 79% in the DNRP. Based on the 95% CIs obtained from the samples of patients included in the medical record review, worst and best case scenarios were assessed. These supplementary analyses showed that the sensitivity of the DSR could in fact be as low as 92% and as high as 99%, whereas the positive predictive value could be as low as 72% and as high as 98%. For the DNRP, the sensitivity could be as low as 73% and as high as 84%, whereas the positive predictive value could vary from 62% to 88%.

Table 1

Sensitivity and positive predictive value for acute stroke in the Danish Stroke Registry and the Danish National Registry of Patients in Denmark, 2009

Registry	Verified positive/total positive by gold standard	Sensitivity % (worst/best case scenario)*	Verified positive/total positive by registry	PPV % (worst/best case scenario)*
Danish Stroke Registry	9,671/10,015	97 (92–99)	9,671/10,707	90 (72–98)
Danish National Registry of Patients	7,877/10,015	79 (73–84)	7,877/10,000	79 (62–88)

Notes:

Reflects sensitivity analyses.

Abbreviation: PPV, positive predictive value.

Of the 156 medical summaries that were reviewed using the second approach, we identified 74 cases of acute stroke. Table 2 shows that based on data from the second approach, the difference in sensitivity between the two registries was even greater, ie, 91% (95% CI 81%–96%) in the DSR and 58% (95% CI 46%–69%) in the DNRP. The negative predictive value was 91% (95% CI 83%–96%) in the DSR compared with 72% (95% CI 62%–80%) in the DNRP. However, the specificity and positive predictive value did not appear to differ between the registries.

Table 2

Sensitivity, specificity, positive predictive value, and negative predictive value for acute stroke in the Danish Stroke Registry and the Danish National Registry of Patients, based on patients hospitalized at four major departments of neurology in Denmark, January 4–10, 2010

Registry	Verified positive/total positive by gold standard	Sensitivity % (95% CI)	Verified negative/total negative by gold standard	Specificity % (95% CI)
Danish Stroke Registry	67/74	91 (81–96)	74/82	90 (82–96)
Danish National Registry of Patients	43/74	58 (46–69)	79/82	96 (90–99)
Registry	Verified positive/total positive by registry	PPV % (95% CI)	Verified negative/total negative by registry	NPV % (95% CI)
Danish Stroke Registry	67/75	89 (80–95)	74/81	91 (83–96)
Danish National Registry of Patients	43/46	93 (82–99)	79/110	72 (62–80)

Abbreviations: CI, confidence interval; PPV, positive predictive value; NPV, negative predictive value.

Discussion

Using two different approaches in this study, we found that the sensitivity appeared to be higher in the DSR than in the DNRP, while the positive predictive value was comparable in the two registries.

The study has some important methodologic strengths and limitations. Using two alternative approaches to assess the validity of the registration of hospitalized patients, we aimed to both ensure results that were representative for the entire country (approach 1) and findings that were based on a thorough assessment of all of the hospitalized potential stroke cases, including patients who had not been captured by any of the registries (approach 2). The study design allowed for an estimation of the negative predictive value as well as sensitivity and specificity, which is otherwise rarely assessed in validation studies due to lack of access to an independent reference data source.18 Further, the medical records were reviewed by a physician, with independent reviews of all difficult cases by two consultants in neurology. Only a negligible number of medical records could not be found or were excluded due to insufficient information. It appeared from our review of medical records in approach 2 that 82 of 156 admissions to the four neurologic wards were not due to an acute stroke. Other diagnoses included transient ischemic attack, sequelae of stroke, subdural hematoma, and intracerebral tumors, implying that patients admitted to these wards did have neurologic symptoms potentially indicating a stroke, which warranted their admission to a neurologic ward.

The limitations of the approaches used in our study should, however, be noted. In approach 1, we relied on the assumption that all hospitalized patients with acute stroke would be registered in the DSR and/or the DNRP, although it is evident that there will be exemptions. Hospitalized patients not registered with an acute stroke may include surgical patients with minor stroke or patients admitted with other serious underlying conditions such as cancer or hematologic illness, or they may be inadequately diagnosed and thus misclassified by the hospital ward where they were initially admitted. Further, the number of patients/records involved in approach 1 was relatively small, precluding us from making otherwise relevant subanalyses (eg, according to type of stroke or patient age). In approach 2, we only included information from four neurologic wards. These wards were characterized by a high patient volume and had agreed to take active part in a larger project. Therefore, they were not necessarily representative of Danish neurologic wards in general. However, the overall high agreement between findings from the two approaches indicates that these limitations did not have any major impact on the study findings. Finally, only one non-stroke specialist read all the medical records.

Our data indicated a higher sensitivity in the DSR than in the DNRP, in addition to a slightly higher positive predictive value in the DSR. This may, in part, be attributed to different ways of collecting and reporting data on acute stroke events to the two registries; for the DSR, data collection and reporting relies on a standardized registration form prospectively filled out by the health professionals responsible for the treatment and care of the patient. The collection of data is a multidisciplinary effort with multiple staff members, including physicians, nurses, physiotherapists, and occupational therapists, being involved with each patient and a detailed, large dataset is recorded for each patient. In addition, yearly audit reports are published by the DSR based on the performance of the individual wards, and these may enhance the commitment of the staff involved to ensure the validity of data reported to the DSR. In contrast, for the DNRP, only a physician is involved in coding of the diagnosis. The physician has typically been involved in the treatment of the patient, but does not necessarily have detailed knowledge about the entire admission or disease history. Further, detailed instructions on the coding of diagnoses in the DNRP are not available, and the coding of diagnosis may potentially be influenced by economic incentives due to use of the Diagnosis-Related Group system. Finally, the staff receive no direct feedback on the reported data, which may negatively influence the interest and sense of ownership of the data.

Generally, previous studies are characterized by use of different gold standards in the assessment of the validity of registries, different definitions of relevant diagnosis codes to validate as well as varying use of diagnostic procedures (such as CT or MRI scans), all of which complicate direct comparisons of existing results. However, several studies have reported a high positive predictive value of stroke in hospital discharge registries,18–28 and a few studies have reported moderate positive predictive value.13,29–32 Fewer studies have examined the sensitivity of discharge registries; a sensitivity of 76%–82% was reported from an Italian hospital discharge register,25 whereas a sensitivity of 88%–93% was found in Swedish and Norwegian hospital discharge registers,13,28,31,33 85%–97% in a Finnish hospital discharge register,26,27 and 76% in an American hospital discharge register.32 To our knowledge, only one study has investigated the validity of patient registration in a stroke-specific registry; using an estimated “true” number of stroke cases in the population as the gold standard, Stegmayr and Asplund reported a sensitivity of 96% and a positive predictive value of 100% in the Northern Sweden MONICA (Multinational MONItoring of trends and determinants in CArdiovascular disease) registry compared with values of 93% and 68%, respectively, in hospital discharge registers.13 These results are in accordance with our findings, although the positive predictive value of the MONICA registry was slightly higher than that of the DSR; this may be partly due to exclusion of patients over 74 years of age in the Swedish study, for whom diagnosing of stroke may be less accurate. This suggests that a greater validity of stroke-specific registries may also apply to registries in other countries. However, further studies are needed to confirm this assumption.

We recommend that further efforts are needed if DNRP is to be used for monitoring of stroke epidemiology and stroke care. Such efforts should probably both involve further refinement of the algorithms used for identifying stroke events in the DNRP (eg, in line with recent experiences from Sweden)28 and improvements in the registration practice with the registration of data becoming a more integrated part of the work flow during clinical work combined with more emphasis on training and instructions to staff involved in the coding of diagnoses to the DNRP. The DSR currently appears to have better data validity than the DNRP and appears to be a useful data source, but continued attention is required in order to improve the positive predictive value further.

Conclusion

Registration of acute stroke in the DSR registry overall appeared to have high validity. However, further efforts are warranted to improve the validity of the DNRP and precautions should be taken if using the registry for administrative, quality improvement, and research purposes.

Table 1

Distribution of random samples of acute stroke diagnoses according to data source and medical record review

	Medical record
	+	−	Total
DSR only	21	3	24a
DNRP only	5	20	25
DSR + DNRP	21	2	23b

Notes:

One patient was excluded due to lack of a Civil Registration Number

two patients were excluded due to lack of diagnostic information in the medical records.

Table 2

Distribution of patients registered with acute stroke according to the Danish Stroke Registry and medical record review

	Medical record
Overall
Danish Stroke Registry		+	−	Total
	+	9,671	1,036	10,707
	−	344	1,378	1,722
	Total	10,015	2,414	12,429
Worst case
Danish Stroke Registry		+	−	Total
	+	7,719	2,988	10,707
	−	672	1,050	1,722
	Total	8,391	4,038	12,429
Best case
Danish Stroke Registry		+	−	Total
	+	10,444	263	10,707
	−	155	1,567	1,722
	Total	10,599	1,830	12,429

Notes: Sensitivity: 9,671/10,015=96.6%; Worst case scenario: 7,719/8,391=92.0%; Best case scenario: 10,444/10,599=98.5%. Positive predictive value: 9,671/10,707=90.3%; Worst case scenario: 7,719/10,707=72.1%; Best case scenario: 10,444/10,707=97.5%.

Table 3

Distribution of patients registered with acute stroke according to the Danish National Registry of Patients and medical record review

	Medical record
Overall
Danish National Registry of Patients		+	−	Total
	+	7,877	2,123	10,000
	−	2,138	291	2,429
	Total	10,015	2,414	12,429
Worst case
Danish National Registry of Patients		+	−	Total
	+	6,715	3,802	10,000
	−	2,332	97	2,429
	Total	8,530	3,899	12,429
Best case
Danish National Registry of Patients		+	−	Total
	+	8,784	1,216	10,000
	−	1,676	753	2,429
	Total	10,460	1,969	12,429

Notes: Sensitivity: 7,877/10,015=78.7%; Worst case scenario: 6,198/8,530=72.7%; Best case scenario: 8,784/10,460=84.0%. Positive predictive value: 7,877/10,000=78.8%; Worst case scenario: 6,198/10,000=62.0%; Best case scenario: 8,784/10,000=87.8%.