Reliability of Past Medical History in a Single Hospital Participating in Get With The Guidelines-Stroke Registry.

Background The GWTG (Get With The Guidelines)-Stroke registry supports clinical research and quality improvement projects that often rely on past medical history elements, the reliability of which remains largely unknown. Here, we evaluated the reliability of specific past medical history elements in a local GWTG-Stroke data set, with particular attention to calculating the CHA2DS2-VASc score. Methods and Results A single-center cohort was identified by querying the Hospital of the University of Pennsylvania's GWTG IQVIA Registry Platform for patients admitted with acute ischemic stroke between January 2017 and December 2020, with a previously known history of atrial fibrillation. Demographics and previously known medical history elements were retrieved from the registry to calculate the CHA2DS2-VASc score. Five neurologists abstracted the same medical history elements from the health records. The κ statistics quantified the reliability of medical history elements and CHA2DS2-VASc score. Four hundred fifty-three patients with acute ischemic stroke and previously known atrial fibrillation were included in the cohort. In comparison with manual reabstraction, registry-based medical history elements were only moderately reliable: congestive heart failure (κ=0.53), hypertension (κ=0.42), diabetes (κ=0.80), prior stroke (κ=0.45), and vascular disease (κ=0.48). However, leveraging these variables to calculate the CHA2DS2-VASc score was more reliable (κ=0.73). Conclusions Previously known medical history elements in the GWTG-Stroke registry were only modestly reliable in this single-center study, suggesting caution should be exercised when relying on any individual history elements in registry-based research. Combining these variables to calculate the CHA2DS2-VASc score was somewhat more reliable. Multicenter data are needed before assuming generalizability.

The GWTG (Get With The Guidelines)‐Stroke registry is an American Heart Association (AHA)‐supported quality improvement program aimed at promoting consistent adherence to evidence‐based treatment guidelines. , , The GWTG‐Stroke registry also supports a wide range of clinical research and quality improvement projects at the local and national level. The registry relies on local data abstraction and completion of a case record form, the reliability of which has largely been established by comparing registry data with manual chart review. However, past medical history warrants unique consideration, because unlike demographics, time points, or medication administration, the key elements of the medical history rely on a provider’s ability to obtain and document a thorough history. Moreover, given the multidisciplinary nature of stroke care, there are often multiple providers writing notes on each patient, which could also lead to inconsistency. Reliability of these data is critical, because registry‐based analyses frequently target a specific population based on the presence of a specific vascular risk factor. , , ,

Past medical history fields can also be leveraged to calculate risk‐stratification scores, such as the CHA2DS2‐VASc score, which estimates the annualized risk of stroke in patients with atrial fibrillation (AF). , Although the CHA2DS2‐VASc score is not directly collected in GWTG‐Stroke, registry data can be leveraged to calculate the CHA2DS2‐VASc score in future studies, including those exploring why some patients with AF are not on anticoagulation at the time of their stroke. Here, we evaluated the reliability of specific past medical history elements in the local GWTG‐Stroke data set, with particular attention to calculating the CHA2DS2‐VASc score.

Methods

This was a single‐center, retrospective cohort study approved by the University of Pennsylvania Institutional Review Board. A waiver of informed consent was granted. The data that support the findings are available from the corresponding author upon reasonable request.

GWTG‐Stroke Database and Generation of CHA2DS2‐VASc Score

The authors queried the Hospital of the University of Pennsylvania’s local GWTG‐Stroke IQVIA Registry Platform to identify all patients admitted between January 2017 and December 2020, with a discharge diagnosis of acute ischemic stroke and AF that was known before the stroke. Patients with newly detected AF during the stroke hospitalization were excluded. Admission date, patient demographics, previously known medical history, and initial National Institutes of Health Stroke Scale score were retrieved from the IQVIA Registry Platform. Vascular disease, as defined by the CHA2DS2‐VASc score, is not explicitly captured by the GWTG‐Stroke case record form, so it was scored based on the presence of at least 1 of the following diagnoses before the index stroke: coronary artery disease/prior myocardial infarction, carotid stenosis, or peripheral vascular disease. This work represents the authors’ independent analysis of local data gathered using the AHA GWTG IQVIA Registry Platform but is not an analysis of the national GWTG data set and does not represent findings from the AHA GWTG National Program. Registry data at the University of Pennsylvania are populated by a team of trained data abstractors, led by a nurse with >15 years of data abstraction experience. The lead abstractor was responsible for the majority of charts but was supplemented by 2 neuroscience nurses and 2 stroke research coordinators. All GWTG abstractors completed standard training. The lead abstractor routinely reviewed charts that others abstracted to ensure interrater reliability was appropriately maintained. Standard instructions for GWTG abstraction indicate that medical history elements should be abstracted from a range of documentation during the index admission. The Hospital of the University of Pennsylvania GWTG database has been maintained for more >15 years.

Chart Abstraction

Manual chart review was performed for each patient identified by the above query. For this reabstraction process, review was limited to documentation during the stroke hospitalization to mirror the GWTG data abstraction. Abstraction instructions are available in Data S1 and closely replicated the standardized instructions used by the trained GTWG abstractors. Components of the previously known medical history were abstracted, focusing on components of the CHA2DS2‐VASc score: congestive heart failure, hypertension, diabetes, prior stroke/transient ischemic attack/thromboembolism, and vascular disease. As per the CHA2DS2‐VASc score, vascular disease was defined as a previous myocardial infarction, peripheral arterial disease, or aortic plaque. Previously known elements of the medical history were defined as conditions known to exist before the admission for the acute stroke. Conditions that are newly diagnosed during the stroke hospitalization are not considered part of the previously known medical history but are separately categorized as newly diagnosed. Separately, coding newly diagnosed variables were used to calculate the discharge CHA2DS2‐VASc score. Reabstraction was divided evenly among 5 neurologists.

Statistical Analysis

Interrater reliability of the registry‐calculated and reabstracted CHA2DS2‐VASc score was assessed by quadratic‐weighted κ statistic. A Bland‐Altman plot depicted the agreement between the 2 CHA2DS2‐VASc scoring methods. A stratified analysis was performed to compare the 5 physicians performing manual reabstraction. Dichotomized CHA2DS2‐VASc score (<2 versus ≥2) and individual past medical history elements were compared by Cohen κ. A McNemar test assessed if a medical history element was more likely to be reported by one particular scoring method. All tests were evaluated at a significance level of P<0.05. Statistical analyses were performed in Stata 15.1 (StataCorp, College Station, TX).

In considering sample size, it was estimated that 80% to 90% would have a CHA2DS2‐VASc score of ≥2. A 10% difference between the 2 scoring techniques was conservatively estimated to ensure a sufficiently large sample size was selected. Setting α to 0.05, a sample of 361 would be adequate to detect a κ of 0.70. At least 100 patients met eligibility criteria annually, so a 4‐year sample was selected.

Results

Four hundred fifty‐seven patients were identified as having a discharge diagnosis of acute ischemic stroke and a previously known history of AF. Four patients were excluded after manual review; 1 had an incorrect diagnosis of stroke, and 3 had inaccessible medical records. The final cohort consisted of 453 patients. Mean age was 75 (±12) years. The cohort was 49% women. Mean admission National Institutes of Health Stroke Scale score was 13 (±9). Fifty‐eight percent were White, 28% were Black, 3% Asian, and 11% undetermined. There were no missing data for age, sex, and previously known medical history. Three percent were missing National Institutes of Health Stroke Scale scores. Table 1 summarizes the medical history and total CHA2DS2‐VASc score based on GWTG registry abstraction and manual reabstraction.

Table 1

Past Medical History: Comparing the GWTG‐Stroke Registry and Manual Abstraction

Medical history element	Based on the GWTG	Based on manual reabstraction	Agreement	κ
Congestive heart failure	23%	32%	81%	0.53
Hypertension	75%	82%	81%	0.42
Diabetes	29%	32%	92%	0.80
Prior stroke	32%	33%	76%	0.45
Vascular disease	33%	29%	78%	0.48

Agreement was assessed by Cohen’s κ. GWTG indicates Get With The Guidelines.

The Figure shows the concordance of registry‐calculated and reabstracted CHA2DS2‐VASc scores. Quadratic‐weighted κ was 0.73. The κ was similar across the 5 reviewers (Figure S1). The scatterplot (Figure – Panel A) reveals a linear relationship between the 2 scoring techniques (r=0.73). Notably, at the low end of the spectrum, the registry‐calculated technique may overestimate the score compared with the reabstracted score, whereas at the high end of the spectrum, the registry‐calculated technique may underestimate the score. A Bland‐Altman analysis (Figure – Panel B) shows that the overall mean difference between the 2 scoring techniques was 0.20 (nearly 0), suggesting high concordance overall, but the 95% CIs for agreement are broad (95% CI, −2.8 to 2.4), indicating substantial disagreement for individual patients. When dichotomizing the pre‐admission total CHA2DS2‐VASc score as <2 versus ≥2, there was 95% agreement between the scoring methods, and the κ was 0.69.

Figure 1

Reliability of calculating the CHA2DS2‐VASc from the GWTG (Get With The Guidelines)‐Stroke registry.

A, The scatterplot depicts the CHA2DS2‐VASc score based on manual reabstraction (x axis) and calculation by GWTG‐Stroke registry data (y axis). Quadratic weighted κ=0.73. The solid black line represents the linear fit line (R=0.73) and the shaded area represents the 95% CIs. B, A Bland‐Altman plot shows that the overall mean difference between the 2 techniques was 0.20 (dashed line). The 95% CIs for agreement extend from −2.8 to 2.4 (shaded area). In both panels, the dot size reflects the number of patients at a given point.

Despite reliability of the overall CHA2DS2‐VASc score, individual past medical history elements were less reliably reported (Table 1). A significant minority of cases had a discrepancy with respect to an individual essential element. Previously known diagnoses of congestive heart failure, hypertension, and diabetes were underreported in the GWTG‐Stroke registry (Table 2). Prior stroke and vascular disease were common sources of disagreement (24% and 23%, respectively). Prior stroke discrepancies were fairly well balanced between the 2 scoring techniques. Vascular disease may have been scored more often by the GWTG scoring technique but this imbalance was nonsignificant (P=0.09).

Table 2

Discrepant Past Medical History Elements

Medical history element	Only noted via GWTG	Only noted via reabstraction	P value
Congestive heart failure	24 (5%)	62 (14%)	<0.001
Hypertension	28 (6%)	60 (13%)	0.0006
Diabetes	12 (3%)	27 (6%)	0.02
Prior stroke	49 (11%)	61 (13%)	0.25
Vascular disease	60 (13%)	42 (9%)	0.08

P values calculated by McNemar test. GWTG indicates Get With The Guidelines.

During manual reabstraction, new diagnoses (during the stroke admission) of vascular risk factors were recorded. New diagnoses were not included in the previously known medical history as reported above. A new diagnosis of congestive heart failure was made in 15% of patients, diabetes in 8%, hypertension in 10%, and vascular disease in 7%. The mean total CHA2DS2‐VASc score was 4.3 on admission and 5.8 on discharge (35% of the study population had a prior history of stroke before the admission).

Discussion

Clinical registries, such as GWTG‐Stroke, provide the infrastructure to support unique clinical research and quality improvement efforts, but data validity is critical. The scale of data collection in GWTG‐Stroke presents challenges at each participating hospital. In this single‐center study, the reliability of most individual past medical history elements was modest at best. The reliability of the registry‐calculated CHA2DS2‐VASc score was somewhat more substantial. It may be reasonable to leverage these registry data to calculate the CHA2DS2‐VASc score if applicable to a given quality improvement or clinical research project, but these findings also suggest exercising caution when relying heavily on certain individual components of the previously known medical history. These findings reflect local data gathered using the AHA GWTG IQVIA Registry Platform but not national‐level data.

The overall CHA2DS2‐VASc score may be more reliable than the dichotomized score (κ=0.73 and 0.68, respectively). This could be explained by a larger number of data elements in the total score, but comparing κ statistics is limited because the overall score was assessed by quadratic‐weighted κ, and the majority of discrepant scores only differed by 1. It is important to note the CHA2DS2‐VASc score discussed here reflects the score at the time of hospital admission, and therefore does not reflect new diagnoses made during the stroke hospitalization. The CHA2DS2‐VASc score before the stroke admission may be important when assessing why some patients with AF are not on anticoagulation at the time of their stroke.

In a national‐level evaluation of data accuracy, Xian et al reported excellent reliability of GWTG‐Stroke registry data but did not report individual history elements. Many registry data fields, such as demographics, time points, and laboratory values have little uncertainty during abstraction. The presence or absence of a medical history element seems straightforward at first pass but in practice relies heavily on a clinician’s ability to obtain a thorough history and document accordingly. Moreover, the GWTG‐Stroke registry abstractors and the manual chart reviewers in this study had access to the same medical records and were given the same instructions for abstraction, so these results may reveal a discrepancy in how clinical documentation is interpreted. The high rate of discrepant past medical history elements should be considered when future studies rely on these variables. Initial review of the summary statistics suggested prior stroke was relatively well balanced between the 2 scoring techniques. However, this apparent balance was a result of discrepancies being common with both scoring techniques, reflected in a relatively weak κ statistic. Heart failure, hypertension, and diabetes were more likely to be scored by manual reabstraction, which raises the possibility that physicians performing reabstraction were more thorough than GWTG abstractors. Alternatively, physicians performing reabstraction may have been more comfortable interpreting unclear provider documentation. Vascular disease, on the other hand, was less likely to be scored by manual reabstraction, which may reflect an overly sensitive strategy for scoring vascular disease based on GWTG data. Multicenter data are required to determine the generalizability of these finding.

These results should be interpreted within the context of several limitations. Most notably, as a single‐center study, these findings may not reflect data reliability at other centers. The retrospective nature of the study may be perceived as a limitation, but this more accurately reflects real‐world use of the registry data. Furthermore, this study did not explore all medical history elements on the case record form, but instead focused on elements that contribute to the CHA2DS2‐VASc score. The authors have no reason to suspect the variables reviewed here are more or less reliable than other medical history variables not reviewed. The manner in which registry data are leveraged to assess vascular disease presents a limitation because the CHA2DS2‐VASc score defines vascular disease as the presence of prior myocardial infarction, peripheral arterial disease, carotid stenosis, or complex aortic plaque. Here, vascular disease was more likely to be scored in the GWTG‐based score, but the reliability of vascular disease was similar to that of more straightforward variables such as hypertension, heart failure, and prior stroke. Assessing reliability or validity is also limited by the gold standard to which the comparison is made. Here, the manual chart review might be perceived as the gold standard, but the registry is similarly populated by manual chart review. Both methods of assessment may suffer from similar limitations, so several reviewers were used in this study to minimize the possibility of idiosyncratic tendencies with record review or interpretation. However, each patient chart was manually reviewed by a single neurologist, so we are unable to assess the reliability or misclassification bias of individual abstractors.

Conclusions

Individual medical history elements in the GWTG‐Stroke registry were only moderately reliable in this single‐center study, suggesting that caution should be exercised when relying heavily on any individual history elements in registry‐based research. Combining these variables to calculate the CHA2DS2‐VASc score was somewhat more reliable. Using GWTG‐Stroke registry data to estimate the CHA2DS2‐VASc score may be particularly reasonable because the score is often used to estimate risk category (ie, low versus high risk), in which case some degree of inaccuracy may be acceptable. Multicenter data are needed before assuming generalizability.

Sources of Funding

This work was supported by the AHA (19CDA‐34630033, C.G.F.) and the National Institutes of Health (K23‐NS110993, C.G.F.).

Disclosures

None.

Data S1

Figure S1

Click here for additional data file.