Methodologic Differences Across Studies of Patients With Atrial Fibrillation Lead to Varying Estimates of Stroke Risk.

BACKGROUND: Guidelines for anticoagulation in atrial fibrillation (AF) assume that stroke risk scheme point scores correspond to fixed stroke rates. However, reported stroke rates vary widely across AF cohort studies, including studies from the same country. Reasons for this variation are unclear. This study compares methodologies used to assemble and analyze large AF cohorts worldwide and assesses potential bias in estimating stroke rates. METHODS AND
RESULTS: From a previous systematic review of AF cohorts, we analyzed studies including at least 5000 patients. We assessed methods used to generate rates of ischemic stroke off anticoagulants, according to a structured inventory of database interrogation methods. Nine studies (497 578 total patients) met our criteria. Overall cohort stroke rates ranged from 0.45% to 7.03% per year. In bivariate study-level analysis, multiple features were associated with higher stroke rates, including AF identified as inpatients versus outpatients (rate ratio 2.60, 95% confidence interval, 1.19, 5.68), and lack of clinical validation of outcome events (rate ratio 4.09, 95% confidence interval, 1.06, 15.70). European studies reported rates more than 4-fold higher than North American studies. International Classification of Diseases (ICD) coding schemes for outcomes varied widely. Multiple high rate features coexisted in the same studies.
CONCLUSIONS: Among AF cohort studies, differences in the composition, method of assembly, determination of clinical features and outcomes, and analytic approach were strongly associated with reported stroke rates. Our study highlights the need for standardized and validated methodologies for AF cohort assembly and analysis to generate accurate stroke rates to better support anticoagulation guidelines for patients with AF.

Clinical Perspective

What Is New?

Accurate estimates of stroke rates off anticoagulants are needed to optimize risk‐based use of anticoagulants in patients with atrial fibrillation (AF).

There are marked differences in reported rates of stroke across different AF cohorts, including those from the same or adjacent geographic regions, implying that study methodology accounts for much of the differences in stroke rates.

These differences undermine the generalizability of CHA2DS2‐VASc score thresholds for anticoagulation recommended by leading guidelines.

We identify study design features explaining some of the variation in reported stroke rates and provide a methodologic framework to standardize measurement of stroke rates in AF.

What Are the Clinical Implications?

Guideline authors should be aware of the variability in reported AF stroke rates and should encourage more standardized measurement of these rates.

Physicians should appreciate the uncertain relationship between CHA2DS2‐VASc scores and absolute stroke rates and should reflect that uncertainty in discussions with patients with AF about anticoagulant therapy.

Introduction

Patients with nonvalvular atrial fibrillation (AF) who take oral anticoagulants (OACs) have a dramatically reduced risk of ischemic stroke,1 but face an increased risk of bleeding, which can be fatal. As a result, the decision to anticoagulate an individual patient should be based on the expected net clinical benefit of OAC (ie, the difference between the reduction in stroke risk and the increase in bleeding risk, weighted by the severity of each of these respective outcomes).2 Because of this balance of benefits and harms, patients at low risk of ischemic stroke may have little or even negative net clinical benefit from OAC treatment. Decision analyses have shown that the threshold of absolute stroke risk for an average individual patient at which OAC treatment yields a net clinical benefit is between 0.9% and 2% per year.3

Current guidelines from the American Heart Association (AHA)/American College of Cardiology (ACC)/Heart Rhythm Society (HRS) and the European Society of Cardiology (ESC) have recommended the use of the CHA2DS2‐VASc stroke risk scoring system.4, 5, 6 The CHA2DS2‐VASc score incorporates patient characteristics to create a point score, with increasing scores corresponding to higher risks of ischemic stroke.7 The most recent ESC and AHA/ACC/HRS guidelines both recommend using OAC at a CHA2DS2‐VASc score of 2 or greater (not including “female” as a risk factor for the ESC threshold).

An underlying assumption of these guidelines is that each CHA2DS2‐VASc point score corresponds to a fixed stroke rate off anticoagulant therapy, which can be extrapolated to a positive or negative net clinical benefit should anticoagulants be taken. To support this assumption, the guidelines cite off‐anticoagulant stroke rates from large cohort studies.7, 8, 9 However, our and others’ previous work has shown wide variation in the reported stroke rates from different cohorts of patients with AF.5, 10 Depending upon AF cohort, a CHA2DS2‐VASc score of 2 would lead to a positive or a negative expected net clinical benefit, calling into question the general applicability of guideline recommendations. The fact that wide variation in stroke rates is seen in studies from the same or adjacent countries argues that variation in study methodology is an important contributor to variation in observed rates. The current study seeks to explain this variation by comparing relevant worldwide AF cohorts according to core methodologic features.

Methods

The data, analytic methods, and study materials will not be made available to other researchers for purposes of reproducing the results or replicating the analyses. The studies analyzed have been previously published and are easily accessible for review. This article is an analysis of previously published data and methods from multiple reports. No new data were collected and institutional review board approval was not required to undertake this analysis of published data.

Study Selection

The current analysis focuses on a subgroup of studies from our previous systematic review of studies reporting stroke rates among patients with AF not taking OAC (Figure 1).10 We excluded randomized controlled trials, which contribute only a small fraction of follow‐up of AF patients off anticoagulants,10 and we focus exclusively on large observational cohort studies where design and analysis methods are more variable. To this end, we restricted our analyses to studies that included at least 5000 patients. These remaining 9 studies represented 1 137 597 person‐years or 95% of total follow‐up from the original set of 24 studies, and also included all cohorts that are repeatedly cited in AHA/ACC/HRS and ESC guidelines.4, 5, 6

Figure 1

Selection of worldwide atrial fibrillation observational cohort studies.

Description of Cohort Methods and Composition

Published articles, online supplements, and cited references (where applicable) were reviewed and the methodologic characteristics of each study were recorded and categorized. The 3 sources of variation in stroke rates across studied cohorts are true differences in rates of stroke, chance variation, and methodologic differences across studies (Figure 2). The large size of cohorts we have included makes chance variation unlikely. We focus on detailed differences in methodologic approach and cohort composition. In Table 1,11, 12, 13 we provide an inventory of methodologic considerations when estimating stroke rates from cohorts of AF patients, particularly those drawn from administrative databases, and their potential for bias. In the current analysis, we assess the most prominent and accessible of these concerns (Figure 2), as follows.

Figure 2

Sources of variation in rates of ischemic stroke reported across AF cohort studies focusing on differences in methodology. AF indicates atrial fibrillation; OAC, oral anticoagulants.

Table 1

Methodologic Considerations for Cohort Studies to Assess Stroke Rates in AF

Category	Considerations	Potential for Bias
Cohort design	Prospective research study vs retrospective (or prospective) analysis of a nonresearch administrative database Inpatient vs outpatient Source population: geographic region, generalizable sample vs census Calendar time	Administrative databases, without validation studies, are more prone to error in assessing presence of AF, comorbidities, and outcome. Direction of bias is unclear Inpatients are sicker. Their inclusion biases towards higher stroke risk Large national databases provide tighter confidence intervals, but results may not generalize to other populations AF stroke rates may be lower in more recent years. Also, the means of diagnosing AF, comorbidities, and stroke may change over time
Determination of AF status	Research study: ECG documented; self‐report—validated by ECG? Medical record diagnosis Administrative code: ICD‐9 or ICD‐10: Inpatient discharge diagnosis vs outpatient code Incident vs prevalent diagnosis Look‐back period to see if AF is new Rule out transient AF Distinguish PAF from more persistent AF Include AF patients with mechanical heart valves Include AF with mitral stenosis Include atrial flutter	Self‐report is prone to false positives and false negatives; ECG documentation removes false positives Accurately reflects clinical diagnosis if validated by ECG Hospital discharge diagnoses are likely to be accurate; outpatient codes less so. Consultant billing or test ordering codes often are false positives—need validation by chart review of sample Stroke risk may be higher at start of AF. Need a look‐back period before the first (index) diagnosis of AF to assure that AF is new Including postoperative (frequently postheart surgery) or postacute illness AF leads to underestimates of stroke risk Lower AF burden likely leads to lower stroke risk Not “nonvalvular” AF, biases towards higher stroke and bleed rates Not “nonvalvular” AF, likely biases slightly toward higher stroke risk Inclusion of atrial flutter will mildly reduce stroke rates
Determination of anticoagulation status	Self‐report Medication list in the medical record Pharmacy database dispensing record INR tests for VKAs Multiple drug insurance coverage plans Distinguish OAC for cardioversion from long‐term OAC	Accurate if patient is health literate; will capture discontinuation May not capture nonadherence or discontinuation Accurate for dispensed medication; may not capture nonadherence or short‐term discontinuation Excellent supplementary information on use of VKAs Patients may be incorrectly typed as off‐OAC if drugs covered on another plan Transient OAC for cardioversion can be confused with discontinuation of OAC prescribed for long‐term treatment
Determination of CHA2DS2‐VASc stroke risk score comorbidities	Self‐report Medical chart diagnoses Outpatient and inpatient diagnosis codes Requirement for diagnosis‐specific medication use or confirmatory test (anti‐hypertensive, HbA1c) Look‐back period; Limited number of codes that can be listed; no. of medical encounters	Accurate if patient is health literate Captures clinical diagnoses although criteria for diagnoses are not standardized Administrative hospital discharge ICD diagnosis codes are more likely to accurately reflect clinical diagnosis. Including outpatient codes will capture more comorbidities and raise CHA2DS2‐VASc score This reduces false positives, likely increases false negatives, increases severity of diagnosis: Leads to biased underestimate of CHA2DS2‐VASc score The longer the look‐back period, the larger the number of diagnoses allowed to be listed, the larger the number of medical encounters, the greater the chance that a diagnosis will be captured in a database
Ascertainment of outcome events	Types of events: ischemic stroke, TIA, systemic embolus, other Hospital vs hospital plus outpatient outcome events ICD diagnosis: which codes; primary vs nonprimary discharge diagnoses; are codes validated? Outpatient medical record diagnosis Patient self‐report Inclusion of outpatient death certificate events Validation of all or a sample of outcome events	The more types of events included in the outcome cluster, the higher will be the rates Restriction to hospital events misses outpatient events ICD diagnoses: less restrictive sets of codes (eg, ICD‐9 436 or ICD‐10 I64), lead to higher sensitivity and event rates but more false positives; inclusion of nonprimary discharge diagnoses leads to overestimation of rates; using only primary discharge diagnoses leads to modest underestimation of rates11 Needs validation by clinical record of event. May identify nonhospitalized events Needs validation by clinical record of event. May identify nonhospitalized event Two types of error in outpatient death certificate diagnoses: 1. Incorrect diagnosis, usually false positive; 2. Incorrect classification of stroke event as a new event, rather than an old event Errors in death certificate diagnoses lead to biased overestimates of rates Validation of outcome code sets increases accuracy, usually countering biased high rates
Analysis of period “at risk” off anticoagulants	Excluding patients who start OAC later in the follow‐up period Blanking period Start of follow‐up Loss to follow‐up Competing risk of death Updating comorbidities	Leads to biased underestimates of rates Avoids events occurring before the diagnosis of AF; may avoid double‐counting of events; misses early events If >1 diagnosis of AF needed, follow‐up should start at the first diagnosis to avoid immortal time bias12 May be informative with patients having a stroke becoming lost to follow‐up—leads to biased underestimates of rates Over longer time periods, nonstroke deaths lead to lower risk of stroke by removing patients at risk13 Over longer follow‐up, patients may develop stroke risk factors. Updating risk factor status reduces underestimating patients’ CHA2DS2‐VASc scores

AF indicates atrial fibrillation; APs, anti‐platelets; ASA, acetylsalicylic acid; ECG, electrocardiogram; HbA1c, glycosylated hemoglobin; ICD, International Classification of Disease; INR, international normalized ratio; OAC, oral anticoagulant; PAF, paroxysmal atrial fibrillation; TIA, transient ischemic attack; VKA, vitamin K antagonist.

Cohort design

We recorded whether the cohort study was a prospective research study or a retrospective analysis of an administrative database, if patients were identified from exclusively or predominantly inpatient settings, the region of the population (North America, Europe, Asia, and Middle East), and the midpoint calendar year of the follow‐up period (Table 2).9, 14, 15, 16, 17, 18, 19, 20, 21

Table 2

Composition and Methods of Assembly of Large Worldwide AF Cohorts, by Overall Annualized Stroke Rate

Cohort	Overall Stroke Rate	Subjects, n	Average Follow‐Up Time	Female Sex (%)	Mean Age (y)	Midpoint Year of Follow‐up	Study Design	Setting of Patient Identification	Prior Stroke (%)	Mean CHA2DS2‐VASc Score
United States—Women's Health Initiative (1993–2010)14	0.45	5981	11.8 y	100	65.9	1997	Prospective, research study	Outpatient	2.6	2.74
Taiwan—National Health Insurance Database Subset (1997–2008)15	1.28	7920	4.5 y	45.9	72	2003	Retrospective, administrative database	Both outpatient and inpatient	4.2	2.5
United States—ATRIA (1996–2003)16	1.97	10 927	2.4 y	45.2	72	2000	Retrospective and prospective, administrative database	Outpatient	8.3	3.09
Israel—Clalit Health Services AF (database established 1998; AF patients studied in 2012)17	2.98	37 358	0.9 y	50.2	72	2012	Retrospective administrative and clinical databases	Both outpatient and inpatient	0	3.47
UKCPRD (1998–2012)18	2.99	60 594	2.8 y	48.7	74.4	2005	Retrospective, administrative database	Both outpatient and inpatient	14.7	3.3
Stockholm Area Database (AF patients identified 2005–2009 and followed in 2010)19	3.29	24 195	1 year	47.4	72.6	2010	Retrospective, administrative database	Both outpatient and inpatient	20.8	3.62
Taiwan—National Health Insurance Research Database (1996–2011)20	3.71	186 570	3.4 y	46	72	2004	Retrospective, administrative database	Both outpatient and inpatient	20.5	3.79
Swedish Atrial Fibrillation Cohort Study (2005–2008)21	4.5	90 490	1.4 y	51	78.4	2007	Retrospective, administrative database	Primarily inpatient	16	3.7
Danish National Patient Registry (1997–2006)8	7.03	73 538	1 ya	51.2	72.8	2003	Retrospective, administrative database	Inpatient	18.2	3.05

AF indicates atrial fibrillation; ATRIA, Anticoagulation and Risk Factors in Atrial Fibrillation; UKCPRD, United Kingdom Clinical Practice Research Datalink.

Reported rates are based on 1 y of follow‐up.

Determination of AF status

We recorded how the diagnosis of AF was ascertained (eg, individual self‐report or coded database diagnosis, most often from the International Classification of Diseases, Ninth Revision [ICD‐9] or Tenth Revision [ICD‐10]),22, 23 whether an ECG was required, and whether the diagnosis was validated by chart review. We noted whether the cohort included patients with recently diagnosed, incident, AF or with prevalent AF, and how long before the AF diagnosis investigators looked to ensure the AF was new (ie, the “look‐back period”). We further assessed whether the study described the type of AF ie, transient24 (eg, following heart surgery), paroxysmal, or more persistent AF and whether patients with mechanical heart valves and mitral stenosis (ie, “valvular” AF) were excluded (Table 3).

Table 3

Determination of AF Diagnosis, Anticoagulation Status, and Comorbidities in Large Worldwide AF Cohorts, by Overall Annualized Stroke Rate

Cohort	Overall Stroke Rate	Method of AF Determination	ECG Required for AF Diagnosis	ASA and Antiplatelet Users Excluded	AF Diagnosis Validated by Chart Review	Incident Versus Prevalent AF	Look‐back Period Pre‐AF Index Date	Transient, Paroxysmal, or More Persistent AF	Method of OAC Determination	Method of Comorbidity Determination
United States—Women's Health Initiative14	0.45	Self‐reporta (ECG confirmed in some)	No	No	No	Both	Not applicable	No distinction	Self‐reporta	Self‐reporta
Taiwan—National Health Insurance Database Subset15	1.28	ICD‐9 codes, not specified, presumably 427.31	No	Yes	No	Unspecified	>2 yb	No distinction	Pharmacy database dispensed prescriptions (Rx)	ICD‐9 codes, not specified
United States—ATRIA16	1.97	Multiple outpatient ICD‐9: 427.31 diagnoses or 1 diagnosis plus ECG reading; mitral stenosis, prosthetic heart valve, and hyperthyroidism excluded	No	No	No	Both	≥5 y	Estimate 20% paroxysmal, transient AF excluded	Pharmacy database: dispensed Rx plus repeated INR tests	ICD‐9 codes, validated in small samples and updated over follow‐up
Israel—Clalit Health Services AF17	2.98	ICD‐9 and International Classification of Primary Care and text reading of electronic medical records: Exact codes not provided	No	No	No	Prevalent	2 y	No distinction	Pharmacy database dispensed Rx	ICD‐9 and International Classification of Primary Care and text reading of electronic medical records: Exact codes not provided; glycated hemoglobin for diabetes mellitus
UKCPRD18	2.99	UKCPRD multiple Read codes	No	No	No	Incident	b	No distinction	UKCPRD prescription written	UKCPRD Read codes, multiple, and ICD‐10 codes
Stockholm Area Database19	3.29	ICD‐10 I48; no mitral stenosis and no mechanical valve surgery	No	No	No	Both	b	No distinction	Pharmacy database dispensed Rx	ICD‐10 codes
Taiwan—National Health Insurance Research Database20	3.71	ICD‐9: 1 hospital discharge diagnosis or 2 outpatient diagnoses, no exclusions other than warfarin and APs	No	Yes	No	Both	b	No distinction	Pharmacy database dispensed Rx	ICD‐9 codes
Swedish Atrial Fibrillation Cohort Study21	4.5	ICD‐10 I‐489 hospital discharge and clinic diagnosis, excluded died during index hospitalization, mitral stenosis, valve surgery	No	No	No	Both	≥18 y	No distinction	Pharmacy database dispensed Rx	ICD‐9 codes
Danish National Patient Registry8	7.03	ICD‐10 I48, excluded aortic or mitral valve disease or valve surgery by codes; also atrial flutter	No	No	No	Both	b	Both	Pharmacy database, note excluded warfarin use up to 180 d before hosp dx of AF—but could have started VKA later in F/U	ICD‐10 codes plus condition‐specific medicationsc Some validation of diagnoses

AF indicates atrial fibrillation; APs, anti‐platelets; ASA, acetylsalicylic acid; ATRIA, Anticoagulation and Risk Factors in Atrial Fibrillation; ECG, electrocardiogram; F/U, follow‐up; ICD, International Classification of Disease; INR, international normalized ratio; Rx, prescription; UKCPRD, United Kingdom Clinical Practice Research Datalink; VKA, vitamin K antagonist.

Self‐report of AF, anticoagulation, and comorbidities during structured interviews at study enrollment and follow‐up.

Increasing look‐back period over calendar time.

All CHA2DS2‐VASc comorbidities were determined by ICD9/10 codes except for diabetes mellitus, which was determined by receipt of a glucose‐lowering drug in the pharmacy database; hypertension, which required the presence of at least 2 antihypertensive medications from different classes; and congestive heart failure, which required both an ICD9/10 code and receipt of a loop diuretic.

Determination of anticoagulation status

We recorded the method to determine whether a patient was on OACs (ie, self‐report, medication list from a medical record, pharmacy dispensing records, and international normalized ratio tests [for vitamin K antagonists]) (Table 3).

Determination of CHA2DS2‐VASc stroke risk score comorbidities

We recorded the method determining the presence of comorbidities that make up stroke risk scores (eg, self‐report, medical chart diagnosis, ICD‐9/10 codes, and disease‐specific medications or test results). Here, too, the length of the look‐back period was relevant, providing an opportunity to identify comorbid conditions at study entry (Table 3).

Ascertainment of outcome events

We recorded the average follow‐up time for each study (Table 2), and the event types in the outcome cluster: ischemic stroke alone or ischemic stroke plus other thromboembolism, including systemic emboli and/or pulmonary emboli, and transient ischemic attacks. We recorded how outcome events were ascertained (ie, by administrative hospital discharge codes [ICD‐9/10], medical record diagnosis, or by patient report [Table 4]).25 We noted which ICD‐9/10 codes were used, and whether the codes were primary or nonprimary discharge diagnoses. We recorded whether death certificate diagnoses were used and whether outcomes were validated by medical record review (Table 4).

Table 4

Outcome Determination and Analytic Strategy in Large Worldwide AF Cohorts, by Overall Annualized Stroke Rate

Cohort	Overall Stroke Rate	Stroke Outcome Cluster	Allow Diagnosis From Death Certificate	Validation or Adjudication of Outcome	Patient Excluded if Started OAC During Follow‐Up	Blanking Period	ICD‐9,‐10 Outcome Codes	Stroke as Primary Discharge Diagnosisa
United States—Women's Health Initiative14	0.45	Ischemic stroke (I‐Stroke)	No	Yes	No	No	Not applicable: self report	Not applicable: self report
Taiwan—National Health Insurance Database Subset15	1.28	I‐Stroke	No	No	Yes	No	ICD codes, not further specified	Not Specified
United States—ATRIA16	1.97	I‐Stroke and systemic embolism	No	Yes	No	No	ICD‐9 hospital discharge diagnosis codes 433, 434, and 436; validated by chart review	Primary discharge diagnosis position only
Israel—Clalit Health Services AF17	2.98	I‐Stroke and TIA	No	No	No	Unclear, 2‐y look‐back for diabetes mellitus	ICD‐9 and International Classification of Primary Care and text reading of electronic medical records Exact codes not provided	Not specified
UKCPRD18	2.99	I‐Stroke	No	No	No	No	Read codes and ICD‐10: I63, I64	Not specified
Stockholm Area Database19	3.29	I‐Stroke	Yes	No	No	No	ICD‐10: I63	Not specified
Taiwan—National Health Insurance Research Database20, 25	3.71	I‐Stroke	No	No	Yes	No	ICD‐9: 433.xx, 434.xx	Not specified
Swedish Atrial Fibrillation Cohort Study21	4.5	I‐Stroke	No	No	Yes	Yes	ICD‐10: I63	Not specified
Danish National Patient Registry8	7.03	I‐Stroke, systemic embolism, and pulmonary embolism	Yes	No	No	Yes, 7 days	I26, I63, I64, I74 Adjusted for anti‐platelet use	Not specified

AF indicates atrial fibrillation; ATRIA, Anticoagulation and Risk Factors in Atrial Fibrillation; ICD, International Classification of Disease; OAC, oral anticoagulants; TIA, transient ischemic attack; UKCPRD, United Kingdom Clinical Practice Research Datalink.

Stroke ICD‐9, ‐10 codes in primary discharge diagnosis position.

Analysis of period “at risk” off anticoagulants

We noted whether the study excluded patients who started OACs after their period at risk off anticoagulants. Furthermore, we recorded whether a blanking period was used (ie, a period immediately following AF diagnosis during which outcome events were not counted [Table 4]).11

Statistical Analysis

We calculated event rates by dividing the number of events by the corresponding follow‐up time. At the study level, we examined factors predictive of stroke rate using Poisson regression with a scale parameter to account for overdispersion in a bivariate analysis. The log of person‐years was included as an offset in the models. Because of the small number of trials included in our analysis, we did not attempt a multivariable model. Results are presented as rate ratio (95% confidence interval). A 2‐sided P value of 0.05 or less was considered statistically significant. All analyses were conducted using SAS version 9.4 (SAS Institute, Cary, NC).

Results

In the 9 AF cohorts with at least 5000 patients not taking OAC8, 14, 15, 16, 17, 18, 19, 20, 21 identified in our systematic review, overall event rates ranged from 0.45% to 7.03% per year (median of 2.99% per year). Event rates differed markedly between national cohorts from adjacent geographic regions (eg, Sweden and Denmark) as well as cohorts from the same country (eg, the 2 Taiwanese cohorts) (Table 2).

Composition of AF Cohorts

The composition of the cohorts and the methods for their assembly are displayed in Table 2, ordered by overall stroke rate. The cohorts with the highest event rates had a retrospective study design, enrolled primarily or exclusively inpatients, and had larger proportions of patients with prior stroke. However, overall cohort stroke rate was not a simple function of stroke risk score. While the 2 cohorts with the lowest event rates also had the lowest mean CHA2DS2‐VASc score,15, 16 the cohort with the highest event rate, the Danish National Patient Registry, had a CHA2DS2‐VASc score lower than cohorts with much lower overall event rates. With the exception of the Women's Health Initiative, mean age and proportion female were similar across cohorts. In these studies, midpoint calendar year did not have a clear association with stroke rate (Table 2).

Determination of AF Diagnosis, Anticoagulation Status, and Comorbidities

While the Women's Health Initiative used self‐report to determine AF, the remaining cohorts used ICD‐9/10 diagnosis codes (Table 3). Though some studies did include AF diagnosis by ECG as 1 entry criterion,15, 17 none required an ECG for inclusion into the study and no cohort required review of original ECGs, and none validated the diagnosis by review of medical charts. Only 1 cohort specifically excluded transient AF and provided an estimate of the proportion with paroxysmal AF.17 Only the United Kingdom Clinical Practice Research Datalink cohort included exclusively incident AF (Table 3).

Most studies established OAC use via pharmacy databases recording dispensed prescriptions, although the Women's Health Initiative used only self‐report (Table 3). One study supplemented pharmacy records with international normalized ratio testing.17 The 2 Taiwan National Health Insurance Research Database studies excluded patients using antiplatelet agents (Table 3). These 2 Taiwanese studies produced notably disparate overall event rates (1.28% per year versus 3.71% per year).

Patient comorbidities used to calculate CHA2DS2‐VASc scores were largely determined by ICD‐9/10 or other database codes in all cohorts except for the Women's Health Initiative, which relied on self‐report for most comorbidities, including hypertension, which was defined as self‐report of taking an antihypertensive medication (Table 3). The ATRIA (Anticoagulation and Risk Factors In Atrial Fibrillation) study validated ICD‐based diagnoses of comorbidities in small sample chart reviews. The Danish National Patient Registry required both an ICD‐10 code as well as use of a loop diuretic in the pharmacy database for the diagnosis of congestive heart failure, use of 2 antihypertensive medications from 2 separate medication classes to diagnose hypertension, and use of a glucose‐lowering drug to diagnose diabetes mellitus. All administrative database studies had average look‐back periods of 2 or more years8, 16, 17, 18, 21, 22 (Table 3).

Determination of Outcome Events and Analysis of Stroke Rates

Ischemic stroke was the primary outcome in all 9 studies (Table 4). The Israel Clalit study also included transient ischemic attack, the Danish National Patient Registry Study included peripheral embolism and pulmonary embolism, and the ATRIA study included systemic embolism. Eight studies used ICD‐9 or ICD‐10 hospital discharge codes to identify outcome events.8, 15, 16, 17, 18, 19, 20, 21, 25 Seven of these did not specify whether nonprimary discharge diagnoses were also included; presumably they were.8, 15, 17, 18, 19, 20, 21, 25 There was substantial variation in the codes used for the outcome of ischemic stroke. While the more specific codes of ICD‐9 433 and 434 and ICD‐10 I63 were uniformly used, more nonspecific codes (eg, ICD‐9 436: “Acute, but ill‐defined, cerebrovascular disease”; and ICD‐10 I64: “Stroke, not specified as hemorrhage or infarction”)22, 23 were also used by several studies (Table 4). Only 2 studies validated outcome events via clinical record review; both of these studies had lower than median overall event rates.15, 17 Two studies used death certificates to capture outcome events; both had higher than median overall event rates8, 20 (Table 4). Three studies excluded patients from analysis if they started OACs later in follow‐up16, 21, 22 (Table 4).

Quantitative Relationship Between Cohort Features and Methodologic Approaches and Reported Stroke Rates

Given the limited number of studies included, there were wide confidence intervals around estimates of rate ratios in our study‐level analyses. Nonetheless, reported stroke rates were clearly associated with several study features (Figure 3). Higher stroke rates were reported from cohorts with a retrospective versus a prospective study design, RR 4.09 (1.06, 15.70) and inclusion of inpatients only versus solely outpatients, RR 2.60 (1.19, 5.68). Lack of outcome event validation was also associated with higher stroke rates, RR 4.09 (1.06, 15.70), but it is worth noting that cohorts lacking event validation were the same as those with a retrospective design. Inclusion of a death certificate diagnosis of stroke as part of outcome event ascertainment was associated with a 78% increase in reported stroke rates, but this RR did not achieve statistical significance. Reported stroke rates were higher in cohorts having higher proportions of patients with prior stroke and higher CHA2DS2‐VASc scores, although these effects mainly reflected the very low outcome rates in the lowest categories. Stroke rates for studies that did and that did not include aspirin users were very similar. In our study‐level analysis, exclusion of patients starting OAC later in follow‐up and use of a blanking period were not significantly associated with rates of stroke. North American cohorts reported the lowest stroke rate, significantly lower than the rate for European cohorts, RR 4.62 (1.00–21.41). In this set of cohorts, there was no clear pattern of stroke rates by calendar year of the study (Figure 3).

Figure 3

Atrial fibrillation cohort features and methodologic approaches associated with reported stroke rates: study‐level analysis. ASA indicates acetylsalicylic acid; OAC, oral anticoagulants.

Discussion

The decision to anticoagulate a patient with AF is based on the expected net clinical benefit of OAC, which is the balance between reduced risk of ischemic stroke and increased risk of bleeding, weighted by the impact of each outcome.2, 3 The expected net clinical benefit is largely determined by the estimated absolute risk of ischemic stroke off OAC for a given individual patient.26, 27 Decision analysis has identified the threshold for a positive net clinical benefit as a stroke risk off anticoagulants of between 0.9% and 2% per year.3 The AHA/ACC/HRS and ESC guidelines recommend the CHA2DS2‐VASc scoring system to guide use of anticoagulants in AF, assuming that CHA2DS2‐VASc point scores correspond to fixed absolute risks of ischemic stroke.4, 5, 6 However, our previous work demonstrated wide variation in overall and point score–stratified stroke rates across multiple large AF cohorts, despite similar high study quality using an objective scoring system.10, 28 Of note, the CHA2DS2‐VASc point score–specific stroke rates observed in several of these cohorts were too low to support the anticoagulation threshold recommended by the AHA/ACC/HRS and ESC guidelines (CHA2DS2‐VASc ≥2, discounting female sex, for the ESC guidelines).4, 5, 6 Marked differences in stroke rates were reported from separate studies from the same national database15, 20 and from national databases from adjacent countries,19, 21 making it unlikely that the observed differences reflected true differences in population stroke rates. Furthermore, the large size of these cohort studies made chance variation an unlikely source of differences. We concluded that many of the differences in reported stroke rates across cohorts were due to differences in research methodology.10

Eight of the 9 studies that we assessed used large insurance or national administrative databases. There are general guidelines for conducting analyses of such databases,29 but these guidelines have not addressed the range of AF‐specific methodologic concerns, highlighted in Table 1, which can have large effects on estimating stroke rates. Friberg et al described the strong dependence of observed AF stroke rates on such analytic choices as ICD‐10 discharge diagnosis position, length of blanking period, and inclusion of transient ischemic attack as an outcome.11 Nielsen et al described the impact of excluding AF patients who later in follow‐up started OAC30 and others have assessed the validity of ICD codes relevant to identifying AF and ischemic stroke.31, 32 In the current study, we provide a more comprehensive perspective, exploring differences in stroke rates according to the full range of core features of study design (ie, characteristics of the cohort, assembly of AF patients, determination of anticoagulant status, assessment of comorbidities, ascertainment of outcome events, and analytic approach). In our study‐level analysis, we found several notable associations of these features with stroke rates (Figure 3). Studies including AF patients identified as inpatients had markedly higher rates of stroke than studies that solely included outpatients. Similarly, retrospective analyses of databases had higher event rates than prospectively conducted studies, and studies that simply included ICD code‐identified events had higher rates than studies that validated events. We found marked regional variation in reported stroke rates. In particular, studies from North America observed far lower rates of stroke than those done in other regions. It seems unlikely that true AF stroke rates in Europe are 4 times higher than those in North America. In fact, important methodologic features overlapped with region; North American studies that we included used prospective designs with event validation; included European studies did not. Finally, we note that there was marked variation in event types included as outcomes (eg, inclusion of transient ischemic attacks), code sets used to identify given outcome events, and whether nonprimary discharge diagnoses were included.

Our study does shed some light on the puzzling difference in reported stroke rates between the Swedish AF21 and the Danish AF Registry cohorts,8 where both studies analyzed comprehensive national databases. Despite having a lower average CHA2DS2‐VASc score, the Danish cohort reported an overall stroke rate 56% higher than the Swedish cohort. The Swedish cohort excluded patients who started OAC later in follow‐up. Nielsen et al have argued that this approach likely excluded some patients who started OAC after sustaining a stroke off OAC, resulting in a biased low observed stroke rate.33 By contrast, the Danish AF cohort study used a more expansive set of outcomes and an additional, more nonspecific code for ischemic stroke (I64 as well as I63). The Danish study also accepted death certificate diagnoses of stroke as valid acute events, a feature associated with a 78%, but not statistically significant, increase in event rates in our study‐level analysis. Presumably, such deaths occurred in an outpatient setting since inpatient strokes would be first identified by an ICD‐10 diagnosis of stroke. Death certificate diagnosis of outpatient stroke may be inaccurate and, furthermore, may identify a prior stroke as the underlying cause of death, rather than an acute event. Another difference in the analyses of these neighboring AF cohorts was in the ascertainment of AF stroke risk factors. The Danish study required use of condition‐specific medications, thereby restricting the diagnoses to more severe conditions. This likely raised the CHA2DS2‐VASc‐specific, though not overall, stroke rates. The quantitative impact of these methodologic differences could only be approached by reconstructing and evaluating the 2 cohorts using the exact same methods.

Administrative databases are an attractive source of information about AF stroke rates because of the size, accessibility, and generalizability of the AF cohorts available. Guidelines for AF highlight the findings of studies of administrative databases in their recommendations for use of anticoagulants.4, 5, 6 On the surface, interrogation of such databases would seem straightforward. However, as we have demonstrated, rate estimates across such cohorts do not agree. Methods of establishing denominator AF populations, assigning comorbid diagnoses, establishing anticoagulant status, defining outcome events, and analytic strategy are highly variable. In studies comparing therapeutic options within a given study, these sources of bias may be balanced in the comparison arms, thereby generating generalizable estimates of relative effect (eg, RRs). Moreover, methodologic variability is particularly problematic when estimating the small absolute annual ischemic stroke rates associated with AF; yet these rates are what are needed to guide prescription of anticoagulants. Our structured inventory of methodologic considerations (Table 1) can provide a framework for standardizing interrogation of administrative databases to generate accurate estimates of AF stroke rates. Current large research registries using high‐quality and comprehensive clinical data and follow‐up can also contribute, although they are subject to referral biases.34, 35, 36

We focused on only 9 AF cohort studies. However, these cohorts accounted for 95% of the total person‐years in our systematic review of studies reporting stroke rates in AF patients off OAC, and are repeatedly cited, particularly in guidelines. The small number of studies limited the statistical power of our study‐level analyses and prevented multivariable modeling. In addition, some features associated with stroke rate overlapped in the same studies. As a result, we cannot identify the strongest independent study‐level determinants of reported stroke risk. Nonetheless, the bivariate effects we identified were strong and support the case that analytic approaches need to be standardized and validated to generate accurate estimates of stroke rates. We believe we have provided a logical inventory of methodologic considerations in conducting AF stroke rate studies, but we acknowledge that other methodologic choices may account for a significant proportion of the differences in reported stroke rates. Finally, our analysis at the study level may obscure true associations at the individual patient level. For example, we found no relationship between a study's reported stroke rates and the calendar year of the study. Yet, evidence from patient‐level analysis suggests that AF stroke rates have been decreasing in recent years.37

Conclusions

Precise and accurate estimates of stroke rates off anticoagulants are needed to optimize risk‐based use of anticoagulants in patients with AF. There are marked differences in reported rates of stroke across different large cohorts of patients with AF, including those from the same or adjacent geographic regions. These differences undermine the generalizability of CHA2DS2‐VASc score thresholds for anticoagulation recommended by leading guidelines. Using a structured approach, we have identified important differences in the methods used to study large AF cohorts and have documented the effects of these differences on estimates of overall and CHA2DS2‐VASc point score–specific stroke rates off anticoagulants. Our findings suggest a framework for a standardized, validated approach to creating and analyzing AF cohorts to produce accurate and generalizable predictions of stroke risk (Table 1). Guidelines should take into account the variation in both reported stroke rates and study methods when recommending a stroke risk score threshold for use of anticoagulants for patients with AF.

Sources of Funding

Dr Quinn was supported by the Harvard Medical School Fellowship in Patient Safety and Quality (Boston, MA). Dr Singer was supported, in part, by the Eliot B. and Edith C. Shoolman Fund of Massachusetts General Hospital (Boston, MA).

Disclosures

Dr Singer has consulted for Boehringer Ingelheim (modest), Bristol‐Myers Squibb (significant), CVS Health (modest), Johnson & Johnson (modest), Medtronic (modest), Merck (modest), and Pfizer (modest), and has received research support from Boehringer Ingelheim (significant), and Bristol‐Myers Squibb (significant). The remaining authors have no disclosures to report.