Reliability of patient-reported outcome measures: Hemorrhage, anticoagulant, antiplatelet medication use.

BACKGROUND: Most antithrombotic medication users are older adults. Patient-reported outcome measures are commonly used in clinical research on antithrombotic medication, such as the diagnosis of intracranial hemorrhage.
OBJECTIVES: To determine the reliability of patient-reported intracranial hemorrhage, anticoagulant and platelet aggregation inhibitor use in the older adult population. PATIENTS/
METHODS: We conducted a secondary analysis of a prospective, observational cohort study of older adults who presented to the emergency department with a fall. The primary outcome was diagnosis of intracranial bleeding. We compared patient-reported intracranial bleeding to structured chart review with adjudication. We also compared patient-reported use of antiplatelet and anticoagulant medication to physician-reported medication use supplemented with structured chart review. We calculated the diagnostic accuracy of the patient-reported outcomes using our comparators as the reference standard.
RESULTS: Exact agreement for patient-reported intracranial bleeds was 95%, with a Cohen's kappa of 0.30 (95% confidence interval [CI], 0.15-0.45). The sensitivity was 36.7% (95% CI, 20.6%-56.1%) and specificity 97.2% (95% CI, 95.8%-98.1%). For anticoagulant medication use, exact agreement was 87%, Cohen's kappa 0.66 (95% CI, 0.63-0.72), sensitivity 84.0% (95% CI, 79.3%-83.8%), and specificity 87.6% (95% CI, 85.1%-89.7%). For antiplatelet medication use, exact agreement was 77%, Cohen's kappa 0.50 (95% CI, 0.44-0.55), sensitivity 68.7% (95% CI, 64.0%-73.1%), and specificity 81.2% (95% CI, 78.0-83.8%).
CONCLUSIONS: Patient-reported outcome and exposure data were unreliable in this study. Our findings have a bearing on future research study design.

Some research studies use participant interviews to classify outcomes or exposures.

We analyzed the reliability of patient‐reported intracranial bleeding and antithrombotic use.

Most participants with intracranial bleeding denied having the diagnosis.

Patient‐reported antithrombotic use differed from physician and chart documentation.

INTRODUCTION

Studies evaluating anticoagulants and platelet aggregation inhibitors generally report both the efficacy and safety of the medication. Intracranial hemorrhage is the most feared complication of antithrombotic therapy and is an important safety outcome for studies evaluating antithrombotic drugs. Antithrombotic drugs are commonly used for the prevention of stroke in atrial fibrillation and treatment or prevention of venous thromboembolism and arterial disease. The majority of people prescribed antithrombotic medications are older adults with atrial fibrillation or venous thrombotic disease. , ,

For research assessing anticoagulants and platelet aggregation inhibitors, the generally accepted standard for baseline data collection and follow up involves patient interviews, including specific questions on medication use and bleeding complications. Cognitive impairment, progressive hearing loss, and other geriatric syndromes may limit the utility of patient‐reported outcome measures in some older adults. As a result, older adults are often excluded from clinical research. , , This is problematic for research pertaining to antithrombotic therapy, since older adults constitute the majority of antithrombotic medication users and aging is associated with increased bleeding risk. ,

The objective of this study was to determine the reliability of patient‐reported intracranial hemorrhage and antithrombotic medication use in the older adult population.

METHODS

Study design

We conducted a secondary analysis using data from a prospective, observational cohort study (NCT03870867) that enrolled older adults (≥65 years) who presented to one of the three emergency departments (Hamilton General Hospital, Hamilton; Juravinski Hospital, Hamilton; Mount Sinai Hospital, Toronto) in Ontario, Canada, after a fall. Patients were interviewed at the time of enrollment and again at follow‐up, 42 days after they were enrolled. We compared patient‐reported outcomes to our medical record review. This study received ethics approval from Hamilton Integrated Research Ethics Board and the Research Ethics Board at Mount Sinai Hospital before commencing. The study had research ethics board approval to follow patients in person for 42 days, only if they were able to give consent. Those who were too unwell or else did not have capacity to give consent were included in the study but were not followed in person (and not included in this analysis).

Measurement of study outcomes

The primary outcome of the original study was diagnosis of intracranial bleeding within 42 days of the index emergency department presentation. Intracranial bleeding was identified by medical record review for all hospitals where the patient was hospitalized during the 42‐day follow‐up period. Charts were systematically reviewed, starting with computed tomography (CT) and magnetic resonance imaging (MRI) brain imaging reports, then emergency department and inpatient records, followed by hospital discharge summaries and clinic letters. Intracranial bleeding was defined as bleeding diagnosed by head CT or MRI within any compartment (epidural, subdural, subarachnoid, intracerebral, intraventricular, or brain contusion) regardless of blood volume. An adjudication panel of three expert physicians reviewed the intracranial bleeding cases, which were identified by medical record review, to confirm the diagnosis of intracranial bleeding. Medical record reviews were performed independently in duplicate. Disagreements were resolved by the local principal investigators.

The study collected baseline data on the use of antiplatelet medications (aspirin, clopidogrel, ticagrelor, and prasugrel) and anticoagulant medications (warfarin, dabigatran, apixaban, edoxaban, rivaroxaban, unfractionated heparin, low‐molecular‐weight heparin, sinthrome, and fondaparinux), all secondary outcomes in this analysis. The treating emergency physician completed a data collection form to indicate the use of antiplatelet and anticoagulant medications. This was supplemented by an in‐depth medical record review by trained research personnel. Documented evidence of antiplatelet or anticoagulant use in the medical record (including emergency physician and nursing documentation, physician admission records, pharmacy reconciliation records, and clinic visits within the past month) or indicated by the treating physician on the data collection form was considered as evidence of medication use. Health record reviews were performed in duplicate by trained researchers, who were blinded to the patient interviews. Disagreements were resolved by the local principal investigators.

Patient‐reported outcome measures

For 42‐day follow‐up, patients were interviewed by a trained research assistant, either by telephone or in person if the patient was admitted in hospital. The interview could take place with the substitute decision maker upon the patient’s request. The interviewer asked about diagnoses of intracranial bleeding and followed a standard script (Appendix 1): “Has a physician told you that you have had bleeding in your head since your initial emergency department visit?”

Patients were also interviewed by a research assistant at the time of enrollment, following a standard interview script that had been piloted before study commencement (Appendix 1). Participants were asked about their use of antiplatelet and anticoagulant medications: “Do you take aspirin, Plavix, or Brilinta”; and “Do you take a blood thinner or an anticoagulant?”

Data analysis

Descriptive statistics were reported using general measures of frequency and central tendency, or proportions. The primary analysis was the reliability between patient‐reported intracranial bleeding events and intracranial bleeds identified through medical record review with adjudication. Secondary analyses included the reliability of patient‐reported antiplatelet and anticoagulant use, as compared to medical record review in combination with physician‐reported data. Cohen’s kappa statistic with 95% confidence intervals (CIs) was used to determine the reliability of all three outcomes. We also reported the sensitivity and specificity of patient‐reported intracranial bleeds, anticoagulant use, and antiplatelet use (using the review of the medical record as the reference standard). The analysis was performed using the Statistical Package for Social Sciences (SPSS), version 26.0 (IBM, Armonk, NY, USA).

RESULTS AND DISCUSSION

From the 1753 study participants, 986 completed their baseline and 42‐day interview, 114 died before their initial or 42‐day interview, 221 could not be contacted at 42 days, 376 were unable to consent for the interviews, and 56 had an intracranial bleed identified before 42 days and did not require a 42‐day interview. A total of 1205 patients provided interview information on one or more of the following: diagnosis of intracranial bleeding or anticoagulant use and/or antiplatelet use. Median age of the patients was 81 years, and the majority were women (see Table 1 for demographics obtained from medical chart review).

TABLE 1

Demographics of the cohort

Description	Number (%) or median (IQR) N = 1205
Age	81 (74‐87)
Male	469 (39)
Hypertension	932 (77)
Liver disease	30 (2)
Diabetes	359 (30)
Renal impairment	124 (10)
Cancer	102 (8)
Stroke or transient ischemic attack	226 (19)
Cognitive impairment	248 (21)
Congestive heart failure	173 (14)

Abbreviation: IQR, interquartile range.

Primary outcome

Of the 1205 participants in this analysis, 986 (82%) participants provided information on intracranial bleeding at a follow‐up interview. Five of these patients said they did not know if they had been diagnosed with intracranial bleeding. Thirty‐eight participants reported being diagnosed with intracranial bleeding within 42 days of their presentation to the emergency department (Table 2). Based on the medical record review, the adjudication panel confirmed that 32 patients had been diagnosed with intracranial bleeding. Exact modal agreement between patient‐reported bleeds and panel adjudication was 95%, with a Cohen’s kappa of 0.30 (95% CI, 0.15‐0.45). When compared to medical record review with adjudication, patient‐reported history of intracranial bleeds had a sensitivity of 36.7% (95% CI, 20.6%‐56.1%) and a specificity 97.2% (95% CI, 95.8%‐98.1%).

TABLE 2

Comparison of patient‐reported intracranial bleeding to medical record review with adjudication

	Intracranial bleed by medical record review with adjudication	No intracranial bleed by medical record review with adjudication	Total
Patient‐reported bleed	11	27	38
Patient‐reported no bleed	19	924	943
Patient did not know	2	3	5
	32	954	986

Secondary outcomes

Of the 1205 participants in this analysis, 1159 (96%) gave baseline information regarding anticoagulant medication use, with 362 reporting routine anticoagulant use (Table 3). Eight participants did not know if they were taking an anticoagulant. The medical record review, in combination with the physician‐reported study forms, found that 309 participants were prescribed anticoagulants. For anticoagulation use, exact agreement was 87%, with a Cohen’s kappa of 0.66 (95% CI, 0.63‐0.72). When compared to combined physician‐reported and health record review, the sensitivity of patient‐reported anticoagulation was 84.0% (95% CI, 79.3%‐83.8%) and specificity 87.6% (95% CI, 85.1%‐89.7%).

TABLE 3

Comparison of patient‐reported anticoagulant use to medical record review/physician‐reported data

	Anticoagulant use by medical record review/physician‐reported	No anticoagulant use by medical record review/physician‐reported	Total
Patient‐reported anticoagulant use	257	105	362
Patient‐reported no anticoagulant use	49	740	789
Patient did not know	3	5	8
	309	850	1159

Of the 1162 patients who gave baseline information about antiplatelet medication use, 423 patients reported they were taking an antiplatelet medication (Table 4). Seven patients did not know if they were taking an antiplatelet medication. The medical record review, in combination with the physician‐reported study forms, confirmed that 413 participants used an antiplatelet medication. Exact agreement was 77%, with a Cohen’s kappa of 0.50 (95% CI, 0.44‐0.55). The sensitivity of patient‐reported antiplatelet use was 68.7% (95% CI, 64.0%‐73.1%) and specificity 81.2% (95% CI, 78.0%‐83.8%).

TABLE 4

Comparison of patient‐reported antiplatelet use to medical record review/physician‐reported data

	Antiplatelet use by medical record review/physician‐reported	No antiplatelet use by medical record review/physician‐reported	Total
Patient‐reported antiplatelet use	283	140	423
Patient‐reported no antiplatelet use	129	603	732
Patient did not know	1	6	7
	413	749	1162

Subgroup analyses of patients >75 years of age and patients with a diagnosis of cognitive impairment did not find a significant difference in accuracy of patient‐reported outcomes as compared to those ≤75 years of age or those without a diagnosis of cognitive impairment (Appendix 2).

In this prospective cohort study of older adults, we found that data collected via in‐person interviews were unreliable. Our findings have implications for the design and methods of future studies on antithrombotic medications and bleeding. In particular, participants who had been diagnosed with intracranial bleeding frequently denied or were unaware of the diagnosis. Researchers should consider identifying intracranial bleeding through medical health record review. Furthermore, 10% to 20% of participants reported antiplatelet and anticoagulant medication use when the treating physician and hospital records did not. Care should be taken when verifying antithrombotic use in older adults, and consideration should be given to cross referencing with additional information sources.

Although intracranial bleeding could result in impaired cognition in older adults, this is an understudied topic without large, well‐powered studies reporting such associations. This biologically plausible explanation might account for participants’ poor recollection of being diagnosed with intracranial bleeding. However, all participants included in this analysis were capable of giving informed consent to participate in the study. Participants were also given the opportunity to have a family member answer interview questions to ensure that our findings exhausted all possible methods for obtaining accurate patient‐reported data. There are no prior studies evaluating the reliability of patient‐reported intracranial pathology. Our findings were similar to those reported in regard to patient‐reported medication. , ,

There are some limitations in our study. We did not study the reliability of other clinical outcomes such as the diagnosis of venous thromboembolism, or gastrointestinal or urological hemorrhage. We assumed that our reference standards (medical record review in combination with adjudication for intracranial bleeding and physician‐reported data for medications) were an accurate representation of reality. In particular, it is possible that we misclassified drug use in some patients, since we considered the patient an anticoagulant or antiplatelet user if this was documented on either the physician‐reported form or the medical record review. We rechecked the medical records for those cases where participants reported intracranial bleeding, including charts from all hospitals where they were patients. We found no information supporting a missed diagnosis of intracranial bleeding. However, there was no similar method to verify whether or not the patient was prescribed antiplatelet or anticoagulant medication. We used open‐ended questions in the interview, asking our patients to classify their medications as anticoagulants, and we used brand names for clopidogrel and ticagrelor. We do not know whether asking for written lists would have been more accurate or whether talking with a caregiver or family member instead of the participant would have given different results.

In conclusion, our analysis found that when interviewed, older adults did not report reliable information on the diagnosis of intracranial bleeding, and there was poor reliability between patient‐reported use of antithrombotic medications and physician‐reported/medical record–documented use of these medications. Future research focusing on antithrombotic medication and bleeding should account for these findings in the study design.

RELATIONSHIP DISCLOSURE

The authors declare no conflicts of interest.

Appendix S1

Click here for additional data file.

Appendix S2

Click here for additional data file.