Inviting patients to identify diagnostic concerns through structured evaluation of their online visit notes.

BACKGROUND: The 21st Century Cures Act mandates patients' access to their electronic health record (EHR) notes. To our knowledge, no previous work has systematically invited patients to proactively report diagnostic concerns while documenting and tracking their diagnostic experiences through EHR-based clinician note review.
OBJECTIVE: To test if patients can identify concerns about their diagnosis through structured evaluation of their online visit notes.
METHODS: In a large integrated health system, patients aged 18-85 years actively using the patient portal and seen between October 2019 and February 2020 were invited to respond to an online questionnaire if an EHR algorithm detected any recent unexpected return visit following an initial primary care consultation ("at-risk" visit). We developed and tested an instrument (Safer Dx Patient Instrument) to help patients identify concerns related to several dimensions of the diagnostic process based on notes review and recall of recent "at-risk" visits. Additional questions assessed patients' trust in their providers and their general feelings about the visit. The primary outcome was a self-reported diagnostic concern. Multivariate logistic regression tested whether the primary outcome was predicted by instrument variables.
RESULTS: Of 293 566 visits, the algorithm identified 1282 eligible patients, of whom 486 responded. After applying exclusion criteria, 418 patients were included in the analysis. Fifty-one patients (12.2%) identified a diagnostic concern. Patients were more likely to report a concern if they disagreed with statements "the care plan the provider developed for me addressed all my medical concerns" [odds ratio (OR), 2.65; 95% confidence interval [CI], 1.45-4.87) and "I trust the provider that I saw during my visit" (OR, 2.10; 95% CI, 1.19-3.71) and agreed with the statement "I did not have a good feeling about my visit" (OR, 1.48; 95% CI, 1.09-2.01).
CONCLUSION: Patients can identify diagnostic concerns based on a proactive online structured evaluation of visit notes. This surveillance strategy could potentially improve transparency in the diagnostic process.

INTRODUCTION

According to the National Academies of Sciences, Engineering, and Medicine (NASEM) report Improving Diagnosis in Healthcare, most people will experience at least one diagnostic error in their lifetime. Lack of adequate data sources and insufficient measurement methods provide a limited view of the problem. The NASEM report highlights the need to monitor the diagnostic process (ie, “a complex, collaborative activity that involves clinical reasoning and information gathering to determine a patient’s health problem”) and to create further opportunities to identify, learn from, and reduce diagnostic errors. Among the report’s recommendations is to increase engagement of patients and families to contribute valuable input to facilitate accurate and timely diagnosis. Prior research indicates that patients can identify concerns related to the diagnostic process and often disclose information that existing measurement methods fail to capture. In the inpatient setting, there has been movement toward understanding patients’ needs for engagement in quality- and safety-focused informatics interventions., However, patients remain on the periphery of diagnostic safety initiatives.

Implementation of the 21st Century Cures Act mandates patients’ access to their clinical notes, diagnostic test results, and other information in their electronic health records (EHRs) beginning in 2021. Increased transparency of medical information and evolving capabilities of patient-facing technologies could provide a novel opportunity to engage patients in safety surveillance. For example, patients have reported medication concerns and incorrect documentation while accessing their providers’ notes online (eg, open notes) through secure web-based portals., In one study, 1 in 5 patients who read their notes found a mistake—many of which were related to the diagnostic process. To our knowledge, no previous work has systematically invited patients to identify concerns about diagnosis in their recent visits. This type of approach can enhance safety surveillance from patients and raise the bar for transparency of the diagnostic process.

Patients also need structured tools for reliable reporting that yield information that is meaningful to clinicians and health systems for improving safety. Our long-term goal is to develop a proactive surveillance strategy that helps identify diagnostic safety concerns in patients. However, methods to study if and how patients can evaluate the diagnostic process via note review have not been developed. Such methods could uncover patients’ feedback about the diagnostic process both when things go well and when they do not. As a first step to achieving the goal of a proactive surveillance safety strategy related to diagnosis, we developed and tested methods to allow patients to systematically identify any diagnostic concerns while accessing and reviewing their recent visit notes and to identify predictors of patient-reported diagnostic concerns.

MATERIALS AND METHODS

Participants and setting

Our study team partnered with Geisinger, a large integrated healthcare organization in central, south-central, and northeast Pennsylvania, spanning 45 counties, mostly rural, serving approximately 4.2 million residents. The study was approved by the local Institutional Review Boards.

To ensure an appropriate target population for testing, we first applied methods to select patients who were more likely to experience diagnostic concerns. We use the term “diagnostic concern” rather than diagnostic error to account for variations in how patients may self-report and perceive diagnostic accuracy and because diagnosis is an evolving process involving uncertainty. In the context of this study, a diagnostic concern is present when a patient disagrees with the statement, “I feel I was correctly diagnosed during my first visit.” We used an electronic trigger algorithm (e-trigger) based on unexpected visit patterns, an approach that has been tested in prior work., E-triggers mine EHR databases to identify signals for likely adverse events that can be confirmed with additional review., The trigger was based on the rationale that an unexpected hospitalization or return to clinic, urgent care, or ED visit after an initial primary care visit may indicate that a diagnosis was missed during the first visit. All planned/previously scheduled office visits or elective admissions were excluded to increase the potential that the visit was “unexpected.” We applied this algorithm in Geisinger’s enterprise data warehouse from October 2019 to February 2020 and identified patients with an index primary care visit followed by an unplanned return visit (such as an emergency department, primary care or urgent care visit, or nonelective hospitalization) within 14 days. We defined this cohort “at-risk” for diagnostic concerns because the diagnosis may have been missed or evolved at the first visit.

Patients aged 18–85 years old were included if they met e-trigger criteria and if they had previously logged on to MyGeisinger, a patient portal based on the Epic EHR, at least once to view clinical notes. We applied the electronic algorithm to Geisinger’s enterprise data warehouse daily to identify patients as soon as they become eligible, thus detecting “at-risk” patients in real time as possible. After confirming eligibility criteria, we contacted patients via a secure electronic message with an invitation to complete a newly developed instrument, the Safer Dx Patient Instrument, about their experience of diagnosis at the time of the index visit, including an assessment of the diagnostic process. A follow-up message was sent to patients who did not respond after 2 weeks. We excluded patient caregivers (n = 17), patients who were unsure (n = 49), or did not respond (n = 1) to whether or not the diagnosis was correct, and one patient who was unable to view the clinician’s notes in MyGeisinger. Patients received a $25 gift card for instrument completion.

Development and testing of the Safer Dx Patient Instrument

We adapted the Revised Safer Dx Instrument, a medical record review tool developed to enable clinicians to determine the presence or absence of diagnostic error for a specific episode of care. The Safer Dx Instrument has been previously validated and applied in multiple settings to evaluate the diagnostic process. When used for review of high-risk records (eg, records flagged by electronic algorithms), the instrument can help clinicians reliably identify missed opportunities in diagnosis. We adapted this instrument to be used similarly by patients.

The Safer Dx Instrument evaluates 5 dimensions of the diagnostic process,: (1) the patient-provider encounter (history, physical exam, symptom presentation, determinations regarding need for further evaluation, testing and/or referral), (2) performance and interpretation of diagnostic tests, (3) follow-up and tracking of diagnostic information over time, (4) subspecialty and referral-specific factors, and (5) patient-related factors. Early drafts of the Safer Dx Patient Instrument were created, reviewed, and revised over multiple meetings among the authorship team and assessed by a psychometrician (CS). A patient advocate and a health literacy expert from Geisinger reviewed multiple versions of the instrument to ensure questions were patient-centered and that questions and formatting were appropriate for the patient population. The items were adapted to assess the 5 dimensions of the diagnostic process from the perspective of the patient, including questions about the reason for the visit, the accuracy of patient symptoms documented, relevance and accuracy of the physical exam documented, testing concerns, adequacy of follow-up instructions, adequacy and completeness of the care plan to address all medical concerns, and assessment of the diagnosis (Table 1).

Table 1.

The Safer Dx Patient Instrument: items for determining presence or absence of diagnostic concern in a primary care encounter

Questions	Response
1. Please tell us why you visited the doctor on this day.	Open ended
2. The notes captured information related to my symptoms accuratelya	Likert1 2 3 4 5
Please describe any inaccuracies:	Open ended
3. The provider conducted a physical exam relevant to my symptoms during my visit.a	Likert
	1 2 3 4 5
4. The notes captured my physical exam accurately.a	Likert
Please describe any inaccuracies:	1 2 3 4 5
	Open ended
5. Did the provider order any tests?a,b	Yes/no
a. Based on the notes reviewed, I had concerns about the test(s) my doctor ordered (eg, lab imaging or any procedure).	Likert
	1 2 3 4 5
Please explain:	Open ended
b. Based on the notes review, I had concerns about my test results.	Likert
	1 2 3 4 5
Please explain:	Open ended
6. The notes reflected what I was supposed to do if my symptoms did not get better.a	Likert
	1 2 3 4 5
7. The care plan the provider developed for me addressed all my medical concerns.a	Likert
	1 2 3 4 5
8. I feel I was correctly diagnosed during my first visit.b	Likert
	1 2 3 4 5
If not, please explain:	Open-ended
a. The notes reflected multiple alternative diagnoses that were discussed with me.	Likert
	1 2 3 4 5
b. I received a new diagnosis at my follow-up visit.	Likert
Please explain:	1 2 3 4 5
9. I trust the provider that I saw during my visit.	Likert
	1 2 3 4 5
10. I did not have a good feeling about my visit.	Likert
	1 2 3 4 5
11. How often do you need to have someone help you when you read instructions, pamphlets, or other written material from the provider or pharmacy?	Likert (1 = never, 2 = rarely, 3 = sometimes, 4 = often, 5 = always)
	1 2 3 4 5

Question related to the dimensions of the diagnostic process.

Skip pattern based on response (1 = disagree, 2 = somewhat disagree, 3 = neither agree nor disagree, 4 = somewhat agree, 5 = agree).

Patients were asked to rate their level of agreement with statements about the diagnostic process using a 5-point scale (agree, somewhat agree, neither agree nor disagree, somewhat disagree and disagree) and were also able to provide open-ended comments on most items. Response choices for one item (“Did the provider order any tests?”) were dichotomous. Skip pattern questions associated with a conditional response were used. A self-reported assessment of the diagnosis was captured through a single question: “I feel I was correctly diagnosed at my first visit.” To assess for diagnostic concerns, patients were given a specific date and instructed to review available notes. We expected patients to use this information to reflect on their visit experience and rely on hindsight knowledge about their care experiences to answer questions. In addition to questions about the diagnostic process, the instrument included items that assessed trust in the clinician, feelings about the visit, and health literacy.

Initial versions of the instrument were reviewed using the think-aloud method with patients at a separate site (n = 5) to assess for clarity. To obtain further feedback, the instrument was also presented to Geisinger’s Patient and Family Advisory Council, the Patient Experience department leadership, and a multidisciplinary system-wide committee focused on improving clinical diagnosis. The instrument was piloted at Geisinger twice (n = 27 and n = 59, respectively). In the first pilot, we identified inconsistencies in the data and reworked the skip patterns, while the second pilot ensured item response variability.

Patient involvement

Patients were consulted throughout the study. Initially, a patient advocate met with the first author to discuss the purpose of the study and how to adapt the instrument to capture the diagnostic process while also being patient-centered. They reviewed multiple drafts of the survey. In addition, the Patient and Family Advisory Council reviewed the instrument and suggestions were incorporated.

Patient review of the visit progress note and response

Patients were instructed to review their clinician’s note from the date of the index visit prior to completing the questionnaire, and if possible, to keep the notes accessible while simultaneously answering questions. Data from the questionnaire was collected and managed using DatStat, a web-based application for collecting and managing research and quality improvement study data within Geisinger. In addition, we extracted patient-related information from the Geisinger enterprise data warehouse that consolidates data from the Epic EHR and other sources. All patient-related variables were those that were confirmed or defined at the time of the index visit date. Information obtained included age at the time of the visit, sex, race and ethnicity, and type of health insurance.

Statistical analysis

The primary outcome was self-identified diagnostic concern (defined as disagreement [eg, disagree, somewhat disagree] with the statement “I feel I was correctly diagnosed during my first visit”). Likert scale response data were treated as continuous data and assumptions of normality and equal variance of residuals were tested and met. Univariate and multivariate logistic regression models were used to identify whether patient-identified diagnostic concerns were associated with responses to other items assessing patients’ perceptions of the 5 different dimensions in the diagnostic process, trust, and feelings about the visit. In addition, we adjusted for potential confounding factors using patient-related variables (ie, age at the time of the visit, sex, race, ethnicity, type of health insurance, and health literacy). Questionnaire responses were examined for any values that were missing, and sampling weights were used to account for missing data and other potential biases.

We used t-tests to compare patient characteristics between patients who identified their diagnosis as correct and those who reported a diagnostic concern. Bivariate correlations were used to determine the relationship between variables and detect any multicollinearity in our data. To further assess the potential for multicollinearity for each independent variable, we also examined variance inflation factor (VIF) scores (<10). All variables in our analyses were entered sequentially in the model using the forward selection approach to determine which variables to add or drop in the model. Our criterion for entry was at P < .05 and remaining variables that did not meet criteria were removed. We used the Hosmer-Lemeshow test to determine the goodness-of-fit of our model. All analyses were conducted using SAS 9.4 (SAS Institute Inc., Cary, NC, USA).

RESULTS

Patient characteristics

The algorithm was applied to 293 566 primary care visits from October 2019 to February 2020 to identify eligible patients. We sent surveys to 1282 patients who met all inclusion criteria and received 486 responses (response rate: 37.9%). Of these, 469 (96.5%) were completed by patients and 17 (3.5%) by caregivers. After applying exclusion criteria (ie, 17 caregivers, 49 patients were unsure and one was unresponsive to the question of whether the diagnosis was correct, and one who did not have access to notes), our final sample consisted of 418 patients. Of 418 patients, 12.2% (n = 51) had a diagnostic concern.

The average age of patients was 48.6 years (SD ±16.0 years) at the time of the visit. Just over three-fourths of patients were female (76.8%), and almost all were White (95.9%), followed by Black or African American (2.9%), Asian (0.7%), and American Indian or Alaska Native (0.5%). Only 2.6% identified as Hispanic or Latinx. Most had private insurance (83.3%), followed by public insurance (14.1%), federal (1.9%), and no insurance (0.7%). Characteristics of patients with and without self-identified diagnostic concerns were similar, with the exception that latter patients were older (P < .001, Table 2). Half of patients reported that their provider ordered tests during the visit (49.8%; n = 208).

Table 2.

Characteristics of Safer Dx Patient Instrument respondents

Variables	Self-identified diagnostic concern (n = 51)	No diagnostic concern (n = 367)	P value
Age at index visit (years), mean (±SD)	40.4 (±13.7)	49.8 (±16.0)	<.001*
Gender			.32
Male	9 (17.6)	88 (24.0)
Female	42 (82.4)	279 (76.0)
Race			.24
White	47 (92.2)	354 (96.5)
Black or African American	2 (3.9)	10 (2.7)
Asian	1 (1.9)	2 (0.5)
American Indian or Alaska Native	1 (1.9)	1 (0.3)
Ethnicity			.75
Hispanic or Latino	1 (2.0)	10 (2.7)
Not Hispanic or Latino	50 (98.0)	357 (97.3)
Insurance			.30
Private	47 (92.2)	301 (82.0)
Public	4 (7.8)	55 (15.0)
Federal	0 (0.0)	8 (2.2)
None	0 (0.0)	3 (0.8)
Health literacy			.31
Never needs help with reading	39 (76.5)	301 (82.0)
Rarely needs help with reading	6 (11.8)	47 (12.8)
Sometimes needs help with reading	4 (7.8)	14 (3.8)
Often needs help with reading	2 (3.9)	5 (1.4)
Always needs help with reading	0 (0.0)	0 (0.0)

Significant at 0.05.

Safer Dx Patient Instrument responses

There were significant differences in responses to what dimensions of the diagnostic process were involved across patients with self-identified diagnostic concerns. Patients who identified diagnostic concerns reported more breakdowns in diagnostic processes: symptoms were not captured adequately [24.5% (n = 12) vs 2.4% (n = 9) for patients reporting correct diagnosis, P < .001], physical exam was not relevant to symptoms [11.8% (n = 6) vs 2.2% (n = 8), P < .001], the notes did not capture the physical exam accurately [19.6% (n = 10) vs 3.5% (n = 13), P < .001], the notes did not reflect what to do if symptoms did not get better [35.3% (n = 18) vs 5.4% (n = 20), P < .00], and disagreement that the care plan addressed all medical concerns [39.2% (n = 20) vs 1.4% (n = 5), P < .001]. Patients identifying a diagnostic concern also indicated lower agreement with items assessing the adequacy of the care plan, trust in the provider, and positive perception of the visit (Table 3). About one-third of patients who identified a diagnostic concern also indicated they received a new diagnosis at the follow-up visit.

Table 3.

Predictors in the Safer Dx Patient Instrument for reported diagnostic accuracy with patient characteristics (n = 418)

Variables	All responses (n = 418)	Self-identified diagnostic concern (n = 51)	No diagnostic concern (n = 367)	P value
Safer Dx Patient Instrument
The notes captured information related to my symptoms accuratelya				<.001*
Agree	348 (83.3)	22 (43.1)	326 (89.3)
Somewhat agree	33 (7.9)	9 (17.7)	24 (6.6)
Neither agree nor disagree	14 (3.3)	8 (15.7)	6 (1.6)
Somewhat disagree	8 (1.9)	5 (9.8)	3 (0.8)
Disagree	13 (3.1)	7 (13.7)	6 (1.6)
The provider conducted a physical exam relevant to my symptoms during my visita				<.001*
Agree	367 (87.8)	31 (62.0)	336 (92.6)
Somewhat agree	27 (6.5)	10 (20.0)	17 (4.7)
Neither agree nor disagree	5 (1.2)	3 (6.0)	2 (0.6)
Somewhat disagree	6 (1.4)	4 (8.0)	2 (0.6)
Disagree	8 (1.9)	2 (4.0)	6 (1.7)
The notes captured my physical exam accuratelya				<.001*
Agree	354 (84.7)	27 (52.9)	327 (89.8)
Somewhat agree	26 (6.2)	8 (15.7)	18 (5.0)
Neither agree nor disagree	12 (2.9)	6 (11.8)	6 (1.7)
Somewhat disagree	8 (1.9)	3 (5.9)	5 (1.4)
Disagree	15 (3.6)	7 (13.7)	8 (2.2)
Did the provider order any tests?a				0.44
Yes	208 (49.8)	28 (54.9)	180 (49.2)
No	209 (50.0)	23 (45.1)	186 (50.8)
Based on notes reviewed, I had concerns about test(s) my doctor ordered (eg, lab, imaging or any procedure)a				0.02
Agree	28 (6.7)	2 (7.1)	26 (14.5)
Somewhat agree	16 (3.8)	5 (17.9)	11 (6.2)
Neither agree nor disagree	9 (2.2)	9 (32.1)	26 (14.5)
Somewhat disagree	5 (1.2)	0 (0.0)	5 (2.8)
Disagree	123 (29.4)	12 (42.9)	111 (62.0)
Based on notes reviewed, I had concerns about my test results				0.02
Agree	38 (9.1)	5 (17.9)	33 (18.9)
Somewhat agree	22 (5.3)	8 (28.6)	14 (8.0)
Neither agree nor disagree	37 (8.9)	5 (17.9)	32 (18.3)
Somewhat disagree	4 (1.0)	1 (3.6)	3 (1.7)
Disagree	102 (24.4)	9 (32.1)	93 (53.1)
The notes reflected what I was supposed to do if symptoms did not get bettera				<.001*
Agree	313 (74.9)	19 (37.3)	294 (80.8)
Somewhat agree	28 (6.7)	5 (9.8)	23 (6.3)
Neither agree nor disagree	36 (8.6)	9 (17.7)	27 (7.4)
Somewhat disagree	14 (3.3)	8 (15.7)	6 (1.7)
Disagree	24 (5.7)	10 (19.6)	14 (3.9)
The care plan the provider developed for me addressed all my medical concernsa				<.001*
Agree	326 (78.0)	10 (19.6)	316 (86.1)
Somewhat agree	48 (11.5)	9 (17.7)	39 (10.6)
Neither agree nor disagree	19 (4.5)	12 (23.5)	7 (1.9)
Somewhat disagree	7 (1.7)	4 (7.8)	3 (0.8)
Disagree	18 (4.3)	16 (31.4)	2 (0.5)
The notes reflected multiple alternative diagnoses that were discussed with me				—
Agree	—	4 (7.8)	0 (0.0)
Somewhat agree	—	6 (11.8)	0 (0.0)
Neither agree nor disagree	—	20 (39.2)	0 (0.0)
Somewhat disagree	—	3 (5.9)	0 (0.0)
Disagree	—	18 (35.3)	0 (0.0)
I received a new diagnosis at my follow-up visit				—
Agree	—	17 (34.0)	0 (0.0)
Somewhat agree	—	8 (16.0)	0 (0.0)
Neither agree nor disagree	—	11 (22.0)	0 (0.0)
Somewhat disagree	—	2 (4.0)	0 (0.0)
Disagree	—	12 (24.0)	0 (0.0)
I trust the provider that I saw during my visit				<.001*
Agree	348 (83.3)	17 (33.3)	331 (90.9)
Somewhat agree	32 (7.7)	6 (11.8)	26 (7.1)
Neither agree nor disagree	19 (4.5)	13 (25.5)	6 (1.7)
Somewhat disagree	7 (1.7)	6 (11.8)	1 (0.3)
Disagree	9 (2.2)	9 (17.7)	0 (0.0)
I did not have a good feeling about my visit				<.001*
Agree	9 (2.2)	9 (17.7)	0 (0.0)
Somewhat agree	7 (1.7)	6 (11.8)	1 (0.3)
Neither agree nor disagree	19 (4.5)	13 (25.5)	6 (1.7)
Somewhat disagree	32 (7.7)	6 (11.8)	26 (7.1)
Disagree	348 (83.3)	17 (33.3)	331 (90.9)

Questions related to the dimensions of the diagnostic process.

Significant at 0.05.

Predictors of patient-reported diagnostic concerns

In univariate analyses, we found the following diagnostic processes to be associated with patient-identified diagnostic concerns: accuracy of patient symptoms documented, accuracy of physical exam according to symptoms, accuracy of the physical exam documented, adequacy of follow-up instructions, and adequacy of care plan to address all concerns. Items related to testing were not statistically significant.

A multivariate logistic regression analysis (Table 4) showed that patients who disagreed with the statement “the care plan the provider developed for me addressed all my medical concerns” were almost 3 times more likely to identify a diagnostic concern (Table 4). Additionally, patients who disagreed with the statement “I trust the provider that I saw during my visit” were 2.1 times as likely to identify a diagnostic concern (95% confidence interval [CI], 1.19–3.71). Patients who agreed with the statement “I did not have a good feeling about the visit” were 1.5 times as likely to identify a diagnostic concern (95% CI, 1.09–2.01). Regarding patient characteristics, patients who are not of Hispanic or Latinx ethnicity were almost 8 times more likely to identify a diagnostic concern compared to patients who are Hispanic or Latinx ethnicity (95% CI, 1.56–039.63).

Table 4.

Predictors in the Safer Dx Patient Instrument for reported diagnostic accuracy

Diagnostic process variable		Univariate logistic regression				Multivariate logistic regression
	n a	Odds ratio	95% CI limits		P value (Pr > |t|)	Adjusted odds ratio	95% CI limits		P value (Pr > |t|)
The notes captured information related to my symptoms accurately	416	2.50	1.84	3.39	<.001*	2.13	0.96	4.73	.06
The provider conducted a physical exam relevant to my symptoms during my visit	413	1.98	1.41	2.77	<.001*	1.05	0.66	1.67	.84
The notes captured my physical exam accurately	415	1.98	1.55	2.54	<.001*	0.39	0.17	0.90	.03
Did the provider order any tests?	417	1.26	0.70	2.27	.44	—	—	—	—
Based on notes reviewed, I had concerns about test(s) my doctor ordered (eg, lab, imaging or any procedure)	207b	0.85	0.68	1.07	.17	—	—	—	—
The notes reflected what I was supposed to do if symptoms did not get better	415	2.00	1.62	2.47	<.001*	1.10	0.63	1.90	.75
The care plan the provider developed for me addressed all my medical concerns	418	4.53	2.97	6.89	<.001*	2.65	1.45	4.87	.002*
I trust the provider that I saw during my visit	415	5.94	3.83	9.21	<.001*	2.10	1.19	3.71	.01*
I did not have a good feeling about my visit	417	2.13	1.76	2.57	<.001*	1.48	1.09	2.01	.01*
Patient characteristics
Age at index visit	418	0.96	0.94	0.98	<.001*	0.98	0.94	1.01	.14
Gender	418
Male (reference group)	97
Female	321	1.47	0.69	3.16	.32	1.75	0.58	5.30	.32
Race	418
White (reference group)	401
Other race	17	2.32	0.72	7.45	.16	0.84	0.28	2.51	.76
Ethnicity	418
Hispanic or Latino (reference group)	11
Non-Hispanic or Latino	407	1.40	0.17	11.28	.75	7.86	1.56	39.63	.01*
Insurance	418
Private	348	2.58	0.89	7.43	.08	0.94	0.18	4.97	.94
Public or Other (reference group)	70
Health literacy	417
Never needs help with reading (reference group)	339
Rarely needs help with reading	53	0.98	0.39	2.47	.002	1.11	0.28	4.44	.005
Sometimes needs help with reading	18	2.20	0.68	7.09	<.001*	2.92	0.75	11.45	<.001
Often needs help with reading	7	3.08	0.57	16.65	<.001*	0.31	0.06	1.77	.45

Sampling weights were used to account for missing responses.

Based on if patient responded “Yes” to having tests ordered in previous question.

Significant at .05.

DISCUSSION

As a first step toward using open notes to engage patients in diagnostic safety surveillance, we tested methods to solicit patients’ feedback about diagnostic concerns on a near real-time basis. At-risk patients were able to review their visit notes, identify breakdowns in the diagnostic process, and self-report any diagnostic concerns. We found that patients who identified diagnostic concerns indicated more concerns about the adequacy of care planning, trust, and reported bad feelings about a visit.

This study builds on the NASEM report recommendations to develop patient-centered methods to measure diagnostic safety. The use of open notes has been associated with improved patient engagement and quality of care, and this work builds on prior OpenNotes efforts to identify patient safety issues in the medical record., When paired with e-triggers to identify patients potentially vulnerable to diagnostic errors, use of the Safer Dx Patient Instrument may enable more proactive patient-centered measurement of patient-reported diagnostic concerns. Notably, this article focused on patient evaluations and perceptions of the diagnostic process, rather than the diagnostic process from a provider perspective. However, patient concerns, satisfaction, and perceptions are influential predictors of subsequent trust in the medical system, as well as follow-up visits and in some cases care outcomes. Systematic evaluation of visit notes by patients has potential to uncover underlying patient-centered contributory factors that affect diagnosis which is otherwise harder to gather. With additional development and testing, this strategy could be used in the future as part of larger organizational initiatives to identify and learn from patient-reported diagnostic concerns and promote organizational learning. For instance, the Safer Dx Patient Instrument could provide near real-time data to healthcare organizations about patients’ experiences of the diagnostic process and intervene to avoid any potential safety issues or to address quality of care issues. Feedback from patients could be useful to both clinicians and health care organizations to improve practice, (ie, patient-clinician communication) and potentially improve diagnostic performance. Such feedback programs are essential for the development of a learning health system to improve patient safety.

Strengths of this approach include providing patients a means to identify where along the diagnostic pathway their concerns may have emerged—symptom accuracy, physical exam relevance and accuracy, follow-up, and care planning. For instance, one dimension of the diagnostic process, care planning (eg, whether the care plan was comprehensive and addressed all medical concerns) was found to be significantly associated with self-identified diagnostic concerns. Discordance between care planning expectations and symptoms/diagnosis is important because if there is no shared understanding about diagnosis with patients, care will not be sensitive to patients’ preferences or patients may be left feeling their diagnostic safety was compromised. A surveillance strategy using the Safer Dx Patient Instrument is sensitive to patients’ experiences by helping uncover negative experiences or patient-perceived care breakdowns that are rarely documented in the medical record and otherwise invisible to the health system because many patients are hesitant to speak up.

We also found that trust and having a bad feeling about a visit were associated with self-identified diagnostic concerns. Patients and their families experiences of misdiagnosis are associated with reduced trust in their current clinicians. Interpersonal trust has long been considered an essential aspect of the patient-physician relationship, and increased trust is associated with better patient outcomes. As Table 3 shows, patient concerns about elements of the diagnostic process, such as whether tests were conducted or whether notes reflected what they described accurately, were less strongly related to perceived diagnostic accuracy than trust in the provider. A shared-decision making model adapted specifically for the diagnostic process may facilitate trust. Trust could be further reinforced through transparency—clinicians could encourage their patients to review their visit notes online and ensure the visit note documentation is comprehensive and accurate and includes relevant discussion around patient values/goals for care. We recommend future research seek to determine predictors of trust in one’s provider since trust appears pivotal for patient perceptions of diagnostic accuracy in our study.

Several limitations warrant discussion. Despite efforts to encourage participating through reminders and an incentive, our response rate was low. The study is limited by small sample size and should be interpreted with caution. Additionally, our sample lacked diversity and included patients who used the portal and were also identified as “at-risk” through one specific algorithm; therefore, the results and frequency of diagnostic concerns may not be generalizable. However, our sample of respondents largely reflected the general demographics of MyGeisinger users. In 2020, close to half of MyGeisinger users are between 18 and 50, are 93% White and non-Hispanic (reflective of the geographic areas of service). While 60% of MyGeisinger users are female, 76.7% of respondents were female. We did not conduct chart reviews on our sample; therefore, we cannot confirm whether the diagnostic concerns identified were safety concerns. We also did not verify whether the index visit and the follow-up visit were related. We cannot account for how patients may have interpreted terms, such as diagnosis, follow-up, and care planning. Further study is required on the relationship between trust and diagnostic concerns. We cannot assess directionality (eg, trust impacts diagnostic concerns or diagnostic concerns impact trust), or if review of the notes impacts trust (ie, evolution of a diagnosis between index visit and follow-up).

Similar to other diagnostic safety studies, hindsight bias remains a possibility and is not limited to not remembering correctly, but rather new information that may shape patients views about their provider retrospectively. It is also possible that some patients did not view their notes as instructed and relied solely on memory. At Geisinger, as of December 2020, only 16% of the more than 2.3 million notes shared by the system through the patient portal were viewed by the patients. However, with the US federal mandate of guaranteed access, patient awareness, and active use of open notes is expected to grow. Future methods must reach a diverse set of patients, and thus we recommend replication and extensions of our findings in stratified samples that include a higher representation of diverse patient populations. Nonresponse bias could not be assessed, and it is possible that nonresponders may have had different experiences of the diagnostic process. Finally, upon further investigation, we observed a probable suppression effect in our model between the item “the notes captured my physical exam accurately” and patients’ trust in their providers. The change of direction in multivariate logistic regression for the item “the notes captured my physical exam accurately” is likely due to relationships between predictors and should not be overinterpreted. We thus recommend future research on open notes to determine when and how perceptions of exam and interaction components relate to patient perceptions of overall outcomes.

CONCLUSION

The movement toward full transparency in 2021 via 21st Century Cures Act is a unique opportunity to leverage patient’s access to their clinician’s notes as a method for patient-centered evaluation of quality and safety of care. The Safer Dx Patient Instrument provides at-risk patients an opportunity to self-report diagnostic concerns based on the evaluation of their visit notes. With additional development, testing, and evaluation with diverse patient populations, the proactive surveillance method we tested can guide patients in assessing the diagnostic process. The methods outlined herein may have the potential to engage patients in safety by encouraging self-reporting of diagnostic concerns and improving transparency of diagnostic processes, thus improving diagnostic safety.

FUNDING

This work was supported by the Gordon and Betty Moore Foundation (GBMF 5498) and partially supported by the Houston Veterans Administration (VA) Health Services Research and Development (HSR&D) Center for Innovations in Quality, Effectiveness and Safety (CIN 13-413). TDG was additionally supported by an Agency for Healthcare Research and Quality Mentored Career Development Award (K01-HS025474). HS was additionally supported by the VA HSR&D Service (IIR 17-127 and the Presidential Early Career Award for Scientists and Engineers USA 14-274), the Agency for Healthcare Research and Quality (R01HS27363), the Gordon and Betty Moore Foundation (GBMF 8838), and the CanTest Research Collaborative funded by a Cancer Research UK Population Research Catalyst award (C8640/A23385). DTC was additionally supported by the Office of Academic Affiliations VA Advanced Fellowship Program in HSR&D.

AUTHOR CONTRIBUTIONS

TDG is the guarantor, initiated the study, supervised the collaborative project, designed the data collection tool, monitored data collection for the entire study, developed the statistical analysis plan, and drafted and revised the paper. DTC designed the data collection tool, developed the statistical analysis plan, cleaned and analyzed data, and drafted and revised the paper. DKU implemented the study, designed the data collection tool, monitored data collection for the entire study, and revised the paper. SK implemented the study, monitored data collection for the entire study, and revised the paper. TMS implemented the study, maintained compliance records, monitored data collection for the entire study, and revised the paper. CSp designed the data collection tool, developed the statistical analysis plan, advised on data analysis, and revised the paper. CSc designed the data collection tool, advised on data analysis, and revised the paper. DT designed the data collection tool, advised on data analysis, and revised the paper. HS designed the data collection tool, developed the statistical analysis plan, advised on data analysis, and revised the paper.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

Click here for additional data file.