A retrospective look at the predictions and recommendations from the 2009 AMIA policy meeting: did we see EHR-related clinician burnout coming?

Clinicians often attribute much of their burnout experience to use of the electronic health record, the adoption of which was greatly accelerated by the Health Information Technology for Economic and Clinical Health Act of 2009. That same year, AMIA's Policy Meeting focused on possible unintended consequences associated with rapid implementation of electronic health records, generating 17 potential consequences and 15 recommendations to address them. At the 2020 annual meeting of the American College of Medical Informatics (ACMI), ACMI fellows participated in a modified Delphi process to assess the accuracy of the 2009 predictions and the response to the recommendations. Among the findings, the fellows concluded that the degree of clinician burnout and its contributing factors, such as increased documentation requirements, were significantly underestimated. Conversely, problems related to identify theft and fraud were overestimated. Only 3 of the 15 recommendations were adjudged more than half-addressed.

INTRODUCTION

On February 17, 2009, President Barack Obama signed the American Recovery and Reinvestment Act, which included the Health Information Technology for Economic and Clinical Health (HITECH) Act. The federal government invested $36 billion in incentives over the next decade to catalyze a massive increase in EHR adoption. Hospitals and medical offices spent many billions more on software and implementation.

Since then, clinician burnout in the US has increased significantly with estimates among physicians often exceeding 50%. Some have even called this an “epidemic.” EHRs are frequently listed as a significant contributor to burnout. Put differently, the current burnout crisis may be, at least in part, an unintended consequence of ubiquitous EHR implementation. AMIA’s EHR-2020 Task Force report described the interrelationship of CMS billing rules and physicians’ documentation burden. Ironically, the misapplication of computer-based documentation tools simultaneously increased the burden and reduced the usefulness of clinical notes—clearly, an adverse unintended consequence of EHR adoption. Hartzband and Groopman recently noted, “The EHR, initially lauded for its potential as a repository of patient information, has become a tyrannical, time-consuming billing tool.”

Following the enactment of the HITECH Act, AMIA focused its 2009 Annual Health Policy Meeting (hereafter called simply the Policy Meeting) on the discussion of unanticipated consequences that could occur with the large-scale, national implementation of HIT resulting from the HITECH Act. (Note: One of the authors, JS, was a cochair of the Policy Meeting.) A critical factor discussed during the Policy Meeting was whether the unintended consequences of EHR implementation could be anticipated or predicted and their effects ameliorated. The meeting participants developed 17 predictions and 15 recommendations. The Policy Meeting included many fellows of American College of Medical Informatics (ACMI). ACMI “…is a college of elected fellows who have made significant and sustained contributions to the field of biomedical informatics.” It is the central body for a community of senior scholars and practitioners who are committed to advancing the informatics field.

Each winter, ACMI hosts a symposium to discuss significant trends in the field. In 2020, the symposium focused on the role of EHRs in clinician burnout. One session focused on the fellows’ current opinions about predictions and recommendations from the Policy Meeting. In this article, we present those collated opinions about: whether the current unintended consequences of widespread EHR implementation were predicted at the Policy Meeting; the degree to which these unintended consequences have contributed to clinician burnout; whether recommendations from the Policy Meeting were implemented; and reflections on other current trends that were not foreseen at the Policy Meeting.

MATERIALS AND METHODS

Demographics and setting

Forty ACMI fellows attended the 2020 symposium, of which roughly 2 dozen participated in a retrospective session. Attendance at the session was not taken and voting was optional, with responses ranging from 18–23 votes per question. However, those who self-identified on the polling app included chairs and senior faculty of informatics departments, directors of informatics institutes, Chief Research Informatics Officers, practicing clinicians, experts in human computer interaction, and corporate executives.

Session structure

The session at the meeting was designed as a collaborative activity intended to provide thought and discussion among the fellows. It utilized a live, modified Delphi process supported by a live polling app (polleverywhere.com) and moderated by 1 of us (JS). The session consisted of 4 phases. The first phase was evaluation of the accuracy of predictions from the Policy Meeting. Each prediction was presented to the fellows at the session and participants rated the accuracy of the prediction on a 7-value Likert Scale. Responses were voluntary. Lower values indicate that the current magnitude of the problem is less than was expected by the informatics community in 2009, the middle score indicates that the magnitude of the problem was in line with expectations, and higher values indicate that the problem turned out to be worse than expected. Unlike a traditional Delphi, there were no rounds. Rather, participants could see the anonymized aggregate votes as they occurred, could discuss the item, and could change their votes. When the votes had stabilized (typically after 1 or 2 minutes), the process moved to the next item. This live method has been shown to produce comparable results to a traditional Delphi process.

The second phase involved several individual questions. The first was a classification of whether the causal relation between EHRs and burnout was direct, indirect, or a mixture of direct and indirect effects. The second question asked whether the contribution of EHRs to the current burnout crisis could have been anticipated in 2009. Scoring for that question used the same response scale as the Policy Meeting: 1) “Totally obvious to everyone that this would happen”; 2) “The experts saw this coming, others did not”; 3) “Had we worked harder, we could have anticipated this”; and, 4) “Total surprise. This was not predictable.” The third question asked the extent to which the informatics community accurately predicted the burnout crisis, using the same 7-value Likert scale as the first phase. The last question addressed the percentage of burnout attributable to EHRs.

The third phase evaluated whether recommendations from the Policy Meeting had been achieved. This used a 5-value Likert Scale, ranging from no action to complete response and resolution of the problem. In the final phase, participants were asked to identify other HIT outcomes over the past decade that were not anticipated by the Policy Meeting. This phase utilized a real-time submission system where participants could up- or down-vote previously submitted items.

This work reflects the collected reflections of domain experts as a public meeting session, as opposed to a prospective research study. Following the meeting, the aggregated results of the session were shared with symposium attendees. All symposium attendees were given the opportunity to join as authors in drafting this manuscript. For these reasons, we present this as a collaborative opinion piece.

RESULTS

Responses to the first phase are summarized in Table 1. The prediction for which the impact was rated most underestimated in 2009 compared to the 2020 reality was, “Behaviors like cut/paste will result in decreased data quality.” The prediction that was most overestimated in 2009 was, “False positives from abuse and fraud detection algorithms will harm clinicians and/or patients.” While none of the predictions from 2009 directly addressed “burnout,” several predictions addressed components, including increased documentation, increased cognitive load, data overload, and physicians taking early retirement. For all of these, participants felt that the problems today are worse than was anticipated in 2009.

Table 1.

Retrospective assessment of predictions from the 2009 AMIA Policy Meeting

Prediction	Number of Responses	Chicken Little. Minimal Impact1	Moderately Less than We Expected2	Slightly Less than We Expected3	Got it about Right about the Magnitude4	Slightly More than We Expected5	Moderately More than We Expected6	Huge Problem. Much Worse than We Expected7	Right Bars Mean Problem Severity Was Underestimated. Left means Overestimated.	Median	Interquartile Range
Behaviors like cut/paste will result in decreased data quality	21	0.0%	0.0%	0.0%	4.8%	14.3%	14.3%	66.7%		7	1
Limitations of current EHRs will cause staff to create a large number of work-arounds	20	0.0%	0.0%	0.0%	25.0%	5.0%	25.0%	45.0%		6	2.25
There will be an increase in documentation and reporting requirements	21	0.0%	0.0%	0.0%	4.8%	9.5%	42.9%	42.9%		6	1
Usability Issues with EHRs will increase cognitive load for clinicians	20	0.0%	0.0%	0.0%	20.0%	0.0%	40.0%	40.0%		6	1
Contract restrictions and other fears will result in organizations not sharing critical information and lessons learned	21	4.8%	0.0%	9.5%	0.0%	28.6%	42.9%	14.3%		6	1
The growth of EHRs will lead to data overload by clinicians	22	4.5%	0.0%	0.0%	27.3%	18.2%	50.0%	0.0%		5.5	2
EHR adoption will lead to increased use of “physician extenders”	22	0.0%	0.0%	0.0%	18.2%	36.4%	31.8%	13.6%		5	1
Alert fatigue will lead to patient harm	21	0.0%	0.0%	9.5%	38.1%	14.3%	38.1%	0.0%		5	2
Problems with system interconnections will lead to patient harm	21	0.0%	19.0%	9.5%	23.8%	33.3%	14.3%	0.0%		5	2
Increased regulation will create barriers to entry and reduce innovation	21	0.0%	14.3%	14.3%	38.1%	4.8%	19.0%	9.5%		4	3
Push to adopt EHRs will lead to early retirements of clinicians	21	0.0%	4.8%	9.5%	38.1%	4.8%	42.9%	0.0%		4	2
Widespread availability of EHRs will increase fraud and abuse	22	4.5%	27.3%	3.5	3
EHR implementation failures will occur at many organizations and lead to staff exhaustion	22	4.5%	36.4%	13.6%	22.7%	13.6%	9.1%	0.0%		3	2
A punitive or regulatory approach to addressing system flaws will stifle this open exchange and will, ultimately, lead to self-protective behavior and inferior systems	20	30.0%	15.0%	25.0%	20.0%	5.0%	5.0%	0.0%		3	3
Patient and provider identity theft will be a significant problem	22	31.8%	36.4%	13.6%	9.1%	9.1%	0.0%	0.0%		2	1
Alert dependence will lead to patient harm	21	38.1%	33.3%	19.0%	4.8%	4.8%	0.0%	0.0%		2	2
False positives from abuse and fraud detection algorithms will harm clinicians and/or patients	18	50.0%	33.3%	5.6%	11.1%	0.0%	0.0%	0.0%		2	1

Note: Predictions are sorted from most underpredicted to most overpredicted, median ranking.

All participants rated the causal relationship between EHRs and burnout as a mixture of direct and indirect effects. Participants were also asked to assess: whether the current burnout crises could have been anticipated (Figure 1A); how well the informatics community did, in fact, anticipate it (Figure 1B); and, the contribution of EHRs to burnout (Figure 1C). Participants concluded that the burnout crisis was moderately anticipatable, and most assessed the contribution of EHRs between 11% and 60%.

Figure 1.

Responses to questions regarding whether the current burnout crisis was a anticipatable (A), whether is was accurately predicted by the informatics community (B), and the contribution by EHRs (C).

The assessment of the Policy Meeting recommendations is summarized in Table 2. Overall, most recommendations were assessed as having resulted in “No significant action” or “Some small amount of work” over the past 11 years. However, 3 recommendations were assessed as half or more complete. Two of these were related to regulation: “Reconcile multiple EMR [Electronic Medical Record] certifications to eliminate conflicts”; and “Avoid a rush to FDA regulation of HIT as a medical device.” A recommendation for AMIA was also rated as more than half accomplished, “Responses to government about legislation or rules must come across as helpful, educational, and oriented toward the public good (not as lobbying for our constituency).” Overall, recommendations regarding a research agenda were assessed as resulting in little, if any, action.

Table 2.

Assessment of response to recommendations

General Recommendation:	Number of Responses	No Significatn Action on this Recommendation1	Some Small Amount of Work2	About Halfway There3	Largely Responded to4	We Are Done. Check It Off.5		Median	Interquartile Range
Develop a framework for sharing of experiences and near misses (eg, Aviation Safety Reporting System)	19	52.6%	42.1%	5.3%	0.0%	0.0%		1	1
Research Recommendations
Determine and disseminate best practices for HIT design	21	19.0%	76.2%	0.0%	0.0%	4.8%		2	0
Determine and disseminate optimal organizational strategies for HIT system implementation	20	10.0%	80.0%	10.0%	0.0%	0.0%		2	0
Conduct research to improve the ability to identify, anticipate, and avoid/mitigate unintended consequences	21	9.5%	66.7%	23.8%	0.0%	0.0%		2	0
Conduct additional cognitive research on the relationship of HIT system design to unintended consequences	21	14.3%	61.9%	19.0%	4.8%	0.0%		2	0
Create a taxonomy related to unintended consequences of HIT implementations	20	5.0%	55.0%	35.0%	5.0%	0.0%		2	1
Government Recommendations:
Fund research aimed at understanding the benefits and risks of government's HIT policies	20	55.0%	45.0%	0.0%	0.0%	0.0%		1	1
Support CER studies of HIT systems and implementations	19	47.4%	47.4%	5.3%	0.0%	0.0%		2	1
Federal leadership to create incentives so that organizations will be more willing and able to share information	20	45.0%	40.0%	15.0%	0.0%	0.0%		2	1
Acknowledge the limitations of HIT. Avoid belief that technology will somehow “fix” healthcare systems ills	21	23.8%	52.4%	23.8%	0.0%	0.0%		2	0
Regulatory Recommendations:
EHR Implementations accredited by a standards-organization like JCAHO	20	65.0%	25.0%	10.0%	0.0%	0.0%		1	1
Reconcile multiple EMR certifications to eliminate conflicts	21	4.8%	14.3%	52.4%	28.6%	0.0%		3	1
Avoid a rush to FDA regulation of HIT as a medical device	21	4.8%	19.0%	14.3%	52.4%	9.5%		4	1
AMIA Recommendations:
More interaction with, and education of, attorneys who often overinterpret rules	19	63.2%	36.8%	0.0%	0.0%	0.0%		1	1
Responses to government about legislation or rules must come across as helpful, educational, and oriented toward the public good (not as lobbying for our constituency)	19	0.0%	26.3%	31.6%	36.8%	5.3%		3	1.5

Note: The wording of recommendations is taken from the Policy Meeting report.

Abbreviations: CER, comparative effectiveness research; EMR, Electronic Medical Record; FDA, Food and Drug Administration; HIT, health information technology; JCAHO, depreciated abbreviation for the Joint Commission.

When asked to identify other HIT outcomes over the past decade that were not anticipated and that had not already been discussed, participants submitted 22 items. Two of these ranked significantly higher than the others: “That after 10 years, EHRs have not significantly improved their UX [user experience]” and “Increasing monopoly of EHR vendors.”

DISCUSSION

The collective opinion of the ACMI fellows participating in this session was that, while many consequences of the HITECH act were foreseen in 2009, the magnitude of the current burnout crisis largely was not. On a brighter note, the problems of rampant identity theft or “false positives from abuse and fraud detection algorithms” have not been as severe as was feared, perhaps owing to advancements in EHR security and regulations.

While the number of biomedical informatics experts included in this exercise was small, all of the participants are ACMI fellows who have at least 1 decade, and average more than 2 decades, of experience in the field. Thus, all of the participants were active in the informatics community when the HITECH Act was passed, and many were already senior leaders. Therefore, they were well-positioned to judge the “pulse” of the community at that time and now. However, this exercise was limited in that participants were only asked to assess the fraction of the burnout crisis attributable to EHRs. It was not intended to separate out the impact of the intrinsic EHR software from the ways that EHRs have been implemented or to assess which EHR components or functions most contribute to burnout. Developing and ranking all of the other contributors to clinician burnout was beyond the scope and available time at the ACMI conference.

While the emergence and magnitude of the current burnout crisis among clinicians was underestimated, concerns with the burden of documentation and suboptimal EHR user interfaces were widespread at the time, and informatics research was attempting to address the burden well before clinician burnout became a major issue. For example, increased documentation burden was identified at the Policy Meeting as a likely consequence. Partly as a result, the 2011 AMIA Policy Meeting focused on clinical data capture and documentation. Subsequent research has addressed the documentation burden. Research by the informatics community has quantified the EHR factors contributing to burnout. Other work has focused on reducing the burden created by excessive alerts., The degree of clinician burnout has become so concerning that it was addressed in a National Academy of Medicine report, to which a number of ACMI members (including author DD) contributed. The concerns about EHR user interfaces were also recognized a decade ago, and research has addressed user interface needs. Workshops involving both informaticists and clinicians have been conducted to reduce clinician burden by enhancing EHR efficiency.

Research alone will not be sufficient to address the current issues. For example, the Policy Meeting recommended both research and regulatory tools to encourage sharing of best practices; however, contractual clauses prevented such sharing. Only recently have the concerns about “gag clauses” and other limitations on sharing best practices begun to be addressed through legislation.

Based on decades of research in the area of EHRs, there were reasons for the informatics community to be concerned about potential adverse consequences of rapid adoption of EHRs. Although concerns about EHR usability, data overload, and alert fatigue were identified at the Policy Meeting, the magnitude of the combined burdens placed on clinician users and the resultant burnout was significantly underestimated. Another observation is that, even though scores of informatics research projects have developed potential approaches to mitigate these problems, too few of those have been translated into real-world solutions. In hindsight, we suggest that implementing more of the 2009 recommendations, such as research on the cognitive burden of commercial EHRs and incentives to share best practices, may have been able to mitigate some of the clinician burnout currently being experienced.

CONCLUSION

In 2009, the informatics community was concerned that the rapid and widespread adoption of EHRs would have unintended consequences. In retrospect, some of those fears were overblown. These included concerns about rampant patient and provider identity theft or widespread patient harm from alert dependence. In contrast, some issues turned out to be significantly worse than anticipated, such as, the impact of cut/paste on EHR data quality or the increase in documentation requirements. While informatics experts did accurately predict a number of the issues that now contribute to clinician burnout, we did not accurately foresee the magnitude of the current crisis. Perhaps equally important, the Policy Meeting included a number of recommendations that may have reduced the severity of HIT-related unintended consequences, including physician burnout. Unfortunately, few of these recommendations were enacted.

FUNDING

Portions of this work were supported by the National Library of Medicine grant number R01 LM006910 (Hripcsak) and the National Center for Advancing Translational Sciences grant number UL1TR001422 (Dr. Starren and use of the Poll Everywhere software).

AUTHOR CONTRIBUTIONS

JS led the in-person lookback exercise and conducted the initial data analysis. All symposium attendees were given the opportunity to join as authors in drafting this manuscript. All other listed authors contributed to interpreting the data, identifying relevant references, drafting and revising the manuscript, and approval of the final version. All authors except DD participated in the lookback exercise.