Evidenced-Based Claims About Evidence.

Presentations on evidence translation in health care often begin by citing the statistic that it takes 17 years for evidence to diffuse into practice. The source of the 17 year estimate is a paper by Balas and Boren published in 2000 in the Yearbook of Medical Informatics.[1] The authors identified nine mostly primary care services supported by evidence from randomized trials and calculated, based on the trials’ publication dates and current levels of use, that it takes 15.6 years for use to reach 50%. Including the time from submission to publication, 1.4 years, brings the implementation lag up to the familiar 17-year figure. Although the paper was published in a relatively obscure venue, it was mentioned in the Institute of Medicine’s Crossing the Quality Chasm report[2] and has been widely cited in the literature on evidence translation ever since. The paper itself has been cited more than 800 times,[3] but this figure understates its influence because some authors reference the Institute of Medicine report instead. The 17-year figure is cited by policy makers (e.g., Agency for Healthcare Research and Quality director Andrew Bindman,[4] former White House Office of Management and Budget director Peter Orszag[5]) and in the popular press.[6]

“Evidence” is a somewhat amorphous term, but Balas and Boren focus on the adoption of treatments following the publication of “landmark trials.” Some treatments are tested in multiple, high-quality trials that produce conflicting results, but these cases are rare. The evidence for many treatments rests on observational studies or decision analyses, but if trials do not affect practice, it is unlikely physicians will respond to studies that are lower in the hierarchy of evidence. The publication of landmark trials produces the cleanest test of the impact of evidence on practice.

Understanding how evidence from trials affects practice is important, not only for identifying opportunities to improve care but also, more broadly, for assessing the performance of the health system. Balas and Boren’s finding suggests that the system is deeply flawed: physicians routinely ignore evidence that could benefit their patients. However, despite its troubling implications, their work has received little scrutiny.

It turns out there are a number of problems with the 17-year estimate and how it is used. Some of their utilization estimates are low and not well-referenced. For example, they assumed that only 20% of diabetic patients had foot exams in 1998. According to the Centers for Disease Control and Prevention, the actual rate was closer to 60%.[7] They use an estimate of mammography rates from 1997, which, when combined with their assumption that rates increase linearly, implies that the mammography rate reached 50% in 1993, when in fact rates reached that level in 1990[8] or earlier.[9]

Physicians can apply evidence only to patients who show up. Most of the utilization measures are calculated using broad-based denominators (e.g., all adults,[10] adult health plan members with diabetes[11]). They include patients who do not have regular check-ups. Consequently, they partly reflect the behavior of patients and do not isolate the impact of evidence on physicians’ decisions. Also, widescale delivery of some services requires an extensive infrastructure, such as manufacturing capacity and distribution systems for vaccines. Diffusion rates for these services do not tell us much about how physicians respond to evidence generally, yet the 17-year estimate is typically quoted without caveat.

Dissemination and implementation science researchers have latched on to and amplified the 17-year implementation lag estimate to highlight the relevance of their field.[12] Their counterparts in behavioral economics have also embraced the notion that practice changes slowly to illustrate the applicability of behavioral economics to medicine.[13] They claim that physicians display belief persistence and confirmation bias—physicians are slow to let go of long-standing beliefs in response to new evidence and irrationally overweight evidence that is consistent with their preconceptions. However, discussions of this phenomenon are often long on theory and short on real-world examples. The examples that are commonly cited to illustrate how physicians discount new evidence, such as prostate-specific antigen screening and breast cancer screening for women under age 50, are notable mainly because changes in screening recommendations were prompted by a reassessment of the existing scant and conflicting evidence base rather than the release of new evidence from a randomized trial.

Counterexamples are discounted or ignored, though there are cases where evidence has led to rapid changes in practice. The Z0011 trial,[14] which randomly assigned breast cancer patients with one to two positive sentinel lymph nodes to undergo a compete axillary dissection or no additional treatment following lumpectomy, is a case in point. There was a large decline in the proportion of breast cancer patients meeting the Z0011 trial inclusion criteria who had eight or more nodes examined (a proxy for the receipt of an axillary dissection).[15] The share of patients who had eight or more nodes removed decreased from over 50% to below 30% in a matter of months. In fact, treatment started changing shortly after the results of the trial were presented at a medical conference, even before the trial was published.

There are characteristics of the procedure that may have facilitated the translation of evidence into practice. The audience for the evidence was a close-knit community of specialists, physicians who performed fewer axillary dissections did not pay a price in terms of fame or fortune, and the procedure is associated with side effects, and so patients faced tangible harms from overtreatment. However, axillary dissection is no more atypical than the treatments and tests commonly used to illustrate the theory that physicians are slow to respond to evidence. In fact, there was concern that inertia and a pro-intervention bias in oncology would lead physicians to ignore the Z0011 trial results.[16] The case of the Z0011 trial suggests that physicians are not always locked-in to established therapies. Sometimes high-quality evidence is sufficient to change practice.

There are of course cases where high-quality evidence has failed to change how physicians treat patients. For example, use of tight glycemic control in patients admitted to intensive care units did not decline after a trial found that it increased mortality.[17] However, these must be considered alongside cases where evidence affected practice.

Rather than trying to divine a one-size-fits-all rule about the speed at which results are translated into practice, researchers should instead try to build a more robust knowledge base about the factors that influence how physicians respond to evidence through observational analyses and interventional dissemination studies. How does the quality of evidence affect the uptake of findings? Are there characteristics of trials that predict rapid uptake? How does uptake vary based on physicians’ training, compensation scheme, practice environment, and other modifiable factors? Are studies that highlight patient harms more influential than ones that emphasize equivalence of a primary endpoint? The answers will help investigators and funders design better trials and dissemination strategies and policymakers better health systems.

In the meantime, the 17-year implementation lag estimate should be laid to rest. There is still a lot of work to be done to make evidenced-based medicine a reality. Acknowledging that sometimes evidence leads to rapid change does not negate that fact.