How Often do Medical Specialties Question the Practices that They Perform? An Empirical, Cross-Sectional Analysis of the Published Literature.

An aspect of overuse is who decides which practices are evaluated for overuse and which of the studies on overuse are published in the medical literature. We sought to examine the frequency with which studies in medical journals questioned an established practice. As a secondary objective, we sought to determine if there was variance among medical specialties. We conducted a retrospective, cross-sectional review of the published literature in 14 medical specialty journals. We included studies from one issue in three high-impact journals (November/December 2020) for each specialty. We assessed whether the study reported on a medical practice, whether it reported on an existing practice, whether the author expressed uncertainty regarding the practice, whether the study was a randomized design, and if the authors encouraged further testing in randomized studies. For all medical specialties combined, we found that 37% (n = 98) questioned existing practices, and 15% (n = 40) either tested the practice in a randomized trial or encouraged future randomized testing of the practice. The medical specialties that questioned their practices the most were gastroenterology (61%; n = 10/18), obstetrics/gynecology (52%; n = 11/21), and cardiovascular (50%; n = 5/10). These findings indicate that, although research is being conducted to examine current medical practices, few studies advocate for randomized testing of these practices, and even fewer actually test them in a randomized fashion. Additionally, the variation across medical specialties suggests areas in which to look for potential practices that are low-value, duplicative, and/or wasteful.

What do we already know about this topic?

Rising medical costs in the US have led to increasing interest in identifying low-value, wasteful, or duplicative medical practices that provide little benefit to the patient but collectively impart substantial healthcare costs. The decision of which practices are evaluated for overuse can influence which practices are later determined to be low-value.

How does your research contribute to the field?

The medical specialties that evaluated and tested existing practices in their field the most were gastroenterology, obstetrics/gynecology, and cardiovascular. Conversely, clinical neurology, rheumatology, and endocrinology were least likely to test and question existing practices.

What are your research’s implications toward theory, practice, or policy?

Medical disciplines with the least introspection may be areas to examine potential low-value, duplicative, and/or wasteful practices.

Introduction

Rising medical costs in the US have led to increasing interest in identifying low-value, wasteful, or duplicative medical practices that provide little benefit to the patient but collectively impart substantial healthcare costs. One estimate suggests that low-value services among the entire Medicare population could cost as much as $8.5 billion. Our research—and work done by other researchers—have identified numerous low-value care practices across all medical specialties.[2,3]

Identifying practices that have already been flagged as wasteful, duplicative, or low-value is one aspect of reducing healthcare costs. Still, another element of overuse is who decides which practices are evaluated for overuse. The Choosing Wisely Campaign has collected lists of low-value practices by medical specialty, but it relies on professional organizations to report which practices should be listed as low-value. As a result, several concerns were raised about the decisions over which services are included as part of the Choosing Wisely Campaign, including the reporting organizations being more apt to report low-value care from other medical disciplines than from their own field. Further, there is wide variability in the impact on healthcare costs in which services are reported (eg, perioperative fasting vs surgery as initial treatment for cancer when effective neoadjuvant is available), as physicians and professional organizations, not payers of medical care, determine services deemed low-value.

Most, if not all, low-value, duplicative, or wasteful services are identified through published studies on the topic, but this depends on which services are studied, how they are reviewed, and which studies are accepted for publication in medical specialty journals. The question then can be asked, do certain medical disciplines question their own practices more than others (ie, do they evaluate existing practices, and do they seek this evaluation in the form of high-quality study design)? As such, we sought to look across medical specialties to see if there are disciplines where this introspection is more or less common in the published medical literature.

Methods

Article Selection

From a list of medical disciplines, we selected the top three medical journals that publish original research, based on impact factor, for each specialty. We used the Scimago Journal and Country Rank (https://www.scimagojr.com/journalrank.php?area=2700), by subject category under “medicine” for determining the journal’s impact factor. We selected the current issue (as of November 24, 2020), which was usually a December 2020 issue but sometimes a November 2020 issue, and made a list of all research articles, original research, research letters, brief reports, commentaries, editorials, and reviews included in that issue of the journal. We excluded letters to the editors, which are almost always critical of a previously published article and do not provide any new findings.

Data Abstraction

For each article that reported new findings or that conducted a formal analysis of findings in the literature (eg, original research, research letters, brief reports, and reviews), we abstracted these data: the overall results of the study (pos = met its endpoint; neg = did not meet its endpoint; or equivocal = both positive and negative results), funding source, study design, whether the article focused on a specific medical practice, and if so, whether the practice was novel or established, whether the practice was particular to that specialty, whether the authors of the study were critical of the practice, whether the authors expressed uncertainty or conflicting expert opinion about the practice if it was established, and whether the study was randomized or if the authors of the article promoted further research of the practice in higher-quality studies (eg, randomized controlled trials). We defined established medical practices as those for which we could find evidence of their use outside of clinical trials. For the majority of the studies, we were able to locate this information in the article’s introduction, discussion, or accompanying editorial. For other practices, we searched Google Scholar to see if observational/non-interventional studies had documented the practice’s use.

For each article that only summarized findings in the literature and did not report new results (eg, commentaries and editorials), we abstracted these data: whether the article was about a specific medical practice, whether the author of the commentary/editorial supports the practice, whether the author of the article promoted further research and evaluation of the practice, and whether the article was related to an original research article or review in the same issue of the journal.

We created a variable indicating whether the study exhibited introspection of the respective practice. Studies/articles we considered were those that concerned an established or existing medical practice specific to their field and were evaluated because of uncertainty regarding the practice. A flowchart of how introspection was determined is presented in Figure 1.

Figure 1.

Flowchart of the questions to determine introspection in published original research articles, research letters, brief reports, and reviews in high-impact medical specialty journals (November/December 2020).

Statistical Analysis

We reported results in frequencies and percentages. Our primary focus was to evaluate these frequencies across all medical specialties, but as a pilot or secondary objective, we examined these by medical specialty. All analyses were performed in R statistical software, version 3.6.1. Because we used publicly available data, which did not include personally identifiable information, an institutional review board’s approval was not required.

Results

Six-hundred eighty-five studies reported results in an original research article, research letter, brief report, or review. Of those, 268 studies reported on a specific medical practice. The five specialties with the greatest number of studies reporting on a medical practice were the following: urology (n = 39), radiation (n = 37), neurosurgery (n = 33), oncology (n = 29), and obstetrics/gynecology (n = 21). Of the studies that reported on a medical practice, 144 reported on an existing practice, while 124 reported on a novel practice. For all articles that were an original research article, research letter, brief report, or review, the most common practice types were (Table 1) drug (n = 86), procedure (n = 78), radiation (n = 22), algorithm (n = 32), and device (n = 20). The most common funding type was public/non-profit (n = 95).

Table 1.

Questioning of existing practices in original articles, brief reports, research letters, and reviews reporting on a medical practice published in high-impact journals (November/December 2020).

	No questioning of existing practices N = 170	Questioning of existing practices a N = 98
Practice type, n (%)
Algorithm, n = 32	28 (17)	4 (4)
Device, n = 20	14 (8)	6 (6)
Drug, n = 86	63 (37)	23 (23)
Exercise, diet, lifestyle, n = 11	11 (6)	0 (0)
Procedure, n = 78	28 (17)	50 (51)
Radiation, n = 22	12 (7)	10 (10)
Screening, n = 11	6 (4)	5 (5)
Testing, n = 6	6 (4)	0 (0)
Funding type, n (%)
Public/non-profit, n = 95	58 (45)	37 (38)
Industry, n = 30	27 (21)	3 (3)
Combination of public/industry, n=10	10 (8)	0 (0)
Not indicated, n = 76	31 (24)	45 (46)
None, n = 15	3 (2)	12 (12)
Study design
Animal/cadaver, n = 4	3 (2)	1 (1)
Case report/series/control, n = 12	10 (6)	2 (2)
Observational cohort, n = 134	78 (47)	56 (57)
Randomized controlled trial, n = 48	37 (22)	11 (11)
Review/meta-analysis, n = 27	14 (8)	13 (13)
Secondary analysis of randomized study, n = 11	6 (4)	5 (5)
Single-arm trial, n = 14	9 (5)	5 (5)
Other, n = 15	10 (6)	5 (5)
Article results
Positive, n = 142	94 (55)	48 (49)
Negative/null, n = 48	22 (13)	26 (27)
Equivocal, n = 78	54 (32)	24 (24)

aQuestioning of existing practices was determined as studies/articles examining an established or existing medical practice specific to their field, and where the authors expressed uncertainty regarding the practice.

For all medical specialties combined, we found that 37% (n = 98) questioned existing practices. The medical specialties that most questioned practices they performed were gastroenterology (67%; n = 12/18), obstetrics/gynecology (52%; n = 11/21), cardiology (50%; n = 5/10), pulmonary and critical care (50%; n = 3/6), and neurosurgery (45%; n = 15/33). Figure 2 shows the percentage of studies reporting on a medical practice and the percentage of studies raising questions or doubts about practices they perform.

Figure 2.

Variation in the percentage of original studies that questioned specialty-specific practices (Nov/Dec 2020).

For all medical specialties combined, we found that 15% (n = 40) questioned existing practices and either evaluated the practice in a randomized trial or encouraged the practice to be tested in a randomized trial. The medical specialties with the highest percentage of studies that questioned a specialty-specific medical practice and either evaluated the practice in a randomized trial or encouraged the practice to be tested in a randomized trial were (Table 2) cardiology (40%; n = 4/10), pulmonary and critical care (33%; n = 2/6), obstetrics and gynecology (29%; n = 6/21), oncology (21%; n = 6/29), and gastroenterology (22%; n = 4/18).

Table 2.

Questioning of existing practices in original articles, brief reports, research letters, and reviews reporting on a medical practice published in high-impact journals (November/December 2020) by medical specialty.

Medical specialty (# studies covering a medical practice/# studies published in 3 high-impact journals) Median impact factor	Questioning of existing practices a n (%)	If the study questions a specialty-specific practice, does it encourage testing in a randomized study? n (%)	Combination of questioning of existing practices and encouraging randomized studies n (%)
All specialties combined (268/646)	Yes: 98 (37)No: 170 (63)	RCT: 23 (9)Yes: 17 (6)No: 58 (22)	Yes: 40 (15)No: 228 (85)
Cardiology (10/19)22.67	Yes: 5 (50)No: 5 (50)	RCT: 3 (30)Yes: 1 (10)No: 1 (10)	Yes: 4 (40)No: 6 (60)
Endocrinology (18/54)16.02	Yes: 3 (17)No: 15 (83)	RCT: 1 (6)Yes: 0No: 2 (11)	Yes: 1 (6)No: 17 (94)
Gastroenterology (18/36)19.82	Yes: 12 (67)No: 6 (33)	RCT: 3 (17)Yes: 1 (6)No: 8 (44)	Yes: 4 (22)No: 14 (78)
Infectious disease (16/61)14.81	Yes: 5 (31)No: 11 (69)	RCT: 1 (6)Yes: 1 (6)No: 3 (19)	Yes: 2 (13)No: 14 (87)
Nephrology (9/25)8.40	Yes: 3 (33)No: 6 (67)	RCT: 1 (11)Yes: 0No: 2 (22)	Yes: 1 (11)No: 8 (89)
Neurology, clinical (6/20)17.13	Yes: 1 (17)No: 5 (83)	RCT: 0Yes: 0No: 1 (17)	Yes: 0No: 6 (100)
Neurosurgery (33/77)4.61	Yes: 15 (45)No: 18 (55)	RCT: 0Yes: 4 (12)No: 11 (33)	Yes: 4 (12)No: 29 (88)
Obstetrics/gynecology (21/67)5.73	Yes: 11 (52)No: 10 (48)	RCT: 5 (24)Yes: 1 (5)No: 5 (24)	Yes: 6 (29)No: 15 (71)
Oncology (29/31)32.96	Yes: 9 (31)No: 20 (69)	RCT: 4 (14)Yes: 2 (7)No: 3 (10)	Yes: 6 (21)No: 23 (79)
Orthopedic (15/45)12.68	Yes: 6 (40)No: 9 (60)	RCT: 1 (7)Yes: 1 (7)No: 4 (27)	Yes: 2 (13)No: 13 (87)
Pulmonary and critical care (6/25)17.45	Yes: 3 (50)No: 3 (50)	RCT: 0Yes: 2 (33)No: 1 (17)	Yes: 2 (33)No: 4 (67)
Radiation (37/71)5.86	Yes: 17 (46)No: 20 (54)	RCT: 2 (5)Yes: 3 (8)No: 12 (32)	Yes: 5 (13)No: 32 (87)
Rheumatology (11/54)9.00	Yes: 1 (9)No: 10 (91)	RCT: 1 (9)Yes: 0No: 0	Yes: 1 (9)No: 10 (91)
Urology (39/61)5.16	Yes: 7 (22)No: 32 (78)	RCT: 1 (3)Yes: 1 (3)No: 5 (13)	Yes: 2 (5)No: 37 (95)

aQuestioning of existing practices was determined as studies/articles examining an established or existing medical practice specific to their field, and where the authors expressed uncertainty regarding the practice.

There were 133 commentaries/editorials, of which 65 discussed a specific medical practice. The five specialties with the highest number of commentaries/editorials reporting on a medical practice were (Table 3) oncology (n = 12), infectious disease (n = 9), neurosurgery (n = 9), radiation (n = 9), and pulmonary and critical care (n = 8). Obstetrics and gynecology and orthopedics did not have any editorial commentaries reflecting on their discipline’s medical practice. The specialties with the highest number of commentaries/editorials reporting on a medical practice critically or skeptically were gastroenterology (67%; n = 2/3), clinical neurology (50%; n = 2/4), pulmonary and critical care (50%; n = 4/8), cardiovascular (40%; n = 2/5), and nephrology (40%; n = 2/5). Rheumatology had only one study with a commentary/editorial about a medical practice, and it questioned a specialty-specific practice.

Table 3.

Questioning of existing practices in commentaries and editorials reporting on a medical practice published in high-impact journals (November/December 2020), by medical specialty and study results linked to commentary/editorial.

Medical specialty (# editorials covering an established medical practice/# editorials/commentaries covering a medical practice/# editorials/commentaries published in 3 high-impact journals)	Introspection a for established practices, by results of study linked to commentary/editorial n (% of established practices)	No introspection for established practices, by results of study linked to commentary/editorial n (% of established practices)
Cardiovascular (2/5/9)	1	1
Endocrinology (2/4/7)	0	0
Gastroenterology (3/3/6)	2	1
Infectious disease (3/9/18)	1	2
Nephrology (2/5/16)	0	2
Neurology, clinical (1/4/8)	1	0
Neurosurgery (6/9/19)	2	4
Obstetrics/gynecology (0/0/2)	0	0
Oncology (6/12/15)	2	4
Orthopedic (0/0/4)	0	0
Pulmonary and critical care (4/8/16)	2	2
Radiation (3/9/18)	1	2
Rheumatology (0/1/2)	0	0
Urology (5/15/24)	1	2

aQuestioning of existing practices was determined as studies/articles examining an established or existing specific medical practice that was specific to their field, and where the authors expressed uncertainty regarding the practice.

Discussion

We found that only a minority of published studies in high-impact medical journals (37%) questioned a medical practice that was established at the time of the study. Further, only 15% of studies concerning a medical practice (40% of studies questioning a medical practice) either tested the practice in a randomized study or encouraged trialists to test the practice in a randomized study. Our results, which are slightly higher than previous estimates, apply to discipline-specific journals rather than general medical journals. Similar to these prior studies, we found that only a small percentage of studies that questioned existing practices were funded by industry (3%), even though as many as half of all clinical trials registered on clinicaltrials.gov are sponsored by industry. We also found that almost half of studies questioning an existing practice examined a procedure, rather than a medication or another type of practice. Industry has a vested interest in supporting research involving novel medical practices, particularly those involving drugs, but there is little incentive for them to fund studies on practices that have already been implemented.

We found notable variation in how often a specialty was willing to question practices they perform. Gastroenterology, obstetrics/gynecology, cardiovascular, and pulmonary and critical care were specialties with a high percentage of articles that not only questioned a specific medical practice but also either tested these practices in a randomized study or encouraged further testing in a randomized study. Conversely, clinical neurology, rheumatology, nephrology, urology, and endocrinology had a lower percentage of introspection and encouragement of randomized studies. We previously reported that for these specialties, there was a low percentage of practices that were considered medical reversals, which may be due to reduced desire to appraise longstanding practices in these specialties.

For specialties that tend to not question the practices they perform, there is a good chance that there are undiscovered examples of overused medical practices. By identifying these areas and specialties where there is less examination of established medical practices, one might find ways to reduce low-value care. For researchers interested in low-value care, these specialties may provide a good opportunity, as low hanging fruit may yet be unaddressed.

We found that studies questioning the effectiveness of utilized practices tended to reflect procedural practices, including radiation treatment and screening, and less of practices regarding lifestyle practices, drugs, and devices. There may be various reasons the latter studies were less likely to question specialty-specific practices. Drugs and devices tended to be novel and funded by industry. In contrast, studies on lifestyle practices, including diet and exercise, often took positive results of observational studies as reason to implement them into practice before randomized studies were done to confirm efficacy. Perhaps these practices are so readily accepted because people feel little to no harm or downsides in implementing them. The situation of randomized studies refuting observational evidence in diet and exercise has occurred multiple times in recent years, and in some cases, they were shown to be harmful in randomized studies.[8,9]

We also found studies that questioned specialty-specific practices tended to report negative or null results compared to studies that did not. This may signal publication bias, where positive studies are more likely to be published unless there is uncertainty regarding the practice.[10,11] Preferential publication may indirectly promote low-value care through fewer negative or null studies on these same practices being published. The current status of publication bias is unknown, in part because meta-analyses often do not evaluate this type of bias.

Limitations

There are two key limitations to our work. First, we utilized only one month’s worth of publications, which limits our capacity to conduct cross-specialty comparisons. To lessen bias, we chose which month to analyze at random. Also, because of the COVID-19 pandemic, an unusually high number of studies were reported on SARS-CoV-2. SARS-CoV-2 studies were published broadly across medical specialties. These studies often did not report on medical practices, and if they did, the practices were novel. Because of this, the results of our analyses are not generalizable to the literature at-large. Second, our analysis relied on the subjective judgment as to whether or not the study questioned existing specialty-specific practices. We attempted to use clear determination rules, and each article was reviewed by two independent reviewers (AH and KP), with a third (VP) reviewing any discrepancies. As such, others may code these articles differently. We encourage these efforts.

Conclusion

In summary, our results suggest that although research is being conducted to examine current medical practices, few studies encourage randomized testing of these practices, and even fewer actually test them in a randomized fashion. Further, the variation across medical specialties suggests areas in which to examine potential practices that are low-value, duplicative, and/or wasteful.