Physician Agreement With Recommendations Contained in a National Guideline for the Management of Incidental Pulmonary Nodules: A Case Study.

Description of the Problem

Since the 1980s, there has been a growing emphasis on the use of evidence-based practice guidelines in medicine.[1] However, national guidelines are based on evidence of variable quality, including non-evidence-based common medical practice. While attempts are made to ensure impartial recommendations during guideline development, if there is lack of sufficient clinical evidence, some recommendations could be the result of inadvertent bias.[2] At times, guidelines from different organizations on the same clinical entity can also contradict each other.[3]

Even if guidelines are based on well-conducted clinical trials or other unbiased high quality evidence, repeated studies have also shown that physician practice is often not concordant with existing guidelines.[4] There is similar lack of radiologists’ adherence to evidence-based imaging guidelines.[5] Several explanations have been proposed for the discrepancy between clinical practice and evidence-based guidelines, including lack of awareness of the guidelines, lengthy and unclear guidelines, decreased willingness to change existing practice patterns, absence of embedded guidelines in clinical workflow, and patient preference.[6]

Lack of physician agreement with portions of clinical guidelines may be an important contributor to why guidelines are often not broadly followed in clinical practice. We therefore conducted a survey of a multispecialty group of expert physicians in our institution to determine their agreement with the 2017 Fleischner Society Guidelines for Management of Incidental Pulmonary Nodules (FSG), the most frequently referenced imaging guidelines for pulmonary nodules.[7]

What We Did

Like most guidelines, the FSG are a collection of recommendations based on multiple different clinical scenarios. The 2017 FSG were comprised of 18 unique recommendations, which for the purpose of this study were each codified into a “Clinical Evidence Logic Statement”, or CELS, defined as a statement regarding the appropriateness of diagnostic imaging for a certain indication. CELS were formatted as “IF… THEN” statements, a previously-described guideline representation format.[8] Representation of recommendations in such a format may simplify the creation of clinical decision support tools to enable and accelerate broad adoption.

For each recommendation, the FSG includes ratings for the strength of evidence supporting the recommendation based on the American Society of Chest Physicians grading system. However, before we assessed radiologist agreement with specific recommendations, two medical librarians from the Harvard Library of Evidence analyzed each CELS independently and graded the recommendation based on the supporting clinical studies using the Oxford Centre for Evidence-based Medicine (OCEBM) Levels of Evidence and the U.S. Preventative Services Task Force I-Scores, an objective and broadly accepted methodology.[8] The purpose of internally grading the strength of evidence behind each recommendation, as opposed to relying solely on the FSG’s own rating, was to assess whether physician agreement with each guideline recommendation varied according to the internally- or FSG-ascertained level of evidence supporting a specific recommendation.

A total of 9 physicians were then surveyed via SurveyMonkey to assess agreement with each of the 18 recommendations, or CELS, contained in the FSG. Physicians were invited to participate in the study based on clinical expertise in pulmonary nodule management and/or experience with FSG recommendations, in order to represent a broad range of specialties that would manage patients with incidental pulmonary nodules. Among the 9 survey participants, there were 3 radiologists with subspecialties in thoracic or cancer imaging, 3 thoracic surgeons, and 3 internal medicine and pulmonology physicians. The physicians were given 18 different clinical vignettes, the associated FSG recommendation, the Harvard Library of Evidence grading of evidence, and the FSGs’ own grading of evidence (Figure 1). Survey participants were asked to agree with the FSG recommendation without modification, agree with modification, or disagree. A free text comment box was also provided.

Figure 1.

Example Survey Vignette and Question

The primary outcome was the percent of physicians who agreed with each FSG recommendation and the corresponding CELS. Physicians who chose to modify the FSG recommendation were considered to have disagreed with the recommendation for our analysis. We compared the proportion of physicians who agreed and disagreed with each of the 18 CELS using a chi-square analysis. We also performed subgroup analysis comparing percent agreement for the different nodule types and risk categories outlined in the FSG.

Outcomes and Limitations

All participants completed the survey. The percent agreement between physicians differed considerably across each of the recommendations, or CELS, ranging from 0% agreement to 100% agreement, with an average agreement rate of 48.2% (Table 1). There was significant variation in the proportion of physicians who agreed with each CELS in the FSG (p<0.01). Rates of physician agreement were not associated with FSG grading of the strength of underlying evidence. For instance, 88.9% of physicians agreed with FSG that a single low risk nodule >8 mm should be managed by a follow-up chest CT at 3 months, PET/CT, or tissue sampling; a recommendation that had an FSG grade of 1A (high quality evidence). In contrast, the rate of physician agreement for follow-up management of multiple subsolid pulmonary nodules with the most suspicious being equal to or larger than 6 mm was also 88.9%, although that recommendation had an FSG grade of 1C (low or very low-quality evidence) (p=0.21). Similar analysis was not performed for the OCEBM grading given that all but one recommendation had the same strength of evidence (5 I, based on expert opinion with insufficient evidence).

Table 1.

Physician Agreement with Individual Recommendations/ Clinical Evidence Logic Statements (CELS) from 2017 Fleischer Society Guidelines for the Management of Incidental Pulmonary Nodules

CELS	Fleischner Guideline	Fleischner Society Recommendation	Oxford Grade	Fleischner Society Grade	Physicians agreeing with statement (%)	P Value
1	Single nodule < 6mm, low risk	No routine CT follow-up.	5-I	1C	22.2	<0.01
2	Single nodule 6 –8 mm, low risk	Chest CT at 6–12 months, then consider CT at 18–24 months	5-I	1C	66.7
3	Single nodule >8 mm, low risk	Consider Chest CT at 3 months or PET/CT or tissue sampling	5-I	1A	88.9
4	Single nodule < 6 mm, high risk	Optional Chest CT at 12 months	5-I	2A	0.00
5	Single nodule 6 – 8 mm, high risk	Chest CT at 6–12 months, then CT at 18–24 months	5-I	1B	33.3
6	Single nodule > 8mm, high risk	Consider Chest CT at 3 months or PET/CT or tissue sampling	5-I	1A	100.0
7	Multiple nodules, most suspicious nodule < 6 mm, low risk	No routine CT follow-up	5-I	2B	11.1
8	Multiple nodules, most suspicious nodule 6 – 8 mm, low risk	Chest CT at 3–6 months, then consider CT at 18–24 months	5-I	1B	66.7
9	Multiple nodules, most suspicious nodule > 8 mm, low risk	Chest CT at 3–6 months, then consider CT at 18–24 months	5-I	2A	66.7
10	Multiple nodules, most suspicious nodule < 6 mm, high risk	Optional Chest CT at 12 months	5-I	Not stated	0.0
11	Multiple nodules, most suspicious nodule 6 – 8 mm, high risk	Chest CT at 3–6 months, then CT at 18–24 months	5-I	Not stated	66.7
12	Multiple nodules, most suspicious nodule > 8 mm, high risk	Chest CT at 3–6 months, then CT at 18–24 months	5-I	Not stated	22.2
13	Single ground glass nodule < 6 mm	No routine CT follow-up	5-NS-contra-dicts	1B	22.2
14	Single ground glass nodule >= 6 mm	Chest CT at 6–12 months to confirm persistence, then CT every 2 years until 5 years	5-I	1B	88.9
15	Single part solid nodule < 6 mm	No routine CT follow-up	5-I	1C	0.0
16	Single part solid nodule >= 6 mm	Chest CT at 3–6 months to confirm persistence. If unchanged and solid component remains <6mm, annual CT should be performed for 5 years	5-I	1B	77.8
17	Multiple subsolid nodules, most suspicious < 6 mm	Chest CT at 3–6 months. If stable, consider CT at 2 and 4 years	5-I	1C	44.4
18	Multiple subsolid nodules, most suspicious >= 6 mm	CT at 3–6 months	5-I	1C	88.9

Subgroup analysis was performed to determine if physicians showed higher rates of agreement with FSG recommendations for specific subgroups of nodules (Table 2). CELS recommendations were grouped according to their corresponding nodule description provided in the FSG. CELS regarding the management of large nodules had the highest physician-FSG agreement at 76.2%, compared to only 30.3% agreement for small and medium nodules (p<0.01). There was no significant difference in rates of physician agreement with FSG recommendations between high- versus low-risk solid nodules, single versus multiple nodules, or solid versus subsolid nodules. While a few specific FSG recommendations had 100% agreement, when FSG recommendations were grouped according to their broader nodule description, there was no unanimous acceptance of the FSG guidelines by our local expert panel.

Table 2.

Physician Agreement by Pulmonary Nodule Subgroup

Subgroup	Clinical Evidence Logic Statements (CELS) categories	Agree (%)	P Value
Patient risk
High Risk Solid Nodules	1–3	59.3	0.14
Low Risk Solid Nodules	4–6	44.4
Number of nodules
Single Nodule	1–6, 13–16	50.0	0.66
Multiple Nodule	7–12, 17–18	45.8
Solid Appearance
Solid Nodules	1–12	45.4	0.40
Subsolid Nodules	13–18	53.7
Nodule Size
Small and Medium Nodules	1, 2, 4, 5, 7, 8, 10, 11, 13, 15, 17	30.3	<0.01
Large Nodules	3, 6, 9, 12, 14, 16, 18	76.2

Our work has several limitations. First, the study surveyed only 9 physicians in a single large academic institution. However, the study was designed to be exploratory and to highlight and test the hypothesis that guideline non-adherence may be partly affected by lack of physician agreement with guideline component recommedations. Despite this important limitation, the survey participants represented a wide variety of specialties that would manage patients with pulmonary nodules, and many of them had extensive experience with the content of the 2017 Fleischner Society Guidelines.

Guidelines are composed of multiple unique recommendations and the strength of the underlying evidence supporting each recommendation can vary considerably. The factors that underlie rates of adherence to guidelines are multiple and include the strength of the underlying evidence, the risks and benefits associated with the disease, and as we discussed in this paper, physician agreement with the underlying clinical recommendations embedded within those guidelines. The issue of local experts disagreeing with ‘national’ guidelines, especially those based primarily on professional society consensus rather than high quality scientific evidence, may be a widespread issue that would benefit from further research.