Sources of variation in physician adherence with clinical guidelines: results from a factorial experiment.

BACKGROUND: Health services research has documented the magnitude of health care variations. Few studies focus on provider level sources of variation in clinical decision making-for example, which primary care providers are likely to follow clinical guidelines, with which types of patient.
OBJECTIVES: To estimate: (1) the extent of primary care provider adherence to practice guidelines and the unconfounded influence of (2) patient attributes and (3) physician characteristics on adherence with clinical practice guidelines.
DESIGN: In a factorial experiment, primary care providers were shown clinically authentic video vignettes with actors portrayed different "patients" with identical signs of coronary heart disease (CHD). Different types of providers were asked how they would manage the different "patients" with identical CHD symptoms. Measures were taken to protect external validity.
RESULTS: Adherence to some guidelines is high (over 50% of physicians would follow a third of the recommended actions), yet there is low adherence to many of them (less than 20% would follow another third). Female patients are less likely than males to receive 4 of 5 types of physical examination (p < .03); older patients are less likely to be advised to stop smoking (p < .03). Race and SES of patients had no effect on provider adherence to guidelines. A physicians' level of experience (age) appears to be important with certain patients.
CONCLUSIONS: Physician adherence with guidelines varies with different types of "patient" and with the length of clinical experience. With this evidence it is possible to appropriately target interventions to reduce health care variations by improving physician adherence with clinical guidelines.

Social and behavioral scientists have begun to identify different nonmedical influences on provider clinical decision making. For a range of common medical problems, clinical decision making appears to be influenced as much by who the patient is (their age, race, or gender), which provider they encounter, and the type of organizational setting in which medical care is provided as it is by the signs and symptoms of the problem itself 1–3. Some studies show that only about half of those who utilize the health system actually receive the recommended processes of medical care 4. Recently, attention has turned to the quality of medical decisions–that is, the extent to which specific providers adhere to appropriate medical practice guidelines (test ordering, prescribing behavior, life style recommendations, referrals, and patient follow-up) for common medical problems presented by specific types of patients 5. This paper focuses on specific sources of variation in physician adherence to clinical guidelines: that is, which primary care providers are likely to evidence appropriate clinical practice, with which types of patients? Evidence on these specific sources of provider variation is required to ensure that interventions to improve the quality of care and reduce health variations are appropriately targeted. First, we sought to identify the extent to which providers adhere to clinical guidelines in their clinical decision making; second, we examine the influence of patient and provider factors on clinical decisions making. Is variation associated with specific patients? Is it associated with particular providers?

METHODS

The objective of this research was to estimate the unconfounded influence (either singly or in combination) of: (a) patient attributes (age, gender, race, and socioeconomic status) and (b) provider characteristics (physician gender and years of clinical experience) on medical decision making when providers are presented “patients” who show identical signs and symptoms strongly suggestive of coronary heart disease (CHD) (a common medical problem). We conducted a factorial experiment that permits estimation of unconfounded main effects and interactions of any 2 of the variables listed above. The experiment focused on a range of clinical decisions as they relate to a case of CHD 6, 7. Research methods are summarized below (see also McKinlay et al 2006, 8.

A version of the videotaped CHD condition (varying by age, race, or gender) was shown to each of the physicians recruited as subjects for the experiment. We recruited a total of 128 Massachusetts licensed, randomly sampled internists and family physicians. We stratified subjects according to gender and level of clinical experience, including eligible physicians until each cell was complete. The “patient” (reluctantly made the appointment) presents with a complaint of “indigestion” and features of chronic atypical chest pain. For the estimation of main effects, a total sample of 128 physicians gives 80% power to detect an absolute difference in means of 25%. For 2-way interactions, it provides 80% power to detect an effect size of .25. The effect size is a ratio of the variability of the hypothesized means divided by the variability of the observations. For 2 means with a difference Δ, standard deviation of subjects s, the effect size Δ/2s9. Immediately after viewing the selected video for the experiment, the experimental subjects completed a semistructured interview. This interview included questions concerning how they would manage the case of CHD depicted in the video in their everyday clinical practice, including their most likely diagnoses, their certainty with respect to the diagnosis, test ordering, prescriptions, lifestyle recommendations, and what other information they might seek. Qualitative techniques were employed to elicit the reasons why decisions were made.

Experimental Stimuli (Scenarios)

Professional actors and actresses were recruited and trained (under experienced physician supervision) to realistically portray a “patient” presenting to a primary care provider with the signs and symptoms of CHD. Sixteen versions of the scenario were videotaped, systematically varying the “patient’s” age (55 vs. 75 years to get some separation between the middle aged and older patients), race (white vs. black), gender and socioeconomic status (lower vs. higher social class—a janitor vs. a teacher). Potentially relevant nonverbal indicators were embedded in the script, such as the “Levine fist.” Each videotaped encounter simulated an initial interview with either an internist or family practitioner and was of 7–8 minutes in duration, reflecting the average length of a consultation (face time) with a primary care physician (not including a physical exam 10).

CHD was selected because: (a) it is among the most common and costly problems presented by older patients to primary care providers 11; (b) it is a relatively well-defined organic medical condition; and (c) it can result in a range of possible diagnostic, therapeutic, and life style actions. A script for the case of CHD was developed from tape-recorded role-playing sessions with experienced, clinically active advisors. “Patients” in the CHD vignette presented with symptoms suggestive, although not pathophenomic, of CHD (including, for example, heartburn, indigestion unrelieved with antacids, new substernal discomfort, which is exertional and resolves after several minutes rest, pain in the back between the shoulder blades, stress, and elevated blood pressure).

Experimental Subjects (Physicians)

To be eligible for selection, an equal number of male and female physicians had to: (a) be internists or family practitioners; (b) have ≤12 years clinical experience (graduated between 1989 and 1996) or ≥22 years experience (graduated between 1965 and 1979) to get clear separation by level of experience; (c) be trained at an accredited medical school in the US; and (d) be currently providing clinical care at least half time. Eligible physicians were randomly sampled from throughout Massachusetts to fill 4 design cells (gender by level of experience). Screening telephone calls were conducted to identify eligible subjects and an hour-long, in-person interview was scheduled (at which time informed consent was obtained). Each physician subject was provided a modest stipend ($100) to partially offset lost revenue and to tangibly acknowledge participation. The response rate was 64.9%.

Assessing the Quality of Medical Care

Assessment of the quality of decision making requires some gold standard against which physician behavior can be compared. We originally planned to derive this standard from 2 main sources: (a) official clinical guidelines promulgated by, for example, AHRQ and AHA; and (b) the recommendations of a respected group of local clinical peers as to what any minimally competent provider should do when encountering the videotaped “patient.” This approach was designed to accommodate the competing interests of different groups by developing a consensus view triangulated from these 2 sources on the most appropriate management of the presenting “patients” on the videotape. Table 1 depicts considerable divergence between the actions listed in the clinical guidelines and the recommendations made by clinically active peers. Furthermore, there was little consensus among the clinical peers as a group about the recommended action for the clinical case. This divergence may partly explain the fact that when asked whether their knowledge of guidelines contributed to their decisions with respect to the “patient” in the videotape, 75% of the physician subjects, said “no”: There were no significant differences in the use of guidelines depending on physician gender (p = 1.0) level of experience (p = .69), or their interaction (p = .69). For the purposes of this paper, clinical guidelines developed by ACC/AHA/ACP-ASIM (Guidelines for the Management of Patients with Chronic Stable Angina 12, 13 for CHD were used as 1 useful gold standard (recognizing there are others) against which physician decision making could be assessed. The clinical actions recommended in these guidelines were grouped into 5 categories, following the logical order of the encounter: information seeking, physical examination, test ordering, drug prescriptions, and life style recommendations. Whereas chest x-ray is not required for all patients with CHD, the guidelines strongly recommend it if there is evidence of congestive heart failure, valvular heart disease, pericardial disease, aortic dissection, or pulmonary disease.

Table 1

The Concordance or Discordance Between 6 Boston Area Clinical Experts and Clinical Guidelines on Key Aspects of Care for CHD

Guidelines	Clinician Agreement
Information seeking
Quality of pain	-
Duration of pain	-
Provoking factors	∅
Reliving factors	-
Patient medical history	√
Family history	-
Physical examination
Heart	-
Lungs	-
Abdomen	∅
Peripheral extremities	-
Vascular neck exam	-
Test ordering
ECG/EKG	√
Hemoglobin	-
Glucose	-
Lipids	-
Chest x-ray	-
Stress test	√
Drug treatments
Aspirin	√
Beta blocker	∅
Short acting nitrates	√
Lifestyle recommendations
Diet or weight	-
Exercise	-
Smoking	-

∅ = 3 physicians in agreement with guidelines

√ = >3 physicians in agreement with guidelines

= <3 physicians in agreement with guidelines

Sources: Gibbons RJ, et al. 12 and Gibbons RJ, et al. 13

Validity of the Experimental Approach

With every study there is some trade off between internal and external validity. The present experimental study has excellent internal validity, but its external validity can be questioned. Four precautionary steps were taken to enhance external validity, that is, whether the responses subjects gave represent the care they would truly provide. First, to achieve clinical authenticity of the scenario, physicians provided expertise during script development and were present during filming, where professional actors played the patient roles. Second, when physician subjects were asked how typical the “patient” viewed on the videotape was compared with patients in their everyday practice, 92% considered them very typical or reasonably typical. Third, the doctors viewed the tapes in the context of their practice day (not at a professional meeting, a course update, or in their home). In other words, it was likely they saw real patients before and after they viewed the “patient” scenario. Fourth, the doctors were specifically instructed to view the “patient” as 1 of their own patients and to respond as they would in their own practice.

RESULTS

We present primarily main effects results. There were a number of significant 2-way interactions, but there was no consistent pattern. The significant 2-way interactions are discussed below when they modify significant main effects.

Results from the experiment are divided into 3 main groups. First, we examined the proportion of physicians who would follow the clinical guidelines for the management of CHD. Second, we focused on the influence of 4 patient attributes (gender, age, race, SES) and their relation to guideline adherence. Third, we examined the influence of 2 provider characteristics (physician gender, age/years of clinical experience). Thus, our experimental approach permits unconfounded estimation of 5 different influences on physician behavior.

The Extent of Guideline Adherence

Figure 1 summarizes the proportion of Massachusetts primary care physicians randomly sampled by design cell who would follow clinical guidelines when encountering the “patient” presenting with symptoms of CHD. A high percentage would obtain a medical history (83%), order an ECG/EKG (88%), examine the heart (74%), lungs (76%), and abdomen (78%), and order a stress test (75%). Compared to our guidelines standard, many physicians would obtain incomplete information on pain (duration and relieving factors), would not examine peripheral extremities and the neck, and would not order a glucose test or chest x-ray. Only a small proportion (6%) would recommend an increase in physical activity. Whereas there is high adherence to some guideline proscriptions (over 50% of primary providers would follow a third of them) there is low adherence to many of them (less than 20% of providers would follow a third of actions) recommended in the guidelines. There was high adherence (>50%) concerning the acquisition of information regarding provoking factors and patient history; the examination of the heart, lungs and abdomen; ordering EGG, hemoglobin, and stress tests; and making dietary recommendations. There was lower adherence (<20%) regarding the acquisition of information regarding the duration of pain, factors that relieve it, and family history; the examination of the extremities and neck; ordering glucose tests and x-rays; and making recommendations to increase exercise level.

Figure 1

The percentage of primary care physicians who follow clinical guidelines when encountering a “patient” with coronary heart disease. X axis = percent of physicians who follow guidelines. Y axis = actions recommended by clinical guidelines.

Influence of Patient Attributes

The influence of the “patient’s” gender on CHD guideline adherence is depicted in Fig. 2. Female patients are significantly less likely to receive 4 of the 5 components of the physical examination: cardiac (heart; 84% of men vs 64% of women), pulmonary (lungs) (87% vs 64%), peripheral vascular (extremities) (18% vs 6%), and vascular neck exam (26% vs 9%). Whereas many of the patient gender differences in CHD guideline adherence do not reach statistical significance, there is general consistency in the results. The female “patients” received less than males on 2/3 of the specific clinical actions suggested in the guidelines (16 of the 23 listed).

Figure 2

The percentage of primary care physicians who follow clinical guidelines when encountering a “patient” with coronary heart disease: patient gender. X axis = percent of physicians who follow guidelines. Y axis = actions recommended by clinical guidelines. Red bars = female. Blue bars = male.

Figure 3 summarizes results concerning the effect of “patient” age (55 vs 75 years) on CHD guideline adherence. The only significant difference concerned the recommendation that the “patient” stop smoking-younger “patients” were twice as likely (40%) to receive such advice as older “patients” (23%, p < .04).

Figure 3

The percentage of primary care physicians who follow clinical guidelines when encountering a “patient” with coronary heart disease: patient age. X axis = percent of physicians who follow guidelines. Y axis = actions recommended by clinical guidelines. Red bars = 55 years old. Blue bars = 75 years old.

No significant differences or consistent patterns were evident with respect to either the race (White vs Black) or socioeconomic status (lower vs higher) of the “patient” and physician adherence to guidelines.

The influence of Physician Characteristics

Figure 4 summarizes the overall (main) effects of physician level of experience as to whether they follow guidelines when encountering a “patient” with CHD. The gender of the physician in the study did not appear related to his or her adherence to the CHD guidelines. A physician’s level of clinical experience, however, did produce several significant findings. Table 2 shows that older/more experienced providers were more likely to inquire about factors provoking chest pain (for older patients), to order a glucose test (for male patients) and to prescribe short-acting nitrates (62.5% of more experienced physicians vs 21.9% of less experienced physicians, p < .0001). There was a consistent pattern (although results did not achieve statistical significance) of younger/less experienced doctors conducting all 5 components of the physical examination, whereas the older/more experienced physicians were more likely to order all 6 of the tests listed.

Figure 4

The percentage of primary care physicians who follow clinical guidelines when encountering a “patient” with coronary heart disease: physician level of experience. X axis = percent of physicians who follow guidelines. Y axis = actions recommended by clinical guidelines. Red bars = more experience. Blue bars = less experience.

Table 2

The Way in Which Physician Experience Interacts with Patient Age and Gender

		Level of Physician Experience (%)		p value
Variable	Patient Age (years)	Less	More
Information Seeking	75	34.4	71.9	.0452
Provoking factors	55	43.8	46.9
	Patient Gender
Test Ordering	Male	0.0	34.4	.0497
Glucose	Female	9.4	18.8

DISCUSSION

In this experiment, we provided physicians with a standardized clinical scenario to examine their clinical decision making, and compared this against 1 recognized standard of clinical guidelines for the evaluation of possible CHD. We found that whereas there was high physician adherence to some of the clinical guidelines for management of CHD, many physicians would not follow others—less than 50% would adhere to 2/3 of the specific recommendations. Moreover, characteristics of both the provider and the patient appear to play a role in adherence to guidelines. The gender of the “patient” appears to be influential: female “patients” received fewer of the actions recommended by clinical guidelines for the diagnostic evaluation of CHD. A “patient’s” age significantly affected a physician’s recommendation to quit smoking. An explanation for this may be found in the reaction of a physician colleague who stated, “I understand this result. I’ve got older patients and I’ve been going on for years about their smoking. They’re never going to quit. I’ve really tried and I’ve given up.” A physician’s years of clinical experience were also associated with significant differences: less experienced doctors conducted more components of the physical examination, whereas more experienced physicians were more likely to recommend diagnostic testing.

The use of clinical guidelines as the gold standard for clinical care was chosen for several reasons: they are a) thought to reflect a consensus opinion based upon the current medical evidence (panels of experts from professional societies), and b) a useful standard because they include history of symptoms, clinical examination, diagnostic testing, and lifestyle and pharmacologic intervention recommendations. It is noteworthy that several parts of the clinical history and physical examination are inadequately addressed, as these are inexpensive and provide potentially critical information to elucidate the problem. The majority of physicians included the two diagnostic tests directly related to diagnosis of CHD, namely EKG and stress testing, whereas there was greater variability on tests for related or alternative diagnoses, such as glucose testing for undiagnosed diabetes, or chest x-ray for alternative diagnosis.

Considering those guidelines directly concerned with diagnosing and treating CHD and unstable or new angina, we find that providers generally follow the testing guidelines well, but are less adherent to those pertaining to historical information, and early treatments with aspirin, beta blockers or short-activating nitrates. Likewise, the general differences in eliciting historic information suggests that these differences are important and could account for some of the gender disparities observed. Guidelines that relate to primary and secondary prevention, or assessment of important comorbidities may be considered less important at the first evaluation, and thus their lower use by physicians is perhaps understandable.

Some of the variability from the guidelines may reflect timing issues, where providers would postpone beta blocker therapy until after EKG and exercise stress testing, when the extent of disease and potential risk of unstable angina is understood. However, the prescription of short-acting nitrates or aspirin are recommended for their potential protection against cardiac events, even in this case of “heartburn” where upper gastrointestinal pathology may account for symptoms. The lack of exercise counseling could reflect caution until the diagnosis is established, but the same would not hold for the 2/3 of providers who would not recommend smoking cessation.

Our findings corroborate and extend the work of others, and indicate that even when there are evidence-based guidelines for the management of a condition as common as suspected CHD, some physicians’ use of guideline activities continues to lag. This has been identified in other conditions as well. For example, regular monitoring of blood sugar is considered essential to the effective treatment of diabetes and prevent complications 14. However, Saaddine and colleagues 15 found that only 29% of diabetic patients reported having their blood sugar tested during the previous year. Another study by McGlynn and colleagues 4 found that 24% of diabetic patients received 3 or more glycosylated hemoglobin tests over a 2-year period. Grant and colleagues 5 found fewer than half of all diabetics with elevated glycosylated hemoglobins had a change in medication, and only 10% of patients with elevated blood pressure readings had a change in management.

Our results show gender differences in the initial evaluation of possible CHD in women. Prior literature suggests reduced rates of revascularization, or delay in care for acute coronary events in women compared to men, as an explanation for higher morbidity and mortality for CHD in women 16–22. However, our findings suggest that gender disparities in evaluation may begin even earlier in the clinical history and physical examination for CHD.

Our findings did not show a main or interaction effect by physician gender. That is, female physicians did not provide more guideline-based care to either male or female patients than did their male counterparts. Similarly, we found no differences based on patient race in recommendations for guideline appropriate CHD care.

The findings based on physician experience are mixed. Whereas more experienced providers ordered more diagnostic testing when the “patient” was older, their less experienced counterparts conducted more thorough physical examinations. The reduced behavioral counsel to older patients in the vignette may reflect a bias that older patients are less likely to adopt smoking cessation and other behavioral change. However, some evidence suggests that older patients are as or more likely to adopt change 23.

Limitations

Several limitations of this study should be noted. First, whereas the rigorous experimental design permits excellent internal validity, external validity remains a threat. Four precautionary steps were taken to hopefully minimize this threat (physicians were involved in script development; study subjects [physicians] were specifically asked how typical the “patient” was compared with patients in everyday practice; subjects viewed the tapes in the context of their practice day; subjects were specifically instructed to view the “patient” as 1 of their own patients). Second, the response rate of 64.9& (while high for a study of US physicians in the present climate) means over a third of those eligible and selected did not agree to participate. This is an unavoidable consequence of the decision to randomly sample in an attempt to increase the generalizability (external validity) of the research findings. Third, the level of adherence to guidelines may depend on which guidelines are selected as the gold standard. The guidelines used in this research were recommended by clinical colleagues as promulgated by a reputable professional organization and considered to have wide visibility among providers. They are clearly only 1 set among many different guidelines developed by numerous groups. Future research could investigate whether any variability in physician adherence is guideline specific.

Implications

The implications of this study rest on the assumption that physicians need to adopt and adhere to evidence-based guidelines in their everyday practice. Evidence-based disease management strategies, including early use of aspirin and short-acting nitrates, cannot reach their full potential if not incorporated into clinical care. The widely reported gender inequalities in coronary heart disease are unlikely to be reduced if improvements in the illness behavior of women (personal risk assessment, symptom recognition, and earlier help-seeking) are not matched by appropriate diagnoses, test ordering, and lifestyle recommendations by providers 24. The question is how to ensure the incorporation of clinical guidelines into everyday practice: that is, how to institutionally support and reimburse physicians’ adherence to their own, or their colleagues’ own, recommendations (possibly pay-for-performance).

Whereas there is considerable literature addressing patient barriers to adherence to treatment recommendations, less is written on the barriers in clinical practice to physician adherence to recommendations. Some authors cite inadequate professional training 25, especially the limitations of training of primary care providers to address so many complex medical conditions. Others cite reimbursement policies that reward procedures over evaluation and management. Other barriers to adherence to guidelines include provider concerns about “cookbook medicine”, perceived regulatory intrusion into practice and unwillingness to buy into the concept of management guidelines. Other barriers may be a lack of systems support, such as electronic medical records, which incorporate guideline management recommendations to ensure uniformity in care; or lack of patient participation in decision support 26. Lastly, the number of guidelines propagated and the lack of concurrence among them make it difficult for providers to find a clear consensus on the best management practices 27.