Is There a Doctors' Effect on Patients' Physical Health, Beyond the Intervention and All Known Factors? A Systematic Review.

Purpose: Despite billions of doctor visits worldwide each year, little is known on whether doctors themselves affect patients' physical health after accounting for intervention and confounders such as patients' and doctors' data, hospital effects, nor how strong that doctors' effect is. Knowledge of surgeons' and psychotherapists' effects exists, but not for 102 other medical specialties notwithstanding the importance of such knowledge.
Methods: Eligibility Criteria: Randomized controlled trials (RCTs), case-control, and cohort studies including medical doctors except surgeons for any intervention, reporting the proportion of variance in patients' outcomes owing to the doctors (random effects), or the fixed effects of grading doctors by outcomes, after multivariate adjustment. Exclusions: studies of <15 doctors or solely reporting doctors' effects for known variables. Sources: Medline, Embase, PsycINFO, inception to June 2020. Manual search for papers referring/referred to by resulting studies. Risk of Bias: Using Newcastle-Ottawa scale.
Results: Despite all medical interventions bar surgery being eligible, only thirty cohort papers were found, covering 36,239 doctors, with 10 specialties, 21 interventions, 60 outcomes (17 unique). Studies reported doctors' effects by grading doctors from best to worst, or by diversely calculating the doctor-attributed percentage of patients' outcome variation, ie the intra-class correlation coefficient (ICC). Sixteen studies presented fixed effects, 18 random effects, and 3 another approach. No RCTs found. Thirteen studies reported exceptionally good and/or poor performers with confidence intervals wholly outside the average performance. ICC range 0 to 33%, mean 3.9%. Highly diverse reporting, meta-analysis therefore not applicable.
Conclusion: Doctors, on their own, can affect patients' physical health for many interventions and outcomes. Effects range from negligible to substantial, even after accounting for all known variables. Many published cohorts may reveal valuable information by reanalyzing their data for doctors' effects. Positive and negative doctor outliers appear regularly. Therefore, it can matter which doctor is chosen.

Strengths and Limitations

A strength of our review is the comprehensive literature search, using a complex and complete combination of terms for the search strategy to identify most of the relevant studies; furthermore, we screened the articles’ list of references and studies citing the article for further eligible studies, with no limitations regarding the language or timeframe. In addition, it is the first systematic review providing detailed and clear reporting of the effect size, and that the doctors’ effect is often substantial.

Conversely, there is a trio of limitations. First, although the scoring of the Newcastle-Ottawa Scale (NOS) for assessing the risk of bias showed that the majority of included studies scored a value of 8 or 9 (9 is the maximum total), that scale has been critiqued for being “difficult to use and [having] vague decision rules”1 leading to poor or fair inter-rater reliability among reviewers. However, The Cochrane Collaboration2 has endorsed its implementation in systematic reviews that include nonrandomized studies. Second, since all of the included studies were set in Europe and North America, our findings may not be applicable to other locations, particularly developing nations. Finally, among our included studies, data was reported too heterogeneously in content and presentation to allow meta-analysis.

What is already known on this topic: psychotherapists and surgeons are well known to have a substantial effect on patients’ physical health. However, the scale of the influence of (non-surgical) medical doctors on patients’ physical health, after accounting for all known confounders, is less understood. In other words, is there a doctors’ effect which there is currently no explanation for?

What this study adds: this systematic review is considered to be the first to address the unexplained doctors’ effect on patients’ physical health, showing that medical doctors can be effect modifiers of interventions. Findings are highly variable, ranging from little effect through to large effects, where the latter can result in significant differences in patient’s physical health outcomes, depending on the doctor, which means that it can matter which doctor is chosen.

Rationale

Each year, patients worldwide visit medical doctors billions of times, with 800 million visits in the United States3 and 150 million visits in Australia4 alone. However, apart from a classic5 1955 essay6 that states “[T]he most frequently used drug in general practice was the doctor himself”, there has been limited research on whether medical doctors, on their own, can represent an intervention or an effect modifier of interventions, ie whether different doctors who use the same intervention have differing patient’s physical health outcomes, even after accounting for all known variables, including doctor demographics and patient risk factors. It is well-known that psychotherapists can have a significant effect on their patients’ mental health, an effect that equals the strength of pharmaceutical interventions and is mentioned in training manuals.7 It is also known that surgeons, after accounting for all known information,8 do have a widely varying effect on patients’ physical health. Therefore, it would be useful to know whether this applies to other medical doctors, as a fundamental question in medical research is what effect the medical practitioner has on patients’ physical health. The doctor certainly has an effect by choosing and applying the intervention, but it is less clear whether the effect goes beyond the intervention, and whether doctors constitute an intervention in their own right.

Research on general doctors’ performance has concluded that it is difficult to assess practice variation among doctors and therefore, it is often not worthwhile to direct quality improvement efforts at this level of medical services.9,10 However, some doctors were found to be more effective than others at employing interventions, owing, for example, to a substantial volume or practice effect in many surgical specialties.11,12 Recent evidence also proposes that patients’ outcomes can be substantially affected by provider expectations.13 In other non-surgical specialties, research conducted on doctors’ effects is scarcer, with evidence limited to primary care,14,15 obstetrics,16 and acute care,17 in which physicians’ factors point to a sizeable effect on patients’ health outcomes. Thus, a significant doctors’ effect detected indicates that there are doctors who perform better than others. Many initiatives aimed at improving medical standards aim to identify underperformers to either remove them from medical practice or propose strategies to improve their standards.18–20 However, there seems to be no systematic review that answers a more basic question: Are there differences among doctors which contribute to creating an effect on patients’ physical health outcomes, even when all known factors have been accounted for?

In a kitchen, it would be obvious that cooks using the same ingredients have widely varying outcomes. In law, practitioners charge widely varying rates, with clients presumably assuming that the most expensive lawyers are so much better than the average lawyer that they are worth their higher fees. No such presumption of substantial differences between doctors seems to exist in medicine as an established research fact.

If we know whether medical doctors can differ widely in their performance, then we can find out under what circumstances the effect is large or small, important, or unimportant. In addition, we can check whether there are positive and negative outliers among doctors, allowing health care services to support the negative outliers to improve, if possible, and to learn from the positive outliers, and, if needed, make sure that they are treated with the care and respect such exceptionally good doctors deserve.

This systematic review gives the answer to precisely this question: What research has been published that shows whether doctors, on their own, have an effect on patients’ physical health outcomes, after taking into account all known information? Known information can consist of patient demographics and risk factors, intervention, doctor demographics such as age, specialization, education and experience, and hospital or area effects such as county or country effects.

This review further looks at the quality of the publications and their heterogeneity, and whether reporting on doctors’ performance can be improved and prepared for meta-analysis. It may seem ambitious to cover 102 non-surgery medical specialties21 in a single publication but such is the paucity of this material – despite the billions of interactions of medical doctors each year – that the number of publications found do fit into a single systematic review. Future reviews may be more focused, but an overarching review is the first step, due to the current lack of any review.

What is the Current State of Research?

In 2002, the British Medical Journal (BMJ) devoted an entire issue to the following question: “What’s a good doctor and how do you make one?”22 assuming that it would be useful to know what a good doctor is. In this special edition, one article presented letters from doctors and others attempting to answer these questions. One quote stated: “There is not a single piece of evidence or the means to measure whether a doctor is good or bad.”23 The editorial of that 2002 issue stated

(…) defining a good doctor, I suggest, lies in degree of difficulty somewhere between defining a good composer and a good human being. In fact, it’s impossible.

Hospitals are known to substantially influence patients’ physical health outcomes and hospital performances regarding patients’ physical health outcomes vary widely.24–29 The same is true for larger entities like regions or countries where mortality rates can differ substantially.30

Recent research has investigated 10 surgical trials, in which the effect size of surgeons was analyzed to assess the surgeon intra-cluster correlation coefficients (ICCs), ie the percentage of the whole patient outcome variation due to the surgeon. It revealed that surgeons alone are responsible for a range of effects on patients’ health outcomes, which vary between different surgical specialties.31

Objectives

This systematic review examines the existing literature on measuring and reporting doctors’ effects on patients’ physical health after adjusting for known factors for medical doctors that are not surgeons. Psychotherapists are here not considered to be medical doctors.

Methods

Eligibility Criteria

This systematic review follows the standards set for Synthesis Without Meta-analysis (SWiM).32 Only studies that investigated actual patients’ physical outcomes were included. Scientific publications that reported patients’ opinions or their satisfaction levels were excluded as these are not patients’ physical health outcomes and often less reliable measurements.33

The study PICO is as follows:

Information Sources and Search Strategy

We conducted a comprehensive search on the following databases: Embase, Medline via PubMed, and PsycINFO, to retrieve pertinent studies that investigate the doctors’ effect on patients’ physical health outcomes, from inception until June 2020. The search strategy was designed and developed for each database by JMC, a search specialist (). In addition, using the references lists of the selected articles and former reviews we manually searched for potentially related studies that may have been missed in the initial literature search. Furthermore, systematic review registries including PROSPERO and Cochrane’s CENTRAL register were searched for similar reviews.

Selection Process and Further Eligibility Criteria

Two review authors independently screened the titles and abstracts of all retrieved records. Any disagreements were resolved via discussions and consultation with a third reviewer. We included any case-control study, retrospective or prospective cohort study, or randomized controlled trial (RCT) that graded individual doctors according to their performance regarding the patients’ physical health outcomes, or where the percentage of the variance in patients’ outcomes is explained by differences between doctors. All outcomes related to patients’ physical health were eligible, for example survival/mortality rate, repair reoperations, hospitalization rates, length of post-procedure stay, readmission rate, post-operative complications, pain, infection rate, embryo transfer rate, blood pressure, cholesterol, and glycemic control. Surgeons were excluded from this review as they were reviewed in a separate paper.8 No restrictions were placed on publication date or language.

We excluded studies that address only doctors’ effects related to specific known doctor-related variables, such as the doctor’s specialty or the volume of procedures performed. Studies including fewer than 15 doctors and cross-sectional studies were also excluded, due to their increased risk of bias.

No authoritative source was found to provide a reference for the smallest number of clusters required for a reliable ICC estimation. Here the number of referred-to clusters is the minimum number of practitioners to warrant inclusion. We used 15 as a minimum number but realize this is somewhat arbitrary (Figure 1).

Figure 1

Flow diagram of selection of included documents.

Data Collection Process and Data Items

We used Endnote 9 for exporting the titles and abstracts of retrieved records, which were then uploaded into Rayyan for screening. Then the potentially eligible records were marked as members of a group in the original Endnote library and their full text documents added to the library for further full-text screening.

From each final included study, CS and a second extractor independently and in duplicate, extracted the relevant data into an excel sheet including the following variables:

Study ID consisting of the first author’s last name and year of publication

Type of study (RCT, Cohort)

Country of origin

Medical specialty

Type of intervention(s)

Patients or procedures

Number of doctors

Number of hospitals or institutions

Outcome type(s)

Number of positive and negative outliers

Authors’ evaluation of significant doctors’ effect Y/N

Multivariate analysis Y/N

ICC (intra-class correlation coefficient)

Risk of Bias Assessment

Two review authors independently assessed the risk of bias of all included cohort studies, using the Newcastle-Ottawa Scale (NOS).34,35

Effect Measures

One pathway of evaluating doctors’ performance is to measure fixed effects, which are covered as a statistical technique by Allison.36 Fixed effects allow the identification of high and low outliers and give an impression of how heterogeneously doctors perform in a particular area. Grading doctors also shows whether the variation in effect is consistent with chance or bigger than that. The metric for the fixed effects in this study is the percentages of negative and positive outliers, as defined and reported per each individual study.

The other method of assessing a doctors’ effect is by measuring random effects, also explained by Allison.36 Random effects measure the variation in patient outcomes that is due to the doctor beyond known factors, such as their level of experience. Likewise, these effects cannot be explained by differences in diagnostic prowess or choosing more or less suitable interventions. Random effects allow the discernment of how much doctors may constitute an intervention in their own right. That measurement is called the intra-class correlation coefficient (ICC). Examples are mortality in intensive care,37 or levels of uncontrolled hypertension,38,39 or high HgA1c levels18,38,40–42 among patients of family medicine doctors or general practitioners.

The ICC is here described as the proportion of patients’ health outcomes that resulted from the doctor’s effect, in the form of a percentage of the total patient outcome variation. The significance of even small ICCs is covered in the Discussion.

Synthesis Methods

The identification of doctors’ effects on patients’ health outcomes is presented in many different ways that can be classified into two methods. Both methods either use hierarchical regression or multilevel mixed effects regression modelling to understand both doctor and higher-level variation.43,44

Percentage of Variation in Patient’s Health Outcome

The percentage of variation in patients’ health outcome owing to the doctor is reported as the intra-class correlation coefficient (ICC). The ICC, which can be identified through post-regression estimation, is a number ranging from 0 to 1 representing the percentage of variation in a particular outcome due to each level in the regression model. Therefore, to enable the allocation of the percentage of variation owing to the doctor, random effects for doctors and occasionally for hospitals or other higher-level aggregators such as county, are included in the studies.38,39,45–47

The regression analyses included patient risk scores and other known confounders such as doctors’ demographics as fixed effects. There was a pronounced variance in the depth and quality of the analysis between different studies, with Papachristofi et al as a high quality example.47 In addition, further extensive literature is available addressing the ICC.48–53

Grading Doctors from Best to Worst

Regarding this approach, doctors are ordered according to the patients’ physical health outcomes, typically with a 95% confidence interval (CI). This CI is calculated using, for example, cluster-robust standard errors,54,55 or other means such as simulation,16 or the delta method.38 Doctors are considered to be outliers when their 95% CI is wholly above or below the mean rate of the patients’ outcomes. Consequently, results are reported by listing the outliers in order, or as a funnel or caterpillar plot,56 with the latter constituting an outcome-ordered forest plot.

Reporting Bias and Certainty Assessment

Since meta-analysis was not applicable, we did not assess the reporting bias nor conducted certainty assessments.

Results

Study Selection

We retrieved 4713 records from electronic searches, reduced to 3778 after removing duplicates, and 119 after screening. Manually searching the reference list of these studies yielded an additional 6750, reduced to 60 after screening. The resulting 179 studies were reviewed in full, yielding 79 accepted studies of which 30 applied to doctors other than surgeons. These 30 studies with 36,239 doctors met our pre-specified criteria for inclusion in the final synthesis (Figure 1).

Study Characteristics

The final 30 included studies either graded individual doctors from best to worst according to their performance (N=9),16,18,57–63 or recorded a residual variation owing to doctors in a multivariate multi-level analysis yielding an ICC (N=11),38–42,45,46,64–67 or both (N=7),37,47,68–72 or used a different way to describe their results (N=3).17,73,74 Jemt et al74 used a different approach but also listed one positive and two negative outliers.

All 30 studies were observational cohort studies that included doctors from multiple specialties, such as general practitioners, family doctors, or primary care physicians (N=11),18,38–42,45,46,59,62,65 anesthesiologists (N=4),47,68,70,71 cardiologists (N=4),58,60,61,67 hospitalists or residents (N=7),17,37,62,63,66,69,72 and one each of dentistry,74 gynaecology,16 pathology,64 paediatrics,46 radiology,73 and reproductive medicine.57 (N=18) studies were conducted in the USA,18,37–42,45,58,59,61,62,64,67–69,72,73 (N=7) in the UK,16,17,47,60,66,70,71 and one each in Canada,63 Italy,57 Netherlands,65 Spain,46 and Sweden.74 The number of included doctors ranges from 21 to 4230. Table 1 summarizes the characteristics of the included studies.

Table 1

Characteristics of Included Studies

Publication	Practitioner	Specialty	Detailed Intervention	Doctors	Patients/ Procedures	Institutions	Outcome	NOS*
Becerra, 201764	Pathologist	Colorectal surgery/Pathology	Lymph node examination after colectomy	814	12,332	187	Suboptimal care	9
Beckett, 201817	Hospitalist	Acute care	Acute care	22	21,570	1	Mortality	8
							Readmission	8
Brown, 201618	GP General Practitioner	Primary care	Diabetes glucose control	133	14,033	84	Avoiding uncontrolled diabetes	9
Cirillo, 202057	OB-GYN doctor or senior residents	Obstetrics	Embryo transfer	32	19,824	1	Ongoing pregnancy	9
Davenport, 202073	Radiologist	Radiography	Headache CT	55	25,596	1	Mortality	9
							Readmission	9
Eijkenaar, 201365	GP General Practitioner	Primary care	Primary care	447	26,684		COPD**-related admissions	8
				537	37,832		Diabetes-related admissions	8
Glance, 201668	Anesthesiologist	Cardiac surgery	Cardiac surgery	357	55,436	40	Major complications or mortality	9
Goodwin, 201369	Hospitalist	Acute care	Acute care	1099	129,491	268	Length of stay	9
				1099	131,710	268	Mortality	9
Gossl, 201358	Cardiologist	Cardiology	Percutaneous coronary intervention	21	7838	3	MACE Major adverse cardiac event inc. death	9
							Mortality	9
Gutacker, 201866	Hospitalist	Emergency Care	AMI Acute myocardial infarction	1746	138,044	148	Length of stay	9
							Mortality	9
							Readmission	9
		Cardiac surgery	CABG*	212	24,505	30	Length of stay	9
							Mortality	9
							Readmission	9
		Pneumonia	Pneumonia	3760	405,671	152	Length of stay	9
							Mortality	9
							Readmission	9
		Stroke	Stroke	1214	144,114	144	Length of stay	9
							Mortality	9
							Readmission	9
		Orthopedic surgery	Hip fracture	1735	156,145	148	Length of stay	9
							Mortality	9
							Readmission	9
			Hip replacement	1325	170,678	229	Length of stay	9
							Mortality	9
							Readmission	9
Hannan, 201767	Cardiologist	Cardiology	Percutaneous coronary intervention	403	27,560	60	Incomplete revascularization	9
Harley, 200516	Gynecologist	Obstetrics	Gynecologists’ performance	143	Not stated	Multiple	7-item composite measure	7
Hofer, 199945	GP General Practitioner	Primary care	Diabetes glucose control	232	3642	3	Hospitalizations	9
Holmboe, 201038	GP General Practitioner	Primary care	Cholesterol control	236	22,526	13 states	Avoiding high cholesterol	9
			Diabetes glucose control	236	22,526	13 states	Avoiding uncontrolled diabetes	9
			Hypertension control	236	22,526	13 states	Avoiding uncontrolled Hypertension	9
Jemt, 201674	Dental Surgeon	Dental Implants	Dental implants	23	8808	1	Implant failure	9
Kaplan, 200959	GP General Practitioner	Primary care	Diabetes glucose control	210	7574		10 quality measures	8
			Cholesterol control	210	7574		10 quality measures	8
Krein, 200240	GP General Practitioner	Primary care	Cholesterol control	258	12,110	9/13	Avoiding high cholesterol	8
			Diabetes glucose control	258	12,110	12/13	Avoiding uncontrolled diabetes	8
Kunadian, 200960	Cardiologist	Cardiology	Percutaneous coronary intervention	261	149,888	48	Mortality	9
Navar-Boggan, 201261	Cardiologist	Cardiology	Hypertension control	47	5979	1	Avoiding uncontrolled hypertension	9
O’Connor, 200841	GP General Practitioner	Primary care	Diabetes glucose control	120	2589	18	Avoiding uncontrolled diabetes	8
Orueta, 201546	GP (Family doctors)	Primary care	Avoidable hospitalization	1193	2,207,175	130	Hospitalization rates	9
	Pediatrician	Primary care	Avoidable hospitalization	286			Hospitalization rates	9
Papachristofi, 201470	Anesthetist	Cardiac surgery	Cardiac surgery	24	18,426	1	Mortality	9
Papachristofi, 201671	Anesthetist	Cardiac surgery	Cardiac surgery	190	110,769	10	Mortality	9
Papachristofi, 201747	Anesthetist	Cardiac surgery	Cardiac surgery	190	107,038	10	Length of stay	9
Prasad-Kerlin, 201837	Hospitalist	Acute care	Mechanical ventilation	345	11,268	104	Mortality	9
Selby, 201039	GP General Practitioner	Primary care	Cholesterol control	1005a	169,156	35	Avoiding high cholesterol	9
			Hypertension control	1049b	232,053	35	Avoiding uncontrolled hypertension	9
Singh, 201572	Hospitalist	Primary care	Primary care	525	48,883	143	Readmission	9
Singh, 201962	GP General Practitioner	Primary care	Primary care	4230	565,579		Hospital readmission	9
Tuerk, 200842	GP General Practitioner	Primary care	Diabetes glucose control	42	1381	1	Avoiding uncontrolled diabetes	8
Verma, 202063	Hospitalist	Acute care	Emergency admissions, inpatient care	135	103,085	7	Length of stay	9
							Mortality	9
							Readmission	9

Notes: If “Institutions” is blank, then the number is not applicable (GPs, General Practitioners, for example), or not given and most likely greater than one. *CABG is coronary artery bypass graft. **COPD is chronic obstructive pulmonary disease. For cholesterol, diabetes, and hypertension management, outcomes were standardized to avoiding high cholesterol/HbA1C/blood pressure. aSingle year numbers. Totals for 6 years are 6,832 doctors, 1,588,407 patients. bSingle year numbers. Totals for 6 years are 6,995 doctors, 2,021,935 patients.

Abbreviation: NOS, Newcastle-Ottawa Scale for assessing risk of bias of cohort studies.

Among the 59 outcomes in the included 30 studies, (N=48) scored 9 stars, (N=10) 8 stars, and (N=1) 7 stars, with a maximum possible score of 9 stars on the Newcastle-Ottawa Scale.34,35 Those of 7 and 8 stars scored either 0 or 1 on the aspect of comparability, whereas the studies with 9 stars scored 2. All included studies scored the maximum points regarding the selection and the outcome criteria (Table 1).

Results of Individual Studies

Altogether 15 studies with 21 outcomes published caterpillar plots or plots that gave the same information.17,18,37,47,57,58,61–63,65,68–72 One paper showed funnel plots.60 Such plots represent and sort the doctors’ performance for a specific patient outcome, usually showing a 95% confidence interval (CI) for each doctor and whether that CI was wholly below or above the mean performance rate. Results varied from no over- or underperformer70,71 up to substantial numbers of both.37,59,61,63,72

Of the 16 studies that show fixed effects, 11 reported one or more exceptional performers after accounting for all known confounders, including doctors’ demographic variables such as their years of experience and volume of procedures/patients, and the at times substantial hospital effects.16,18,37,47,58–61,63,69,74 Two papers57,62 found only negative outliers. Three papers found no positive or negative outliers.68,70,71 (Table 2).

Table 2

Publications by Outcome and Numerical Results

Outcome	Practitioner	Specialty	Publication	ICC^	Outliers %
					Negative	Positive
Avoiding high cholesterol	GP General Practitioner	Primary care	Holmboe, 201038	12.0%
			Kaplan, 200959	9.0%
			Krein, 200240	1.0%
			Selby, 201039	1.9%
Avoiding uncontrolled diabetes	GP General Practitioner	Primary care	Brown, 201618		6.0%	6.8%
			Holmboe, 201038	9.0%
			Kaplan, 200959	33.0%
			Krein, 200240	0.0%
			O’Connor, 200841	0.8%
			Tuerk, 200842	2.0%
Avoiding uncontrolled hypertension	Cardiologist	Cardiology	Navar-Boggan, 201261		6.4%	12.8%
	GP General Practitioner	Primary care	Holmboe, 201038	9.0%
			Selby, 201039	1.9%
Complications
MACE Major adverse cardiac event inc. death	Cardiologist	Cardiac surgery	Gossl, 201358		0.0%	4.8%
Major complications or mortality	Anesthesiologist	Cardiac surgery	Glance, 201668	0.5%	0.0%	0.0%
Hospitalizations
COPD**-related admissions	GP General Practitioner	Primary care	Eijkenaar, 201365	2.5%
Diabetes-related admissions	GP General Practitioner	Primary care	Eijkenaar, 201365	0.6%
Hospitalizations	GP General Practitioner	Primary care	Hofer, 199945	1.0%
			Orueta, 201546	6.1%
	Pediatrician	Primary care	Orueta, 201546	10.3%
Length of stay	Anesthetist	Cardiac surgery	Papachristofi, 201747	0.2%	2.1%	0.5%
	Hospitalist	Acute care	Goodwin, 201369	2.6%	19.5%	18.0%
			Verma, 202063		18.5%	14.8%
		Cardiac surgery	Gutacker, 201866 (Heart attack)	6.5%
			Gutacker, 201866 (CABG*)	5.2%
		Pneumonia	Gutacker, 201866	2.1%
		Stroke	Gutacker, 201866	1.5%
		Orthopedic surgery	Gutacker, 201866 (Hip fracture)	3.2%
			Gutacker, 201866 (Hip replacement)	12.7%
Mortality	Anesthetist	Cardiac surgery	Papachristofi, 201470	0.1%	0.0%	0.0%
			Papachristofi, 201671	0.3%	0.0%	0.0%
	Cardiologist	Cardiology	Gossl, 201358		0.0%	4.8%
			Kunadian, 200960***	0.2%	0.0%	0.4%
	Hospitalist	Acute care	Beckett, 201817* (different presentation)
			Goodwin, 201369	0.8%	1.5%	0.6%
			Prasad-Kerlin, 201837	1.8%	22.6%	25.5%
			Verma, 202063		1.5%	5.2%
		Cardiac surgery	Gutacker, 2018 (Heart attack)	1.4%
			Gutacker, 201866 (CABG*)	0.9%
		Pneumonia	Gutacker, 201866	1.2%
		Stroke	Gutacker, 201866	1.1%
		Orthopedic surgery	Gutacker, 201866 (Hip fracture)	1.2%
			Gutacker, 201866 (Hip replacement)	0.3%
	Radiologist	Radiography	Davenport, 202073
Pregnancy	Reproductive doctor	Obstetrics	Cirillo, 202057		3.1%	0.0%
Readmission	Hospitalist	Acute care	Beckett, 201817 (different presentation)
			Verma, 202063		0.7%	3.0%
		Cardiac surgery	Gutacker, 201866 (Heart attack)	0.4%
			Gutacker, 201866 (CABG*)	0.8%
		Pneumonia	Gutacker, 201866	0.4%
		Primary care	Singh, 201572	15.0%	12.8%	12.5%
			Singh, 201962		0.02%	0.00%
		Stroke	Gutacker, 201866	0.8%
		Orthopedic surgery	Gutacker, 201866 (Hip fracture)	0.7%
			Gutacker, 201866 (Hip replacement)	2.5%
	Radiologist	Radiography	Davenport, 2020** (different presentation)
Suboptimal care	Pathologist	Colorectal surgery	Becerra, 201764	22.5%
Success or failure
Implant failure	Dental Surgeon	Dental Implants	Jemt, 201674		8.7%
Incomplete revascularization	Cardiologist	Cardiac surgery	Hannan, 201767	12.0%
Multiple measures
10 measures	GP General Practitioner	Primary care	Kaplan, 200959		27.8%	43.8%
7-item composite measure	Gynecologist	Obstetrics	Harley, 200516		6.3%	2.1%

Notes: The leftmost column is by patients’ physical outcome with summarized outcome bold. GP is General Practitioner or Primary Care Physician. ^ICC is Intra-class correlation coefficient that shows percentage of variance owing to the practitioner in the form of percentage of total variance after taking into account all known confounders. *CABG is coronary artery bypass graft. **COPD is chronic obstructive pulmonary disease. Outliers are listed for studies which ordered the doctors according to their effect on patient’s physical health outcomes from worst to best or vice versa. The percentages listed represent those practitioners whose 95% CI is wholly below or above the mean. Beckett et al,17 Davenport et al,73 and Jemt et al74 presented their data in a way that only fits partially or not at all in the table. Blank entries under ICC or Outliers mean that the Publication did not report those measures. All studies that reported Outliers except Verma et al63 adjusted their results for other factors like patient risk. *Beckett et al17 reported fixed effects before and after case-mix but no 95% confidence intervals. **Davenport et al73 reported fixed effects with no effect on mortality but other, more indirect measures. ***Kunadian et al60 did not publish the number of cardiologists or outliers directly, though Figure 2 in the paper is a funnel plot. The paper’s reference 7 provides the original data which shows 146,775 cases with 261 cardiologists of whom 7 were underperformers and 5 outperformers. With one exception, all cardiologists with fewer than 31 cases were grouped as “all others”. Cardiologists had one entry in the table for each hospital they worked in. The authors of this systematic review calculated the ICC for this dataset to be 0.17%, 95% CI 0.11%, 0.26%. Mortality is 945/146,775 or 0.64%.

Figure 2

Intra-class correlation coefficients (ICCs) by paper, intervention, and patient outcome. *COPD is chronic obstructive pulmonary disease. **PCI is percutaneous coronary intervention.

A few papers (N=18) presented a random effect, reported in many different ways, which express the intra-class correlation coefficient (ICC), ie the variation due to the doctor as a percentage of the whole variation in patient physical health outcomes, with that variation calculated while accounting for all available patient, doctor, or institution variables.37–42,45–47,62,64–71 Reported random effects ranged from approximately zero (ICC of 0.0%) to substantial (ICC up to 33%, median of 1.9%, mean of 3.9%, inter-quartile range 1.0–4.2%) (Figures 2 and 3, Table 2).

Figure 3

Boxplot of ICC (intra-class correlation coefficient).

Only cholesterol,38–40 diabetes,40–42 and hypertension38,39 control outcomes had more than one study each for the same medical specialty and intervention. ICCs range from 0% to 2%, except Holmboe et al38 who found much higher ICCs of 12% and 9%. The main difference between this and the other studies is that Holmboe’s cohort consisted of doctors who volunteered to participate (Table 2). In nine instances, the ICC was between 9% and 33%.

Reporting Bias, Syntheses, and Certainty of Evidence

Not applicable since there was no statistical synthesis of the results.

Discussion

The findings from this systematic review indicate that doctors have an effect on patients’ physical health, even after taking into account all known variables or confounders. This effect ranges from zero to substantial with nine instances where the doctor was associated with at least 9% of the total variation in patient health.

In terms of the effect of even small ICCs, a randomized controlled trial75 that established the prophylactic value of aspirin was halted early as it was considered to be unethical to withhold aspirin from the control group, even though aspirin only accounted for 1% of the variability in outcomes, ie the trial was halted for a treatment with an ICC of 1%. Further, even a “small” doctors’ effect makes a substantial difference in patient health as that difference is applied billions of times each year in each doctor-patient interaction. The value and importance of even small ICCs is further outlined in these three publications.7,76,77

At times doctors can be identified whose performance is substantially above or below the average performer. Therefore, a possible answer to the question, “What’s a good doctor and how do you make one?”78 is, “A good doctor is a doctor with significantly better patient physical health outcomes than the average doctor.” In addition, a possible answer to, “and how do you make one?” could be,

Good doctors already exist and can be identified. Unless good doctors’ abilities are wholly innate, more good doctors can be made by learning from those who already are good doctors, and exceptionally good doctors also exist.

The key here is that an effect with an unknown cause has been identified. The cause could be anything unmeasured in the included cohort studies, such as doctors’ communication skills, their level of care for patients, their physical or mental health, the time they give to a patient, their ability to listen to a patient, their diagnostic ability (as a more suitable intervention is more likely to yield better outcomes), their ability to perform under stress etc. This is an avenue for further research.79,80

It is noteworthy that no included study identifying exceptionally good doctors made recommendations on how to use this resource. The substantial number of positive outliers are at times not mentioned in the text, only shown in the graph. The closest to an investigation of high performers was presented by Brown et al18 who found that in diabetes control, high performing doctors were more likely to be female and underperforming doctors’ patients were more likely to be male. Goodwin et al69 found that hospitalists’ patients’ length of stay did not affect other patient outcomes. In other words, hospitalists whose patients had shorter lengths of stay in hospital had the same outcome as patients of hospitalists who were underperformers, but no further investigation was undertaken. As one contributory factor to doctors’ performance, recent research has proposed that even health care provider expectations can have a substantial placebo effect on patient outcomes, ie patient outcomes can be affected through “social transmission”.13

Many of the publications excluded for this systematic review among the approximately 10,000 studies were large-scale cohort studies where doctors’ effects were attributed to one or more characteristics. However, this attribution was done without reporting the variation in patients’ physical health outcomes that was due to the doctor after accounting for all known risk factors. It would be relatively simple to re-analyze these and other already cleaned up and prepared datasets for such a residual effect. Publishing ICCs, ie the amount of variation due to doctors in a consistent way, will make future meta analyses possible. The authors have prepared a methodological review for this purpose.81

To the authors’ surprise, re-analyzing existing data is not useful for many randomized controlled trials as no data register the authors contacted had any way to identify trials that showed a doctors’ effect. Further, a clinical trial specialist privately told the authors that in large randomized trials, with many treatment centers, only the center identifier and not the individual doctor identifier is recorded, making it difficult or impossible to extract a doctors’ effect from the data even though it would substantially affect the sample size needed for clinical trials when there are differences among medical doctors, as this would subsequently affect the RCTs statistical power.82

Research that addresses the doctors’ effect on patients’ health outcomes seems to be a form of investigation that is in its infancy. There are no established standards on how to report a doctors’ effect, and results are heterogeneous indeed.

The authors found very little systematic research on the probability that doctors, in their own right, may be an intervention whose effect on patients’ outcomes can be measured and be more or less effective. This is surprising since there is a well-known clustering effect with patients who have the same doctor tending to have more similar outcomes than patients of a different doctor.31,83 Likewise, it is well established in psychotherapy that psychotherapists, in their own right, can constitute an intervention, which is independent of the actual intervention used.7,84

Summary

Given the increasing difficulty with identifying effective new interventions85–87 and the increasing cost of research, it may be worth looking beyond the intervention to the other two components of a medical treatment, viz. the doctor and the patient. If there are substantial differences between doctors in patients’ physical health outcomes, then identifying those doctors who perform well below or well above average could be a relatively simple way to increase the standards of healthcare. This could be done by bringing low performers closer to average and by learning from high performers, which could provide improved healthcare at a relatively low cost. It would certainly be another option for policy makers: to improve the performance in their healthcare system beyond evaluating existing and potential new interventions for suitability.

Once outstanding performers have been identified,16,18,37,58,59,61,72 it may be possible to have them as role models, mentors, or teachers of other practitioners. Current literature considers standards to still be elusive88 and identifies outstanding teachers of medicine by acclaim rather than any objective standards.89 Once identified, excellent role models have been associated by Wright et al89 as “stressing the importance of the doctor-patient relationship in one’s teaching and teaching the psychosocial aspects of medicine” – ie they stress the doctor-patient relationship aspects that go beyond identifying and applying the intervention. Other characteristics may have contributed to exceptional performances, such as their ability to employ easy-to-emulate techniques like putting the patient at ease, willingly listen to the patient to the end, a harmonious lifestyle, a strong sense of purpose, or that they are very rarely exhausted, or have higher expectations of the effectiveness of their intervention,13 or any of a myriad of other possibilities.

The benefit of research investigating outstanding performers could be large as the differences between exceptional and average performers may be substantial, when simple choices made, or techniques used at work or out of work, that contributed to the outstanding performance then become available to other practitioners. As an exceptional performer is often no more expensive to employ than an average or below average performer, there could be very substantial beneficial effects on public health if many other doctors are given the possibility to improve.

Previous attempts at improving standards of care through profiling have run into difficulties. Krein et al40 in 2002 argued that despite large profiling campaigns of individual healthcare providers in order to contain costs and improve quality of care, the evidence of effecting change that way has been mixed, expensive, adversely affected careers, tended to ignore the systems the healthcare providers worked in, and, when done badly, profiling can be meaningless, providing incentives that worsen the quality of care.

A word of caution is that in a number of studies the raw patient physical health outcome numbers showed very large differences between doctors but this difference was strongly reduced or even eliminated after taking into account other factors such as patient risk or patients’ demographics.57,60,63 Even after a risk assessment it may be clear that many members of the worst performing group of doctors produce substandard work but the data available lacks statistical power and precludes identifying individuals with certainty. In such a case, disciplining or evicting individual practitioners may not be justifiable without further investigation. However, the more available data there is for each practitioner, the higher the possibility to misuse such data or to disempower practitioners by limiting the opportunity to use their ability and experience or by adding more and more rules and regulations.

Conclusions and Implications

Doctors have an effect on patients’ physical health for many interventions and outcomes and after accounting for all known data such as doctor demographics and patient risk. This effect ranges from negligible to substantial and therefore, it is worth investigating further whether these effects and their scale persist for other medical specialties and interventions, which at present is not clear due to the small number of studies found and the lack of consistency in their measurements. Many available RCTs and cohort studies could be reanalyzed to address and estimate the doctors’ effects.81 When grading doctors by patients’ physical health outcomes, it is at times possible to identify positive and negative outliers whose confidence interval ranges wholly above or below the average performance. Therefore, it can matter greatly which doctor is chosen.

Population	P	Medical Doctors That are Not Surgeons
Intervention	I	Any
Comparison	C	Not applicable
Outcome	O	Practitioners’ effect on patients’ physical health outcome