Is There a Surgeons' Effect on Patients' Physical Health, Beyond the Intervention, That Requires Further Investigation? A Systematic Review.

Objective: To find and review published papers researching surgeons' effects on patients' physical health. Clinical outcomes of surgery patients with similar prognoses cannot be fully explained by surgeon skill or experience. Just as there are "hospital" and "psychotherapist" effects, there may be "surgeons" effects that persist after controlling for known variables like patient health and operation riskiness.
Methods: Cohort studies and randomized controlled trials (RCTs) of any surgical intervention, which, after multivariate adjustment, either showed proportion of variance in patients' physical health outcomes due to surgeons (random effects) or graded surgeons from best to worst (fixed effects). Studies with <15 surgeons or only ascribing surgeons' effects to known variables excluded. Medline, PubMed, Embase, and PsycINFO were used for search until June 2020. Manual search for papers referring/referred by resulting studies. Risk of bias assessed by Cochrane risk-of-bias tool and Newcastle-Ottawa Scale.
Results: Included studies: 52 cohort studies and three RCTs of 52,436+ surgeons covering 102 outcomes (33 unique). Studies either graded surgeons from best to worst or calculated the intra-class correlation coefficient (ICC), the percentage of patients' variation due to surgeons, in diverse ways. Sixteen studies showed exceptionally good and/or bad performers with confidence intervals wholly above or below the average performance. ICCs ranged from 0 to 47%, median 4.0%. There are no well-established reporting standards; highly heterogeneous reporting, therefore no meta-analysis. Discussion: Interpretation: There is a surgeons' effect on patients' physical health for many types of surgeries and outcomes, ranging from small to substantial. Surgeons with exceptional patient outcomes appear regularly even after accounting for all known confounding variables. Many existing cohort studies and RCTs could be reanalyzed for surgeons' effects especially after methodological reporting guidelines are published.
Conclusion: In terms of patient outcomes, it can matter which surgeon is chosen. Surgeons with exceptional patient outcomes are worth studying further.

Introduction

What is already known on this topic: Previous research has shown associations between characteristics of surgeons, such as their level of surgical experience, and patient health outcomes. It is unclear whether surgeons have an influence on patients’ physical health that has not been captured by known variables and how large that influence is.

What this study adds: This study is the first systematic review of unexplained surgeons’ influence on patients’ physical health. Findings are highly variable, depending on the type of outcome and surgery that can result in substantial differences in patient health outcomes between surgeons.

Rationale

If you want to find a good surgeon, an internet query will provide advice from many sources.3–5 There are also databases of what the database provider considers to be the best surgeons, calculated from raw death and complication rates plus other doctors’ recommendations.6 Surgeons themselves have given their opinion on what makes a good or outstanding surgeon,7–11 with Barry Jackson’s essay perhaps being the most comprehensive.12 However, this information mostly relies on personal experiences, although Jackson’s essay does mention “First-class outcomes after allowing for case-mix”. Existing evidence suggests that some surgeons are more effective at applying interventions than others as there is, for example, a substantial volume effect, ie case volume, and years of practice effect in a number of surgical specialties.13,14 In fact, there are few studies where author-selected outstanding practitioners are investigated,15–17 with only Schenck et al mentioning surgeons.

It is well established that there is a hospital effect, ie that hospitals have a substantial influence on patients’ health outcomes and that there are wide variations in patients’ health outcomes between hospitals.18–23 There is also substantial research on a therapist effect in psychotherapy with wide variations among therapists, so much so that this finding has made it into training material for psychotherapists.24,25 Recent research also suggests that provider expectations could have a causal role in treatment effectiveness.26 At the same time, the placebo effect, which can be substantial,27 including in surgery28–31 with some dissent for orthopedic surgery,32 is suggestive of a surgeon’s effect. The placebo effect shows that even with an inert or inactive intervention, there is an effect on patients. It is possible that part of this effect is due to the surgeon administering the placebo, usually a type of sham surgery.28 However, there are currently no well-established standards on how to assess surgeons as an intervention in their own right or as an effect modifier of a given intervention. Recent research has endeavored to analyze the effect size of surgeons33 by investigating 10 surgical trials for surgeon intra-cluster correlation coefficients (ICCs) and found that surgeons have a range of effects on patient health that differ between surgical specialties.

In the study by Udyavar et al34 of 2149 surgeons performing 569,767 emergency surgeries it was shown that in five out of seven types of surgery, surgeons were responsible for 23% to 47% of the variability in patient mortality. This difference in outcome could not be explained by the choice of treatment, prognostic or diagnostic factors, patient clinical or demographic factors, hospital-level factors, or surgeon volume. To date studies such as Udyavar et al have not been synthesized. In this systematic review we have addressed this gap in the literature.

Objectives

This systematic review aims to identify and evaluate all the research to date examining the effect of surgeons on patient physical health outcomes after known variables have been accounted for. It is part of a larger research project that includes a systematic review of non-surgical practitioners, and a methodological study on how to report practitioners’ effects on patients’ physical health.

Methods

Eligibility Criteria

A systematic review was conducted following Synthesis without meta-analysis (SWiM) guidelines.35 This review limits itself to studies that investigated actual patients’ physical outcomes and excluded studies that focused on patients’ opinions or satisfaction levels, with the rationale that these outcomes are often a more ambiguous way to measure surgeons’ effects.36

The PICO is as follows:

Information Sources and Search Strategy

We initially searched three databases: Medline via its PubMed interface, Embase, and PsycINFO from inception to June 2020 to identify relevant studies that investigate the influence of surgeons on patients’ physical health outcomes. The search strategy used for each database is reported in and was designed by JMC, a specialist in this area. In addition to the electronic search of databases, we further manually searched the references lists of the eligible articles and previous systematic reviews to identify potentially relevant studies that did not appear in the literature search. The following systematic review registries were searched for similar reviews: PROSPERO and Cochrane’s CENTRAL register. One study was suggested by a reviewer.37

Selection Process and Further Eligibility Criteria

Two reviewers independently screened titles/abstracts for inclusion. Any discrepancies were resolved by discussion or in consultation with a third reviewer.

Study designs considered for inclusion were retrospective and prospective observational studies, case-control studies, and randomized controlled studies, where either the proportion of variance in patient outcomes explained by differences between practitioners, ie practitioners’ random effects, are measured, or the difference between the individual practitioners is highlighted, ranging from best to worst, ie practitioners’ fixed effects are measured. Any medical practitioner except psychotherapists were included. At this stage both surgeons and doctors who were not surgeons were included, however this paper only includes studies of surgeons. All other medical doctors are reviewed in a separate paper.

Any patient’s physical health-related outcome was eligible, examples of which are repair reoperations, readmission rate, survival/mortality rate, embryo transfer rate, length of hospital stay, infection rate, estimated blood loss, recurrence rates, pain, and other post-operative complications. There were no date or language restrictions.

We excluded studies that only ascribed a surgeons’ effect to particular surgeon-related variables, such as volume of procedures performed or specialty of surgeon; studies with fewer than 15 surgeons; cross-sectional studies, ie surveys of doctors or patients, as they had an increased risk of bias; and two studies that mentioned fixed or random effects but did not actually list the effects either graphically or in numerical form.38,39

The authors could not find a recommendation for the minimum number of clusters in a study for a systematic review – in this case the minimum number of practitioners. We took 15 practitioners as the smallest cluster size but appreciate that this is an arbitrary number. (Figure 1).

Figure 1

Flow diagram of selection of included documents.

Data Collection Process and Data Items

Titles and abstracts were collected using Endnote 9 and uploaded into Rayyan for inclusion or exclusion where the two reviewers independently screened titles and abstracts. The resulting eligible studies were marked as members of a group in the original Endnote library and their full text documents were added to the library.

CS and a second extractor independently and in duplicate extracted the relevant data from each eligible study and collected the following variables using Excel:

Unique publication identifier consisting of first author and year

Surgical specialty

Type of study (RCT, Cohort)

Type of intervention (can be multiple)

Outcome type (multiple)

Significant surgeons’ effect as per authors’ evaluation Y/N

Number of surgeons

Number of patients or procedures

Number of hospitals/institutions

ICC (intra-class correlation coefficient) Number/NS

Multivariate analysis Y/N

Number of negative and positive outliers

Country of origin

Study Risk of Bias Assessment

Two reviewers independently used the Cochrane risk-of-bias assessment tool2 for the three included randomized controlled studies (Figures 2 and 3), and the Newcastle-Ottawa Scale (NOS) for the included cohort studies.40,41

Figure 2

First risk of bias chart for the three randomized controlled trials included.

Figure 3

Second risk of bias chart for the three randomized controlled trials included.

Effect Measures

The metric for the fixed effects is the percentage of positive and negative outliers as defined in the individual study reports. The metric for the random effects is the variance due to the practitioner or the intra-class correlation coefficient, defined as the variation in patient outcome due to the practitioner as a percentage of the total variation.

Synthesis Methods

As the data are highly heterogenous and there are no established standards on recording doctors’ effects or surgeons’ effects, no statistical synthesis was used. There were 14 surgical specialties plus two papers covering multiple surgeries, 50 separate interventions and 31 separate outcomes.

The surgeons’ effect on patients’ physical health is described in two ways, using multilevel mixed effects regression modelling or hierarchical regression to understand both surgeon and system-level variation.42,43

Percentage of Variation in Patient Outcome Due to the Surgeon in the Form of the Intra-Class Correlation Coefficient (ICC)

Post-regression estimation gives the ICC, which as a number ranging from 0 to 1, gives the percentage of variation in outcome due to each level in the regression model. For example, in a three-level model of patients clustered per doctor, who in turn were clustered within hospitals, each level has an ICC with the total ICCs adding up to 1. In order to realize this, the studies included random effects for surgeons, and at times hospitals or other aggregators, such as county.

Patient risk scores and other available variables like surgeon demographic data or year of intervention were included as fixed effects in the regression analysis. The quality and depth of the analysis varied greatly between papers. Confidence intervals for the ICC were not reported.44 A high quality study is Papachristofi et al.45 There is also substantial other research on the ICC.46–51

Grading Surgeons from Best to Worst

In this approach surgeons are ranked by their patient results, usually with a 95% confidence interval and either the raw, unadjusted scores are reported, or patient risk scores and/or surgeon demographic variables and other data, such as year of operation, are included in the model. In the majority of cases the method to calculate the interval is not mentioned, though there are exceptions52–59 and surgeons whose 95% confidence intervals rank wholly above or below the mean rate of outcomes are considered to be outliers. Reporting is done by listing the count of outliers, or graphically through a caterpillar or a funnel plot,60 with a caterpillar plot being an outcome-ordered forest plot.

Reporting Bias and Certainty Assessment

Due to there being no synthesis, reporting bias and certainty assessments were not undertaken.

Results

Study Selection

Overall, 4713 records were identified from electronic records, in addition to 6461 from other sources. After removing the 1224 duplicates, 10,239 studies underwent screening for eligibility. Then, full-text versions were retrieved for 471 records. One study was added by a reviewer. Finally, after exclusion of ineligible articles, 55 studies of more than 52,436 surgeons were included in the final synthesis.

Study Characteristics

The 55 studies that are included reported 102 outcomes, 33 of which are unique. Of the outcomes, 28 (20 studies)55–57,59,61–76 graded individual surgeons’ performance from best to worst; 38 (12 studies)34,77–87 recorded an ICC due to surgeons in a multivariate multi-level analysis; 14 (8 studies)44,45,53,58,88–91 recorded both; 20 (13 studies)37,52,92–102 provided a non-standard description of fixed effects; and 1 provided an ICC plus a non-standard description of fixed effects.103 One study104 graded surgeons from best to worst in one outcome (complications) and used a non-standard fixed effects description for another outcome (mortality).

Of the 55 studies, three were randomized controlled trials,37,80,93 and 52 were observational cohort studies. The studies included various surgical specialties or aggregates thereof, including 8 or more specialties,81,91 breast surgery,59,73,74,101 cardiac surgery,44,45,52,53,56,65–67,70,89,90,99,100 colorectal surgery,34,61,68,71,78,85,92,103,104 ENT surgery,75 gastrointestinal surgery,34,83 general surgery,34,37,57,77,79,82,86,93,95,96,102 obstetrics,80 ophthalmology,97 orthopedic surgery,55,76,84,102 rectal surgery,72,98 spinal surgery,58 trauma surgery,87 and urology.62–64,69,88,94,102 38 studies were conducted in the USA, 10 in the UK, two in Austria and Sweden, one each in Canada, France, and Germany. The volume of included surgeons ranged from 17 to 14,598. The characteristics of the included studies are summarized in Table 1.

Table 1

Characteristics of Included Studies

Publication	Specialty	Detailed Intervention	Surgeons	Patients/Procedures	Institutions	Outcome	NOS **
Anderson, 201652	Cardiac surgery	Norwood operation	NS	2880	35	Mortality	9
Aquina, 2015a92 pg e163	Colorectal surgery	Colorectal resection	NS	158,596	NS	C. difficile infection	9
Aquina, 2015b61	Colorectal surgery	Upper GI cancer resection	223	14,875	99	Blood transfusion, wound infection, pneumonia, sepsis	9
Aquina, 2016103	Colorectal surgery	Colorectal resection	3481	125,160	210	Blood transfusion, wound infection, pneumonia, sepsis, intra-abdominal abscess	9
Aquina, 201777	General surgery	Inguinal hernia operation	1572	124,416	260	Reoperation	9
		Ventral hernia operation	2012	78,267	256	Reoperation	9
Arvidsson, 200593	General surgery	Hernia operation	25	1068	7	Recurrence	RCT
Becerra, 201778	Colorectal surgery	Lymph node examination in colectomy	1503	12,332	187	Suboptimal care	9
Begg, 200262	Urology	Radical prostatectomy	159	10,737	72	Postoperative complications	9
						Incontinence	9
						Late urinary complications	9
						Mortality	9
Bianco, 200563	Urology	Radical prostatectomy	159	5238	NS	Complications	9
						Incontinence	9
						Late urinary complications	9
Bianco, 201064	Urology	Radical prostatectomy	54	7725	4	Cancer recurrence	9
Bolling, 201065	Cardiac surgery	Mitral valve repair/replacement	1088	28,507	639	Mitral valve repair rates	9
Bridgewater, 200367	Cardiac surgery	Coronary artery surgery	23	8572	4	Mortality	9
Bridgewater, 200566	Cardiac surgery	Aortic valve surgery	25	1097	4	Mortality	8
		Coronary artery surgery	25	9066	4	Mortality	8
Burns, 201168	Colorectal surgery	Colorectal surgery	1557	246,469	156	Reoperation	9
Cromwell, 201369	Urology and Gynecology	Urinary-genital tract fistula	490	1194	129	Reoperation	8
Dagenais, 201988	Urology	Partial nephrectomy	19	1461	1	Estimated blood loss	9
Duclos, 201279	General surgery	Thyroid surgery	28	3574	5	Hypoparathyroidism	9
						Recurrent laryngeal nerve palsy	9
Eastham, 200394	Urology	Radical prostatectomy	44	4629	2	Positive surgical margins	9
Eklund, 200937	General surgery	Inguinal hernia repair surgery	48	1275	NS	Recurrence	RCT
Faschinger, 201195	General surgery	Cataract surgery	17	36,329	1	Capsule rupture	9
Fountain, 200480	Obstetrics	Hysterectomy, Abdominal	43*	876	28*	Complications	RCT
		Hysterectomy, Vaginal	43*	504	28*	Complications	RCT
Gani, 201581	[See on right]	8 (cardiac, GI* surgery, trauma, HPB*, BME*, thoracic, transplant, vascular)	56	22,559	1	Readmission	9
Glance, 200653	Cardiac surgery	Cardiac surgery	138	51,750	33	Mortality	9*
Glance, 201689	Cardiac surgery	CABG*	241	55,436	40	Major complications or mortality	9
Grant, 200870	Cardiac surgery	Cardiac surgery	31	14,637	4	Mortality	9
Healy, 201771	Colorectal surgery	Minimally invasive colectomy	97	3118	46	Complications	8
		Open colectomy	97	2078	46	Complications	8
Hermanek, 199972	Rectal surgery	Rectal carcinoma resection	43	1121	7	Mortality	9
Hermann, 200296	General surgery	Primary surgery for benign thyroid disease	20	16,443	1	Recurrent laryngeal nerve injury (RLNI)	8
Hoffman, 201782	General surgery	General surgery	1128	183,283	601	Complications	9
Huesch, 200956	Cardiac surgery	CABG*	398	221,327	75	Mortality	8
Hyder, 201383	Gastrointestinal surgery	Pancreatoduodenectomy	575	1488	298	Readmission	9
Johnston, 201097	Ophthalmologist	Cataract surgery	404	55,515	12	Posterior capsule rupture (PCR)	8
Justiniano, 201998	Rectal surgery	Rectal carcinoma resection	345	1251	118	Mortality	9
Kaczmarski, 201973	Breast surgery	Breast-conserving surgery	5337	291,065	NS	Reoperation	9
Kissenberth, 201884	Orthopedic surgery	Rotator cuff repair	57	1703	NS	Single Assessment Numeric Evaluation (SANE) score	8
Landercasper, 201974	Breast surgery	Breast-conserving surgery	71	3954	NS	Reoperation	9
LaPar, 201499	Cardiac surgery	Mitral valve repair/replacement	93	4194	17	Lack of repair	8
Likosky, 2012100	Cardiac surgery	CABG*	32	11,838	8	Postoperative low-output failure	9
Luan, 201957	General surgery	Bariatric surgery	38	1277	21	Complications	9
Martin, 201358	Spinal surgery	Lumbar fusion	298	6091	43	Complications	9
Martin, 201358	Spinal surgery	Lumbar fusion	298	6091	43	Reoperation	9
McCahill, 201259	Breast surgery	Breast-conserving surgery	54	2206	4	Reoperation	9
Papachristofi, 201490	Cardiac surgery	Cardiac surgery	18	18,426	1	Mortality	9
Papachristofi, 201644	Cardiac surgery	Cardiac surgery	127	110,769	10	Mortality	9
Papachristofi, 201745	Cardiac surgery	Cardiac surgery	127	107,038	10	Length of stay	9
Quinn, 201891	All surgeries	All surgeries	2724	123,141	51	Any morbidity	9
						Death or serious morbidity	9
						Mortality	9
						Readmission	9
						Reoperation	9
						Surgical site infection	9
Rudmik, 201775	ENT surgery	Endoscopic sinus surgery	43	2168	NS	ESS* revision rate	9
Schumacher, 2017101	Breast surgery	Breast-conserving surgery	93	3470	56	Reoperation	7
Shih, 201585	Colorectal surgery	Colectomy	345	5033	24	Complications	9
Singh, 201876	Orthopedic surgery/Neurosurgery	Spine surgery	3987	39,884	NS	Length of stay	8
						Readmission	8
Thigpen, 201855	Orthopedic surgery	Rotator cuff repair	34	995	1	ASES* performance score	8
Udyavar, 2018a34	Colorectal surgery	Colectomy	2149*	569,767*	225*	Complications	9
						Mortality	9
						Readmission	9
	Gastrointestinal surgery	Peptic ulcer disease	2149*	569,767*	225*	Complications	9
						Mortality	9
						Readmission	9
		Small bowel resection	2149*	569,767*	225*	Complications	9
						Mortality	9
						Readmission	9
	General surgery	Appendectomy	2149*	569,767*	225*	Complications	9
						Mortality	9
						Readmission	9
		Cholecystectomy	2149*	569,767*	225*	Complications	9
						Mortality	9
						Readmission	9
		Laparotomy	2149*	569,767*	225*	Complications	9
						Mortality	9
						Readmission	9
		Lysis of adhesions	2149*	569,767*	225*	Complications	9
						Mortality	9
						Readmission	9
		Overall	2149	569,767	225	Complications	9
						Mortality	9
						Readmission	9
Udyavar, 2018b87	Trauma surgery	Trauma surgery	175	65,706	31	Mortality	9
Udyavar, 201986	General surgery	Emergency surgery	5816	215,745	198	Complications	9
Xu, 2016104	Colorectal surgery	Colectomy	276	2525	44	Complications	9
						Mortality	9
Xu, 2019102	General surgery	Laparoscopic cholecystectomy	2476	1,884,842*	NS	Complications	8
	Urology	Other transurethral prostatectomy	1663*	1,884,842*	NS	Complications	8
		Radical prostatectomy	1663*	1,884,842*	NS	Complications	8
	Orthopedic surgery	Cervical spinal fusion	10,459*	1,884,842*	NS	Complications	8
		Lumbar spinal fusion, anterior column	10,459*	1,884,842*	NS	Complications	8
		Lumbar spinal fusion, posterior column	10,459*	1,884,842*	NS	Complications	8
		Total hip arthroplasty	10,459*	1,884,842*	NS	Complications	8
		Total knee replacement	10,459*	1,884,842*	NS	Complications	8

Notes: *The values are for the whole study population. Values for each subgroup were not reported.

Abbreviations: **ASES, American Shoulder and Elbow Surgeons; BME, breast, melanoma, and endocrine surgery; CABG, coronary artery bypass grafting; ESS, endoscopic sinus surgery; EQ-5D, quality of life via the Euro-Qol; GI, gastro-intestinal; HPB, hepatopancreatic biliary; NOS, Newcastle-Ottawa Scale for assessing risk of bias of cohort studies; NS, not stated, the number is not given and most likely greater than one.

Risk of Bias in Studies

For the cohort studies, of 97 outcomes in 52 studies, (1) scored 7 stars, (21) 8 stars and (75) 9 stars out of a maximum of 9 stars on the Newcastle-Ottawa Scale.40,41 All studies scored the maximum points on the selection criteria and the outcome criteria. Those with 7 and 8 stars scored either 0 or 1 on comparability while the 9-star studies scored 2 (Table 1). The detailed risk of bias assessment of the three randomized controlled trials, using Cochrane RoB, is described in Figures 2 and 3, and .

Results of Individual Studies

Altogether 10 studies published caterpillar plots59,61,64,67,71,74–76,89,91 and five studies presented funnel plots.65,66,68–70 The plots showed the performance of surgeons for a particular patient outcome, usually sorted by performance, providing a 95% confidence interval for each surgeon and indicating whether that confidence interval was wholly above or below the average performance. Results ranged from no over- or underperformer62,66,67,69,70,91 to substantial numbers of both.59,61,62,64,72,73,75,76

Of the papers that reported fixed effects, 15 recorded exceptional performers after taking account of all known variables, including demographic variables of the practitioners, such as experience, volume of patients/procedures, and hospital effects (which themselves can be substantial).55,59,61–66,68,71,73–76,91 Other studies (n=22) published a random effect, worded many different ways, that showed the Intra-Class Correlation (ICC) effect.34,44,45,53,57,58,77–91,103 The random effects reported ranged from zero (ICC of 0.0%) to substantial (ICC of 10% or higher). (Tables 2 and 3, and Figures 4 and 5).

Table 2

Publications by Outcome and Numerical Results

	Outcome	Specialty	Detailed Intervention	Publication	ICC^	Outliers %
						Negative	Positive
Complications	Any morbidity	All surgeries	All surgeries	Quinn, 201891	2.2%	0.18%	0.22%
	Blood transfusion, wound infection, pneumonia, sepsis	Colorectal surgery	Upper GI cancer resection	Aquina, 2015b61		13.0%	28.0%
	Blood transfusion, wound infection, pneumonia, sepsis, intra-abdominal abscess	Colorectal surgery	Colorectal resection	Aquina, 2016103	24.3%	Other	Other
	C. difficile infection	Colorectal surgery	Colorectal resection	Aquina, 2015a92 pg e163		Other	Other
	Capsule rupture	General surgery	Cataract surgery	Faschinger, 201195		Other	Other
	Complications (postoperative)	Colorectal surgery	Colectomy	Shih, 201585	14.0%
				Udyavar, 2018a34	2.3%
				Xu, 2016104		3.3%	NS
			Minimally invasive colectomy	Healy, 201771		10.3%	7.2%
			Open colectomy	Healy, 201771		9.3%	5.2%
		Gastrointestinal surgery	Peptic ulcer disease	Udyavar, 2018a34	0.03%
			Small bowel resection	Udyavar, 2018a34	0.02%
		General surgery	Appendectomy	Udyavar, 2018a34	0.2%
			Bariatric surgery	Luan, 201957		2.6%	15.8%
			Cholecystectomy	Udyavar, 2018a34	0.1%
			Emergency surgery	Udyavar, 201986	27.3%
			General surgery	Hoffman, 201782	6.2%
			Laparoscopic cholecystectomy	Xu, 2019102		Other	Other
			Laparotomy	Udyavar, 2018a34	0.1%
			Lysis of adhesions	Udyavar, 2018a34	0.0%
			Overall (Emergency general surgeries)	Udyavar, 2018a34	0.1%
		Obstetrics	Hysterectomy, Abdominal	Fountain, 200480	7.4%
			Hysterectomy, Vaginal	Fountain, 200480	0.5%
		Spinal surgery	Lumbar fusion	Martin, 201358	2.6%	3.7%	0.0%
		Urology	Other transurethral prostatectomy	Xu, 2019102		Other	Other
			Radical prostatectomy	Begg, 200262		8.0%	3.0%
				Bianco, 200563 *		7.5%	2.5%
				Xu, 2019102		Other	Other
		Orthopedic/Neuro-surgery	Cervical spinal fusion	Xu, 2019102		Other	Other
			Lumbar spinal fusion, anterior column	Xu, 2019102		Other	Other
			Lumbar spinal fusion, posterior column	Xu, 2019102		Other	Other
			Total hip arthroplasty	Xu, 2019102		Other	Other
			Total knee replacement	Xu, 2019102		Other	Other
	Death or serious morbidity	All surgeries	All surgeries	Quinn, 201891	2.0%	0.15%	0.15%
	Estimated blood loss	Urology	Partial nephrectomy	Dagenais, 201988 ##	14.4%	10.5%	10.5%
	Hypoparathyroidism	General surgery	Thyroid surgery	Duclos, 201279	32.0%
	Incontinence	Urology	Radical prostatectomy	Begg, 200262		9.0%	3.0%
				Bianco, 200563 *		9.4%	2.5%
	Late urinary complications	Urology	Radical prostatectomy	Begg, 200262		13.0%	14.0%
				Bianco, 200563 *		13.2%	14.5%
	Major complications or mortality	Cardiac surgery	Cardiac surgery	Glance, 201689	1.76%	3.3%	1.7%
	Posterior capsule rupture (PCR)	Ophthalmologist	Cataract surgery	Johnston, 201097		Other	Other
	Recurrent laryngeal nerve injury (RLNI)	General surgery	Primary surgery for benign thyroid disease	Hermann, 200296		Other	Other
	Recurrent laryngeal nerve palsy	General surgery	Thyroid surgery	Duclos, 201279	10.0%
	Surgical site infection	All surgeries	All surgeries	Quinn, 201891	4.5%	0.29%	0.07%
Length of stay		Cardiac surgery	Cardiac surgery	Papachristofi, 201745	2.79%	11.8%	14.2%
		Orthopedic/Neuro-surgery	Spine surgery	Singh, 201876		10.0%	7.2%
Mortality		All surgeries	All surgeries	Quinn, 201891	1.4%	0.0%	0.0%
		Cardiac surgery	Aortic valve surgery	Bridgewater, 200566 *		0.0%	0.0%
			CABG	Huesch, 200956 ***		1.2%	Other
				Glance, 200653	5.9%	3.3%	8.7%
				Grant, 200870 *		0.0%	0.0%
			Cardiac surgery	Papachristofi, 201490	2.79%	16.7%	0.0%
				Papachristofi, 201644	4.0%	15.0%	6.3%
			Coronary artery surgery	Bridgewater, 200367		0.0%	0.0%
				Bridgewater, 200566 *		0.0%	16.0%
			Norwood operation	Anderson, 201652		Other	Other
		Colorectal surgery	Colectomy	Udyavar, 2018a34	22.9%
				Xu, 2016104		Other	Other
		Gastrointestinal surgery	Peptic ulcer disease	Udyavar, 2018a34	47.3%
			Small bowel resection	Udyavar, 2018a34	23.1%
		General surgery	Appendectomy	Udyavar, 2018a34	6.9%
			Cholecystectomy	Udyavar, 2018a34	3.5%
			Laparotomy	Udyavar, 2018a34	33.2%
			Lysis of adhesions	Udyavar, 2018a34	35.5%
			Overall (Emergency general surgeries)	Udyavar, 2018a34	32.7%
		Rectal surgery	Rectal carcinoma resection	Hermanek, 199972		9.3%	16.3%
				Justiniano, 201998		Other	Other
		Trauma surgery	Trauma surgery	Udyavar, 2018b87	8.7%
		Urology	Radical prostatectomy	Begg, 200262		0.0%	0.0%
Readmission		8 (Cardiac, GIS, Trauma, HPB, BME, Thoracic, Transplant, Vascular)	8 (Cardiac, GIS, Trauma, HPB, BME, Thoracic, Transplant, Vascular)	Gani, 201581	2.8%
		All surgeries	All surgeries	Quinn, 201891	0.7%	0.0%	0.0%
		Colorectal surgery	Colectomy	Udyavar, 2018a34	3.1%
		Gastrointestinal surgery	Pancreatoduodenectomy	Hyder, 201383	0.3%
			Peptic ulcer disease	Udyavar, 2018a34	6.8%
			Small bowel resection	Udyavar, 2018a34	2.9%
		General surgery	Appendectomy	Udyavar, 2018a34	3.5%
			Cholecystectomy	Udyavar, 2018a34	3.0%
			Laparotomy	Udyavar, 2018a34	6.0%
			Lysis of adhesions	Udyavar, 2018a34	4.9%
			Overall (Emergency general surgeries)	Udyavar, 2018a34	2.3%
Reoperation	Reoperation	All surgeries	All surgeries	Quinn, 201891	3.8%	0.04%	0.0%
		Breast surgery	Breast-conserving surgery	Kaczmarski, 201973		17.5%	3.7%
				Landercasper, 201974		5.7%	4.3%
				McCahill, 201259		13.0%	31.5%
				Schumacher, 2017101		Other	Other
		Colorectal surgery	Colorectal surgery	Burns, 201168 **		0.7%	4.5%
		General surgery	Hernia operation	Arvidsson, 200593		Other	Other
			Inguinal hernia operation	Aquina, 201777	40.5%
			Ventral hernia operation	Aquina, 201777	14.0%
		Spinal surgery	Lumbar fusion	Martin, 201358	9.0%	#	#
		Urology	Urinary-genital tract fistula	Cromwell, 201369 **		0.0%	0.0%
Suboptimal care		Colorectal surgery	Lymph node examination in colectomy	Becerra, 201778	7.9%
Success or failure	ASES score	Orthopedic surgery	Rotator cuff repair	Thigpen, 201855		5.9%	8.8%
	Cancer recurrence	Urology	Radical prostatectomy	Bianco, 201064		8.3%	36.1%
	ESS revision rate	ENT surgery	Endoscopic sinus surgery	Rudmik, 201775		16.3%	4.7%
	Mitral valve repair rates	Cardiac surgery	Mitral valve repair/replacement	Bolling, 201065		6.6%	7.4%
				LaPar, 201499		Other	Other
	Positive surgical margins	Urology	Radical prostatectomy	Eastham, 200394		Other	Other
	Postoperative low-output failure	Cardiac surgery	CABG	Likosky, 2012100		Other	Other
	Readmission	Orthopedic/Neuro-surgery	Spine surgery	Singh, 201876		0.1%	0.03%
	Recurrence	General surgery	Inguinal hernia repair surgery	Eklund, 200937		2.1%
Score	ASES^ score	Orthopedic surgery	Rotator cuff repair	Thigpen, 201855		5.9%	8.8%
	Single Assessment Numeric Evaluation (SANE) score	Orthopedic surgery	Rotator cuff repair	Kissenberth, 201884	44.0%

Notes: ^ASES score is American Shoulder and Elbow Surgeons (ASES) performance score; ICC is intra-class correlation coefficient and shows percentage of variance due to practitioner as percentage of total variance after accounting for all known variables. Outliers are listed for papers where the surgeons were ordered in their effect on patients’ physical health from best to worst or vice versa. The percentages listed are those practitioners whose 95% confidence interval is wholly below or above the mean. Outliers listed as “Other” sorted their surgeons by physical patient effect but used a different way to present their data. Common examples are a caterpillar plot without confidence intervals or a bar chart. *99% confidence interval to define outliers used. **99.8% confidence interval to define outliers used. ***90% confidence interval to define outliers used. #Graph too small to calculate positive or negative outliers. ##Dagenais et al88 also shows precisely 0.00 between-surgeon variance for length of stay, glomerular filtration rate (GFR) preservation, positive margins, chronic kidney disease (CKD) upstaging, Clavien grade ≥ 1 complications, and 30-day readmission. Operative time had an ICC of 33.4%.

Table 3

ICC Summary Statistics

ICC n=53 Outcomes
Minimum	0.001%
Maximum	47.3%
Average	10.2%
Median	4.0%
IQR	2.2–14.0%
Standard deviation	0.13

Figure 4

Boxplot of ICC (intra-class correlation coefficient).

Figure 5

Intra-class correlation coefficients (ICCs) by paper, intervention, and patient outcome.

Only for a. complications after colectomy and b. mortality after cardiac surgery was there more than one study included that reported an ICC. As these are the only outcomes with multiple ICCs, a more detailed analysis follows:

For colectomy, Shih et al85 reported an ICC of 14.0% and Udyavar et al34 an ICC of 2.3%. Udyavar defined complications as any of “pulmonary embolism, sepsis, myocardial infarction, acute renal failure, and cardiac arrest” while Shih defined a much longer list of items as complications, including surgical site infection; wound disruption; multiple types of infection; unplanned intubation; transfusion; multiple stroke or clotting diagnoses; multiple heart issues; renal complications or failure; extended coma or mechanical ventilation; nerve damage; failure of the graft or prosthesis; bowel obstruction; and anastomotic leak. For mortality after cardiac surgery three studies44,53,90 reported an ICC of 2.8% to 5.9% (Table 2).

Results of Syntheses, Reporting Biases and Certainty of Evidence

Not applicable as there was no synthesis.

Discussion

In this review, the objective was to determine whether there is a surgeons’ effect on patients' physical health that is apparent even after accounting for all known variables, such as level of experience. Included studies graded surgeons in order of performance or listed the proportion of variation that is due to practitioners after taking account of all known variables. All but three studies were cohort studies. The other three studies were randomized controlled trials. Findings showed substantial heterogeneity that may be related to type of surgery and type of outcome. After accounting for surgeons’ experience, patients’ risk, and all other known variables, there remained at times substantial differences in patients’ physical health outcomes between surgeons. More than a quarter of all studies (15 out of 55) showed high-volume outliers whose performance is well above the average. In contrast, there were types of surgery/intervention/outcome combinations that showed little evidence of a surgeons' effect on patients' physical health. These findings are somewhat consistent with the substantial body of research on a therapist effect in psychotherapy showing a wide variation in patient outcomes.

With two exceptions the authors only found one study per combination of surgical specialty, intervention and patient outcome. The first exception was two studies covering complications after colectomy and they had very different ICCs of 14.0%85 and 2.3%.34 It seems the much wider definition of “complication” in Shih led to a bigger influence of surgeons on the outcome and therefore a higher ICC. The second exception was for mortality after cardiac surgery with three studies44,53,90 reporting an ICC of 2.8% to 5.9%. Clearly, standardized definitions of physical patient outcomes would assist comparisons across studies.

A major limitation of the evidence identified in this review is that there is currently no standard way to report surgeons’ performance on patient's physical health. What does get reported can be divided into either grading individual surgeons by performance or calculating the percentage of variation in patients’ outcome that is due to the surgeon after all known variables have been taken into account. Both types of reporting are worded in many different ways, making discovery of such research difficult as can be seen in that more than 10,000 publications had to be reviewed.

A further limitation is that only for very few papers the primary purpose was to report surgeons’ performance after taking account of all known variables. Much of the time the reviewed publications’ authors emphasized other aspects of healthcare.

Summary

In terms of this systematic review, it was revealed that surgeons’ performance data on physical patient health is available to the authors of many published research studies. However, this data is in most cases either not at all reported or only in a limited way. This data could easily be included in an article prepared for publication as the data is already available and often requires minimal or no extra analysis to provide it in the format recommended in the methodological review that is reviewed for publishing. Publishing this data will also allow these studies to be part of future meta-analyses, gaining further dissemination of the work.

It seems that the possibility that surgeons are an intervention in their own right, an intervention that can be more or less effective and an intervention whose effect can be measured, is an area where there has been little systematic research. This is despite the fact that in psychotherapy it is well established that doctors (therapists) constitute an intervention in their own right, independent of the actual intervention they use.24,25

Furthermore, if the intervention is held constant, then surgeons are an effect modifier whose strength varies substantially depending on the intervention and the patients’ physical health outcome measured.

If it can be established when and how much surgeons constitute an intervention or a substantial effect modifier in their own right, independent of the intervention they use, then this opens up the possibility that this intervention (surgeons) can be systematically managed and improved to the benefit of patients, the surgeons themselves, and the entire health system.

None of the studies that identified outstanding surgeons61,64–66,68,71,74–76,89,91 made any recommendations on how to use this potential quality improvement resource. So far, we see little or no evidence in the literature that even when exceptional performers have been quantitatively identified, these exceptional performers are used as role models or as research subjects for qualitative research in order to find out what makes them exceptional.

A key point of this systematic review is that the authors specifically looked for studies that showed a surgeons’ effect for which there was no explanation, ie a residual effect after all known information had been included in the statistical analysis. Therefore, the cause of the surgeons’ effect measured is, by the definition of the research question for this systematic review, not known. This leaves open the question whether the cause is unknowable, or if there are one or more causes that could be identified in future research.

If we want to know what makes a good surgeon beyond the well-founded opinions of surgeons12 or those who work with surgeons – and how to train surgeons to be good surgeons – then the first step beyond all the current measures taken to train surgeons could be to reliably identify outstanding surgeons. Consequently, we can find out if their ability can be passed on to others and, if yes, to lift the overall standard of healthcare by transferring their exceptional ability to other surgeons. This is especially so as identifying data is already available in the many datasets consisting of medical records, some of which were accessed in the cohort studies covered here.

Exceptional performances may be due to personality characteristics that may be hard or impossible to emulate, or we may find out that the surgeons employ easy to emulate techniques like connecting with patients, or simply have higher expectations of patient outcomes,26 or we may find that they live stress-resistant lives, or that they are rarely exhausted, or any other of a myriad of possibilities. If research that investigates exceptional performers identifies simple techniques or choices made at work, or out of work, that could be emulated relatively easily by many other surgeons, then this could lead to fewer complications and more successful surgeries, and there could be large beneficial effects on healthcare costs and patient health.

However, the misuse of identifying supposed underperformers, for example by disciplining or evicting practitioners whose performance appears substandard but who are not statistical outliers or whose performance appears substandard due to a small number of high-risk patients, or due to other confounders like incomplete case-mix or risk score data, is a danger that can cause substantial harm to the surgeons. Further, an old saying is that what gets measured gets managed.105 If more data is available for each surgeon, then this data can be misused to disempower practitioners by adding more and more rules and regulations, and by giving practitioners less opportunity to use their experience and ability. Such data can also be misused in being available online, especially with insufficient explanations of proper usage; or being very much out of date, as is the case for two publicly available databases of surgeons whose data in 2021 only went until 2013106 and 2014.6 Moreover, giving surgeons key performance indicators of patient outcomes could be an unwarranted intrusion into the doctor/patient relationship and lead to surgeons avoiding high-risk patients, as even a few such patients can skew an individual surgeon’s patient outcome statistics, confirmed anecdotally here.66 However, this fact is denied if patients’ risk was accounted for.107 Hence crude performance data should not be published.67

Strengths and Limitations of This Study

The strength of this work lies in the broad search of the literature, the condensed and clear reporting of effect size, and the importance behind the finding that the surgeons’ effect at times has a significant effect size, as big as many non-surgical interventions themselves. The search term strategy used to identify studies was a complex and complete combination of terms that should have identified most of the relevant published studies. Furthermore, the references list of relevant articles and studies citing these articles were screened. This review was not limited by language or by timeframe.

On the other hand, there are at least three broad limitations. First, the Newcastle-Ottawa Scale (NOS) was used for quality assessment with the majority of studies scoring between 8–9 (9 being the maximum total); however, the NOS has been critiqued for being “difficult to use and [having] vague decision rules”1 which derived from poor or fair inter-rater reliability between reviewers. However, it is important to note that associations between individual quality domains or overall quality score and effect estimates were not found. Moreover, the NOS has been endorsed by The Cochrane Collaboration2 for its implementation in systematic reviews of non-randomized studies.

Second, as all of the review’s studies were conducted in North America and Europe, it is unclear whether the findings can be generalized to other regions, particularly in developing nations.

Finally, while the outcome data was heterogeneous and did not enable a meta-analysis, there was also heterogeneity regarding surgical specialty, type of intervention, and type of outcome. Thereby, it is difficult to draw conclusions and synthesize studies with inconsistent outcome measures, and these characteristics have often been found attributable to a lack of a high level of evidence on the specific research subject.

Conclusions and Implications

Even after accounting for surgeons’ experience, patients’ risk and all other known variables there remain sometimes substantial differences in patients’ physical health outcomes between surgeons. Therefore it can matter which surgeon is chosen. At times it is possible to identify high-volume outliers whose performance is well above the average, and it could be worthwhile to study these surgeons to see whether their excellence can be passed on to their peers. It is evident that there are currently no well-established standards on how to assess surgeons as an intervention in their own right, thus systematic approaches to establishing standardized measures are needed, and researching the surgeons’ effect on patients’ physical health is still in its early stages.

Population	P	Surgeons
Intervention	I
Comparison	C
Outcome	O	Surgeons’ effect on patients’ physical health outcome