How do you tell whether a change in surgical technique leads to a change in outcome?

PURPOSE: Surgeons routinely evaluate and modify their surgical technique to improve patient outcome. It is also common for surgeons to analyze results before and after a change in technique to determine whether the change led to better results. Simple comparison of results before and after surgical modification may be confounded by the surgical learning curve. We developed a statistical method applicable to analyzing before/after surgical studies.
MATERIALS AND METHODS: We used simulation studies to compare different statistical analyses of before/after studies. We evaluated a simple 2-group comparison of results before and after the modification by the chi-square test and a novel bootstrap method that adjusts for the surgical learning curve.
RESULTS: In the presence of the learning curve a simple 2-group comparison almost always showed an ineffective surgical modification to be of benefit. When the surgical modification was harmful, leading to a 10% decrease in the success rate, 2-group comparison nonetheless showed a statistically significant improvement in outcome about 80% of the time. The bootstrap method had only moderate power but did not show that ineffective surgical modifications were beneficial more than would be expected by chance.
CONCLUSIONS: Simplistic approaches to the analysis of before/after surgical studies may lead to grossly erroneous results under the surgical learning curve. A straightforward alternative statistical method allows investigators to separate the effects of the learning curve from those of the surgical modification. Copyright (c) 2010 American Urological Association Education and Research, Inc. Published by Elsevier Inc. All rights reserved.