Common pitfalls in statistical analysis: Clinical versus statistical significance.

In clinical research, study results, which are statistically significant are often interpreted as being clinically important. While statistical significance indicates the reliability of the study results, clinical significance reflects its impact on clinical practice. The third article in this series exploring pitfalls in statistical analysis clarifies the importance of differentiating between statistical significance and clinical significance.

One of the common problems faced by readers (and authors!) of medical articles is in the interpretation of the word “significance.” The term “statistical significance” is often misinterpreted as a “clinically important” result. The confusion stems from the fact that many people equate “significance” with its literal meaning of “importance,” whereas in statistics, it has a far more restrictive connotation. This article explains the idea of the statistical significance and differentiates it from clinical relevance or importance, which is an entirely different concept. In the previous article, in this series, we looked at different ways of expressing statistical significance (“P” values versus confidence intervals).[1] Measures of statistical significance quantify the probability of a study's results being due to chance. Clinical significance, on the other hand, refers to the magnitude of the actual treatment effect (i.e., the difference between the intervention and control groups, also known as the “treatment effect size”), which will determine whether the results of the trial are likely to impact current medical practice. The “P” value, frequently used to measure statistical significance, is the probability that the study results are due to chance rather than to a real treatment effect. The conventional cut off for the “P” value to be considered statistically significant is of 0.05 (or 5%). What a P < 0.05 implies is that the possibility of the results in a study being due to chance is <5%.

In clinical practice, the “clinical significance” of a result is dependent on its implications on existing practice-treatment effect size being one of the most important factors that drives treatment decisions. LeFort suggests that the clinical significance should reflect “the extent of change, whether the change makes a real difference to subject lives, how long the effects last, consumer acceptability, cost-effectiveness, and ease of implementation”.[2] While there are established, traditionally accepted values for statistical significance testing, this is lacking for evaluating clinical significance.[3] More often than not, it is the judgment of the clinician (and the patient) which decides whether a result is clinically significant or not.

Statistical significance is heavily dependent on the study's sample size; with large sample sizes, even small treatment effects (which are clinically inconsequential) can appear statistically significant; therefore, the reader has to interpret carefully whether this “significance” is clinically meaningful. A study published in the Journal of Clinical Oncology compared overall survival in 569 patients with advanced pancreatic cancer who were randomised to receive erlotinib plus gemcitabine versus gemcitabine alone.[4] Median survival was found to be “significantly” prolonged in the erlotinib/gemcitabine arm (6.24 months vs. 5.91 months, P = 0.038). The P = 0.038 means that there is only a 3.8% chance that this observed difference between the groups occurred by chance (which is less than the traditional cut-off of 5%) and therefore, statistically significant. In this example, the clinical relevance of this “positive” study is the “treatment effect” or difference in median survival between 6.24 and 5.91 months – a mere 10 days, which most oncologists would agree is a clinically irrelevant “improvement” in outcomes, especially when considering the added toxicity and costs involved with the combination.

Most journals now endorse the use of the CONSORT statement for reporting of parallel-group randomized trials, which emphasizes the need for reporting of the estimated effect size and its precision (such as 95% confidence interval) for each primary and secondary outcome.[5] Readers should bear in mind that interpretation of study results should take into account the clinical significance by looking at the actual treatment effect (with confidence intervals) and should not just be based on “P” values and statistical significance.