Common pitfalls in statistical analysis: Intention-to-treat versus per-protocol analysis.

During the conduct of clinical trials, it is not uncommon to have protocol violations or inability to assess outcomes. This article in our series on common pitfalls in statistical analysis explains the complexities of analyzing results from such trials and highlights the importance of "intention-to-treat" analysis.

INTRODUCTION

In interventional studies, a subset of participants often do not conform to the protocol. These “protocol violations” can be of various types: One or more participants for some reason do not receive the respective interventions to which they were randomized, inadvertently receive an intervention meant for the other trial arm, receive a prohibited concomitant intervention, or are not available for assessment of the planned outcome either because of loss to follow-up or for another reason.[1] During the analysis of the trial results, the researcher is tempted to exclude such “nonconforming” participants. The motivation is not one of deceit, but of integrity, ensuring that comparisons are made between those participants in each trial arm who strictly adhered to the planned treatment so that the true efficacy of one intervention over the other can be assessed.

However, despite the above apparent attractiveness of this approach, such exclusion poses multiple problems.[1] These include:

It violates the principle of randomization. In a 2-arm study, randomization ensures comparability of the two groups, i.e., balanced for known and unknown confounders or prognostic factors, only as they were originally randomized. When some participants in either or both the groups are excluded, the remaining participants in the two groups can no longer be considered as balanced. The problem becomes larger as the number of exclusions increases

At times, the noncompliance is related to a particular intervention or to disease severity. For instance, the inability to complete the scheduled treatment or appearance of unacceptable side effects may be more frequent in patients with severe disease. In addition, these may occur more often in the active treatment arm than in the placebo arm. Hence, exclusion of the participants who do not complete the treatment or follow-up as planned would lead to differential exclusion of patients with severe disease in the treated group, with the residual group unlikely to resemble the original group obtained at randomization. This may make the treatment look better than it actually is

Exclusion of participants in one or both groups, particularly if their number is large, may lead to a significant reduction in sample size and hence in study power

Exclusions can introduce a bias. Often the decision to exclude a particular participant is controlled, at least to some extent, by the investigator, who may be tempted to exclude patients who are not doing well in a particular arm

The purpose of a trial is to assess the proportion of persons in a group who may be expected to benefit from a particular treatment. Those who do not complete treatment can of course not be expected to benefit from it. The proportion of responders among those who complete treatment thus provides an exaggerated estimate of treatment effect – this does not accurately reflect the beneficial effect that may be expected in clinical practice among those who are prescribed this particular treatment.

To obviate (or minimize) these problems, it is recommended that “intention-to-treat (ITT) analysis” be used. The principle of ITT analysis is that all participants should be analyzed in the group to which they had been randomized, i.e. as if they had received the intervention which they were supposed to receive, irrespective of the treatment actually received. Fisher defines ITT as analysis which “Includes all randomized patients in the groups to which they were randomly assigned, regardless of their adherence with the entry criteria, regardless of the treatment they actually received, and regardless of subsequent withdrawal from treatment or deviation from the protocol.”[2]

The use of ITT analysis ensures maintenance of comparability between groups as obtained through randomization, maintains sample size, and eliminates bias. In addition, results obtained in such analysis more closely represent clinical practice, dealing with “effectiveness” of the intervention rather than “efficacy.” In view of these advantages, ITT is today considered as a defacto standard for analysis of clinical trials, though a minority school of thought believes that this approach is too conservative.[1]

In contrast, per-protocol (PP) analysis refers to inclusion in the analysis of only those patients who strictly adhered to the protocol. The PP analysis provides an estimate of the true efficacy of an intervention, i.e., among those who completed the treatment as planned. However, as discussed above, its results do not represent the real life situation and it is likely to show an exaggerated treatment effect.

The CONSORT guidelines for reporting of “parallel group randomized controlled trials” recommend that both ITT and PP analyses should be reported for all planned outcomes to allow readers to interpret the effect of an intervention.[3]

Of course, there are some special situations. For instance, in noninferiority trials, the use of PP analysis is considered particularly important.[4] A detailed discussion of this is beyond the purview of this piece, but will be done in a subsequent article in this series.

A randomized trial published recently in the New England Journal of Medicine compared early (intervention arm) versus delayed (standard arm) introduction of allergenic foods into the diet of breast-fed children.[5] The primary outcome was the development of allergy to any food between 1 and 3 years of age. Results from the ITT analysis (1162 participants) showed no difference between groups for the primary outcome (intervention arm: 32/567 [5.6%] versus standard arm: 42/595 [7.1%]; P = ns). However, a PP analysis (732 participants) showed a significantly lower frequency of food allergy in the intervention arm versus the standard arm (5/208 [2.4%] vs. 38/524 [7.3%]; P = 0.01). It is interesting to note that only 32% (208/652) of the participants in the intervention arm adhered to the protocol as compared to 88% (524/595) of the participants in the standard arm. The authors have offered several explanations for this lack of compliance. Importantly, and in our opinion rightly, they gave precedence to the results of the ITT analysis over those of the PP analysis, and concluded that “the trial did not show the efficacy of early introduction of allergenic foods.”

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.