Methodological challenges in indirect treatment comparisons: spotlight on a recent comparison of long-acting injectable aripiprazole versus paliperidone palmitate in the treatment of schizophrenia.

In a recent study, an indirect treatment comparison was performed to examine the relative efficacy and tolerability of aripiprazole once monthly and paliperidone palmitate once monthly. The authors concluded that the results may suggest relative advantages for aripiprazole once monthly over paliperidone palmitate once monthly in the short-term treatment of schizophrenia. However, the validity of the study is compromised as an indirect treatment comparison using extant data may violate important assumptions. Other methodological issues identified further highlight the challenges of performing indirect treatment comparisons.

Indirect treatment comparisons can advance clinical care by offering information to fill gaps in the knowledge of the relative effects of two or more treatment options (Bucher ; Jansen ). These studies attempt to approximate direct comparisons when head-to-head studies are lacking or have not sufficiently resolved uncertainties about relative treatment effects (Bucher ; Jansen ). However, even under ideal conditions, indirect treatment comparisons may not definitively resolve these uncertainties, and therefore, their results must be interpreted with caution and considered in the context of the assumptions required to ensure their validity (Kim ). These studies are prone to numerous potential methodological challenges, which increase the degree of uncertainty (Jansen ).

The International Society for Pharmacoeconomics and Outcomes Research Task Force published a two-part report detailing Good Research Practices when carrying out indirect treatment comparison studies (Hoaglin ; Jansen ). The report highlights three basic assumptions critical to the validity of these studies: all trials included must be comparable in terms of potential effect modifiers, such as trial or patient characteristics (assumption of similarity); there must be no substantial heterogeneity between trial results in pairwise comparisons (assumption of homogeneity); and there must be no relevant discrepancy or inconsistency between direct and indirect evidence (assumption of consistency) (Hoaglin ; Jansen ). However, use of a valid random-effects model can appropriately account for the increased uncertainty introduced by heterogeneity, but a heterogeneous result can still be difficult to interpret. Others have weighed in similarly: ‘The results of examination of similarity, homogeneity, and consistency must be handled appropriately. If there has been a major violation of any of these assumptions, this may even mean that no worthwhile indirect comparison can be performed’ (Kiefer ).

Pae report an indirect treatment comparison study of efficacy and tolerability between aripiprazole once monthly (AOM) and paliperidone palmitate once monthly (PP). The authors concluded that the results ‘may suggest that there may be relative advantages for AOM over PP in the short-term treatment of schizophrenia’. However, several methodological issues, beyond those cited by the authors, compromise the validity of the study and highlight more generally the challenges of performing indirect treatment comparisons. The most critical threats to the study's validity arise from violations of the assumption of similarity. Other methodological issues pertain to the selection of trials to be included in the indirect comparisons.

In particular, it is questionable whether the assumption of similarity can be supported, given the differences identified between the placebo-arm patients from PP and AOM studies. Differences in terms of dropout rates, change in symptom severity, and baseline symptom severity are indications that treatment modifiers, such as study populations and settings, were not similar. A large difference was found between placebo-arm patients who dropped out because of treatment-emergent adverse events [14.3% (73/510) for PP and 0.8% (13/172) for AOM] and all-cause early discontinuations [63.5% (324/510) for PP and 50.6% (87/172) for AOM] (Pae ; Table 1). Baseline symptom severity, as assessed by the mean Positive and Negative Syndrome Scale (PANSS) total score, in placebo arms was markedly different between the studies [89 (13.5) for PP studies and 104 (11.0) for AOM]. Greater symptomatic improvement, as measured by the PANSS score, occurred for patients treated with placebo during the AOM study compared with those treated with placebo during PP studies [−11.7 (20.9) vs. −2.2 (21.9), respectively] (Table 1). Baseline symptom severity, as assessed by the PANSS total score, is predictive of effect size in response to antipsychotic treatment, with higher scores being associated with greater effect sizes (Furukawa ). At baseline, the difference in the mean PANSS scores among the combined PP studies and the AOM study is 14.2 using all subject data (Pae ), and the difference is 15.0 units when only the placebo subjects are examined (Table 1). The authors acknowledge this difference, but do not consider it meaningful (Pae ). However, smaller differences in total PANSS scores are associated with meaningful effects on functional outcomes, hospitalization duration, and healthcare costs (Glick ; Leddy-Stacy and Rosenheck, 2016). In addition, the difference of 14.2 points approaches the minimum PANSS score change of 15.3 points associated with a clinically meaningful difference on the basis of an analysis using Clinical Global Impression score anchoring (Hermes ). In other words, this magnitude of difference seems difficult to ignore.

Table 1

Comparison of study characteristics for the placebo arms

Many AOM and PP randomized-controlled trials were excluded in the analysis by Pae . This is understandable meta-analyses must sacrifice a degree of data completeness for the sake of being able to combine and compare data that would otherwise not be possible. A significant concern with the Pae analysis is the failure to account for whether the excluded data have clinical significance and whether the included studies and specific study results are representative of the treatment effects of interest.

A total of four PP studies were included in the Pae analysis. Two of the four studies used low-dose initiation regimens that are not recommended for clinical practice (Gopal ; Nasrallah ). As the efficacy results from these two earlier, developmental studies were marginal, the initiation regimen was revised to achieve higher plasma concentrations (i.e. two administrations, 1 week apart, in the deltoid muscles, using higher doses). The revised initiation regimen is currently recommended worldwide in product labeling to negate the requirement for additional oral antipsychotic supplementation during initiation, and thus represents the standard of care.

In 2010, Pandina and colleagues published the primary study using a revised two-dose initiation regimen. The study was not included in the Pae analysis, but was eligible for inclusion: it is a 13-week, double-blind comparison of adults with acutely exacerbated symptoms of schizophrenia who were randomized in a 1 : 1 : 1 : 1 ratio to paliperidone palmitate at 25, 100, or 150 mgEq doses or placebo, assessing the mean change in the PANSS total score from baseline. The study is included in the PP prescribing information and can be readily accessed through PubMed. Instead, the data used were from a post-hoc subgroup analysis of the Pandina study, which included patients with moderate-severe symptoms (Alphs ). Although one could argue that the selection of this post-hoc defined subgroup should make the study population more similar to that of the AOM study population, the disease severity in the included Alphs and colleagues data is still about 10 PANSS points less than that in the AOM study. Furthermore, of the two treatment arms within the post-hoc analysis by Alphs , the higher dose arm (150 mgEq) – the arm showing the greatest effect – was excluded (essentially creating a subgroup of a subgroup) (Pae ). The effect of these inclusion and exclusion criteria applied by Pae and colleagues is to lower the overall estimate of PP efficacy, relative to considering a broader set of PP studies.

The PP study in acute schizophrenia by Kramer was also excluded (Pae ). The rationale for this exclusion is not provided, but it also appears to fulfill all eligibility criteria as defined by Pae and colleagues: a 9-week, double-blind, comparison of adults with acutely exacerbated schizophrenia who were randomized to paliperidone palmitate or placebo, assessing a primary endpoint of the mean change in the PANSS total score from baseline. Coincidentally, the Kramer and Pandina represent data from the PP development program with the strongest effect sizes. The exclusion of these two PP studies shifts the results in favor of AOM. Inclusion of the studies with the earlier, suboptimal initiation regimens is also relevant to early dropout findings as dropout rates would be inflated in the context of suboptimal efficacy.

In the absence of prospective, randomized, head-to-head clinical trials, a statistical comparison using matched patient-level pooled data may be a feasible and informative technique to provide a preliminary comparison of two medications. Using patient-level data, one may create comparison groups that are similar with respect to the distribution of a number of demographic and baseline characteristics so that indirect comparisons would be valid. Data points from many of the Janssen and colleagues clinical studies and almost all the paliperidone palmitate clinical trial data are available through the Yale University Open Data Access (YODA) Project, . One of the main intentions of this project is to promote strong scientific collaboration for greater public good.

The indirect treatment comparison study by Pae sought to answer questions on the comparative efficacy between AOM and PP in the short-term treatment of schizophrenia. They suggest possible advantages of AOM over PP on the basis of their findings. However, our close reading of the paper gave rise to methodological concerns that we believe affect the strength and interpretability of their conclusions.