A cautionary note on a recently proposed sensitivity analysis for unmeasured confounding.

Introduction

Unmeasured confounding is a common threat to the validity of observational studies. In practice, confounding bias may arise from both confounders that are completely unmeasured and from confounders that are measured with error. Recently, Kasza, Wolfe and Schuster, hereafter KWS, proposed a method for assessing the impact of unmeasured confounding on causal effect estimates in settings where both the exposure and outcome are binary. Effectively, this method amounts to specifying two sensitivity analysis parameters that together with the observed data distribution completely determine the value of the target causal effect. Varying these two parameters along a grid of plausible values will result in a range within which the true value of the causal effect can reasonably be assumed to lie. When measured confounders are present, KWS suggest performing inverse probability of treatment weighting (IPW) to adjust for these, and computing all the corresponding quantities in the weighted population. We would like to draw attention to the fact that, in general, the target causal quantity in the original population need not be equal to the corresponding causal quantity in the weighted population. As a consequence, proceeding as suggested by KWS will in fact not lead to a characterization of the original population, as intended. While the sensitivity analysis parameters computed for the weighted population do carry some information about the causal estimate in the original population, they lose their original interpretation, so that proposing a grid of plausible values becomes a more difficult task.

This note is organized as follows. We first review the method proposed by KWS in the absence of measured confounders, and show why it fails when measured confounders are present and adjusted for by IPW. We then show what information can be recovered from the sensitivity analysis parameters in the weighted population and suggest two modifications to the original procedure in order to obtain a valid sensitivity analysis. We illustrate this by reanalyzing the data on abciximab and death used by KWS. Sampling variability issues are ignored throughout.

No measured confounders

Following KWS we let and be the binary exposure and outcome of interest, and we let be the potential outcome upon exposure to level . The target parameters are the counterfactual probabilities and , or some contrast of these, such as the causal risk difference or the causal risk ratio . Identification of these parameters requires three assumptions: positivity, consistency and exchangeability. The positivity assumption states that for all possible levels of , the probability of receiving level of the exposure is non-zero:

The consistency assumption states that, if a subject is factually exposed to level , then the potential outcome is equal to the factual outcome :

The exchangeability assumption states that, for all levels of the potential outcome is statistically independent of the exposure :

Under positivity, consistency and exchangeability, the counterfactual probability is equal to , the probability of the outcome among those who factually received level so that the causal risk difference and ratio are equal to the observed risk difference and ratio, respectively.

The positivity assumption holds in most study designs, and the consistency assumption is often taken as an axiom. The exchangeability assumption is more questionable, since it would be violated whenever there is confounding of the exposure and the outcome. To assess the impact of unmeasured confounding on causal effect estimates, KWS proceed as follows. First, they use the law of total probability to rewrite as where the second equality follows from the consistency assumption. On the right-hand side of this expression, only the counterfactual probability is unobserved. Then, they define the ‘confounding function’ which measures the amount of deviation from exchangeability. Note that exchangeability holds if and only if for . For each fixed value of one can compute . Thus, KWS propose to use as a sensitivity analysis parameter and vary it over a range of plausible values in a sensitivity analysis, thereby obtaining a range of plausible values for .

We end this section with a minor technical remark. KWS actually define as . Thus, our definition of agrees with KWS for , but corresponds to in KWS for . We have redefined in this way to enable a more concise notation in the following sections.

Measured confounders

When measured confounders are present, KWS propose to adjust for these by IPW, and then carry out the sensitivity analysis outlined in the previous section in the weighted population. Presumably, their rationale for this procedure is that the weighting, if properly done, can be expected to bring the population closer to the ideal scenario where any remaining deviation from exchangeability is solely due to unmeasured confounding, so that it may be reasonable to focus on a narrower range of values for the sensitivity analysis parameters than before the weighting.

To explain the problem with this approach, we introduce some additional notation. We let be the set of measured confounders that are adjusted for by IPW. For notational convenience we will assume that only contains categorical variables. When there are continuous elements in , the corresponding sums and probabilities will have to be replaced with integrals and probability densities in the formulas below. The corresponding positivity assumption now requires that for all such that and .

The weighting changes the distribution of variables, both observed and counterfactual. To distinguish between probabilities in the original population and probabilities in the weighted population we use for the former and for the latter. Similarly, we use for the confounding function in the weighted population, i.e.

Under conditional exchangeability given, which states that, for all levels of the potential outcome is statistically independent of the exposure , given : we indeed have that .

Following KWS one would specify a range of plausible values for , map each value of into a value for , and plug this into the right-hand side of equation (1) together with , and . The problem is that, by using probabilities in the weighted population on the right-hand side of equation (1), we are no longer computing the target parameter in the original population, , but in the weighted population, , and these are not generally equal. It can be shown (see the Supplementary Material, available as Supplementary data at IJE online) that they are related through

It can also be shown (see the Supplementary Material, available as Supplementary data at IJE online) that the absolute difference may be as large as 1/2. Thus, by using the method proposed by KWS we may over- or under-estimate by as much as 1/2. We remark that when conditional exchangeability (equation 2) holds, then it is indeed true that , but if we are willing to assume conditional exchangeability, then performing sensitivity analysis is unnecessary.

Another issue that further accentuates the problem with KWS’ proposal to use as a sensitivity analysis parameter is that, even though conditional exchangeability implies , the converse is not true; we give a numerical example of this in the Supplementary Material, available as Supplementary data at IJE online. In particular, knowing that is not enough to identify , since conditional exchangeability may still be violated. Thus, in contrast with, is not an intuitively straightforward reference point, as it no longer uniquely corresponds to the scenario where conditional exchangeability holds.

Alternative sensitivity analyses with confounding functions

Even if the confounding function cannot be used as proposed by KWS, it can be shown that does contain some information about the counterfactual probability , which, in principle, makes it possible to use it in a sensitivity analysis. Specifically, in the Supplementary Material, available as Supplementary data at IJE online, we show that, for a fixed value of , is bounded by and

Thus, one may carry out a sensitivity analysis by varying over a specified range of plausible values, and use the relations in equations (3) and (4) to compute lower and upper bounds for for each value of in the specified range.

The lower and upper bounds for are non-increasing functions of . Thus, one may obtain ‘global’ lower and upper bounds for by replacing in equations (3) and (4) with the largest and smallest value of in the specified range, respectively.

We make a technical remark. First, it can be shown (see the Supplementary Material, available as Supplementary data at IJE online) that the bounds in equations (3) and (4) are not tight, and may include values of that are not logically possible, given the observed data. Thus, the alternative sensitivity analysis outlined above may give an ‘unnecessarily wide’ range of values for . However, we conjecture that the derivation of tight bounds as functions of is a difficult optimization problem.

A more direct way to modify the original procedure so as to obtain a valid sensitivity analysis is to define confounding functions for each stratum of

Then where , and the sensitivity analysis for will comprise one parameter for each stratum of .

If can be assumed to be equal to a constant across all levels of then the expression in equation (6) simplifies to

We may thus carry out a sensitivity analysis by varying over a range of plausible values, and convert this to a range of plausible values for through the relation in equation (7).

Note that conditional exchangeability (equation 2) holds if and only if for and all levels of , which further holds if and only if for and all levels of . Hence, in contrast to there is a natural reference point for . Furthermore, whereas specification of only translates to bounds for , specification of completely determines . However, we caution the readers that it may be hard to determine a reasonable range for , since this scalar depends on both the unknown confounding function and the conditional exposure distribution . One way to solve this problem is to consider all possible values for , as outlined in the following section. A related problem is that it may be hard, even for subject-matter experts, to determine whether the assumption of constant is plausible, in particular when is high-dimensional.

Range restrictions for the sensitivity analysis parameters

It is worth mentioning that the observed data distribution imposes constraints on all of the sensitivity analysis parameters mentioned above, and should thus inform the choice of grid along which these parameters are to be varied. In particular, since , we have that . For the same reason, and for all . It follows that where the lower bound is obtained by letting approach , and the upper bounds is obtained by setting . If we assume to be equal to a constant for all levels of , then it can be shown (see the Supplementary Material, available as Supplementary data at IJE online) that:

In practice, these bounds for may be estimated by fitting regression models for and , and using the fitted models to construct predictions , and for each observed level . The bounds for are then estimated as

Re-analysis of the data on abciximab and death

KWS analyzed a publicly available dataset, ‘lindner’, from the package ‘twang’ in R. This dataset contains information on 996 patients who received an initial percutaneous coronary intervention (PCI) at the Christ Hospital in Ohio in 1997, and were followed for at least 6 months by the staff of the Lindner Center. Patients who were thought to have a more severe condition were treated with abciximab (a platelet aggregation inhibitor). For details on these data we refer to KWS and to the help files for the ‘twang’ package.

KWS aimed to estimate the causal effect of abciximab on mortality , while adjusting for a set of potential confounders , including sex, height, diabetes (yes/no), an indicator for recent acute myocardial infarction, left ventricular ejection fraction, the number of vessels included in the PCI (0–5) and an indicator for the insertion of a coronary stent. They fitted a logistic regression model relating abciximab to these confounders, and used the fitted model to construct weights, which were subsequently used for IPW. In this weighted population we have that and . Thus, under the assumption of conditional exchangeability, given the adjusted confounders, the causal risk ratio is equal to . KWS then carried out their proposed sensitivity analysis in the weighted population for the causal risk ratio, considering values for and in the range 0.5–2. The contour plot in Figure 1 by KWS illustrates this sensitivity analysis. In this contour plot, the computed risk ratio ranges from approximately 0.1 to 0.4, thus supposedly giving fairly strong evidence for a protective effect of abciximab.

Figure 1

Lower and upper bounds for P(Y = 1) as functions of c(a), for a = 0 (left panel) and a = 1 (right panel), for the ‘lindner’ dataset.

However, as pointed out above, this risk ratio does not correspond to the effect of abciximab in the original population, but to the effect in the weighted population, and these can be very different. We thus reanalyzed the data with the alternative sensitivity analysis proposed above, using the sensitivity analysis parameter and the same logistic regression model for the weights as KWS. Figure 1 displays the result. In this figure, the lower and upper curves represent the lower and upper bounds for as functions of , for (left panel) and (right panel). We observe that, in the considered range for , the lower bounds for and are constant and equal to 0.015 and 0.011, respectively. The upper bound for ranges from 0.056 to 0.178, and the upper bound for ranges from 0.019 to 0.041. Thus, the global bounds for are given by (0.015, 0.178), and the global bounds for are given by (0.011, 0.041). Finally, we obtain global lower and upper bounds for the causal risk ratio as 0.011/0.178 = 0.06 and 0.041/0.015 = 2.7, respectively. These results show that, if we believe that and may be anywhere in the range 0.5–2, then the causal risk ratio may be anywhere in the range 0.06–2.7, which is substantially wider than the range obtained with KWS’ original sensitivity analysis.

We next carried out the alternative sensitivity analysis proposed above, using the sensitivity analysis parameter . To this end we fitted a logistic regression model for death on abciximab and the measured confounders. This model, together with the logistic regression model for abciximab, gives bounds for and equal to (0.77, 1.35) and (0.98, 1.07), respectively, through the relation in equation (8). Using the fact that then leads to a range for the causal risk ratio equal to 0.14–0.26, which is substantially narrower than the range obtained with KWS’ original sensitivity analysis.

The reason why the two sensitivity analyses give strikingly different results is that the second analysis makes much stronger assumptions than the first. The first analysis only assumes that KWS’s sensitivity parameters and are confined to the range 0.5–2, whereas the second analysis additionally assumes that the function is constant across levels of the measured confounders . Determining whether or not this assumption is plausible requires subject-matter expertise, and we thus refrain from making a strong recommendation as to which result to trust. R code for all analyses is provided in the Supplementary Material, available as Supplementary data at IJE online.

Discussion

The problem of unmeasured confounding is fundamental in causal inference from observational data. KWS propose a method for dealing with the basic case of binary exposure and binary outcome. However, transferring this sensitivity analysis strategy to the case where measured confounders are present is not a straightforward task. This is characteristic of a range of sensitivity analysis methods, in which adjusting for measured confounders often leads to a rapid increase in the number of sensitivity parameters needed,. In that case, additional assumptions often have to be made in order to keep the analysis manageable and interpretable. By examining the sensitivity analysis approach proposed by KWS we have illustrated the fact that performing inverse probability of treatment weighting to adjust for measured confounders effectively changes the distribution of the potential outcomes, and as such, analyzing the weighted population does not result in a straightforward characterization of the original population.

We have presented two possible modifications to remedy this, each based on different assumptions, which we have described in detail. The first modification provides bounds for the causal effect of interest, without making any additional assumptions. These bounds are not tight, and are expected to result in a fairly wide interval for the causal effect of interest, in general. The second modification additionally assumes that the function is constant across levels of the confounder, and is expected to result in a narrower interval, in general. The choice between these analyses hinges on whether or not the additional assumption of the second analysis can be deemed plausible or not, which must be determined on a case-by-case basis, by subject-matter experts.

In line with KWS we have focused on settings where both the exposure and the outcome are binary. Epidemiological studies often have other types of variables, such as continuous exposures and outcomes, and time-to-event outcomes. Both of our modifications of KWS’s sensitivity analysis can be used for time-to-event outcomes, if the outcome is dichotomized as the indicator of having the event before a certain point in time. Our first modification requires an estimate of the outcome probability ; to properly account for censoring this can be estimated with the Kaplan–Meier method, for exposed and unexposed separately. Our second modification requires an estimate of the conditional (on measured covariates) outcome probability ; this can be obtained from, for instance, a fitted Cox proportional hazards model or a flexible parametric model. We recognize the extension to truly continuous exposures and outcomes as an important but challenging topic for future research.

Supplementary data

Supplementary information is available at IJE online.

Conflict of interest

None declared.

Click here for additional data file.