Assessing the prior event rate ratio method via probabilistic bias analysis on a Bayesian network.

BACKGROUND: Unmeasured confounders are commonplace in observational studies conducted using real-world data. Prior event rate ratio (PERR) adjustment is a technique shown to perform well in addressing such confounding. However, it has been demonstrated that, in some circumstances, the PERR method actually increases rather than decreases bias. In this work, we seek to better understand the robustness of PERR adjustment.
METHODS: We begin with a Bayesian network representation of a generalized observational study, which is subject to unmeasured confounding. Previous work evaluating PERR performance used Monte Carlo simulation to calculate joint probabilities of interest within the study population. Here, we instead use a Bayesian networks framework.
RESULTS: Using this streamlined analytic approach, we are able to conduct probabilistic bias analysis (PBA) using large numbers of combinations of parameters and thus obtain a comprehensive picture of PERR performance. We apply our methodology to a recent study that used the PERR in evaluating elderly-specific high-dose (HD) influenza vaccine in the US Veterans Affairs population. That study obtained an HD relative effectiveness of 25% (95% CI: 2%-43%) against influenza- and pneumonia-associated hospitalization, relative to standard-dose influenza vaccine. In this instance, we find that the PERR-adjusted result is more like to underestimate rather than to overestimate the relative effectiveness of the intervention.
CONCLUSIONS: Although the PERR is a powerful tool for mitigating the effects of unmeasured confounders, it is not infallible. Here, we develop some general guidance for when a PERR approach is appropriate and when PBA is a safer option.

INTRODUCTION

From healthcare to econometrics to the social sciences, the findings of observational studies almost always suffer from bias due to confounding. Numerous methods to control for confounding have been developed, and for each of these, dealing with unmeasured confounders usually presents the trickiest challenge.

One method that has been used to address confounding due to unmeasured variables is prior event rate ratio (PERR) adjustment, developed by Weiner and colleagues and shown to perform well in reproducing the results of a Scandinavian randomized controlled trial (RCT) with an observational study conducted using UK Electronic Medical Record (EMR) data.1 Subsequent work by Weiner et al showed a similarly good performance of PERR applied to observational studies in replicating other RCTs.2, 3, 4 Uddin et al5 studied the performance of PERR and identified situations in which PERR adjustment increases rather than decreases bias. Further development and refinement, including derivation from PERR adjustment of a pairwise Cox likelihood function, was carried out by Lin and Henley.6

Here, we endeavor to understand when PERR yields meaningful bounds on the size and/or direction of the treatment effect of interest, even if it cannot be guaranteed to remove (or even reduce) bias. We begin with a Bayesian network7, 8 representation of the causal pathways in a standard observational study (Figure 1). We use Bayesian network analysis to define exact analytic expressions for the joint probabilities of study variables. (Uddin et al used Monte Carlo simulation to approximate the joint probabilities.) We examine the behavior of the system and identify circumstances where the PERR method overestimates and underestimates the true effect of a treatment. As an illustrative example, we then apply our approach to performing a probabilistic bias analysis (PBA) of a study of the relative effectiveness of high‐dose (HD) versus standard‐dose (SD) seasonal influenza vaccine in the Veterans Affairs (VA) patient population.9

Figure 1

Bayesian network, or probabilistic directed acyclic graph, with dichotomous variables denoting baseline period event , treatment , observation‐period event , time‐varying unobserved confounder {, and the rate ratios (RRs) describing the associations among them. The RRs are defined in Equations (10) to (18) [Colour figure can be viewed at http://wileyonlinelibrary.com]

THE PERR METHOD

Let us suppose we are conducting a cohort study in which the rate of the outcome of interest is measured not just during the observation period after the treatment but also during a baseline period before the treatment. We denote a possible event occurring during the baseline period as , a possible event occurring during the observation period as and a possible treatment as . The variables are dichotomous, so, for example, an individual with experienced an event in the baseline period, did not receive the treatment and did not experience an event in the observation period. To streamline the notation, we will write as , etc. We use to denote the probability, over a time interval , that an event occurs in an individual. For example, is the probability per time that an individual experiences an event in the observation period, conditional on having received treatment. Measured at the population level, becomes the average incidence rate, ie, events per person per time . We denote the incidence rate as . Since the two quantities are numerically identical (assuming that both are defined over the same unit of time, and that individuals are independent), we will use them interchangeably. The incidence rate ratio (RR) in the baseline period is then whereas, in the observation period, it is

Now, let us suppose that we find for the event to differ from unity by a statistically significant amount. This suggests that the treatment and control arms are unbalanced with respect to the distribution of one or more determinants of the event. Assuming that there are no systematic measurement errors and that all measurable confounders have already been controlled for, we are led to suspect that there is residual unmeasured confounding by indication. The PERR method attempts to correct for the imbalance and thus recover the “true” RR of the treatment through normalizing by

THE CAUSAL MODEL

We begin with a Bayesian network (or probabilistic directed acyclic graph [DAG])7 depicting, from the perspective of a single individual, the potential causal associations in our study. We include an unmeasured dichotomous confounder, . We define distinct values for the confounder in the baseline period () and the observation period (). We further divide into and to allow for the possibility that the relationship between and is bidirectional (ie, the state of influences the state of , and the state of influences the state of ). We also allow for the possibility of a direct causal connection between baseline event and treatment . One can show (see the work of Greenland et al7) that this DAG has a null adjustment set and that there exist no instruments or conditional instruments that might permit instrumental variable regression.

We write down a set of equations that describe this causal diagram. The effect of each directed edge is expressed as an RR whose value depends on the state of the variable corresponding to the edge's originating vertex. The model equations describing the population‐level incidence rates (or equivalently, individual‐level probabilities ) over a time period of the occurrence of , , and are where , , , , are constants on [0,1]. The operator Π, ∈ , is the closest‐element mapping from real numbers to [0,1], ie,

The s are defined as follows: where the 's are constants >0.

Uddin et al used Monte Carlo simulation to obtain approximate conditional rates/probabilities from this model. Instead, we will use Bayesian network analysis (see, eg, the work of Pearl8) to calculate exact probabilities, as follows.

The incidence rate with which a given set of values of , and is realized on the above network is their joint probability

Joint rates/probabilities of subsets of the variables are calculated by summing the joint probabilities over all possible combinations of the remaining variables, for example,

Conditional rates/probabilities can likewise be calculated, for example, where

We implement these calculations in the R language.10

THE PERFORMANCE OF PERR ADJUSTMENT

In an observational study, the following incidence rates will be measured:

, the rate of treatment across the study population;

, the rate of the event in the treatment arm during the observation period;

, the rate of the event in the control arm during the observation period;

, the rate of the event across the whole study population during the observation period (where ).

If patients have also been observed during a baseline period, then we further have measurements of the following:

, the rate of the event in the treatment arm during the baseline period (ie, the rate among those who are later treated);

, the rate of the event in the control arm during the baseline period (ie, the rate among those who are not later treated);

, the rate of the event across the whole study population during the baseline period (where ).

The RRs between treatment and control arms in the observation and baseline period are given by Equations (1) and (2), respectively, while the PERR estimate of the treatment effect is given by Equation (3).

The true, unbiased effect of treatment on the probability of event is given by (see Equation (18)). The question we wish to answer for our causal model is, under what circumstances does PERR succeed in decreasing bias, and under what circumstances does it actually increase bias? Furthermore, under what circumstances is the estimate either < or >? For specificity, we take events to be harmful, thus the smaller , the more effective the treatment . Therefore, when , the effectiveness of is overestimated (overoptimistic PERR), whereas if , it is underestimated (pessimistic PERR).

We begin by computing the behavior of the in comparison to the true effect under different scenarios.

Unobserved confounder effect is present and affects the baseline and observation period event probability equally (ie, ) and also affects the probability of treatment (ie, ), while is fixed at 1. Baseline period events do not affect the probability that the confounder is present (ie, ). Figure 2 shows , the PERR, and true effect as a function of . As we can see, even though diverges from the true effect, the PERR is exactly equal to the true effect.

Figure 2

The conventional rate ratio (RR) (), the prior event rate ratio (PERR) estimator, and the true RR (=, set to 0.7) in the case where and . is held fixed at 1.5, and . Note that the PERR and the true RR are coincident

Retaining the effect of the unobserved confounder, we allow , the effect of the occurrence of a baseline event on the likelihood of an observation period event, to vary. Results are shown in Figure 3. As can be seen, increasing above 1 makes the PERR progressively more pessimistic (ie, increases above 1), though the effect is relatively weak and bounded.

Figure 3

Performance of the prior event rate ratio (PERR) estimator as a function of . The ratio of the PERR to the true effect, ie, , is plotted. A ratio greater than 1 thus corresponds to an overestimate of the true RR and, hence, an underestimate of the true treatment effect. Scenarios with (Scenario a), = 2 (Scenario b), = 3 (Scenario), and = 4 (Scenario d) are shown. As before, we set , , and fix , . In each case, the overestimate of the true rate ratio (RR) by the PERR initially becomes greater with , then reaches a maximum, and decreases again. The height of the maximum increases with . When , the maximum PERR overestimate of the true RR is 15% and occurs when

We allow to differ from . Results are shown in Figure 4. When , (ie, the effect of the confounder on likelihood of an event is weaker in the baseline period than in the observation period), the PERR is pessimistic; when the converse is true, , then the PERR is overoptimistic. As we will see later (Section 5 and Appendix A), one reason that these two values may differ is if the effect of treatment in an individual changes depending on whether the confounder is present or absent.

Figure 4

Accuracy of the prior event rate ratio (PERR) estimator as a function of , with held fixed at 3. In Scenario a, , whereas in Scenario b, . As before,

We allow to differ from 1; see Figure 5. Above 1, even modest values of suffice to make the PERR substantially overoptimistic. Conversely, as is decreased below 1, PERR rapidly becomes more pessimistic. Note that cannot exceed , otherwise would exceed 1 for individuals having a baseline event .

Figure 5

Performance of the prior event rate ratio (PERR) estimator when differs from 1. In Scenario a, there are no confounders, ie, , and as before, . In Scenario b, the confounder is present (), with and

We reverse the directionality of the baseline period relationship between confounder and event: while a pre‐existing confounder does not affect the probability of a baseline event , an individual without pre‐existing may develop as a result of experiencing . Figure 6 shows that, as the strength of the association increases, PERR becomes increasingly overoptimistic. Depending on the pre‐existing confounder incidence rate , and on , may or may not have a lower limit >0.

Figure 6

Performance of the prior event rate ratio (PERR) estimator when the directionality of the causal relationship between baseline confounder and is reversed, ie, (presence of does not affect probability that is present), (presence of increases the probability that is present), and as above, . Scenario a: . Scenario b: . Scenario c: is increased from 1.5 to 3

We allow for the possibility that an individual either loses or gains the confounder between baseline and observation period. We see from Figure 7 that both scenarios act to make PERR overoptimistic; by how much depends on and .

Figure 7

Performance of the prior event rate ratio (PERR) estimator when the value of is allowed to differ from the value of (in all cases ). Scenario a: and , so that , (ie, probability to gain confounder is 0.25). Scenario b: As in Scenario a but with increased from 1.5 to 3. Scenario c: (ie, probability of losing confounder is 0.5)

We thus see that the PERR works just as intended as long as the association between confounder and events has the same strength in the baseline and observation periods. The more confounding differs between the periods, though, the poorer the PERR does as an estimator of the true effect. Whether the strength of confounding increases or decreases with time is important, because this determines whether the PERR overestimates or underestimates, respectively, the true effect of the intervention.

We also see that the PERR is overoptimistic when there is a causal connection directed from baseline period event to treatment , either directly or via confounder . Finally, PERR is overoptimistic when individuals are able to either gain or lose the confounder over the course of the study period.

Probabilistic bias analysis

We have investigated ways in which using the PERR to control for unobserved confounding can fail. However, if one has sufficient information to be able to place limits on the strengths of the causal associations among the study subjects' set of attributes {, and the incidence rates of the unobserved confounders, one may still be able to obtain useful constraints on the true treatment effect via PBA.11

The approach is straightforward: one chooses prior probability distributions for the incidence rates of the unobserved confounder, and for each of the factors governing the associations among the attributes, except for the treatment effect itself, . One then performs iterations of drawing a set of values from these distributions. For each set, one computes the value of needed to realize the observed rates , , , , and . In this way, one obtains a posterior distribution of possible values of the true treatment effect. The number of iterations should be sufficiently large that increasing it further does not appreciably change the shape of the posterior distribution. Here, we implement all of the above in R and perform 50 000 iterations for each scenario.

APPLICATION TO AN OBSERVATIONAL STUDY

We apply our method to a study of HD influenza vaccine effectiveness, relative to SD vaccine, against influenza and influenza‐associated outcomes that was conducted within the VA patient population by Young‐Xu et al.9 In this study, a difference in the rate of hospitalization for pneumonia and influenza (P&I; ICD‐9 codes 480‐488) in the HD and SD arms was found in the baseline period even after matching on patient comorbidities, suggesting residual confounding by indication. This was addressed through use of PERR adjustment. We will apply the above PBA methodology to assess the robustness of this approach in this particular study.

For the baseline period, the study reported event rates (in units of events per 10 000 person‐weeks) of and . For the observation period, the rates were , . Thus, the RRs of HD versus SD arms in the baseline and observation periods were and , respectively. This made the unadjusted relative effectiveness , suggesting (contrary to evidence from its RCT12) that the HD vaccine was less effective than SD. However, the fact that the RR in the baseline period differed significantly from 1 () suggested confounding by indication. The PERR estimate of the RR was for an HD to SD relative effectiveness where the confidence interval was obtained using the assumption of Poisson‐distributed events. We apply the causal diagram of Figure 1 to this study, with the following interpretation: starting in the baseline period, an unobserved confounder that is present in part of the patient population—we can think of it as frailty not captured in the patients' medical records—potentially causes one or more of the following:

an elevated likelihood of hospitalization for pneumonia/influenza during the baseline period;

a modified likelihood of receiving HD rather than SD vaccine via confounding by indication: the presence of increases the likelihood that the healthcare provider (HCP) will identify the patient as being at elevated risk, and this may then affect the HCP's decision whether to prescribe SD or HD;

an elevated likelihood that the patient will also possess the confounder in the observation period, ie, (a frail patient is likely to remain frail);

if the confounder does carry over into the observation period, then an elevated likelihood of hospitalization for pneumonia/influenza during the observation period; and

a reduced immune response to vaccination, resulting in a reduction in the protective effect derived from both HD and SD.

Point 5 requires further explanation. It has been shown that frailty can substantially reduce the effectiveness of influenza vaccine against influenza‐associated hospitalization.13 Furthermore, it has been shown that the relative efficacy of HD versus SD does not vary significantly between frail and nonfrail individuals.14 We thus make the assumption that frail individuals receive a weakened vaccine effect, such that the RR due to vaccination is multiplied by a factor for both HD and SD vaccination. It can be shown (see Appendix A) that this is equivalent to increasing by the same factor, which, in turn, suggests that . In our PBA in the following (and in some of the additional analyses in Appendix B), we use the parameterization . From Section 4, we know that, when , this puts us in the regime of a pessimistic PERR, ie, one that will underestimate the relative effectiveness of HD, all other things being equal.

In the context of this study, the edge directed from to in Figure 1 represents the possibility that the HCP's choice of vaccine is directly influenced by whether or not a patient was hospitalized for pneumonia/influenza during the baseline period. P&I hospitalization can also influence vaccine choice by the causal path from to to : hospitalization may cause frailty (instead of/in addition to the other way around), and the presence of this newly acquired frailty may then influence vaccine choice. As we saw in Section 4, both the direct and indirect path can cause the PERR to be overoptimistic.

To investigate the possibility of a direct link from to , we conducted an interview study among nurse infection preventionists in 25 Veterans Health Administration (VHA) facilities to gain understanding of the decision‐making process governing the selection of HD versus SD for a given patient within VA facilities. The study is described in Appendix C. In no case was previous hospitalization for pneumonia/influenza reported as a criterion for preferentially administering HD. On the contrary, in two (8%) of the facilities, recent hospitalization for pneumonia was reported as a possible contra‐indication for HD administration. This suggests that .

We conduct our probabilistic bias analyses by performing Monte Carlo simulations consisting of 50 000 realizations each (some of these are discarded due to having infeasible combinations of inputs). Each realization proceeds via a “draw and adjust” procedure, as follows.

A set of observed values for , , , and is drawn from the results of the study of Young‐Xu et al, assuming, as they do, that event counts are Poisson‐distributed.

All DAG parameters apart from , , and are drawn from their respective prior distributions, and any required relationships among parameters are imposed.

is chosen such that is reproduced.

is chosen such that , the study HD vaccination rate, is reproduced.

is chosen such that is reproduced.

The output of each realization is the posterior distribution of , the true treatment effect, where .

We choose uniform distributions for all input parameters. In Appendix B, we examine the effect of applying progressively more constraints on the input parameter ranges and relationships among them (Table B1). Corresponding posterior distributions of are shown in Figure B1. As can be seen, under minimal assumptions (Analysis 1), PERR is very likely to significantly overestimate . The distribution of  becomes tighter and/or moves further to the right in each successive analysis, and for Analyses 6 and 7, PERR is more likely to underestimate than overestimate the true effect size.

Table B1

Setup and results of the set of seven probabilistic bias analyses (PBA) conducted on the Veterans Health Administration study of Young‐Xu et al.9 Starting with minimal assumptions about the relationships among , , , , , and , we add assumptions (described in the leftmost column) one by one and apply the corresponding constraints on the input parameters. Cells corresponding to tightened constraints are shown in light gray, and darker gray for the one instance of a constraint further tightened

	PBA input parameters					Results
Assumptions	RR of direct	Probability	Relationship	RR of effect	Probability	rVEHD,true	Comparison to rVEHD,PERR=
	effect of	that frail	between	of baseline	that nonfrail	(%)	25(2; 43)%
	baseline	subject	frailty effect	hospitalization	subject		Common	Two‐sample
	hospitalization	(baseline)	on	on baseline	(baseline)		language effect	Hodges‐
	on probability	becomes	hospitalization	frailty:	becomes		size (CLES):	Lehmann
	of HD receipt	nonfrail	in observation	(Fy1 → C1,b)	frail		probability that	estimate of
	(Fy1 → X)	(observation)	and baseline		(observation)		PERR	median
		( Pc_2‾|c_1b)	period:		scaled by		overestimates	difference (%)
			( f≡F_c2→Y2F_c0→Y1)		baseline		true effect, ie,	(positive values
					frailty rate		P(rVEHD,PERR	are PERR
					( F_new≡P_frailr_c1a)		>rVEHD,true)	overestimates)
1) Minimal assumptions	[0.1, 10]	[0, 1]	no relationship	[1, 10]	[0, 1]	−31(−184; 30)	0.957	57
2) No direct effect of	1	[0, 1]	no relationship	[1, 10]	[0, 1]	−10(−46; 19)	0.968	34
baseline hospitalization
on probability of receiving HD
3) Subjects do not recover	1	0	no relationship	[1, 10]	[0, 1]	0(−37; 26)	0.927	26
from frailty
4) Frailty effect on	1	0	[1, 10]	[1, 10]	[0, 1]	9(−18; 37)	0.814	16
hospitalization is at least
as strong in observation
as baseline period
5) In baseline period, frailty	1	0	[1, 10]	[1, Fc0 → Y1]	[0, 1]	15(−17; 57)	0.682	9
effect on hospitalization
is at least as strong as
hospitalization
effect on frailty
6) Previously nonfrail	1	0	[1, 10]	[1, Fc0 → Y1]	0	26(−12; 68)	0.459	−1
subjects do not
become frail in
observation period
7) In baseline period,	1	0	[1, 10]	1	0	38(−13; 74)	0.245	−14
hospitalization
has no effect on
frailty

Figure B1

Results of probabilistic bias analysis to assess the true relative effectiveness of high‐dose (HD) versus standard‐dose (SD) influenza vaccine, , for the scenarios in Table B1 [Colour figure can be viewed at http://wileyonlinelibrary.com]

We now seek to identify specific constraints appropriate for this study. Table 1 gives a set of constraints, informed by published literature, on the relationship between hospitalization and frailty state transitions (see the work of Gill et al15) and on hospitalization rates, including for P&I, of high‐risk and low‐risk individuals (see the work of Mullooly et al16). In using the latter, we assume that the RR of P&I hospitalization of high‐risk versus low‐risk subjects can be used as a proxy for that of frail versus nonfrail subjects. The results of the PBA are given in Table 2, shown graphically in Figure 8, and compared to the PERR estimate in the study of Young‐Xu et al.9 Both distributions have a similar lower 95% CI bound, but both the median and upper bound of the PBA posterior distribution are higher than those of the PERR estimate. In Table 1, we compare the distributions using two different metrics: the common‐language effect size (CLES)17 and the two‐sample Hodges‐Lehmann estimator18 for the difference between two populations. The CLES gives the probability that a sample drawn from the distribution of is greater than a sample drawn from the distribution of . In other words, it gives the probability that PERR overestimates the true effect size. The two‐sample Hodges‐Lehmann estimator is a nonparametric estimator of the median difference between a pair of samples drawn from the two distributions. Both comparisons suggest that the PERR estimate in the study of Young‐Xu et al is more likely to have underestimated than overestimated the true relative effectiveness of HD vaccine.

Table 1

Input parameter ranges for probabilistic bias analysis (PBA) of the study of Young‐Xu et al,9 together with literature sources/rationales

	PBA input parameters
	RR of direct	Probability that	Probability that	Relationship	RR of effect of	Probability that
	effect of baseline	nonfrail subject	subject frail in	between frailty	baseline	subject nonfrail
	hospitalization	becomes frail	baseline period	effect on	hospitalization on	in baseline period
	on probability of	within baseline	becomes	hospitalization in	baseline frailty:	becomes frail in
	HD receipt (Fy1 → X)	period (rc1b)	nonfrail in	observation and	(Fy1 → C1b)	observation
			observation	baseline period:		period (rc2)
			period ( P(c_2‾∣c_1b))	( f≡F_c2→Y2F_c0→Y1)
Constraints	[0.9, 1.1]	[0.011, 0.024]	[0.0006, 0.0036]	[1, 10]	[1.06; 1.66]	[0.011; 0.024]
Source	Veterans Health	15	15	Appendix A, 13, 14	15	15
	Administration
	Survey study,
	Appendix C
Comments	Survey	6‐month	6‐month	f > 1 follows	Hazard ratio of	6‐month
	(Appendix C)	incidence rate of	incidence rate of	from assumption	transition rate	incidence rate of
	suggests Fy1 → X ≤ 1; we	transition from	transition from	that frail	from frail to nonfrail	transition from
	conservatively	nonfrail to frail	frail to nonfrail	individuals derive	state after	nonfrail to frail
	assume RR that varies	state (their Table 1)	state (their Table 1)	less protection	hospitalization,	state (their Table 1)
	by +/− 10% about 1			from vaccine	vs. no intervening
				than nonfrail	hospitalization
					(their Table 2)

Table 2

Output of the probabilistic bias analysis (inputs described in Table 1): posterior distribution of the true treatment effect, and comparison to the prior event rate ratio (PERR) estimate in the study of Young‐Xu et al9

rVEHD,true(%)	Comparison to rVEHD,PERR = 25(2;42)%
	Common‐language effect size:	Two‐sample Hodges‐Lehmann
	probability that PERR overestimates true	estimate of median difference (%) (a
	effect, ie, P(rVEHD,PERR > rVEHD,true)	positive value is a PERR overestimate)
34 (0; 42)	0.28	−11

Figure 8

Output of the probabilistic bias analysis (inputs described in Table 1): posterior distribution of the true treatment effect (red), with the prior event rate ratio estimate of Young‐Xu et al9 (blue) shown for comparison. HD, high dose; SD, standard dose [Colour figure can be viewed at http://wileyonlinelibrary.com]

()

( )

effect, ie,

DISCUSSION

In multiple studies,1, 2, 3 PERR adjustment has been demonstrated to perform well in controlling for bias due to unmeasured confounding. The method is also appealing due to its relative simplicity, both conceptually and in implementation. However, it has been shown that this method may in some cases increase rather than decrease bias.5 We further explored PERR performance and used Bayesian network calculations applied to the causal diagram representation of a previously published observational study of the relative effectiveness of HD versus SD influenza vaccine in the US VA patient population. Using PBA, we showed that, in applying the PERR estimator to control for unmeasured confounding, this particular study is more likely to have underestimated rather than overestimated the true effect size.

This is not to argue that the PERR should be categorically discarded in favor of PBA on Bayesian networks. Under appropriate conditions, the PERR alone will suffice. With reference to the causal diagram of Figure 1, the PERR can safely be used if all of the following apply:

A direct causal association between baseline period event and treatment can be ruled out. Such an association can masquerade as an unmeasured confounder, and naïve application of the PERR in this case acts to increase rather than decrease bias.

There is no variation in the strength of the confounder effect between baseline and observation period.

Individuals can neither gain nor lose the confounder during the study period.

A bidirectional relationship between confounder and baseline‐period event can be ruled out; that is, presence of a baseline event does not affect the probability that the confounder is present.

No direct causal association between baseline period event and observation period event exists. In practice, this is the least critical constraint, since the effect of such an association on the accuracy of the PERR estimator is modest and bounded.

If (i) or (ii) is violated, but one knows the directionality of the relationship, then, depending on the study question, the PERR may still be able to provide a useful constraint on the treatment effect.

This work is subject to a number of limitations. To begin with, although the causal diagram we use is relatively generic, it by no means constitutes an adequate description of every possible type of real‐world study. For example, loss of study subjects due to mortality or dropout is not taken into account. In our example of the study of Young‐Xu et al,9 no subjects were lost during the baseline period, and less than 3% were lost during the observation period (unpublished data). Uddin et al5 did consider loss of subjects and showed that this will cause the PERR to overestimate the treatment effect. For a loss rate of less than 5%, they showed underestimation by well below 1%. However, any study in which mortality/dropout plays a more significant role will need to add this effect into the model. Also, because PERR involves repeat measurements of the same population, correlation effects may be present. In the study of Young‐Xu et al, this issue, as well as the possibility of cluster effects at the VA facility level, was addressed by conducting a sensitivity analysis using generalized estimating equations. Even so, our analysis abstracts time dependence into simply a baseline and an observation period. Studies in which time evolution of subjects is more complex will need to utilize a correspondingly more complex DAG, for example, one in which baseline and observation are broken down into multiple subperiods. It should also be emphasized that, in our PBA re‐analysis of the study of Young‐Xu et al, the sources we use to constrain PBA inputs are derived from a targeted literature search, not an exhaustive literature review. More broadly, any analysis of this type can only be as valid as the DAG that informs it. The construction of a DAG ultimately relies on expert opinion, and no amount of expert opinion can guarantee that a causal link will not be misspecified or overlooked.

Despite the above limitations, our study offers an approach to understanding under what scenarios the PERR is likely to provide an unbiased estimate of the treatment, and a methodology to bound the bias when it is present.

DISCLOSURES

E. Thommes, R. van Aalst, J. Lee, and A. Chit are employees of Sanofi Pasteur. S. Mahmud and Y. Young‐Xu have received funding from Sanofi Pasteur in the context of this and previous studies. J. Snider is an employee of and holds equity in Precision Health Economics, which has received consulting fees from Sanofi Pasteur for this and previous studies.

DATA AVAILABILITY STATEMENT

The computer code used to generate the results of this study is included in the published article's supplementary information files.

SIM_8435‐Suppl‐0001.zip

Click here for additional data file.

Table C1

Study participants contacted and interviewed (data collected) by high dose (HD) influenza vaccine group

Proportion of		Contacted		Data Collected
HD vaccine	Selected VHA	Medical	Interviewees	Medical	Interviewees
administered*	Medical Centers	Centers	(potential)	Centers	(consented)
A: 0%‐4%	41	19	36	6	9
B: 5%‐29%	17	17	37	8	9
C: 30%‐69%	12	12	27	6	8
D: 70%‐100%	12	12	25	5	8
TOTAL	82	60	118	25	34

During the 2014‐2015 season; VHA: Veterans Health Administration.

Table C2

High dose (HD) influenza vaccine quantities ordered and used during 2015‐2016 and 2016‐2017 among study participants (Veterans Health Administration facilities) by group

Proportion of	Season 1: 2015‐2016**				Season 2: 2016‐2017**				Season 2‐Season 1 Difference***
HD vaccine	N	Ordered	Used	% Used	N	Ordered	Used	% Used	N	Ordered	Used
administered *
A: 0%‐4%	6	16 150	12 753	79.0%	6	29 400	24 208	82.3%	6	13 250 (58.2%)	11 455 (62.0%)
B: 5%‐29%	5	29 100	26 806	92.1%	3	10 050	9950	99.0%	4	−50 (−0.5%)	3644 (13.0%)
C: 30%‐69%	6	1120	1075	96.0%	6	18 000	16 371	91.0%	4	12 880 (170.4%)	12 496 (170.6%)
D: 70%‐100%	3	23 000	19 993	86.9%	4	42 290	41 356	97.8%	3	3000 (12.2%)	5073 (22.5%)
TOTAL	20	69 370	60 627	87.4%	19	99 740	91 885	92.1%	17	29 080 (44.1%)	32 668 (42.7%)

During the 2014‐2015 season.

Includes those facilities reporting ordered and used amounts.

Includes those facilities reporting ordered and/or used amounts for both seasons.

Table C3

Mechanisms of acquiring high dose (HD) influenza vaccine among study participants (Veterans Health Administration facilities) for the 2015‐2016 and 2017‐2018 seasons by group

Characteristic		Proportion of HD vaccine administered during the 2014‐2015 season
		A: 0%‐4%	B: 5%‐29%	C: 30%‐69%	D: 70%‐100%	Total
		N = 6	N = 8	N = 6	N = 5	N = 25
HD quantity decision makers	Pharmacy	5 (83.3%)	6 (75.0%)	4 (66.7%)	3 (60.0%)	18 (72.0%)
	Infection Control	1 (16.7%)	2 (25.0%)	1 (16.7%)	1 (20.0%)	5 (20.0%)
	Multidisciplinary committee	2 (33.3%)	2 (25.0%)	3 (50.0%)	3 (60.0%)	10 (40.0%)
	Department Chiefs/
	Individual Providers	2 (33.3%)	2 (25.0%)	2 (33.3%)	1 (20.0%)	7 (28.0%)
Barriers to ordering	Cost	1 (16.7%)	1 (12.5%)	0	0	2 (8.0%)
	Lack of evidence	0	3 (37.5%)	1 (16.4%)	1 (20.0%)	5 (20.0%)
	Prior waste	1 (16.7%)	1 (12.5%)	0	0	2 (8.0%)
Shortage in at least one season?		2 (33.3%)	3 (42.9%)	3 (50.0%)	3 (60.0%)	10 (40.0%)
Unused vaccine in at least one season?		4 (66.7%)	2 (28.6%)	2 (33.3%)	3 (60.0%)	11 (44.0%)
Reason for unused vaccine	Staff unaware of supply	2 (33.3%)	0	1 (16.7%)	0	3 (12.0%)
	Adherence to administration	1 (16.7)	0	0	1 (20.0%)	2 (8.0%)
	criteria
	Other (ie, extra supply
	received from other facility,
	lack of demand, low
	vaccination levels, and poor
	documentation)	1 (16.7%)	1 (12.5%)	1 (16.7%)	1 (20.0%)	4 (16.0%)
Ordered more for 2017/2018 season		3 (60.0%)	3 (42.9%)	1 (25.0%)	3 (60.0%)	10 (40.0%)

Table C4

Policies and practices for distributing high dose (HD) influenza vaccine among study participants (Veterans Health Administration facilities) for the 2015‐2016 and 2017‐2018 seasons by group

Characteristic		Proportion of HD vaccine administered during the 2014‐2015 season
		A: 0%‐4%	B: 5%‐29%	C: 30%‐69%	D: 70%‐100%	Total
		N = 6	N = 8	N = 6	N = 5	N = 25
Standard nursing order/protocol		6 (100%)	6 (75.0%)	4 (66.7%)	4 (80.0%)	20 (80.0%)
	Applies?	6 (100%)	4 (50.0%)	3 (50.0%)	5 (100%)	18 (72.0%)
Facility‐wide policy	Age (65+)	6 (100%)	3 (37.5%)	3 (50.0%)	5 (100%)	17 (68.0%)
	Immuno‐
	compromised/HIV	1 (16.7%)	3 (37.5%)	0	0	4 (16.0%)
	Chronic	1 (16.7%)	1 (12.5%)	1 (16.7%)	0	3 (12.0%)
Department‐specific policy/practice	Applies?	2 (33.3%)	2 (33.3%)	3 (50.0%)	0	10 (40.0%)
	CLC/HBPC	1 (16.7%)	5 (62.5%)	3 (50.0%)	0	9 (36.0%)
	Acute care	1 (16.7%)	2 (25.0%)	0	0	3 (12.0%)
	Specialty Clinics	0	3 (37.5%)	0	0	3 (12.0%)
Provider discretion	Applies?	5 (83.3%)	7 (87.5%)	5 (83.3%)	3 (60.0%)	20 (80.0%)
	MD	5 (83.3%)	7 (87.5%)	5 (83.3%)	3 (60.0%)	20 (80.0%)
	NP/PA	4 (66.7%)	7 (87.5%)	4 (66.7%)	3 (60.0%)	18 (72.0%)
	RN	2 (33.3%)	6 (75.0%)	1 (16.7%)	1 (20.0%)	10 (40.0%)
	LPN	0	1 (12.5%)	0	0	1 (4.0%)
Target patients at high‐risk not covered by policies		6 (100%)	6 (75.0%)	3 (50.0%)	3 (60.0%)	18 (72.0%)

Table C5

High‐risk patient criteria for high dose (HD) influenza vaccination administration among study participants (Veterans Health Administration facilities) for the 2015‐2016 and 2017‐2018 seasons by group

Characteristic		Proportion of HD vaccine administered during the 2014‐2015 season
		A: 0%‐4%	B: 5%‐29%	C: 30%‐69%	D: 70%‐100%	Total
		N = 6	N = 8	N = 6	N = 5	N = 25
	65+	6 (100%)	6 (75.0%)	5 (83.3%)	5 (100%)	22 (88.0%)
Age	80+	1 (16.7%)	0	0	0	1 (4.0%)
	85+	0	0	0	1 (20.0%)	1 (4.0%)
Home‐Based Primary Care patient		2 (33.3%)	1 (12.5%)	1 (16.7%)	0	4 (16.0%)
Inpatient (acute care)		1 (16.7%)	1 (12.5%)		0	2 (8.0%)
Community Living Center resident		0	5 (62.5%)	3 (50.0%)	1 (20.0%)	9 (36.0%)
Being seen inf high risk clinic (eg, geriatrics, Infectious Diseases,		1 (16.7%)	2 (25.0%)	0	0	3 (12.0%)
HIV, pulmonary, nephrology, oncology, and rheumatology)
Immunocompromised		3 (50.0%)	2 (25.0%)	3 (50.0%)	1 (20.0%)	9 (36.0%)
HIV positive		1 (16.7%)	2 (25.0%)	0	0	3 (12.0%)
	HIV/AIDs w/low CD‐4 meter	1 (16.7%)	0	0	0	1 (4.0%)
Renopathic condition		1 (16.7%)	1 (12.5%)	0	0	2 (8.0%)
	on dialysis	1 (16.7%)	0	0	0	1 (4.0%)
Diabetes		1 (16.7%)	2 (25.0%)	1 (16.7%)	0	4 (16.0%)
	uncontrolled	1 (16.7%)	0	0	0	1 (4.0%)
Respiratory issues		1 (16.7%)	1 (12.5%)	0	0	2 (8.0%)
	Asthmatic	1 (16.7%)	1 (12.5%)	0	0	2 (8.0%)
Chronic pulmonary condition		1 (16.7%)	1 (12.5%)	1 (16.7%)	1 (20.0%)	4 (16.0%)
	Chronic obstructive pulmonary disease	1 (16.7%)	1 (12.5%)	1 (16.7%)	0	3 (12.0%)
Cardiovascular condition		1 (16.7%)	1 (12.5%)	3 (50.0%)	0	5 (20.0%)
	Heart disease	0	0	2 (33.3%)	0	2 (8.0%)
	Congestive heart failure	1 (16.7%)	1 (12.5%)	1 (16.7%)	0	3 (12.0%)
Neurological condition		0	1 (12.5%)	0	0	1 (4.0%)
Neuropathic condition		0	1 (12.5%)	0	0	1 (4.0%)
Other/unspecified chronic condition		0	1 (12.5%)	0	0	1 (4.0%)
Blood‐borne infection		0	0	1 (16.7%)	0	1 (4.0%)
Flu in the previous season		0	0	1 (16.7%)	0	1 (4.0%)

Table C6

Contra‐indication criteria for high dose (HD) influenza vaccination administration among study participants (Veterans Health Administration facilities) for the 2015‐2016 and 2017‐2018 seasons by group

Characteristic	Proportion of HD vaccine administered during the 2014‐2015 season
	A: 0%‐4%	B: 5%‐29%	C: 30%‐69%	D: 70%‐100%	Total
	N = 6	N = 8	N = 6	N = 5	N = 25
Reported contra‐indication	3 (50.0%)	3 (37.5%)	3 (50.0%)	2 (40.0%)	11 (44.0%)
History of allergic reaction to influenza vaccine or its components	3 (50.0%)	1 (12.5%)	2 (33.3%)	2 (40.0%)	8 (32.0%)
History of allergic reaction to eggs	1 (16.7%)	0	0	0	1 (4.0%)
Allergy to latex or thimerosal	0	1 (12.5%)	0	0	1 (4.0%)
Already immunized	1 (16.7%)	0	1 (16.7%)	0	2 (8.0%)
History of Guillain‐Barre syndrome	1 (16.7%)	0	0	1 (20.0%)	2 (8.0%)
Not feeling well	1 (16.7%)	0	0	0	1 (4.0%)
Moderate to severe acute illness	1 (16.7%)	0	1 (16.7%)	0	2 (8.0%)
Presence of fever	1 (16.7%)	2 (25.0%)	1 (16.7%)	1 (20.0%)	5 (20.0%)
Chemotherapy of radiation therapy	1 (16.7%)	1 (12.5%)	0	0	2 (8.0%)
Pre‐op or pre‐procedure for invasive procedures/surgeries	1 (16.7%)	0	0	0	1 (4.0%)
Patient being administered Coumadin	0	1 (12.5%)	0	0	1 (4.0%)
Hospice care	0	1 (12.5%)	0	0	1 (4.0%)
Person without capacity to make an informed decision	0	1 (12.5%)	0	0	1 (4.0%)
Physician discretion	0	0	1 (16.7%)	0	1 (4.0%)
Recent pneumonia hospitalization	0	0	2 (33.3%)	0	2 (8.0%)