The number needed to treat adjusted for explanatory variables in regression and survival analysis: Theory and application.

The number needed to treat (NNT) is an efficacy index commonly used in randomized clinical trials. The NNT is the average number of treated patients for each undesirable patient outcome, for example, death, prevented by the treatment. We introduce a systematic theoretically-based framework to model and estimate the conditional and the harmonic mean NNT in the presence of explanatory variables, in various models with dichotomous and nondichotomous outcomes. The conditional NNT is illustrated in a series of four primary examples; logistic regression, linear regression, Kaplan-Meier estimation, and Cox regression models. Also, we establish and prove mathematically the exact relationship between the conditional and the harmonic mean NNT in the presence of explanatory variables. We introduce four different methods to calculate asymptotically-correct confidence intervals for both indices. Finally, we implemented a simulation study to provide numerical demonstrations of the aforementioned theoretical results and the four examples. Numerical analysis showed that the parametric estimators of the NNT with nonparametric bootstrap-based confidence intervals outperformed other examined combinations in most settings. An R package and a web application have been developed and made available online to calculate the conditional and the harmonic mean NNTs with their corresponding confidence intervals.

THE NNT

Introduction

The number needed to treat (NNT) is an index that is widely used in efficacy analysis and cost‐effectiveness analysis in randomized clinical trials, as well as in epidemiology and meta‐analysis. , , , , , It is assumed that the outcome is dichotomous and may be beneficial or nonbeneficial. The NNT is the average number of patients that have to be treated in order to observe one less adverse outcome, or alternatively, the NNT can be defined as the average number of patients that are needed to treat in order to attain one more beneficial outcome due to treatment. These are two equivalent definitions since avoiding an adverse effect can be defined as a treatment benefit. Let the probability of treatment benefit be ; hence the NNT was initially defined by Laupacis et al as . Consider now that the probability of a beneficial outcome for treated patients is composed of two additive components: the probability of a beneficial outcome due to treatment (ie, the probability of treatment benefit), plus the probability of a beneficial outcome that is not due to treatment. Therefore, the probability of treatment benefit is defined as , that is termed the absolute risk difference (ARD). Although the NNT is a commonly used efficacy measure, it is not without limitations. Among the limitations of the NNT are difficulties in its interpretation, , , , , a bi‐modal sampling distribution and unbounded disjoint confidence intervals (CIs). There are five main criticisms of the statistical properties of the NNT estimator. First is the singularity at 0 of the inverse of the ARD. Most of the criticism is directed toward this issue and its consequences, which include the difficulty to construct and interpret its CIs. , In particular, Grieve applied Bayesian analysis to construct CIs with respect to the original definition of the NNT. Rohmel proposed that if the ARD is statistically significant and does not contain zero, it is appropriate to invert the CIs. A comprehensive review of the NNT's statistical limitations can be found in Hutton. Recently Vancak et al resolved the pitfall of singularity at 0 by introducing a modification to the original definition of the NNT. The modified NNT is We adopt this modified version of the NNT. Laupacis et al proposed estimating this unadjusted NNT by replacing the unknown probabilities of a beneficial outcome in each arm with the corresponding proportions of beneficial outcomes. Therefore, the second criticism pointed to the bias of Laupacis' estimator with respect to the true NNT. Third, the NNT is well defined only for dichotomous outcomes. However, in many clinical settings, the outcome is nondichotomous. Therefore, a dichotomization of the outcome is required, which has limitations such as loss of information. , Dichotomization often relies on the definition of the minimal clinically important difference (MCID) that is denoted by . Therefore, for nondichotomous outcomes, without loss of generality, we define the beneficial outcome as , where if the beneficial outcome occurs, and 0 otherwise. Fourth, the NNT does not account for time‐dependent outcomes and thus can be misleading. , Fifth, the interpretation and the definition of the NNT are debatable since different clinical scenarios may result in the same NNT. Grieve, and Hutton challenged the common claim that the NNT is an easily interpretable index by emphasizing the statistical properties of the estimator of the NNT that are commonly neglected. Suissa and Smeeth et al present its miscalculations in various settings, while Kristiansen et al present misinterpretation of the original NNT using an empirical study. We suggest that Vancak et al's modification of the NNT makes it easier to understand and interpret the index, and the this article clarifies the appropriate use of the NNT with time‐dependent outcomes.

In Section 2, we introduce the conditional NNT, which is an NNT that is conditioned on a given value of the explanatory variables. Next, we present the harmonic mean NNT (hereafter, harmonic NNT), which is defined by applying the function as defined in (1) to the marginal probability of treatment benefit. This presentation is followed by the derivation of these NNTs' corresponding asymptotically unbiased and efficient estimators, alongside their asymptotic distributions. In this section, we discuss two primary examples: logistic regression and linear regression. In Section 3, we derive the NNT accommodated to right‐censored data conditioned and unconditioned on explanatory variables. Subsequently, we define the harmonic NNT accommodated to right‐censored data. Then, we derive their corresponding asymptotically unbiased estimators alongside the asymptotic distributions. In this section, we discuss two main examples that are based on the Kaplan‐Meier and Cox model. For all estimators we provide asymptotically correct CIs using four different methods: transformation, delta method, nonparametric and parametric bootstrap. In Section 4, we present a simulation study to illustrate the conditional and the harmonic NNTs in the examples discussed above. Notably, we compare point estimators, the lengths, and the coverage rates of the four CIs methods. Finally, we refer readers to the nntcalc R package and the corresponding web application to calculate the conditional and the harmonic NNTs with their corresponding CIs. The R package and the web application are made available for users online. Detailed proofs appear in the Appendix.

ADJUSTED NNT IN REGRESSION ANALYSIS

Adjustment of the NNT

The need for adjustment of the NNT was recognized in parallel with the increased popularity of this index. The first known attempt to adjust the NNT was by Riegelman and Schroth. Ebrahim, and Misselbrook and Armstrong, further acknowledged the need for a specified adjustment and made the distinction between the overall and the conditional NNT. In particular, Ebrahim acknowledged the need to condition the NNT on both time and explanatory variables. There, regression was advocated for that purpose, however, practical tools were not presented. Altman and Andersen conducted initial research on accommodating the NNT to right‐censored data based on the Kaplan‐Meier estimators. A detailed criticism of the use of NNT in the context of survival analysis and time‐to‐event data was presented by Kristiansen and Dorte, and further by Snapinn and Jiang. The main points of criticism were that the NNT may vary substantially over time, and hence convey different information as a function of the specific time‐point of its calculation. Moreover, Snapinn and Jiang showed examples where the information conveyed by the NNT may be incomplete or even contradictory compared to the traditional statistics of interest in survival analysis. Ola et al adjusted the NNT for a particular explanatory variable in the context of cost‐effectiveness analysis and redefined it as the cost needed to treat. More comprehensive research on adjustment of the NNT for explanatory variables was conducted by Bender and Blettner and Austin.

Conditional and harmonic NNT in regression analysis

Bender and Blettner and Austin adjusted the NNT for explanatory variables in logistic regression. In particular, Bender and Blettner and Bender et al adjusted the NNT in the context of cohort data. In this situation, the covariates may have a different distribution in the exposed and the unexposed arms. Therefore, a distinct NNT was proposed for each arm. Notably, to adjust the NNT to covariate distributions in the control (unexposed) arm, they renamed it the Number Needed to be Exposed (NNE). Furthermore, to account for the effect direction, they divided the NNE into NNEH and NNEB, where H stands for a harmful effect (ie, negative valued NNE), and B for a beneficial effect (ie, positively valued NNE). In these two works, the authors suggested to use the multivariate delta method for the CI construction of the adjusted NNE. In addition to the NNE, they presented the Exposure Impact Number (EIN) to adjust the NNT to the covariates' distribution in the treated (exposed) arm. , Later on, Bender and Vervolgyi, presented the estimation of harmonic NNT, based on the fit of the logistic regression model in the context of randomized clinical trials. In all these works, the authors used the multivariate delta method to compute the CIs of the index. In the context of time‐to‐event, Altman and Andersen proposed inversion of the CIs of the absolute risk reduction to construct CIs for the NNT. Austin introduced the harmonic NNT and suggested using the nonparametric bootstrap for construction of its CI.

Definition of the conditional NNT

In this subsection we introduce the adjusted NNT for explanatory variables and/or covariates in the context of regression analysis. Consider the following scenario: At the baseline, for every clinical trial study participant, a vector of background measurements is taken. Each clinical trial study participant is randomly allocated either to the control or the treatment arm . Usually the explanatory variables include sex, age, baseline severity and treatment. Randomization ensures that all baseline covariates of patients in both treatment groups share the same joint parent distribution. The outcome variable , defined as in Section 1, is a scalar function of the baseline and the endpoint measurements. Without loss of generality, let the beneficial outcome be defined as ; hence, conditioning on the allocated arm is defined as , where . Define the marginal and the conditional probability of beneficial outcome in the th arm by and , respectively. Consequently, we define the conditional probability of treatment benefit given a covariate as where is the support set of the covariate . Hence, we define the conditional NNT for every in as The conditional NNT allows us to calculate the NNT for every possible value of the covariate . Equation 4 below establishes the connection between the conditional NNT and the harmonic NNT. Let the NNT be as defined in (1) and the NNT as defined in (3). Then, Equation (4) states that, since an expectation of the conditional probability of treatment benefit w.r.t. results in the average (marginal) probability of treatment benefit , therefore, the harmonic NNT can be calculated by applying as defined in (1) to . Moreover, as is a convex function for all in , by applying Jensen's inequality it is evident that Averaging the NNT instead of averaging the will result in biased NNT. Moreover, it may lead to a wrong conclusion since the distribution of can be dominated by its extreme values. Consider, for example, a scenario where there is a possible realization of the covariates such that , while for every other realization , , . Therefore, NNT: thus the NNT computed as a mean on the NNT scale equals infinity, that is, . However, if , consequently, the NNT computed as a mean on the risk scale is finite, that is, NNT .

Estimation of NNT

This section introduces the parametric approach to conditional NNT estimation. Assume that the conditional probability of treatment benefit can be described via a vector of unknown parameters . Assume that the probability of beneficial outcome in the th arm is , for . Consequently, , and by (3), NNTNNT. Notably, is the average treatment effect that is commonly a nonlinear function of the covariates . The fundamental problem of causal inference states that it is impossible to observe both and within the same clinical trial study participant, as the participant is allocated to either the treatment or the control arm. Therefore, we observe only one of the two possibilities, while the missing one can be estimated using a model. Thus, we estimate NNT by replacing the unknown parameters with their corresponding point estimators.

For the estimation of the harmonic NNT, the marginal distribution of is required. Usually such a distribution is unavailable; therefore we propose to estimate using the corresponding sample average and then applying as defined in (1) to the result, that is, Namely, for the harmonic NNT, we use a parametric model to estimate the conditional risk difference given a set of covariates , then average it over the empirical distribution of the covariates, and finally take its inverse. Next we present two examples to compute the conditional and the harmonic NNTs: logistic regression and linear regression.

Example 1: NNT in logistic regression

Let be the vector of explanatory variables with a support set , where . Assume that the response variable is dichotomous, that is, . Without loss of generality, let the beneficial outcome be defined as . Assume that given realization , and an allocated arm , follows a Bernoulli distribution with probability , where In this model, , where , for . It can be shown that the conditional probability of treatment benefit is The MLE of the NNT is attained by applying to and replacing the unknown parameters with their corresponding MLEs. The MLE of the harmonic NNT is calculated by applying to the corresponding sample average of as defined in (6).

Example 2: NNT in normal linear regression

Let be the vector of explanatory variables with a support set , where . Assume that the response variable , given a realization and an allocated arm , follows a normal distribution . In this model, , where , for , and the formal model is It can be shown that the conditional probability of treatment benefit is The MLE of the conditional NNT is attained by applying to and replacing the unknown parameters with their corresponding MLEs. The MLE of the harmonic NNT is calculated by applying to the corresponding sample average of as defined in (6). Note that for these two examples, model parameters can be shared by both arms. In other words, and can be modeled either separately with different parameters and or jointly with common parameters that are shared by models of the two arms. These specifications change neither the estimation nor the asymptotic properties of the NNT estimators.

Theoretical properties of the NNT estimator

To this end, the following assumptions are required.

The response variable , given and , follows a parametric model with a true parameter . In this case, the true conditional NNT is given by NNT.

There are observations in the th arm, , and is the total number of observations. Assume that , as .

The probability of treatment benefit, , is a differentiable function w.r.t. ; hence the composite function where as defined in (1), is also a differentiable function w.r.t. for , and .

The standard regularity conditions of the MLEs asymptotics hold.

, and .

Let be the MLE of , its fisher information matrix, and . Then,

and are the MLEs of the conditional NNT and the harmonic NNT, respectively.

NNT , and NNT.

For every , such that ,

,

where and are the gradients of NNT and NNT, respectively, evaluated at .

For a detailed proof of Theorem 1 please refer to the Appendix. Notably, for every such that , NNT. Therefore, the MLE of NNT converges almost surely to infinity. In such a case, discussion of the NNT distribution is meaningless.

CIs of harmonic NNT and conditional NNT

In this subsection we present four different methods of CI construction. These methods will be further used to construct asymptotically‐correct CIs for the conditional and the harmonic NNT. These results are summarized in Theorem 2.

Approach 1: Transformation

Let be a sequence of independent identically distributed (i.i.d) random variables with and . Denote . By the univariate Central Limit Theorem (CLT) where , and . For a monotonically decreasing function , for example, the one defined in (1), the univariate‐CLT‐based asymptotically‐correct level CI for is where is the th quantile of the standard normal distribution.

Approach 2: Delta method

Let be a sequence of independent and identically distributed dimensional random vectors, with and , where is a positive definite covariance matrix of . Denote . By the multidimensional CLT where . Let be a differentiable function over the parametric set with a nonzero gradient . Consequently, the delta‐method‐based asymptotically‐correct level CI for is where is the Euclidean norm.

Approach 3: Nonparametric bootstrap

Assume that the original sample is of realizations of a random vector . The constant of interest is a real valued scalar function at some point , that is, . The bootstrap algorithm takes samples with replacements from the aforementioned observations. This is repeated times with samples, each of size . For each of these samples , an estimator is calculated. Using the and the quantiles of the empirical distribution of we obtain a nonparametric‐bootstrap‐based asymptotically‐correct level CI

Approach 4: Parametric bootstrap

Assume that the original sample consists of observations, which are assumed to be realizations of a parametric distribution that depends on the unknown vector of parameters . This vector is estimated by . Assume that the sample distribution of the estimator is . The parametric bootstrap algorithm takes the estimator and its sampled covariance matrix as a replacement of , and the corresponding covariance matrix . Then we sample times, where is some large positive integer, from . For each of the samples of parameters , a function , for , is calculated, where is a scalar function. This results in a sample of estimators of . Using the normality of this sample, the parametric‐bootstrap‐based asymptotically‐correct level CI is where is the bootstrap sample standard deviation. An advantage of the parametric bootstrap is that in order to obtain an effective CI, usually significantly fewer samples are required, compared to its nonparametric counterpart.

Let assumptions A.1‐A.5 hold, and let , and be the MLE of NNT and NNT, respectively. Therefore, for a fixed , where , the asymptotically‐correct transformation‐based, delta‐method‐based, nonparametric‐bootstrap‐based, and parametric‐bootstrap‐based ‐level CIs are given in ( ), ( ), ( ), and ( ), respectively. In this case, and , for NNT and NNT, respectively. The function is defined in ( ), and is the inverse of the observed Fisher information matrix evaluated at , that is, .

Note that since NNT for all , the lower confidence limit of NNT can be truncated at 1 with no effect on the coverage rate.

HARMONIC NNT AND CONDITIONAL NNT IN SURVIVAL ANALYSIS

Laubender and Bender, Austin, and Yang and Yin presented and elaborated on an NNT for right‐censored data in the Cox proportional hazard and parametric accelerated failure models. Laubender and Bender , derived the harmonic NNE and EIN in the framework of the Cox model, with four types of CIs. Three types were based on the resampling method conditional on the covariates: the normal approximation, the basic bootstrap and the percentile bootstrap. The fourth type was a multivariate delta‐method CI based on the theory of martingales. Yang and Yin proposed an alternative measure to the NNT that is based on the restricted mean survival time instead on the absolute risk difference. All these works analyzed model‐specific scenarios with naturally dichotomous outcomes without providing a general modeling framework. The singularity point of the original definition often led to categorization of the NNT into different measures as a function of the allocated arm and the sign of its estimator, and resulted in CIs that consist of a union of two disjoint infinite intervals. In addition, different authors used varying approaches and terminology to define and compute the conditional and the harmonic NNTs.

Definition of NNT and NNT

Let the outcome variable be the time to event or time until death. For a fixed time point , define the beneficial outcome as . Let the cumulative distribution of be , and define the survival function as . In survival analysis we may not observe the realization due to loss of follow‐up or for other reasons. Namely, we assume that all clinical trial study participants start at time 0, but some are censored before the end of the trial. Therefore, we observe , where is right‐censoring time. The beneficial outcome, given allocated arm , is , where . The marginal probability of a beneficial outcome in the th arm is ; thus, the marginal probability of treatment benefit is Consequently, we define the NNT as Let be the risk factors. Define the conditional NNT as where , and is the function defined in (1). Similarly to the harmonic NNT, the harmonic NNT is calculated by applying to , where the expectation is taken w.r.t. .

Let be an unspecified survival function of the th arm, , at a fixed time point . To estimate the NNT, we need a suitable estimator of , which is also the marginal probability of a beneficial outcome in the th arm. Let be the total number of clinical trial study participants that are observed at the baseline time point . The next example and the subsequent theorems present the nonparametric MLE of NNT and its theoretical properties.

Example 3: NNT using the Kaplan‐Meier nonparametric MLE

Let be the product limit Kaplan‐Meier's nonparametric MLE; formally, for any fixed time we have where is the number of clinical trial study participants in risk at time , and is the total number of failures at time in the th arm. Therefore, we define the NNT to be the estimator of NNT by applying as defined in (1) to the difference between the corresponding Kaplan‐Meier's estimators. Formally, where . Notably, NNT can be stratified by more than two levels. Furthermore, when there is no censoring, it can be shown that NNT coincides with Laupacis NNT, where the beneficial outcome is defined as . See Lemma 2 in the Appendix for detailed proof. Similar results for the parametric logistic regression and the parametric linear regression approaches are presented in Section 2.4.

Theoretical properties of the NNT's estimator

The asymptotic properties of NNT's parametric estimator are presented in Theorem 1, by replacing with . Therefore, we focus only on the theoretical properties of the nonparametric MLE. For further theorems we need another assumption.

Let be the follow‐up time interval, where is a time point such that there is at least one more observation to its right in each of the arms.

Let assumptions and hold, and assume that the survival time is independent of the right‐censoring time , and . Then, for any fixed time point in

is the nonparametric MLE of NNT .

.

For every fixed , such that , , where is the asymptotic variance of NNT .

For a detailed proof of Theorem 3 please refer to the Appendix. We proceed to the construction of CIs that are based on the large‐sample distribution of the NNT.

For every fixed , such that , the asymptotically‐correct transformation‐based, delta‐method‐based, nonparametric‐bootstrap‐based, and parametric‐bootstrap‐based ‐level CIs are given in ( ), ( ), ( ), and ( ), respectively. In this case, , and is NNT , where is the estimated variance of that is based on Greenwood's formula.

Estimation of the NNT

If the structure of is assumed to be fully parametric, then the estimation is done in the same manner as in the parametric models of conditional NNT, that is, by replacing the unknown parameters with their estimators. Therefore, we focus our attention only on the semiparametric structures. We start with a definition of the hazard function. The hazard function is the instantaneous rate of occurrence of the event. Formally, Therefore, the probability of beneficial outcome in the th arm, , can be expressed as a function of the cumulative hazard function up to time , , that is, where . Next, define to be the hazard function given the risk factors , and an allocated arm . Therefore, the conditional probability of beneficial outcome in the th arm is . The most commonly‐used model for the conditional hazard function is Cox's proportional hazard model.

Example 4: NNT in Cox regression

Let be the vector of risk factors with a support set . Assume that given a realization , and an allocated arm , the hazard function is Namely, the baseline hazard has a nonparametric structure, while the adjustment for the risk factors follows a parametric model. In this model, , where is the cumulative baseline hazard function up to time , , and , for . The conditional cumulative hazard function in the th arm is . Therefore, using (22), the conditional probability of beneficial outcome in the th arm is where is the baseline survival probability. Following (2), the conditional probability of treatment benefit is The semiparametric estimator of the conditional NNT, is attained by applying the function from (1) to the estimated , where is replaced with its maximum partial likelihood estimators (MPLE), and with Breslow's nonparametric MLE. The partial likelihood does not depend on the baseline hazard function , which allows estimating the model coefficients without dealing with the nonparametric baseline hazard, which is estimated afterward using the calculated MPLEs. The calculated semiparametric estimator of the conditional NNT we denote by NNT since it is based on the Cox proportional hazard regression model and is estimated via the Cox partial likelihood method. Under this model, the estimator of the harmonic NNT, which is denoted by NNT, is calculated by applying to the corresponding sample average of as defined in (6).

In this section we present the asymptotic analysis of the conditional NNT semiparametric estimator and its corresponding CIs. The generalization of the presented results to other survival models is straightforward using the principles presented below. Thus, we will focus only on the Cox model. To this end, additionally to A.1‐A.6., we require the next two assumptions.

is bounded uniformly in the neighborhood of .

for all finite and positive .

Assume that A.1 to A.8 hold, and that the true model follows the Cox proportional hazard structure with an unspecified baseline hazard, such that and are independent conditionally on the risk factors . Let and be the MPLE and Breslow's nonparametric MLE, respectively. Then, for any fixed time point , and for all ,

NNT NNT , and NNT NNT .

For every fixed time point , such that ,

where and are the gradients of NNT and NNT , respectively, evaluated at , is the Fisher information matrix, and .

For a detailed proof of Theorem 5 please refer to the Appendix. The construction of the CIs is done in the same manner as in Theorem 2, and presented in Theorem 6.

Let assumptions A.1 to A.8 hold. For every , and a fixed time point , such that , the asymptotically‐correct transformation‐based, delta‐method‐based, nonparametric‐bootstrap‐based, and parametric‐bootstrap‐based ‐level CIs are given in ( ), ( ), ( ), and ( ), respectively. In this case, , and , for NNT and NNT , respectively. The sample covariance matrix is estimated using Fisher's observed information matrix that is based on Cox MPLEs of .

For a detailed derivation of the sample variance of NNT in Theorem 6, please refer to the Appendix. For every fixed , both empirical processes , and converge weakly to a corresponding Gaussian process. These processes have zero mean and covariance structures that can be derived from lemma 6.1 in Tsiatis.

SIMULATION STUDY

In this section, a simulation analysis of the aforementioned four examples (logistic regression, linear regression, Kaplan‐Meier, and Cox regression) to compute the conditional and the harmonic NNTs and their corresponding 95% level CIs is presented. The harmonic NNTs were calculated, as described in (6), using the empirical distribution of the covariates. For all settings, sample sizes of 200, 400, and 800 were implemented, with 400 iterations for each sample size. The simulation results of the point estimators and the lengths of the corresponding finite CIs are summarized using boxplot charts. In addition, tables of the CIs' mean coverage rates are presented. This simulation can be replicated using the R package that has been developed and made available for users from the author's GitHub repository. Setting I illustrates the logistic regression model with a continuous explanatory variable. Setting II illustrates the Kaplan‐Meier and the Cox proportional hazard models under the Weibull distribution. Setting III (please see Appendix) illustrates the linear regression model with a continuous explanatory variable. For all settings, the true harmonic NNT was computed by Monte Carlo numerical integration.

Setting I: Logistic regression

In this setting, the explanatory variable is a normally distributed variable with and . The response variable , given a realization and an allocated arm , follows a Bernoulli distribution with probability , where , and . Without loss of generality, the beneficial outcome is defined as . The conditional NNT can be calculated for every . However, for this illustration we chose only three representative values: 1.5, 2, and 2.5. The true values of NNT, NNT, and NNT are 3.32, 4.33, and 6.46, respectively. The true value of the harmonic NNT, computed over the full covariate distribution used in the simulation, is 4.54. The conditional NNT's, , point estimators and the corresponding CIs with their mean coverage rates are illustrated in Figures 1 and 2, and Table 1. The point estimators of the harmonic NNTs with the corresponding CIs and their mean coverage rates are presented in Figures 3 and 4, and Table 2.

FIGURE 1

Parametric MLEs of the Conditional NNT, for , in the logistic regression model (for a formal definition of NNT in the logistic regression model see Equation (7)), for

FIGURE 2

CI lengths by construction method (DL, delta method; NBS, nonparametric bootstrap; PBS, parametric bootstrap; TR, transformation) of the conditional NNT, , in the logistic regression model, for . The box‐plots of certain transformation‐based CIs were not displayed, since they are either infinite or too large, and thus distort the figure

TABLE 1

Setting I: conditional NNT, for , in the logistic regression model (for a formal definition of NNT in the logistic regression model see Equation (7))

	NNT(1.5)				NNT(2)				NNT(2.5)
N	DL	TR	NBS	PBS	DL	TR	NBS	PBS	DL	TR	NBS	PBS
200	1.00	1.00	0.94	0.93	1.00	1.00	0.95	0.93	1.00	1.00	0.94	0.59
400	1.00	1.00	0.94	0.93	1.00	1.00	0.94	0.92	1.00	1.00	0.95	0.80
800	1.00	1.00	0.94	0.94	1.00	1.00	0.94	0.94	1.00	1.00	0.94	0.92

Note: Mean coverage rates of the pointwise CIs by construction method (DL, delta method; NBS, nonparametric bootstrap; PBS, parametric bootstrap; TR, transformation), and sample sizes of .

FIGURE 3

Harmonic NNT in the logistic regression model, for . Parametric MLE; NNT MLE. Laupacis' nonparametric MLE; NNT L

FIGURE 4

CI lengths by construction method (DL, delta method; NBS, nonparametric bootstrap; PBS, parametric bootstrap; TR, transformation) of the harmonic NNT in the logistic regression model, using the parametric (NNT MLE) and the nonparametric (NNT L) MLEs, for . For the parametric MLE, the box‐plots of transformation‐based CIs were not displayed, since they are either infinite or too large, and thus distort the figure

TABLE 2

Setting I: harmonic NNTs in the logistic regression model

	NNT_MLE				NNT_L
N	DL	TR	NBS	PBS	DL	TR	NBS
200	1.00	1.00	0.95	1.00	0.86	0.95	0.95
400	1.00	1.00	0.95	1.00	0.94	0.96	0.96
800	1.00	1.00	0.94	0.94	0.85	0.91	0.91

Note: Mean coverage rates of the pointwise CIs by construction method (DL, delta method; NBS, nonparametric bootstrap; PBS, parametric bootstrap; TR, transformation), and sample sizes of . Based on the parametric maximum likelihood estimator, NNT, and the nonparametric maximum likelihood estimator, NNT.

Setting II: Survival analysis

In this setting, the data were generated with the survsim R package. Specifically, the explanatory variable is a normally distributed variable with and . The allocated arm is a binomial random variable with probability of 0.5, and . The response variable , given a realization and an allocated arm , follows the Weibull distribution with , where , , , and . Consequently, the conditional hazard function is such that , and . The beneficial outcome is defined as . The conditional NNT can be calculated explicitly for every realization , and any time point . We chose a representative time point . For each time point , we chose three representative values of : , , and . The censoring mechanism follows the Weibull distribution with , and . The overall resulting censoring rate was approximately . Particularly, the event rate in the control arm was approximately , and in the treatment arm was approximately . However, the mean follow‐up time in the treatment arm was 5.75 time units, while in the control arm 1.03 time units. The true values of the conditional NNT, are 9.9, 6.18, and 4.40, for of , , and , respectively. The true value of the harmonic NNT(8), computed over the full covariate distribution used in the simulation for this time point, is 4.43. The point estimators of the conditional NNT and the corresponding CIs with their mean coverage rates are illustrated in Figures 5 and 6, and Table 3. The point estimators of the harmonic NNT(8) with the corresponding CIs and their mean coverage rates are presented in Figures 7 and 8, and Table 4. For Setting III of Simulation study that illustrates the linear regression model, please refer to the Appendix.

FIGURE 5

Semiparametric estimators of the conditional NNT, for , and , using the Cox regression model (for a formal definition of NNT in survival analysis see Equation (23)), for

FIGURE 6

CI lengths by construction method (DL, delta method; NBS, nonparametric bootstrap; PBS, parametric bootstrap; TR, transformation) of the conditional NNT, for , and , using the Cox regression model, for . The box‐plots of certain transformation‐based CIs were not displayed, since they are either infinite or too large, and thus distort the figure

TABLE 3

Setting II: conditional NNTs, for , and , in the Cox regression model (for a formal definition of NNT in survival analysis see Equation (23))

	NNT(8|−2.5)				NNT(8|−2)				NNT(8|−1.5)
N	DL	TR	NBS	PBS	DL	TR	NBS	PBS	DL	TR	NBS	PBS
200	1.00	0.99	0.96	0.91	1.00	1.00	0.96	0.90	0.99	0.99	0.95	0.91
400	1.00	1.00	0.94	0.86	1.00	1.00	0.94	0.82	1.00	0.99	0.94	0.86
800	1.00	1.00	0.95	0.83	1.00	1.00	0.94	0.80	0.99	1.00	0.94	0.81

Note: Mean coverage rates of the pointwise CIs by construction method (DL, delta method; NBS, nonparametric bootstrap; PBS, parametric bootstrap; TR, transformation), and sample sizes of .

FIGURE 7

Harmonic NNT in the Cox regression model, for , and . Cox semiparametric estimator NNT; NNT‐COX(8). Kaplan‐Meier nonparametric MLE NNT; NNT‐KM(8)

FIGURE 8

Pointwise CI lengths by construction method (DL, delta method; NBS, nonparametric bootstrap; PBS, parametric bootstrap; TR, transformation) of the harmonic NNT(8) in survival analysis, based on the semiparametric estimator (NNT(8) Cox) and the nonparametric MLE (NNT(8) KM), for . For the nonparametric NNT(8), the box‐plots of transformation‐based CIs were not displayed, since they are either infinite or too large, and thus distort the figure

TABLE 4

Setting II: harmonic NNT, for , in the Cox regression model

	NNT_COX(8)				NNT_KM(8)
N	DL	TR	NBS	PBS	DL	TR	NBS
200	0.99	0.97	0.94	0.96	1.00	1.00	0.94
400	1.00	0.99	0.95	0.92	1.00	1.00	0.95
800	0.99	0.99	0.94	0.92	1.00	1.00	0.93

Note: Mean coverage rates of the pointwise CIs by construction method (DL, delta method; NBS, nonparametric bootstrap; PBS, parametric bootstrap; TR, transformation), and sample sizes of . Based on the semiparametric estimator, NNT, and the nonparametric maximum likelihood estimator, NNT, for .

Simulation summary

In all settings: For the conditional NNTs, the parametric estimators converge to their true values. For the harmonic NNT, where the model is correctly specified, both the parametric and the nonparametric estimators converge to the true value of the parameter. The correctly specified parametric estimators appear to be more stable compared to the nonparametric alternative.

In all settings, the transformation‐based CIs appear to be consistently larger compared to other alternatives. Frequently, infinitely large. Consequently, their coverage rates were frequently . This can be explained by the behavior of the function as defined in (1), in the vicinity of . Namely, for small positive values of , behaves as ; hence large standard errors of the estimated probability of treatment benefit result in wide CIs. All the three other methods of CIs construction mitigate this sensitivity by taking the standard error of the transformation itself, rather than transforming the standard error of the estimated probability of treatment benefit.

For all settings the bootstrap‐based CIs, either parametric or nonparametric, were the most efficient CIs. Specifically, the parametric method mostly produced the tightest CIs, however with a certain extent of undercoverage, while the nonparametric‐bootstrap‐based CIs tended to be slightly larger and more accurate. The delta‐method‐based CIs were usually larger with a perfect or near perfect coverage rate. In addition, similarly to Laubender and Bender, we observed that bootstrap‐based CIs are computationally demanding tasks that consume considerable processing time.

For all settings, the larger the true NNT, the more biased and less stable were the point estimators and thus the CIs were less accurate. This stems from the convexity of as defined in (1). The larger bias resulted in less accurate CIs. Since all CIs are symmetric w.r.t. the point estimators, their limits were also biased upwards, which resulted in lower coverage rates. The larger the true NNT, either conditional or harmonic, the larger sample size that is required in order to obtain more accurate point estimates and associated confidence limits.

The overall simulation results are consistent with the theoretical considerations (in Sections 1, 2, and 3). Although all CI types are asymptotically‐correct, the parametric CIs exploit the asymptotic normality of the MLE. In particular, the asymptotic normality of the ML estimators of NNT is derived from the asymptotic normality of the regression coefficients. There are three layers of approximation. The first layer is for the regression coefficients themselves since we use sample sizes under 1000 which may represent some RCTs in certain domains of research. The second is for the normality of the estimated probability of treatment benefit which is a nonlinear function of the regression coefficients. The third is for the normality of the estimated NNT itself, since it is a convex transformation of the probability of treatment benefit. The normal approximation of the sample distribution of the NNT may thus require very large sample sizes to be accurate. The nonparametric BS‐based CIs do not use the asymptotic normal distribution, rather just the empirical quantiles of the NNT's sample distribution. Therefore, they are not sensitive to deviations from the normality assumption. CIs constructed by nonparametric bootstrap conformed to their nominal confidence coefficient in the scenarios described.

SUMMARY AND DISCUSSION

We have introduced a systematic framework to model and estimate the conditional and the harmonic NNT in the presence of explanatory variables, in various models, with dichotomous and nondichotomous outcomes. The conditional NNT was illustrated in a series of four examples: Logistic and linear regressions, alongside Kaplan‐Meier and Cox‐regression models. We established the relationship between the conditional and the harmonic NNT in the presence of explanatory variables. We introduced four different methods to calculate the asymptotically‐correct CIs for both NNT measures. Additionally, we conducted a simulation study to provide a numerical illustration of the theoretical results with the four examples. The results indicate that the parametric MLE with nonparametric bootstrap‐based CIs are the preferable estimators in all settings. For large NNT values, the point estimators were more biased and less stable. Transformation‐based CIs tend to be too wide or even infinite with a perfect coverage rates. The delta‐method CIs were usually finite, however larger than the bootstrap counterparts, with perfect or near perfect coverage rate. The bootstrap‐based CIs were usually smaller, with the parametric bootstrap‐based CIs frequently suffering from undercoverage, and the nonparametric attaining the expected coverage rates. For smaller NNT values, , the bootstrap‐based CIs, both parametric and nonparametric, performed well with approximately expected coverage rates, while the transformation‐based and delta‐method based CIs were usually much larger with perfect coverage rates. An R package and a corresponding web application to calculate the conditional and the harmonic NNTs with their corresponding CIs, has been developed and made available for users online.

The NNT is not without limitations. NNT is an index for presenting results and not analyzing data. As pointed out by several authors, , the NNT is a one‐dimensional index for conveying particular information and not a magical number to summarize comprehensive data analysis. Hence, it should not be the only statistic presented in a summary of analysis, and the user should acknowledge its caveats. However, the popularity of the NNT in various applications indicates the usefulness of this index and the need for an easy‐interpretable statistic to convey the efficacy of an intervention. Unfortunately, as no clear methodological recommendations regarding the use of NNT have been formulated, calculations and interpretations of the NNT are sometimes misleading and even erroneous. Our work aims to address these problems by providing concise statistical analysis, recommendations, and practical tools for appropriate use of the NNT in clinical trials.

Some scenarios that may arise in certain real‐world data applications were not addressed in this article. Such scenarios include complex patterns of missing data and irregular or less frequent data structures, for example, longitudinal studies and time series with informative or nonmonotonic missingness. Another possible aspect that can be addressed is bias correction of the point estimators of the conditional and harmonic NNTs. These are directions for future research. Nonetheless, we have demonstrated how the NNT with its corresponding asymptotically‐correct CIs can be effectively estimated in various widely‐used statistical models, and provided the users with an R package and a web application to implement these calculations.

TABLE A1

Setting III: conditional NNT, for , in the linear regression model with normally distributed error (for a formal definition of NNT in the linear regression model see Equation (9))

	NNT(1.2)				NNT(1.3)				NNT(1.4)
N	DL	TR	NBS	PBS	DL	TR	NBS	PBS	DL	TR	NBS	PBS
200	0.91	0.93	0.93	0.87	0.91	0.93	0.93	0.87	0.92	0.93	0.94	0.92
400	0.91	0.94	0.93	0.89	0.91	0.94	0.92	0.89	0.91	0.94	0.93	0.91
800	0.96	0.96	0.95	0.96	0.96	0.96	0.96	0.96	0.96	0.96	0.96	0.96

Note: Mean coverage rates of the pointwise CIs by construction method (DL, delta method; NBS, nonparametric bootstrap; PBS, parametric bootstrap; TR, transformation), and sample sizes of .

TABLE A2

Setting III: harmonic NNTs in the linear regression model with normally distributed error

	NNT_MLE				NNT_L
N	DL	TR	NBS	PBS	DL	TR	NBS
200	0.90	0.95	0.92	0.89	0.73	0.82	0.83
400	0.93	0.95	0.92	0.92	0.97	0.99	0.98
800	0.98	0.97	0.95	0.98	0.82	0.90	0.90

Note: Mean coverage rates of the pointwise CIs by construction method (DL, delta method; NBS, nonparametric bootstrap; PBS, parametric bootstrap; TR, transformation), and sample sizes of . Based on the parametric maximum likelihood estimator, NNT, and the nonparametric maximum likelihood estimator, NNT.