The nonparametric maximum likelihood estimator for middle-censored data.

In this note, we consider data subjected to middle censoring where the variable of interest becomes unobservable when it falls within an interval of censorship. We demonstrate that the nonparametric maximum likelihood estimator (NPMLE) of distribution function can be obtained by using Turnbull's (1976) EM algorithm or self-consistent estimating equation (Jammalamadaka and Mangalam, 2003) with an initial estimator which puts mass only on the innermost intervals. The consistency of the NPMLE can be established based on the asymptotic properties of self-consistent estimators (SCE) with mixed interval-censored data (Yu et al., 2000, Yu et al., 2001).

Introduction

Middle censoring occurs when a data point falls inside a random censoring interval whereby it becomes unobservable. For some individuals the exact values are available while for others the corresponding intervals of censorship are observed. We mention two situations where middle censoring occurs. (i) In a follow-up study, if the childhood learning center where the observations are being taken, is closed for a period, due to an external emergency such as the outbreak of severe acute respiratory syndrome (SARS). (ii) In a clinical trial, where the clinic where the observations are being taken, is closed for a period, due to an external emergency such as the outbreak of war or a strike. For situations (i) and (ii) where during a fixed time interval (this fixed interval is indeed, a random interval (denoted by (U,V)) relative to individual's lifetime) the observation was not possible. If some children (or patients) develop a skill (or disease) of interest during this time, we are not able to observe the exact age T of these children (or patients) at the time of skill (or disease) development, rather only the information that the event of interest occurred during a certain time interval (U,V). At first glance, middle censoring, where a random middle part is missing, appears as complementary to the idea of double censoring in which the middle part is what is actually observed. However, a careful reflection and analysis shows them to be quite different ideas; see Jammalamadaka and Mangalam (2003) for details.

Let T , i=1,…,n, be a sequence of i.i.d. random variables with distribution function F 0. Independent of T 's, let (U ,V ), i=1,…,n, be i.i.d. extended real-valued random variables with joint distribution function such that . The ith observation is said to be censored if . We observe , i=1,…,n, where X =T if and X =(U , V ) if . For indentifiability of F 0, we assume thaton any interval [a,b], for which . Observe that if A 0(x)=1 on any interval where F 0 has a positive mass, then censoring occurs with probability 1 on such an interval. As a consequence, there will be no observations on the interval and that prevents us from distinguishing any two distributions which are identical outside [a,b] but differing only on [a,b].

In many censoring situations, if we were to try to estimate the distribution function via the EM algorithm the resulting equation takes the formas described by Tsai and Crowley (1985), where E is the empirical distribution function and X denotes the observed data. This equation was first introduced and referred to as self-consistency equation by Efron (1967). A solution of (1.2) is called a self-consistent estimator (SCE) of F 0. In different types of censoring, the relationship between nonparametric maximum likelihood estimator (NPMLE) and SCE has been studied by various authors. In the case of right censoring the product-limit estimator (Kaplan and Meier, 1958) is the NPMLE, and Efron (1967) showed that it is also self-consistent. In the double-censoring case, Mykland and Ren (1996) provided a necessary and sufficient condition for an SCE to be an NPMLE. In the middle censored cases the self-consistent estimator (SCE) (see Jammalamadaka and Mangalam, 2003) satisfies the following equation: Jammalamadaka and Mangalam (2003)  showed that the NPMLE satisfies the self-consistency equation (1.3). They also pointed out that an SCE provides only a local maximum of the likelihood equation and may not be an NPMLE. Furthermore, they showed that the NPMLE will have all its mass on the uncensored observations except when it so happens that a censored interval contains no uncensored observation. The consistency of the SCE was established by Jammalamadaka and Mangalam (2003) for the special case when either U or V is degenerate. Jammalamadaka and Iyer (2004) proposed an approximation to the distribution function F 0 (denoted by ) for which a modified self-consistent estimator was obtained. They established the asymptotic properties of and provided an upper bound of the difference between F 0 and . Mangalam et al. (2008) showed that under condition (A) every censoring interval contains at least one uncensored observation, i.e. implies that there exists j such that and , the solution of Eq. (1.3) will be unique, and as a consequence will be equal to an NPMLE. Mangalam et al. (2008) proposed a technique for obtaining the NPMLE by dividing the original problem into subproblems.

In this note, we aim to establish connections between the middle-censoring and interval censoring by investigating the self-consistency algorithm. In Section 2, we shall demonstrate that the NPMLE of F 0 can be obtained by using the EM algorithm of Turnbull (1976) or the self-consistent estimating equation (Jammalamadaka and Mangalam, 2003) with an initial estimator which puts mass only on the innermost intervals. Furthermore, we establish the consistency and asymptotic normality of the NPMLE by using the results of Yu et al., 2000, Yu et al., 2001.

The NPMLE

Self-consistency

Turnbull (1976) characterized the NPMLE in the presence of interval censoring and truncation. Frydman (1994) later corrected Turnbull's characterization. Here, we consider the case when there is no truncation. Let if L =R and if . For i=1,…,n, let L =R =T if and (L ,R )=(U ,V ) if . Thus, we observe the middle-censored data A 1,…,A . Following Turnbull (1976), Frydman (1994) and Alioum and Commenges (1996), we consider nonparametric estimation of F 0 using the independent observation A 's. Since Turnbull's EM algorithm can be used to tackle the case when A is a single point set, the connection between middle censoring and interval censoring can be established. Based on the notations defined above, the likelihood is proportional towhere denotes the probability that is assigned to the interval by F 0. We define an NPMLE as , where denotes the class of distribution functions such that . For censored data, using graph theory, Gentleman and Vandal (2001) presented methods for finding the NPMLE of F 0. For censored and truncated data, Hudgens (2005) employed a graph theoretical approach to describe the support set of the NPMLE of F 0.

Define innermost intervals H , j=1,…,J, induced by A 1,…,A to be all the disjoint intervals which are non-empty intersections of these A 's (e.g. is an intersection of A 's) such that or H for all i and j. Let the endpoints of the innermost intervals be q and p , j=1,…,J, where Peto (1973) showed that the NPMLE of F 0 assigns weight, say , to the corresponding innermost intervals H 1,…,H only. Thus, it suffices to maximizewhere and is the usual indicator function. The goal is to maximize likelihood (2.2) subject to the constraintsandThus, we can limit our search to the space given by constraints (2.3), (2.4). We shall use to denote the parameter space that is given by constraints (2.3), (2.4), i.e. To find the maximum likelihood estimate of the vector s, we use an EM algorithm as follows.

E-step: Let the expected value of be denoted by . Then under s, M-step: In the maximization step, we treat the expected values as observed. The overall proportion of failures in the interval H isThe EM algorithm iterates between Eqs. (2.5), (2.6) after selecting initial estimates such that , i.e., computes , updates s by , and repeats until convergence. The resulting self-consistent estimate of s, which is a solution of simultaneous equation , is exactly the Turnbull's (1976) self-consistency algorithm as follows:where Let denote the estimators obtained from (2.7). Using standard convex optimization techniques, Gentleman and Geyer (1994, see p. 619) provided easily verifiable conditions, i.e. Kuhn–Tucker conditions, for the self-consistent estimator to be a maximum likelihood estimator.

Based on the estimators , an estimator of F 0(t) can be uniquely defined for by , but is not uniquely defined for t being in an open innermost interval (q ,p ) with . To avoid ambiguity we define if and . Next, we shall show that the estimator satisfies self-consistent equation (1.3), i.e.

The NPMLE satisfies Eq. (2.8).

First, notice that for each either q =p or and there is no uncensored observation in (q ,p ) if . Furthermore, can be written as Consider an initial estimator , which puts mass only on . Let denote the first step estimator. Without changing the innermost intervals and likelihood function, we can transform data by moving all right censored points between p and q to p . Similarly, move all left censored points between p and q to q (see Li et al., 1997). Hence, we have . Since , we have Hence, F (1) also puts mass only on . Next, we consider the following two cases:

Case  1: q =p . When q =p , we haveNowwhich can be written asHence, Eq. (2.10) is equivalent to Eq. (2.9).

Case  2: . When , since there is no uncensored observations in (q ,p ), we haveNow,First, we have . Next, note that given an interval (L ,R ) and , we either have or . Hence, we have . It follows thatHence, Eq. (2.12) is equivalent to Eq. (2.11). The proof is completed.  □

This conclusion is the same as in Theorem 1 of Jammalamadaka and Mangalam (2003). However, our proof is based on the EM algorithm of Turnbull (1976). It is obvious that if condition (A) holds then q =p for all j, and the solution will have all its mass on the uncensored observations. Furthermore, if we start with an initial estimator which puts weight 1/J on q =p for uncensored observations and on (q +p )/2 for censored observations, we can obtain an NPMLE by using Eq. (2.8). However, similar to interval-censored data, the self-consistent NPMLE of F 0 is not uniquely defined for if . An SCE with an initial estimator which puts weight on intervals other than (q , p ) can lead to a less efficient estimator (not NPMLE).

Large sample properties

In this section, we shall investigate large sample properties of . First, we introduce mixed interval censored (MIC) data. A data set is called a MIC data when it consists of both exact observations and case 2 interval censoring data (i.e. ). Mixed IC data arises in clinical follow-up studies where a tumor maker (e.g., Ca125 in ovarian cancer) is available, a patient whose marker value is consistently on the high (or low) end of normal range in repeated testing is usually under close surveillance for possible relapse. If such a patient should relapse, then the time to clinical relapse can often be accurately determined. However, if a patient is not under close surveillance, and would seek assistance only after some tangible symptoms have appeared, then time to relapse would be subject to case 2 interval censoring. For MIC data, several models have been proposed, and the asymptotic properties of the NPMLE have been investigated under the assumption that either the censoring variables take on finite many values (see Huang, 1999, Yu et al., 1998, Yu et al., 2000), or the censoring and survival distributions are strictly increasing and continuous and they have “positive separation” (see Huang, 1999, Assumption (A3)). For MIC data, define (Y ,Z ) as a pair of extended random censoring times ( allowed) with , and T is independent of (Y ,Z ). Yu et al. (2000, see (2.1)) considered a mixture interval censorship model to characterize MIC data as follows:For middle censored data, since neither right censoring nor left censoring occurs, model (2.13) is reduced toReplacing (Y ,Z ) and (Y ,Z ] in (2.14) with (U ,V ), we obtain the model for middle-censored data as follows: Hence, although the sampling scheme of MIC data seems to be quite different in character from that of middle-censored data described in Section 1, the resulting observations (L , R ) would reduce to the observations from middle-censoring data when there is no left or right censoring.

Let S denote the set of all support points of variables T . Similarly, define S and S for V and U , respectively. Suppose that . Then (i) The NPMLE is strongly consistent, i.e. , and (ii) converges in distribution to a normal variate.

Let Q denote the empirical version of the joint distribution function of (L ,R ) (i=1,…,n). It follows that Eq. (2.8) can be written aswhere Q is the empirical version of Q. In (2.15), if F(t)=F(r−)=F(l), then we encounter in the integrand. In this case, we define . Notice that Eq. (2.15) is exactly the same as Eq. (2.3) of Yu et al. (2000) and Eq. (2.2) of Yu et al. (2001), which is a self-consistent equation of F 0 for the model in Yu et al., 2000, Yu et al., 2001 with mixed interval-censored (MIC) data. By Theorems 2.1, 2.2 and 3.1 of Yu et al. (2001), the strong consistency and asymptotic normality of and are established.  □

Discussion

We have demonstrated how middle-censored data relate to mixed interval-censored data. With some modification of the definition for intervals (q , p )'s, we can obtain the NPMLE of distribution function by using EM algorithm of Turnbull (1976) or self-consistent estimating equation (Jammalamadaka and Mangalam, 2003) with a proper initial estimator. The consistency and asymptotic normality of the NPMLE can be established based on the asymptotic properties of self-consistent estimators (SCE) with mixed interval censored data (Yu et al., 2000, Yu et al., 2001).