A Computationally Efficient Labeled Multi-Bernoulli Smoother for Multi-Target Tracking.

A forward-backward labeled multi-Bernoulli (LMB) smoother is proposed for multi-target tracking. The proposed smoother consists of two components corresponding to forward LMB filtering and backward LMB smoothing, respectively. The former is the standard LMB filter and the latter is proved to be closed under LMB prior. It is also shown that the proposed LMB smoother can improve both the cardinality estimation and the state estimation, and the major computational complexity is linear with the number of targets. Implementation based on the Sequential Monte Carlo method in a representative scenario has demonstrated the effectiveness and computational efficiency of the proposed smoother in comparison to existing approaches.

1. Introduction

Multi-target tracking (MTT) has been widely used in many engineering fields including aerospace surveillance, biomedical analytics, autonomous driving, indoor localization, robotic networks and so on [1,2,3,4,5]. In the applications, both the number of targets and their states may vary in time, and the measurements are obscured by clutter and missed detection [2,3]. Traditionally, approaches to MTT are built on the base of appropriate data association methods such as typically joint probabilistic data association [1] or multiple hypothesis tracking [6,7]. A novel approach has been developed by Mahler based on the random finite set (RFS) theory [2] which has attracted substantial attention in the last decade. Simply speaking, the RFS methods model the target states and the measurements into the RFSs explicitly and have gained tremendous interest in recent years [8]. A variety of RFS filters have been proposed, including probability hypothesis density (PHD) filter [9,10], cardinalized PHD (CPHD) filter [11], Cardinality Balanced multi-Bernoulli (CBMeMBer) filter [12], generalized labeled multi-Bernoulli (GLMB) filter [13,14] and labeled multi-Bernoulli (LMB) filter [15]. Compared with the filtering that refers the current state, the smoothing [16,17,18,19] typically refers to estimating the target state of the past time using all measurements till the current time which has a better accuracy than filtering but also suffers from a higher computational cost. Therefore, it is of great significance in practice to develop an RFS smoother that is computationally efficient, reliable and accurate.

The Bernoulli smoother was first proposed in [20,21]; it performs better than the Bernoulli filter, but adapts to, at most, one target. For the multi-target problem, the forward–backward PHD smoothers are proposed in [17,18,22,23,24]. It has been shown in [17] that the PHD smoother can improve the accuracy of position estimation as compared with the PHD filter, but does not necessarily gain better cardinality estimation. The CPHD smoother is proposed in [25] which uses an approximate scheme to overcome the intractability of the classic CPHD smoother [26] but bears a complicated algorithm structure. A forward–backward multi-target multi-Bernoulli (MeMBer) smoother is proposed in [27] which also does not necessarily improve the cardinality estimation. However, these smoothers, as well as the corresponding filters, do not provide the information for each track. Therefore, labelled RFS-based filters and smoothers have been developed [28,29,30,31,32], for generating track estimates which is also the focus of this paper.

The GLMB smoother is proposed in [32], which is the first exact closed form solution to the smoothing recursion based on labeled RFS but has not been implemented in practice due to the overcomplicated data associations. Recently, Chen has pointed out the challenge to form tracks since the optimal solution is not as simple as labeling [33]. Relevantly, a multi-scan GLMB filter based on the smoothing-while-filtering framework is proposed in [34] for better labeling. However, the truncation of the multi-scan GLMB filter needs to solve an NP-hard multi-dimensional assignment problem. In short, these existing labeled smoothers have a high computational complexity even if it is practically implementable. Furthermore, we point out that, what is also related to the framework of smoothing-while-filtering [34] is the joint smoothing and filtering [19,35,36] based on which so far, however, only a single target is considered.

In this paper, we derive a forward–backward LMB smoother for multi-target tracking. Preliminary and limited results have been published in [37]. This paper provides additional results, complete proofs, and additional experiments. The proposed smoother consists of two parts regarding the forward LMB filtering and backward LMB smoothing, respectively. While in the former we apply the standard LMB filter [15], the key contribution of our work lies in the backward smoothing algorithm design. We prove that the proposed backward LMB smoothing is closed under the LMB prior for the standard multi-target system models and the backward smoothed density of each track is similar to the Bernoulli backward smoothed density [20]. Superior to the approximate parameteric/Gaussian (mixture) approximation [38], the Sequential Monte Carlo (SMC) method is a powerful method for representing arbitrary/non-Gaussian models [4]. Based on the SMC method, the proposed smoother reduces both the state error and the cardinality error as compared with the PHD smoother [17], the MeMBer smoother [27] and the LMB filter [15], and has a lower computational complexity as compared with the PHD smoother and the MeMBer smoother.

The rest of the paper is organized as follows. Basic definitions of labeled RFS and the multi-target Bayes forward–backward smoother are reviewed briefly in Section 2. The proposed forward–backward LMB smoother is detailed in theory in Section 3 and implemented based on the SMC in Section 4, respectively. Simulation results are presented in Section 5 before we conclude in Section 6.

2. Background

2.1. Notation

Single target states are denoted by lowercase letters, for example, x and . Multi-target states are denoted by uppercase letters, for example, X and . x and X are the unlabeled state representations. and are the labeled state representations. State spaces are denoted by blackboard bold letters, for example, a label space which contains a countable number of distinct labels is denoted as , and a unlabeled target state space is denoted as . A labeled single target state has the form , where and . A labeled multi-target state has the form , where denotes a labeled single target state in , denotes the cardinality (or the number of targets) of a multi-target set, and .

The projection is to make and . Then define a generalized Kronecker delta function and inclusion function as:

Define . The inner product of and on a labeled single target space is defined as

The multi-target exponential form of is defined as

2.2. GLMB and LMB RFS

The GLMB RFS is a Labeled RFS constructed by labeled multi-target states. The distribution of an GLMB RFS [13] is exactly closed under the prediction and update of the multi-target Bayes filter. The GLMB distribution is denoted as where is a discrete index set, is the density of the track ℓ, is the nonnegative weight of the hypothesis , and , .

The distribution of an LMB RFS with the parameter set is given by [15] where where denotes the existence probability of the track ℓ, denotes the density, and denotes the weight of the hypothesis . Note that is a discrete countable space and the number of labels in equals to the number of the Bernoulli components (with non-zero existence probability) in the LMB RFS.

An LMB RFS is a special case of an GLMB RFS, and the densities and the existence probabilities of different tracks in an LMB RFS are both uncorrelated. Two properties associated with an LMB RFS are as follows. More specifically, the cardinality distribution [2] is given by where is the elementary homogeneous symmetric function of degree n in variables.

If

The proof of Lemma 1 is given in the Appendix A.

2.3. Multi-Target Bayes Forward–Backward Smoother

The recursion of multi-target Bayes forward–backward smoother is shown in Figure 1 which consists of forward Bayes filtering and backward Bayes smoothing. The multi-target state at t is , where and denotes the label space of the targets at t (including those born prior to t). The measurement set at t is . The prediction and update for the forward Bayes filtering can be denoted as [2] where is denoted as the predicted multi-target density from to t, is denoted as the multi-target Markov transition density at t, is denoted as the multi-target posterior density at t, and is denoted as the multi-target likelihood function at t. The integral in the equations is the set integral. If the backward smoothed density from t to k () is denoted by which is initialized with , the backward Bayes smoothed density at can be written as [32]

Figure 1

The recursion of the multi-target Bayes forward–backward smoother.

2.4. Multi-Target Motion and Measurement Models

Let denote the surviving probability of the target from to t. denotes the single target Markov transition density from to t. denotes the multi-target state at while denotes the survival multi-target state from to t, where and . Considering the survival targets only, multi-target Markov transition density is [2,13]

It is assumed that the newborn targets are an LMB RFS denoted as , and , where denotes the label space of the newborn targets at t. denotes the birth probability of target at t and denotes the corresponded density. The density of newborn targets can be denoted as [2,13]

The multi-target state at t can be denoted as where and are disjoint and independent. From Lemma 1, the joint multi-target Markov transition density is written as [2,13]

Multi-target likelihood function is given as [2] where denotes the detection probability of target at t. denotes the probability that target generates the measurement z if detected. The intensity function (or the PHD) of the Poisson clutter is . The association function has the property that implies . denotes the set of all association functions and . When , the target is missed in detection. When , denotes the measurement associated with target . denotes the false alarms at time t.

3. LMB Smoother

In this section, we detail the proposed LMB smoother and discuss the cardinality estimation. The LMB smoother framework can be simply depicted as shown in Figure 2 which consists of forward LMB filtering and backward LMB smoothing. The forward LMB filtering used in our approach is the standard LMB filter [15]. The backward LMB smoothing each time step has -step backward smoothing recursions where denotes the fixed lag.

Figure 2

The proposed LMB smoother framework.

3.1. Forward LMB Filtering

3.1.1. Prediction

Given that the multi-target prior at time is an LMB (distribution) parameterized as and the density of the newborn targets is an LMB parameterized as , then the predicted multi-target density at t remains an LMB which can be denoted as [12,15] where

3.1.2. Update

Assume that the predicted multi-target density at t is an LMB with the parameter set , namely,

Under the likelihood function of (17), the updated multi-target posterior density is an GLMB (Strictly speaking, it’s a -GLMB which is a special case of an GLMB [13]) which can be denoted as [13] where denotes the class of , denotes the label set of a hypothesis, is the set of the association function , and

In the update of the forward LMB filtering, the multi-target posterior density matches the first-order moment of (24) for computing simplification, i.e., [15] where

The approximate posterior density (29) of the forward LMB filtering preserves the first-order moment of the posterior density (24). It is proved in [29,31] that the approximate posterior density (29) in the LMB class minimizes the Kullback-Leibler divergence (KLD) relative to the posterior density (24).

3.2. Backward LMB Smoothing

In this subsection, the backward LMB smoothing is derived. Our derivation directly relies on the multi-target backward smoothing recursion (13), and is different from the derivation of the backward GLMB smoothing [32] which uses the backward corrector recursion [18] as an intermediate process to avoid the set integral of the quotient of two GLMBs.

Given that the multi-target posterior

where

The proof of Proposition 1 is given in Appendix B. It is observed that the smoothed density of the track ℓ has the same form as the smoothed Bernoulli density when it doesn’t take into account the newborn target [20]. Therefore, the proposed LMB smoother can be deemed as an extension of the Bernoulli smoother [20] to multiple targets. From (33), the existence probability of the track ℓ relates to , and . From (34), the density of the track ℓ contains two terms, where one term only relates to and which preserves the forward filtering state at , and the other term relates to the backward smoothing.

The proposed LMB smoother owns a good computationally efficiency to two strategies: First, the newborn tracks are uncorrelated with the backward smoothing since the newborn tracks cannot be alive prior to the birth time, resulting in a simple label space. Second, the existence probabilities and the probability densities of different tracks for an LMB RFS are uncorrelated, and then each track component can be calculated separately, so the computational complexity of the backward smoothing is linear with the number of the tracks.

The proposed LMB smoother is also approximate Bayes optimal because the LMB family is closed under the prediction operation and the backward smoothing operation. Although the LMB family is not closed under the update operation, the first-order moment approximation in the LMB class minimizes the KLD relative to the posterior density.

4. SMC Implementation and Algorithm Analysis

This section presents the SMC implementation and the state extraction, and analyze the algorithm complexity of the proposed LMB smoother.

4.1. SMC Implementation

Prediction: Consider the multi-target posterior at is . In the SMC implementation, is represented by a set of weighted particles . The density of the newborn targets at t is , where is also represented by a set of weighted particles . Through the prediction of (18)–(22), the predicted multi-target density at t is where is calculated by (20) and is also represented by . That is, where is chosen as the importance sampling density.

Update: Denoting the predicted density as , it can be written as the form of (6) given by , where is denoted by (7) and . is represented by a set of weighted particles . The K-shortest paths algorithm [14] is used to truncate the predicted multi-target density in order to reduce the number of hypotheses. The multi-target posterior density given in (24) is denoted by . We compute and as follows.

can also be represented by a set of particles where and

The weight of the hypothesis is given as where is an association function and . is a hypothesis and . To avoid computing all the hypotheses and their weights, we only reserve a specific number of largest weights and the ranked optimal assignment algorithm [14] is applied to truncate the multi-target posterior density.

The multi-target posterior density of the forward LMB filtering matches the first-order moment of (24) which can be denoted as an LMB with the parameter set where is given by (30) and is represented by a set of weighted particles with where . The number of particles for increases rapidly and resampling [39] is needed.

Backward smoothing: From the forward LMB filtering, the multi-target posterior density is at , where is approximated by . The predicted multi-target density from to k is an LMB and the existence probability of track ℓ is where . The multi-target backward smoothed density from t to k () is denoted as , where is represented by a set of weighted particles .

Proposition 1 implies that the multi-target backward smoothed density from t to is also an LMB given by where is calculated by (33) and is represented by a set of weighted particles . The detailed formula of is given as where and

Note that the predicted density (22) cannot be directly used as in (49) for smoothing. Because the forward LMB filtering performs the resampling procedure [39] in each filtering step, the particles of from (initial smoothed density) are different from those of . We need to estimate for each particle , as in (51).

4.2. Backward Smoothing and State Extraction

The pseudo code of the proposed backward smoothing algorithm is given in Algorithm 1. The forward filtering is up to t and the lag of the backward smoothing is . We need to store multi-target posterior densities from to t and multi-target predicted densities from to t for -step backward smoothing recursions. The backward smoothed density is initialized with . In the SMC implementation, pruning, truncation and track cleanup are required and the label set varies with the time, so is replaced with , where denotes the label set of the corresponded density and . Therefore, we have to represent the label set of backward smoothing at which eliminates the labels of newborn tracks at k and the labels of the pruned tracks in the forward filtering.

Note that, in our approach, resampling [39] can be applied either as the final step of the forward filtering or after the backward smoothing, which lead to insignificant difference in performance. Target states are extracted from the output . That is, the target number N is first estimated by where is the cardinality distribution (9). Then tracks with the largest existence probabilities are extracted and the target states are mean of the corresponding probability densities.

4.3. Algorithm Complexity

The major computational cost of the LMB smoother is due to backward LMB smoothing. The computational complexity of the backward LMB smoothing is which can be obtained from the four for-loops of Algorithm 1, where N is the number of tracks (or targets) and L is the number of particles per track. The computational complexity of the innermost two parallel for-loops is , so the computational complexity of the backward LMB smoothing is multiplying by the outermost two for-loops.

The structures of backward smoothing for PHD smoother [17] and MeMBer smoother [27] are similar to that of LMB smoother, and the difference is that the particles of all tracks are used for backward smoothing for PHD smoother and MeMBer smoother while the particles of each track are used for their respective backward smoothing in our LMB smoother. We can obtain that the major computational costs for the PHD smoother (Here, we consider the classic SMC implementation [17] instead of the fast SMC implementation [23]) and the MeMBer smoother are both approximately to . Therefore, the computational complexity of the proposed LMB smoother is lower than those of PHD smoother [17] and MeMBer smoother [27].

5. Simulation Result

A nearly constant turn (NCT) model is considered which has varying turn rate together with noisy range and azimuth measurements [12,13]. The state of the target with label ℓ at t is denoted as , where , denotes the planar position and velocity of the target, and denotes the turn rate. The NCT model can be written as where and are the state noises of velocity and turn rate, respectively. and where denotes a Gaussian density with mean and variance , and represents an i order identity matrix. The noise of velocity uses m/s and the noise of turn rate uses rad/s. The state transition matrix and noise transition matrix are given as follows, respectively,

The state transition model is denoted by (53) and (54). The sampling interval is s. The target survival probability is . Newborn targets of each time step is an LMB RFS which can be represented as where , , , and . The mean of track is , the mean of track is , and the variances of both are . The unit of position is m, the unit of velocity is m/s, and the unit of the turn rate w is rad/s.

The measurement equation is where the measurement noise is with , m, and rad. The observation region is m rad with the detection probability . The density of the Poisson clutter is as the average number of clutter measurements is 10 per scan.

The number of particles per hypothesized track is set to 1000. We prune the tracks with a weight smaller than . The lag of the smoother is . The OSPA metric [40] with cut-off parameter m and order parameter is used. We compare the proposed LMB smoother with the LMB filter [15], the PHD filter [9], the PHD smoother [17], the CBMeMBer filter [12] and the MeMBer smoother [27] over 100 Monte Carlo trials.

Figure 3 and Figure 4 show the results of the LMB smoother in one trial, in the plane and in x and y coordinates over time, respectively. The number of targets changes over time due to target birth or death, and there are at most five targets in the scenario. The track of each target is smoothing.

Figure 3

The true and estimated trajectories of targets. Different trajectories estimated by the LMB smoother are denoted by different color dots where ‘∘’ denotes the initiations and ‘Δ’ denotes the terminations of the trajectories.

Figure 4

The true and estimated trajectories of targets in x and y coordinates, respectively.

More specifically, Figure 5, Figure 6 and Figure 7 show the cardinality estimation for different methods over 100 Monte Carlo trials. We can see from Figure 5 that the estimated cardinality mean converges to the true cardinality most of time for all methods. Figure 6 shows the errors of the cardinality mean which is given by the estimated cardinality mean minus the true cardinality. At s, 10 s, 40 s and 60 s, there is one or two newborn targets at each time step and the cardinality errors are negative for the LMB filter, because of newborn target detection delays. At s, 81 s and 91 s, one target disappears and the cardinality errors are positive for the LMB filter, because of the detection delay of the target death. At k = 64∼66 s, 78∼80 s and 88∼90 s, the cardinality mean errors are negative for PHD and MeMBer smoother, because of premature deaths of targets. Figure 7 shows the standard deviation of the cardinality estimate for different methods. The standard deviations of the PHD and the MeMBer smoother are larger than those of LMB filter and LMB smoother. In short, the LMB smoother can accurately estimate the cardinality (except for a detection delay at s) and yields the best accuracy.

Figure 5

The true and estimated cardinalities of the PHD smoother, the MeMBer smoother, the LMB filter and the proposed LMB smoother.

Figure 6

The estimated cardinality errors over time.

Figure 7

The estimated standard deviations of the cardinality over time.

Figure 8 shows the average OSPA errors of the PHD smoother, the MeMBer smoother, the LMB filter, and the LMB smoother with 100 Monte Carlo trials. It can be seen that the OSPA of the LMB smoother is less than those of the other methods almost all the time. We also give the average OSPA errors of different methods in Table 1. It can be seen that all three kinds of smoothers can effectively reduce OSPA localization components as compared with the corresponding filters. However, the PHD smoother and the MeMBer smoother do not necessarily reduce the OSPA cardinality components, whereas the proposed LMB smoother can improve the cardinality estimation significantly.

Figure 8

OSPA errors yielded by different methods.

Table 1

Average OSPA miss distances of different methods for all time.

Method	Total	Localization	Cardinality
	OSPA (m)	Component (m)	Component (m)
PHD filter	34.3821	26.4094	7.9727
PHD smoother	27.3199	18.6526	8.6673
CBMeMBer filter	30.2842	22.3566	7.9276
MeMBer smoother	22.0721	14.6664	7.4056
LMB filter	25.6800	22.6574	3.0226
LMB smoother	15.1762	14.4932	0.6830

Figure 9 shows the average execution time of different methods over 100 Monte Carlo trials. Summing the time of every time step, we get the times per simulation for the PHD smoother, the MeMBer smoother, the LMB filter, and the LMB smoother which are 1341 s, 1537 s, 156 s and 356 s, respectively. The proposed LMB smoother have a higher computational complexity than the LMB filter, but obviously lower than the PHD smoother and the MeMBer smoother. The results comply with our theoretical analysis.

Figure 9

The average execution time of each time step for different methods.

6. Conclusions

The paper derives a computationally efficient forward–backward LMB smoother which is closed under the backward smoothing operation and has the advantages for maintaining the independence of different tracks and their track outputs. Both numerical and simulation analyses have demonstrated that the proposed LMB smoother can effectively improve the tracking performance as compared to the PHD smoother, the MeMBer smoother and the LMB filter and have a lower computational complexity as compared with the PHD smoother and the MeMBer smoother. We should point out that our smoother cannot solve the problem of track fragmentation [34] when the label of a track changes before the track ends. It is our future work to investigate the curve/track fitting approach [19,35,36] for improving the continuity and smoothness of the estimated tracks.