A Novel Hard Decision Based Simultaneous Target Tracking and Classification Approach.

Methods dealing with the problem of Joint Tracking and Classification (JTC) are abundant, among which Simultaneous Tracking and Classification (STC) provides a modularized scheme solving tracking and classification subproblems simultaneously. However, there is no explicit hard decision on the class label but only soft decision (class probability) is provided. This does not fit many practical cases, in which a hard decision is urgently needed. To solve this problem, this paper proposes a Hard decision-based STC (HSTC) method. HSTC takes all the decision error rate, timeliness, and estimation error into account. Specifically, for decision, the sequential probability ratio test is adopted due to its nice properties and also the adaptability to our situation. For estimation, by utilizing the two-way information exchange between the tracker and the classifier, we propose flexible three tracking schemes related to decision. The HSTC tracking result is divided into three parts according to the time of making the hard decision. In general, the proposed HSTC method takes advantage of both SPRT and STC. Finally, two illustrative JTC examples with hard decision verify the effectiveness of the the proposed HSTC method. They show that HSTC can meet the demand of the problem, and also has the performance superiority in both decision and estimation.

1. Introduction

Target tracking and classification are critical in battlefield surveillance systems [1,2,3,4,5,6,7,8]. Traditionally, they are treated separately using their respective data and techniques: tracking is usually based on kinematic data while classification relies on attribute data. Meanwhile, since they involve continuous and discrete valued uncertainties respectively, their solutions are different. Recently, solving these two problems jointly has attracted much attention. This is preferable since in many practical problems involving both tracking and classification, the two problems are interrelated: tracking may affect classification by providing flight envelope information for different classes while classification may affect tracking via selecting appropriate class-dependent kinematic models. It is easy to realize that for such problems, tracking and classification should be handled jointly, and the performance of both could be improved by effectively utilizing their mutual-effect.

For solving the Joint Tracking and Classification (JTC) problem, many existing methods ignore the interdependence between tracking and classification [9,10] or only considering the one-way dependence [11,12]. Specifically, in the former, tracking and classification proceed separately. The tracker utilizes the kinematic information for state estimation and the classifier uses identity information for classification, respectively. In the latter, tracking and classification are handled within two-stages considering their mutual-effect to some extent. This can be further classified into two types: tracking then classification (T-then-C) [11,13,14,15,16] and classification then tracking (C-then-T) [12,17]. In T-then-C, it is considered that identity relies heavily on accurate state estimation. The basic idea is that each target class has its own motion model which can be seen as priors. Thus after tracking, the estimated kinematic state is compared with the flight envelops of known classes for class identification. While in C-then-T, it is considered that target type knowledge could help improve tracking performance by, e.g., assisting correct data association and selecting appropriate target models.

Some other researches realize the importance of two-way dependence and one prevailing strategy is the density based method in Bayesian framework, which aims to obtain the posterior joint density-probability of target state and class [18,19,20]. However, this results in heavy computational complexity due to its essence of density inference. Actually, JTC is a point estimation-based rather than density-based problem, and density estimation is even harder than the joint problem itself. We should avoid intermediate subproblems that are even harder than the original problem [21]. For the JTC problem, a novel Joint Decision and Estimation (JDE) framework proposed by Li [22] has provided a good joint solution for problems involving interdependent decision and estimation. It has the potential of arriving at the globally optimal solution. The power of this framework has been demonstrated in [21,23,24,25]. JDE framework belongs to another new and superior framework, which is not considered in this paper. For multiple targets, [26,27] propose a variational approach to simultaneous tracking and classification of multiple objects, whose superiority has been effectively verified. In this paper, however, we consider the problem of single target joint tracking and classification without measurement uncertainties.

In [28], a Simultaneous Tracking and Classification (STC) method was proposed within a modularized scheme. It derives the tracker and classifier simultaneously by utilizing the marginal pdf-pmf of the target state and class directly. A multiple-model filter and a Bayesian classifier are presented together using multisensor data. Under the linear Gaussian assumption, the STC tracking and classification results are obtained based on point inference rather than density inference.

In many existing literatures, there is no explicit hard decision on the class label but only soft decision (class probability) is provided. However, in many real applications, an explicit hard decision is urgently needed because decision usually leads to action. Take air warning system as an example. Suppose that there are two targets in the field of interest, a fighter and an airliner. Our goal is to recognize and track the two targets simultaneously using data from multiple sensors. Giving an alarm signal if a fighter is recognized. In this case, a hard decision is urgently needed. A wrong decision may either lead to false alarm or miss, both of which will bring loss. Another common example is a ground target tracking and identification system. Suppose that two targets, a tank and a school-bus are moving in the crossroads with different motion models, we want to destroy the tank and let the school-bus pass by. Hard decision is necessary and urgent since we want to destroy the tank as soon as possible.

In view of the above, a hard decision is critical in a joint tracking and classification problem. For such a decision, both the timeliness of decision-making and the corresponding decision error should be taken into account. Sequential testing is natural for classification where observations are obtained sequentially. A sequential test consists of a stopping rule and a final decision, whose performance is usually measured by the average sample number (ASN) given allowable decision error probabilities. In view of the above, we propose to use the well known sequential probability ratio test (SPRT) [29] due to its nice properties: SPRT is optimal in the sense that it minimizes the ASN under both hypotheses simultaneously among all tests of the same allowable error probabilities [30]. Besides, SPRT is also adaptable to our JTC problem with hard decision. In many JTC problems, the true target class remains constant. This consists with the basic assumption of SPRT, in which the true hypothesis is assumed to be unchanged over time.

For tracking, considering that tracking and classification are interactive in JTC, better tracking schemes require handling the tracking problem by utilizing their two-way effect. Thus, this paper proposes flexible decision-related tracking strategies. According to the time instant of the hard decision, i.e., before hard decision, at hard decision, and after hard decision, we provide three different tracking results. Finally, all the decision error, decision timeliness, and the estimation performance are considered. In general, the proposed tracking schemes not only meet the demand of the JTC problem but also has nice properties. It takes advantage of the STC method in handling the mutual-effect between decision and estimation within the point inference framework, furthermore, it accounts for the relationship between the hard decision and estimation.

The main contributions of this paper are as follows: (a) We clearly formulate the practical JTC problem in which both the estimation of the target state and the hard decision on the target class are required. (b) A Hard decision-based STC (HSTC) method is creatively proposed with SPRT being the decision criterion. HSTC not only satisfies the practical requirements but also superior in performance. It takes advantage of both SPRT and STC, and adapts to our problem. It considers the decision error, decision timeliness, estimation accuracy, and also utilizes the mutual information exchange between tracking and classification. (c) Two illustrative examples demonstrate the superiority of the proposed HSTC method.

This paper is organized as follows. Section 2 formulates the JTC problem with hard decision. Section 3 reviews the existing STC method. Section 4 as the main part of the paper proposes the HSTC method. Both the HSTC classification and tracking results are presented. In Section 5, two numerical examples are used to illustrate the effectiveness of the HSTC method. Conclusions are made in Section 6.

2. Problem Formulation

In this paper, we consider a simple but representative joint tracking and classification problem. Suppose there is only one target in the field of interest, and it has two possible classes: a fighter and an airliner. We want to track the target and identify its class using all available data. Assume that different target classes have different maneuverabilities: the fighter has larger maneuverability than the airliner, and there is no measurement origin uncertainty. Then, denote by the target state and the target class, our goal is to obtain jointly using , where denotes the measurements up to time k.

Assume that a target of class i has possible models and the motion model corresponding to class i can be written as where the total number of target class is , denotes the target state at time k, and is the jth motion model of a class i target at time k.

Denote the model set for class i as

The transition probability matrix (TPM) for class i is and the initial model probability for class i is

Assuming that the target state evolves according to the following linear dynamics: where is the deterministic input at time k, is assumed to be zero-mean white Gaussian process noise with known variance , is the state transition matrix, and are the gain matrices of input and process noise, respectively, and the superscript denotes a quantity for the jth model of a class i target.

The kinematic measurement is denoted by . Under the linear measurement assumption, we have where is the measurement matrix, and is assumed to be zero-mean white Gaussian noise with known variance .

3. Existing Simultaneous Tracking and Classification Method

To obtain the target state and class jointly, the most fundamental solution is to calculate the joint density-probability of the target state and class [28]:

Most existing methods calculate first and then marginalize it to obtain the target state and class:

To obtain the joint density , many numerical calculation methods are adopted. Different from these, in the STC method [28], the simultaneous pdf-pmf: is used directly rather than calculate first and then marginalize to get and . With this method, both the tracker and the classifier can be obtained simultaneously by accounting for the mutual information exchange between them [28]. Note that under the linear Gaussian assumption, STC requires only the first two moments of the joint distribution of the target state and class. This avoids the complex density calculation, and makes it computation efficient.

4. Hard Decision Based Simultaneous Tracking and Classification

As is explained in Introduction, how can we solve such a joint tracking and classification problem involving hard decision? For decision, a hard and timeliness decision is needed, which explicitly determine the target class. For estimation, the effect that the hard decision may exert should be taken into account. In the following, they will be presented in detailed.

4.1. HSTC Decision

To begin with, sequential testing procedures are preferable since measurements come sequentially. A sequential test contains a stopping rule and a final decision to achieve a trade-off between sampling size and decision accuracy. For JTC problem with hard decision case, the class label should be identified as quickly as possible and the decision error should be as small as possible.

As is well known that SPRT has the nice property [29] that when the type I and type II errors are controlled, the expected sample size under both hypotheses are simultaneously minimized among all the tests. This matches the requirements for our problem exactly. Besides, as mentioned in the Introduction part, the target type is always the same with no class switching problem over time. SPRT fits this situation perfectly. Thus it is adopted in the decision part of our HSTC method.

Overview of SPRT

In this paper, we assume that hypothesis and target class are one-to-one correspondence:

Following SPRT, once a piece of new data arrives, the class-likelihood will be output by the filters, given by: in which is the measurement likelihood at time k, which will be presented in detail later. is a normalization factor, and is the class probability based on .

Before the hard decision is made, only the probability of target class is available. Therefore, the HSTC classifier is given by (6). Following SPRT, to make a hard decision, we need to calculate the log-likelihood ratio (LLR), given by

Assume that there is no prior information about target class, we have , and thus the initially. Based on this, the class-likelihood ratio is equivalent to the class posterior probability ratio when it is updated recursively. That is why we use the posterior class probability here, which is available at every step from the soft decision.

Based on the above, the decision rule in HSTC is given by where

Remark 1: In this paper, we consider the target classification with two possible classes, as is presented in the problem formulation part. Therefore, we adopt the SPRT for hard decision due to its nice properties in handling the binary hypothesis testing. Actually, the proposed HSTC method can extended to multi-classes by replacing the SPRT method with Armitage test [

4.2. HSTC Estimation

Since estimation is closely related to decision in the JTC problem, we should fully consider their mutual influence. In the following, we first analyze the HSTC estimation regardless of the hard decision. Then, we explore novel estimation schemes, which are closely related to the hard decision.

4.2.1. Analysis

With the quadratic estimation cost , where , and based on the denotations in the previous part, the Bayesian tracking result is given by where is the posterior class probability of , which will be presented later. is the state estimate under class , defined by

Here, is state estimate based on model under class , and is the corresponding probability of model under class . To improve the overall tracking performance of the , the well-known interacted multiple model (IMM) approach is adopted here. Details about the IMM estimator is presented in Table 1.

Table 1

One cycle of IMM estimator.

1. Model-conditioned re-initialization (for i=1,2,⋯,M):
Predicted mode probability:	μk|k−1(i)≜P{mk(i)|Z^k−1}=∑_jπ_jiμk−1(j)
Mixing weight:	μk−1j|i≜P{mk−1(j)|mk(i),Z^k−1}=π_jiμk−1(j)/μk|k−1(i)
Mixing estimate:	x¯k−1|k−1(i)≜E[x_k−1|mk(i),Z^k−1]=∑_jx^k−1|k−1(j)μk−1j|i
Mixing covariance:	P¯k−1|k−1(i)=∑_j[Pk−1|k−1(j)+(x¯k−1|k−1(i)−x^k−1|k−1(j))(x¯k−1|k−1(i)−x^k−1|k−1(j))]μk−1j|i
2. Model-conditioned filtering (for i=1,2,⋯,M):
Predicted state:	x^k|k−1(i)=Fk−1(i)x¯k−1|k−1(i)+Gk−1(i)w¯k−1(i)
Predicted covariance:	Pk|k−1(i)=Fk−1(i)P¯k−1|k−1(i)(Fk−1(i))^′+Gk−1(i)Qk−1|k−1(i)(Gk−1(i))^′
Measurement residual	z˜k(i)=z_k−Hk(i)x^k|k−1(i)−v¯k(i)
Residual covariance	Sk(i)=Hk(i)Pk|k−1(i)(Hk(i))^′+Rk(i)
Filter gain	Kk(i)=Pk|k−1(i)(Hk(i))^′(Sk(i))^−1
Updated state	x^k|k(i)=x^k|k−1(i)+Kk(i)z˜k(i)
Updated covariance	P_k|k=∑_i[Pk|k(i)+(x^_k|k−x^k|k(i))(x^_k|k−x^k|k(i))^′]μk(i)
3. Model probability update (for i=1,2,⋯,M):
Model likelihood:	Lk(i)≜p[z˜k(i)|mk(i),Z^k−1]=N(z˜k(i);0,Sk(i))
Mode probability:	μk(i)≜μk(i)Lk(i)∑_jμk(j)Lk(j)
4. Estimation fusion
Overall estimate:	x^_k|k=∑_ix^k|k(i)μk(i)
Overall covariance:	μk−1j|i≜P{mk−1(j)|mk(i),Z^k−1}=π_jiμk−1(j)/μk|k−1(i)

Under the linear Gaussian assumption, and for class , both the model-based state estimate and the likelihood can be obtained using the Kalman filter. Based on the hybrid system consists of the evolution model (2) and the measurement model (3), the estimator at time k contains two main steps: prediction and update. Specifically, the predicted state and the corresponding mean square error (MSE) is: Then, with new measurement available, we can update the state and its corresponding MSE, as follows: Here, the updated state work as in our HSTC method.

In addition to the state estimate, another important issue is the measurement likelihood in (6). Under the linear Gaussian assumption, can by calculated by: in which the measurement residual is given by and the covariance is

Remark 2: Generally, without considering the hard decision, the HSTC estimation is actually a layered multiple model solution with information fusion on two levels. The outer layer is the combination estimates results from different classes. The inner layer is the estimate under each target class. With this HSTC method, both tracker (

4.2.2. Hard Decision Related HSTC Estimation

Denote by the hard decision time. As decision is closely related to estimation in HSTC, with respect to , we split the whole time horizon into three parts: before , at and after .

Before

Before the hard decision is made, both tracking and classification are concerned. The HSTC classifier and tracker and given in (6) and (10), respectively. For a clearer description, we present them again:

HSTC classifier:

HSTC tracker:

Intuitively, estimation (for tracking) is a combination of the estimation results conditioned on all possible classes while decision (for classification) is a soft probability. In general, before hard decision is made, tracking and classification are obtained simultaneously by effectively using their mutual-effect. Specifically, classification affect tracking through the posterior class probability while tracking affect classification through .

At

At , the hard decision is made, which is to “select” one class from all possible classes following SPRT. Accordingly, estimation is conditioned on decision, given by: given by (11) with i being the selected class.

If the decision at is correct, we can get better tracking performance since only the true models are used for estimation and the computational burden will also significantly lowered. Although incorrect decision will deteriorate the estimation performance, on average, estimation would perform good since the decision performance is guaranteed by SPRT. This verifies that tracking and classification are closely linked.

After

After the hard decision is made, we just need to do target tracking. Once the hard decision is made, it is unchangeable. Therefore, from on, the HSTC estimation is and decision will always be . In other words, terminating combination of the class-dependent estimates, and output the estimate corresponding to the hard decision .

Remark 3: Note that our HSTC method always have dual goals: tracking and classification. Before the hard decision is made, we take advantage of the two-way dependence between tracking and classification to achieve better performance of both. Once the hard decision is made, we output the decision and the corresponding estimation result simultaneously. After the hard decision, decision will not change and estimation is the one under decided class. It is worthwhile to mention that the HSTC algorithm significantly differs from the decision-based maneuvering target tracking (MTT) problem. MTT is a pure tracking problem, in which decision is secondary and assists estimation to achieve better tracking performance.

Remark 4: In this paper, we consider the simultaneous tracking and classification of the single target without measurement uncertainties. Thus, there is no data association problem, which is of paramount importance for multiple target tracking and classification. Actually, the proposed HSTC method can be also extended to the case for multiple target tracking and classification by adopting some data association techniques, i.e., joint probabilistic data association (JPDA). Once we determine the state estimate under each target class and the corresponding likelihood of the target class, the proposed HSTC still works.

5. Illustrative Examples

In this Section, two simple but representative joint tracking and classification problem with hard decision are presented for illustration. Suppose there is only one target with two possible types and , i.e., a fighter and an airliner. Our goal is to simultaneously track and identify the target, and take corresponding actions after the hard decision is made. Note that this paper considers the military application of the JTC problem. It is difficult to obtain real data in a battlefield environment. Therefore, we use the simulated data in the following.

It is worthwhile to mention that in this paper, target tracking is based on radar data, which differs from visual tracking. In visual tracking, the number of measurements (e.g., pixels in a frame) in much larger and the noise is much lower than in our case, so a lot of specific object information can be extracted (features including color, edges) extracted from the images [23,32,33]. However, this information cannot be obtained for tracking using radar data, where only dozens of position measurements may be available.

In example 1, classes differ from each other in maneuverability. The single model HSTC method is adopted using only kinematic measurements. In example 2, classes differ from each other in both maneuverability and attribute. The multiple model HSTC method is adopted using both kinematic and attribute measurements.

5.1. Example 1

With the linear motion assumption, the target state evolves according to the following evolution model: The only difference between the two classes is the control input . Specifically, class 1 has while class 2 has . Here, g denotes one gravitational acceleration.

The kinematic measurement model is given by

The standard deviations of the process noise and measurement noise were set as m/s and m/s, respectively. Sampling time s. The simulation results were based on the average of 5000 Monte Carlo runs. The ground truth was generated from a Gaussian distribution with m m/s and . The simulation results were obtained by 5000 Monte Carlo runs, and the true target class was 1.

Table 2 shows the decision performance of the proposed HSTC method. It can be seen that the actual error rate is smaller than the controlled value. For the controlled type I and type II error , the actual error rate is only ; for , it is only . The third column shows the time instant when the hard decision is made in HSTC, which is critical in joint tracking and classification with hard decision. It took HSTC only s on average to identify the class label for and only s on average for . In general, Table 2 verifies that on the premise of ensuring the decision error rate less than the controlled value, the hard decision can be made timely.

Table 2

Simulation results of HSTC.

α,β	Actual Error Rate	Average Decision Time
0.01	0.0086	45.2548 s
0.03	0.0256	34.7372 s

Table 3 shows that HSTC saves about thirty-five percent of the calculation compared with STC, which is consistent with the theoretical analysis in the previous part. In HSTC, when the hard decision is made on class 1, all the calculation related to class 2 will be terminated, and thus the computational burden will surely be reduced. This makes HSTC more applicable in practice.

Table 3

Average computational load per run ().

STC	HSTC
1.3289×10^−2 s	0.8677×10^−2 s

Figure 1 shows the estimation performance of HSTC. The tracking errors of STC and HSTC are both between those of using classes and , and as time goes on, they approach the performance under true class quickly. Note that STC is slightly better than HSTC finally, resulting from decision errors. Actually, STC only pursues the tracking performance. As the weighted sum of estimates under both classes, STC has the optimal estimation performance in the sense of minimum mean square error. However, HSTC has dual goal: classification (make a correct and quick decision) and tracking (have small estimation error), and we need to achieve good classification and tracking performances simultaneously.

Figure 1

RMS error of position and velocity.

It is worthwhile mentioning that Example 1 is adopted as a simple scenario for illustration. In this example, each target has only one type of maneuvering mode, which is the only characteristic that distinguishes the targets. We do not adopt the multiple model approach because in this simple case, the errors caused by inappropriate modeling will be eliminated, and thus it can only result from the HSTC method. Actually, when multiple models are taken into account, similar conclusions can be obtained, which will be shown next.

Remark 5: This example shows that the proposed HSTC method satisfies the requirements of practical JTC problems involving hard decision. What is more important is that the superiority of HSTC is also demonstrated in three aspects: a) the decision error rate is within the controlled value; b) the hard decision can be obtained timely; c) the HSTC estimation also performs good, which is close to the optimal MSE estimation.

5.2. Example 2

In this example, we consider a JTC problem with hard decision using both kinematic and attribute measurements. There is no unified model description of a target feature. For example, radar cross section (RCS) is a function of the orientation of the target and its range, size, and type. Infrared (IR) can provide the shape features such as target spatial distribution and target area. Electronic support measurement (ESM) exploits target EM emission to supply target identity information. In this paper, we adopt the ESM measurement similar to [24]. Each target has several types of emitters, which can be used as an attribute feature. At every time instant, an emitter is either “on” or “off”, and this state switches according to a Markov chain. More details can be found in [24].

Denote by the kinematic measurement and the attribute measurement, respectively. is the target attribute feature, which contains all the attribute information and can evolve over time. Then, the joint measurements can be modeled by the conditional pdf-pmf

For simplicity, assume that the two measurement processes are conditionally independent:

This is usually reasonable since only depends on the target state and only depends on the target feature .

Each class is modeled by multiple dynamic models. Target motion model is of the same form as before except that for class i belongs to model set , , where . The initial model probabilities for each class are and . For each run the true target was still class 1, and the truth state was still generated from a Gaussian distribution as in Example 1. For the attribute evolution and measurement model, suppose class 1 has emitter and class 2 has emitter . To describe the independently emitter usage process, the transition probability matrices for these two classes are the same as and in [28]. The measurement process is declare declare equals to for and 0.2 otherwise. For more details about the evolution and measurement of the attribute data, please refer to [24].

The IMM estimation is adopted in this example. In each class, the models switch according to the transition probability matrix (TPM). The TPM for class 1 target is given by for class 2 target is given by

The results were based on 5000 Monte Carlo runs. Sampling time s, the covariance of process noise and measurement noise were (m/s) and (m/s), respectively. Controlled type I and type II errors .

Table 4 shows the decision performance of the proposed HSTC method using both kinematic and attribute measurements. When compared with Table 2, Table 4 shows that given the same type I and type II errors, HSTC using both kinematic and attribute measurements can make a decision much quicker with smaller actual error rate. Take as an example, it takes HSTC with multisensor data only s to identify the class label with only actual error rate, while s with actual error rate for HSTC with only radar measurement. This demonstrates that the performance of classification is significantly improved with the help of the attribute measurements.

Table 4

Simulation results of HSTC (kinematic + attribute).

α,β	Actual Error Rate	Average Decision Time
0.01	0.0002	7.6310 s
0.03	0.0010	6.2720 s

Table 5 shows that HSTC can save nearly half of the computational load compared to STC. However, when comparing Table 5 with Table 3, i.e., the computational load with kinematic data only, we can find that both the computation complexity of STC and HSTC increase due to the introduction of the attribute measurements.

Table 5

Average computational load per run (kinematic + attribute) ().

STC	HSTC
2.3034×10^−2 s	1.238×10^−2 s

Figure 2 shows the tracking performance of HSTC using both kinematic and attribute measurements. Compared with Figure 1, similar rule can be found in Figure 2. However, with the help of attribute measurements, all tracking error curves converge to their steady state values more quickly. Besides, the gap between the steady state value and the bound is much smaller in Figure 2 than in Figure 1.

Figure 2

RMS error of position and velocity in Example 2.

Remark 6: Generally speaking, the HSTC method can meet the demand of the joint tracking and classification problem with hard decision. It can explicitly output the hard decision of the target class. Meanwhile, it accounts for both the timeliness of hard decision, the decision error, and also the corresponding estimation performance.

6. Conclusions

To solve the practical JTC problem with hard decision, this paper proposes a Hard decision-based STC (HSTC) method, in which SPRT is adopted for decision making. The proposed HSTC method takes advantage of both the SPRT and the STC method. Specifically, on the premise that decision error rate is within the acceptable range, the HSTC method can make an explicit hard decision about the class label as soon as possible. For estimation, different estimation strategies are provided by fully accounting for the mutual information exchange between tracking and classification.

In general, with the proposed HSTC method, all the decision timeliness, decision error, and the estimation performances are considered. Simulation verifies that HSTC cannot only meet the demands of JTC problem with hard decision but also performs good in both tracking and classification. In this paper, we only consider the case of single target with binary target classes. For multiple targets simultaneously detection, tracking and classification with multi-classes is under further research.