A novel social distancing analysis in urban public space: A new online spatio-temporal trajectory approach.

Social distancing in public spaces plays a crucial role in controlling or slowing down the spread of coronavirus during the COVID-19 pandemic. Visual Social Distancing (VSD) offers an opportunity for real-time measuring and analysing the physical distance between pedestrians using surveillance videos in public spaces. It potentially provides new evidence for implementing effective prevention measures of the pandemic. The existing VSD methods developed in the literature are primarily based on frame-by-frame pedestrian detection, addressing the VSD problem from a static and local perspective. In this paper, we propose a new online multi-pedestrian tracking approach for spatio-temporal trajectory and its application to multi-scale social distancing measuring and analysis. Firstly, an online multi-pedestrian tracking method is proposed to obtain the trajectories of pedestrians in public spaces, based on hierarchical data association. Then, a new VSD method based on spatio-temporal trajectories is proposed. The proposed method not only considers the Euclidean distance between tracking objects frame-by-frame but also takes into account the discrete Fréchet distance between trajectories, hence forms a comprehensive solution from both static and dynamic, local and holistic perspectives. We evaluated the performance of the proposed tracking method using the public dataset MOT16 benchmark. We also collected our own pedestrian dataset "SCU-VSD" and designed a multi-scale VSD analysis scheme for benchmarking the performance of the social distancing monitoring in the crowd. Experiments have demonstrated that the proposed method achieved outstanding performance on the analysis of social distancing.

Introduction

The COVID-19 pandemic is ravaging the world, which has sadly caused a significant loss to human life, and a great negative impact on society and the economy. On 30 January 2020, the World Health Organisation (WHO) declared that the outbreak of COVID-19 constitutes a Public Health Emergency of International Concern (PHEIC) (Statement on the second meeting of the International Health Regulations (2005) Emergency Committee regarding the outbreak of novel coronavirus (2019-nCoV, 2021). On 13 January 2021, it reported that there have been 90,054,813 confirmed cases of COVID-19, including over 1,945,610 deaths globally (WHO Coronavirus Disease (COVID-19) Dashboard, 2021). The rapid spread of COVID-19 is mainly through close contact from people to people, and asymptomatic carriers can also spread the virus to others (Aguilar, Faust, Westafer, & Gutierrez, 2020). Due to the high density and mobility of the urban population and the complexity of the urban environment, the spread of the pandemic has been exacerbated to some extent, which brings severe challenges to the construction, governance and sustainable development of cities.

For the epidemic diseases, measures that are taken to prevent and control infections include vaccination, treatment, quarantine, isolation, and prophylaxis (Dias, Queiroz, & Martins, 2020). However, the vaccine for COVID-19 has not yet entered the promotion stage, and the more contagious coronavirus variant has been detected. In this scenario, one effective way to control or slow down the spread of coronavirus is to make sure people maintain social distancing in public places. There exists some work in the literature studying the impact of social distancing on the progression of the coronavirus (Cacciapaglia and Sannino, 2020, Dias et al., 2020, Hellewell et al., 2020, Prem et al., 2020, Rahmani and Mirmahaleh, 2020, Sun and Zhai, 2020). Using Wuhan as a case study, Prem et al. (2020) stated that physical distancing based non-pharmacological interventions have a high potential for flattening the peak of COVID-19 and reducing the overall number of cases. Cacciapaglia and Sannino (2020) demonstrated that social distancing measures are more efficient than border control in delaying the epidemic peak. Sun and Zhai (2020) researched the efficacy of social distancing and ventilation effectiveness in preventing COVID-19 transmission. With the current epidemic unlikely to end in the short term, keeping a safe social distancing1 from others in public spaces and workplaces is one of the key measures for maintaining a low risk of infection.

In recent years, with the deepening of the concept of “smart sustainable cities” (Bibri and Krogstie, 2017, Silva et al., 2018), countries around the world have deeply integrated information technologies with the various needs of urban development. In this context, some research work has explored ways to prevent and respond to the ongoing COVID-19 pandemic by using urban infrastructures and emerging technologies (Bhattacharya et al., 2020, He et al., 2020, Loey et al., 2020, Nguyen et al., 2020, Silva et al., 2020, Zhou et al., 2020), especially in the aspect of automatic social distancing monitoring in public places (Ahmed et al., 2020, Cristani et al., 2020, Khandelwal et al., 2020, Nguyen et al., 2020, Punn et al., 2020, Sathyamoorthy et al., 2020, Shorfuzzaman et al., 2021, Yang et al., 2020). This helps to enhance the resilience and sustainability of cities. The construction of smart cities has also resulted in an explosive growth in video data taken from public spaces. Compared with other big data utilised in existing researches, the video data contains wealthy spatial and temporal information about humans. Exploiting video data to study and analyse human trajectories can more precisely mine human activities in various complex scenes, which is an excellent supplement to non-visual big data and has unique advantages. Therefore, during the pandemic, it is of great theoretical significance and research value to measure and analyse the social distancing between pedestrians based on their spatio-temporal trajectories using surveillance videos in public places and take appropriate epidemic prevention measures according to the crowd gathering situations. This research topic is called Visual Social Distancing (VSD), which refers to approaches relying on video cameras and other imaging sensors to analyse the proxemic behaviour of people (Cristani et al., 2020).

In this paper, a new VSD method based on the human spatio-temporal trajectory has been proposed to quantify and analyse the social distancing between pedestrians in public spaces. The contributions of our work are summarised as follows:

A new hierarchical association based online and real-time multi-pedestrian tracking method is proposed to obtain pedestrians’ trajectories, which can effectively reduce the number of identity switches while achieving overall competitive performance.

A new VSD method based on spatio-temporal trajectories is proposed, considering both the Euclidean distance between sampling points of trajectories from a local perspective and the Frchet distance between trajectories from a holistic view.

A multi-scale social distancing analysis scheme is proposed, including four evaluation metrics, which can evaluate the crowd gathering situations from various time scales respectively.

The rest of the paper is organised as follows. Section 2 gives the related work of VSD and Multiple-Object Tracking. Section 3 provides the detailed contents of the proposed approaches. The related experimental results and discussions are illustrated in Section 4. The conclusion of the given work is presented in Section 5.

Related work

Here we introduce the related work of VSD and Multi-Object Tracking respectively.

VSD problem

Some new research work has been conducted to study the VSD problem for COVID-19 (Ahmed et al., 2020, Cristani et al., 2020, Khandelwal et al., 2020, Nguyen et al., 2020, Punn et al., 2020, Sathyamoorthy et al., 2020, Shorfuzzaman et al., 2021, Yang et al., 2020). For instance, Cristani et al. (2020) proposed a VSD method based on body pose estimation. In each frame, the body pose detector is used to detect visible people. In the corresponding bird’s eye view (top view), each detected pedestrian is regarded as the centre of the circle, and the safe distance as the radius. Then the VSD issue is converted to a sphere collision problem. Yang et al. (2020) proposed a VSD and critical social density detection system to avoid overcrowding by modulating inflow to the region-of-interest (ROI). Shorfuzzaman et al. (2021) used deep learning-based object detection models to detect individuals and implement social distancing monitoring. The Landing AI Company2 developed a social distancing detection tool by detecting pedestrians in real-time video streams and measuring the distance between pedestrians in the corresponding bird’s eye view frames. These methods have made some useful contributions to VSD in the pandemic, but most of them are based on frame-by-frame pedestrian detection rather than pedestrian tracking over a period of time. Although there existed some VSD studies leveraging both detection and tracking approaches (Ahmed et al., 2020, Sathyamoorthy et al., 2020), the tracking algorithms in these methods were employed for tracking detected people and associating them with assigned IDs rather than for trajectory-based social distancing measuring. As a consequence, these frame-by-frame distance metric based VSD methods fall in the category of the detection-based VSD, while the proposed VSD method based on spatio-temporal trajectories distance metric belongs to the trajectory-based VSD. To the best of our knowledge, this research work is the first attempt to address the VSD issue in a dynamic and spatio-temporal manner.

The distinction between the detection-based VSD and the trajectory-based VSD is shown in Fig. 1. The detection-based VSD detects and calibrates the positions of pedestrians, and measures the distance between them frame-by-frame in the bird’s eye view. The trajectory-based VSD tracks pedestrians and calibrates trajectories, and measures the distance between corresponding calibrated trajectories in the three-dimensional spatio-temporal coordinate (adding a time axis ). The detection-based VSD method is from a static and local perspective, while the trajectory-based VSD method is from a dynamic and spatio-temporal perspective. However, during the pandemic, the issue that should be considered is the continuous measurement and analysis of social distancing rather than a specific moment. Therefore, it is more sensible to investigate the VSD problem based on the spatio-temporal trajectory over a time period.

Fig. 1

The distinction between the detection-based VSD and the trajectory-based VSD.

Multi-object tracking

The Multi-Object Tracking (MOT) (Luo, Xing, Zhang, Zhao, & Kim, 2015) task is a fundamental research topic in the field of computer vision, which is widely applied to smart surveillance, autonomous driving, security and other areas. MOT is also an underpinning technique for the trajectory-based VSD. In recent years, with the dramatical improvement of detectors (Redmon and Farhadi, 2017, Redmon and Farhadi, 2018, Ren et al., 2015), Tracking-by-Detection (Al-Shakarji et al., 2018, Bae and Yoon, 2017, Bewley et al., 2016, Chu et al., 2017, Sanchez-Matilla et al., 2016, Wojke et al., 2017) has become the mainstream paradigm of MOT. For Tracking-by-Detection, objects are detected and localised in each frame firstly, and then tracking is conducted by using data association to link detections into trajectories. Therefore, the tracking performance is highly dependent on the performance of the detector and the data association method. Also, MOT can be divided into online tracking (Al-Shakarji et al., 2018, Bae and Yoon, 2017, Bewley et al., 2016, Chu et al., 2017, Sanchez-Matilla et al., 2016, Wojke et al., 2017) and offline tracking (Brendel et al., 2011, Dehghan et al., 2015, Milan et al., 2013, Son et al., 2017). Online tracking refers to data association based only on the past and the current frames, while offline tracking refers to data processing by exploiting all the frames or batch frames. Because offline tracking methods demand the entire set of videos to be obtained in advance, they are less favoured by real-time tasks compared to their online counterparts.

With regard to online Tracking-by-Detection, the traditional methods include the Multiple Hypothesis Tracking (MHT) (Reid, 1979) and the Joint Probabilistic Data Association Filter (JPFAF) (Fortmann, Bar-Shalom, & Scheffe, 1983). Recently, deep learning based methods enhance the tracking performance in complex scenes drastically. The Person of Interest (POI) (Yu et al., 2016) method introduced the high-performance detection and deep learning based appearance feature into the context of MOT. Depending on Convolutional Neural Network (CNN) based detection, the Simple Online and Realtime Tracking (SORT) (Bewley et al., 2016) method utilised the Kalman filter for frame-by-frame prediction and the Hungarian method for data association, by calculating intersection-over-union (IOU) distance as the assignment cost for the association. The Deep SORT (Wojke et al., 2017) method further introduced deep appearance features and motion features on the basis of SORT for assignment costs calculation. Also, some research work in the literature focuses on researching hierarchical data association to improve the reliability of association. Bae and Yoon (2017) proposed a hierarchical association method based on the tracklet confidence, which built optimal tracklets by sequentially linking tracklets and detections using the high and low confidence association. Al-Shakarji et al. (2018) proposed a three-step cascade scheme for efficient data association.

The proposed approaches

The proposed approaches consist of two main steps: (1) a hierarchical association based online multi-pedestrian tracking method to obtain trajectories of pedestrians; (2) a trajectory-based social distancing measurement and analysis method to evaluate social distancing situations between pedestrians in public spaces.

Hierarchical association based online multi-pedestrian tracking

Considering the real-time requirement of the trajectory-based VSD task, it is necessary to design a simple and highly real-time tracking method. Inspired by SORT (Bewley et al., 2016) and Deep SORT (Wojke et al., 2017) method, we utilise the Kalman filter and the Hungarian algorithm (Kuhn, 1955) to address the MOT task. Besides, we design a new hierarchical data association scheme to ensure tracking performance and fewer ID switches.

The states of the tracklets and the transition mechanism

For online Tracking-by-Detection, the essence is a frame-by-frame data association based on detection responses pre-generated by the detector. The detection responses in the current frame are assigned to the existing tracklets (the tracklet is a part of the trajectory formed during the tracking process) according to the data association method. However, the issues of misdetection, occlusion, the appearance and the disappearance of tracking objects lead to many challenges of the MOT task. To tackle these challenges, we adopt a hierarchical data association method based on the states of the tracklets to address multi-pedestrian tracking. According to the number of consecutive associated frames, the states of the tracklets are classified into four categories, namely initial, tentative, stable and deleted. The transition mechanism of the four states of the tracklets is shown in Fig. 2.

Fig. 2

The transition mechanism of the four states of the tracklets.

The initial-state: the initial-state is defined for a new detection that cannot be associated with any existing tracklet. In this state, it is regarded as a new tracklet. If it is associated successfully in the next frame, its state will become as tentative. Otherwise, it will be deleted.

The tentative-state: the tentative-state is a state when the tracklet in initial-state is successfully associated in the next frame. In this scenario, if the tracklet in tentative-state continues to be successfully associated for over consecutive frames (the threshold is a small positive integer), its state will progress to stable. Otherwise, it will be deleted.

The stable-state: the stable-state is defined as a state when the tracklet in tentative-state is successfully associated for over consecutive frames. Once the state of the tracklet becomes stable, it can only be deemed to be finished and deleted when it fails to be associated for over consecutive frames (The threshold is a positive integer significantly greater than ). Besides, the stable-state can be further divided into two stages, the first association stage and the second association stage. These two stages employ different feature metrics for assignment costs calculation during the association (Section 3.1.3).

The deleted-state: under the following conditions, the state of the tracklet will be defined as deleted-state: (1) when the tracklet is in initial-state or tentative-state, and it fails to be associated in the next frame; (2) when the stable-state tracklet fails to be associated for over consecutive frames.

The setting of the initial-state and the tentative-state can effectively address the misdetection issue. This is because the tracklet of the misdetected object in the initial or tentative state will be deleted once the association fails. The stable-state setting takes into account the impact of occlusion during tracking. The tracklet may fail to be associated during the tracking process due to occlusions. Under this situation, the object could not be considered to have disappeared, nor should the tracklet be deleted immediately. The setting of the threshold is beneficial to improve the completeness of tracking. And the two association stages of the stable-state helps to improve the reliability of data association. The deleted-state setting is to reduce unnecessary calculations.

AAM-Softmax appearance feature descriptor

The distance between appearance features is taken into account in the analysis of data association. Inspired by the Cosine Softmax method (Wojke and Bewley, 2018, Wojke et al., 2017) and the Additive Angular Margin Softmax (AAM-Softmax) method (Deng et al., 2019, Jie et al., 2019), an AAM-Softmax appearance feature descriptor network is designed to obtain well-discriminative appearance features of pedestrians. Before being applied in the online tracking task, the network is trained offline by using a large-scale person re-ID dataset Market1501 (Zheng et al., 2015) with 12,936 training images of 751 identities, which facilitates deep metric learning in a pedestrian tracking context.

We mainly use convolutional layers and residual blocks (He, Zhang, Ren, & Sun, 2016) to construct the architecture of the proposed descriptor network (shown in Fig. 3). The pedestrian images with size 128 × 64 × 3 are input into the CNN-based architecture, including two convolutional layers (each layer has 32 kernels with size 3 × 3 and stride 1), a max-pooling layer (pooling size is 3 × 3 and stride is 2) and six residual blocks with two stacked layers. Through the CNN architecture, the feature maps with size 16 × 8 × 128 can be obtained. Then after a Global Average Pooling (GAP) layer, a Batch Normalisation (BN) layer and a Normalisation layer, the descriptor network finally outputs a feature vector with 128 dimensions. In the training phase, the ID of each sample is utilised as a training label, and each embedded feature is input into the fully connected layer followed by the AAM-Softmax classifier, performing supervised learning by the AAM-Softmax loss.

Fig. 3

The architecture of the AAM-Softmax appearance feature descriptor network.

The Softmax classifier is widely used in deep classification tasks, with loss function as Eq. (1): where is the input feature (due to the Normalisation layer in the descriptor, is equal to 1), and is the class label of . is the th column of the weight matrix and is the bias. The target logit term is presented as , is the batch size and is the number of classes. Based on the Softmax loss, the improvement can boost the ability to learning discriminative features effectively. Specifically, the bias is set to 0, and normalisation is imposed on () to project it onto the unit sphere. So the term is equal to , where is the angle between and . The -normalised Softmax loss is presented as Eq. (2): where is the target logit, is the feature scale hyper-parameter. Then by imposing an additive angular margin to the target logit, the AAM-Softmax loss (Deng et al., 2019) formulation can be written as Eq. (3):

The additive angular margin penalty makes the decision boundaries more stringent and separated, enhancing the similarity of intra-class features and the disparity of inter-class features simultaneously, which facilitates to improve the discriminative capability of features effectively. Since the AAM-Softmax loss only imposes an additive angular margin constraint in the angular space, it neither increases the structural complexity of the network nor the number of trainable parameters. When performing the online MOT task, the pre-trained descriptor network is exploited as the feature encoder to obtain discriminative features of pedestrians for the appearance distance metric in the subsequent data association process.

Assignment problem

During the data association process, the assignment costs between tracklets and detections are the basis of association. In this paper, based on different states of the tracklets, different metric methods are utilised to calculate the assignment costs and hierarchical associations are conducted. For each tracklet, except the initial-state tracklet, an appearance feature gallery will be generated, containing the historical appearance features backtracking from the current frame, where is the th -normalised historical appearance feature, denotes the current frame and is the maximum capacity of the gallery. The appearance metric between the tracklet and the detection forms an important part of the distance metric. It can be derived by calculating the minimum cosine distance between all historical appearance features of the tracklet and the appearance feature of the detection, written as Eq. (4): where is the th tracklet (except for the initial-state tracklet), is the th detection and is the appearance feature of . represents the appearance metric between and . Due to the normalisation operation (, ), the cosine similarity can be written as the inner product form .

(1) Assignment costs calculation based on the state.

The initial-state: for the initial-state tracklet, since there is no appearance feature gallery, only position information can be used for data association. Firstly, the standard Kalman filter is used to predict its moving state. Then the IOU distance between the prediction bounding box and the detection bounding box is calculated as the motion metric, presented as Eq. (5): where denotes the th initial-state tracklet and is the prediction bounding box of ; denotes the th detection and is the bounding box of . , and represent the distance metric, the IOU distance and the IOU value between and respectively. is the minimum constant that makes all the IOU distances are not less than 0.

The tentative-state: the tentative-state tracklet already generates an appearance feature gallery , but at this time the features in the gallery are limited. So, at this stage, the IOU distance between the prediction and the detection is still used as the motion metric for association, whilst the appearance metric is only used as a threshold for filtering and discarding infeasible detections. If the appearance metric is greater than the threshold, the candidate detection will be excluded, with no possibility of being associated. The assignment cost is expressed as Eq. (6): where and denote the th tentative-state tracklet and the th detection respectively. , and represent the metric distance, the IOU distance and the appearance metric (obtained by Eq. (4)) between and respectively. is the appearance threshold.

The stable-state: for the stable-state tracklet, the data association process contains two stages. In the first association stage, the Mahalanobis distance is utilised as the motion metric, written as Eq. (7): where and represent the th stable-state tracklet and the th detection respectively. is the Mahalanobis distance between and , and is the measurement space of . In this stage, different metric methods are adopted for different states of the camera (moving or stationary) due to the motion information caused by the camera’s movement. For the moving situation, the appearance metric is mainly considered, and the motion metric is only used for filtering. If the motion metric is greater than the threshold, the candidate detection will be discarded. The assignment cost is written as Eq. (8): where , (calculated by Eq. (4)) and (calculated by Eq. (7)) denote the metric distance, the appearance metric and the motion metric between and respectively. is the Mahalanobis threshold. For the stationary situation, the motion metric and the appearance metric are both taken as the joint metric and integrated into a unified form through the hyper-parameter , presented as Eq. (9):

Due to occlusions or other reasons, the appearance features of the object may change dramatically, causing the stable-state tracklet to fail to be associated in the first association stage. Hence, the second association stage is added for this situation. Instead of considering the appearance metric, only the motion metric calculated by IOU distance is utilised for data association in the second stage.

(2) Hierarchical data association and the states’ update.

The stability of the tracklet determines its confidence. Therefore, according to the order of confidence from high to low, the corresponding states are the stable-state, the tentative state and the initial state, respectively. Based on this confidence order, a hierarchical association method is designed to divide the entire data association stage into three levels. The flow chart of the proposed hierarchical data association is shown in Fig. 4. Specifically, the first association of the stable-state tracklets is performed. Then, those tracklets that fail to be associated subsequently enter the second association stage. After that, the tracklets in the initial-state or the tentative-state are considered to be associated with the remaining detections. According to the states of the tracklets, the corresponding metrics are calculated, and the associations between the tracklets and the detections are conducted by using the Hungarian algorithm (Kuhn, 1955). For the tracklet in initial-state or tentative-state, if it fails to be associated, it will be deleted. This way of dealing with unstable objects can facilitate the tracker to filter the incorrectly detected objects to a certain extent thus to improve the performance efficiency. For the tracklet in the stable-state, if it fails to be associated for over consecutive frames, it will be deleted, which can increase the completeness of the trajectory. For unassociated detection, it will be considered as a new tracklet and will be assigned with a new ID.

Fig. 4

The flow chart of the proposed hierarchical data association.

The description of a tracklet includes the ID, the position information (the coordinates of the bounding box), the appearance feature and its state. After each association, the tracklets will be updated. For the successfully associated tracklet, the coordinates of the tracking bounding box are updated according to its new position; the new appearance feature in the current frame will be added to the appearance feature gallery. But when occlusion occurs, the confidence of the appearance feature decreases due to the introduced noise. To tackle this issue, the IOU values between the prediction bounding box of the tracklet with all detection bounding boxes are calculated and a threshold is set. If there exists an IOU value greater than , the appearance feature gallery is not updated. Besides, the state of the tracklet should be updated as well according to its current state and the transition mechanism (Section 3.1.1).

As a summary, the entire process of online multi-object tracking based on hierarchical data association is illustrated in Fig. 5. For the th frame, firstly the detector is used to conduct multi-object detection. Then, according to different states of the tracklets, different assignment cost calculation methods are adopted to associate the tracklets with the detections hierarchically. After data association, it is required to update the appearance feature galleries and the states of the tracklets. Then the updated tracklets will be associated with the candidate detections in the -th frame.

Fig. 5

The entire process of online multi-object tracking based on hierarchical data association.

Trajectory-based social distancing measurement and analysis

The surveillance videos are taken from arbitrary perspective views, it is necessary to transform the original perspective video into the bird’s eye view to perform distance measurement. This is carried out by utilising the perspective transformation matrix. As the calibration is conducted for the transformation of the ground plane, the bottom–centre point of the tracking bounding box of every trajectory in each frame is transformed into the bird’s eye view as the sampling point of the trajectory. Then, the re-parameterisation time information is added to ensure cannot be backtracked, and the spatio-temporal trajectories of pedestrians are represented in the three-dimensional coordinates space (, and ). For addressing the VSD problem, the discrete Frchet distance (Eiter & Mannila, 1994) is utilised to measure the distance between each spatio-temporal trajectory pair. Finally, the social distancing between pedestrians in the real world can be estimated by multiplying the metric distance with the scaling factor.

Trajectory transformation and distance measurement

The essence of calibration is to map the original video into the bird’s eye view by performing the perspective transformation. The formula of the perspective transformation is presented as Eq. (10): where (, ) and (, , 1) are the Cartesian coordinate and the homogeneous coordinate of the trajectory respectively in each original frame. is the perspective transformation matrix. (, , ) is the calibrated homogeneous coordinate in the bird’s eye view. Its corresponding Cartesian coordinate (, ) can be obtained as Eq. (11):

For calibration, we first need to select a rectangular reference area in the shooting scene. Due to the arbitrary angle of the camera, the reference area appears as a quadrilateral in the original perspective view. Since the video is captured by a single camera (monocular camera), the calibration method is to map the quadrilateral in the original video to the bird’s eye view to re-form a rectangle with the same aspect ratio. Using the four pairs of vertex coordinates of the quadrilateral and the rectangle, the perspective transformation matrix can be calculated by Eq. (10). Then, the calibrated coordinates in the bird’s eye view of each trajectory’s sampling points can be calculated through Eq. (11).

The Frchet distance (Eiter & Mannila, 1994) is to determine the distance between each spatio-temporal trajectory pair and in the calibrated space by taking into account location and time ordering, defined as Eq. (12): where is the distance function of the space . and represent the spatial position of and at time respectively. and are continuous and non-decreasing re-parameterisation. The discrete Frchet distance (Eiter & Mannila, 1994) is an approximation of the continuous Frchet distance. Firstly, the two trajectory curves and are discretised and represented as the sequences with and sampling points, presented with and respectively. Because the distance metric is performed in the bird’s eye view, the corresponding transformed sequences and need to be obtained by Eq. (11). A coupling between and is a sequence of distinct pairs from and , written as Eq. (13): where and , and , and for all , we have or , or . The length of the coupling is the length of the longest link in , presented as Eq. (14): We use Euclidean distance to calculate , and the discrete Frchet distance between and in the bird’s eye view is defined as Eq. (15) By Multiplying the metric distance in the bird’s eye view with the scaling factor , the social distancing in the real world can be estimated. On the one hand, the Euclidean distance is used to measure the distance between each sampling point pair of trajectories from a local perspective, presented as Eq. (16): On the other hand, from a holistic view, the discrete Frchet distance is exploited to measure the distance between trajectory pairs, presented as Eq. (17):

Social distancing analysis

Here we design a multi-scale social distancing analysis scheme to evaluate the social distancing situations in public spaces from multiple time scales. The scheme includes the following four evaluation metrics:

The Average Ratio of Pedestrians with Unsafe Social Distancing (ARP-USD): if the distance between pedestrians, calculated by Eq. (16), is below the minimum acceptable distance, we believe that the pedestrians are at an unsafe distance at this moment. The ARP-USD metric is the mean proportion of the number of pedestrians with an unsafe distance in respect to the total number of people over a period of time in the public space. Given a video with frames, for the th frame, is the total number of tracking persons, is the position point of the th pedestrian. The set of pedestrians with unsafe social distancing in the th frame is represented as Eq. (18): where are the calibrated points of in the bird’s eye view, is the Euclidean distance, is the scaling factor, is the safe distance threshold, and is the number of the elements in . If , the ratio of pedestrians with an unsafe distance in the th frame can be written as , so the ARP-USD is calculated as Eq. (19): where is the number of the frames with .

The Number of Trajectory Pairs with Unsafe Social Distancing (NTP-USD): if the distance between the trajectory pair, calculated by Eq. (17), is below the safe distance , we consider the trajectory pair to be at an unsafe distance. The NTP-USD metric is the number of the stable-state trajectory pairs with unsafe social distancing. Assuming that the number of the stable-state trajectories is ( is dynamically updated) and the th stable-state trajectory is represented as , the set of the stable-state trajectory pairs with unsafe social distancing can be formulated as Eq. (20): where are the mapped trajectories of in the bird’s eye view, where is the discrete Frchet distance and is the scaling factor. is the number of the elements in , indicating the value of the NTP-USD.

If the discrete Frchet distance of the trajectory pair is less than the safe distance threshold, it means that the distance of each sampling point pair of the two trajectories has been less than the safe distance for the entire measurement process. In another word, the two pedestrians have continuously violated social distancing for a period of time. Therefore, based on spatio-temporal trajectories, the NTP-USD metric measures the overall number of trajectory pairs with an unsafe distance in public spaces, which reflects the situation of social distancing violations in the public area over a period of time.

The Number of Pedestrian Pairs with Continuous Unsafe Social Distancing (NPPC-USD): Concerning the spread of pandemics, the longer people stay at an unsafe distance, the higher the risk of infection. Therefore, the duration of pedestrians staying within an unsafe distance is an essential factor that should be taken into account. The discrete Frchet distance used in NTP-USD is to measure the similarity of trajectories for a given duration holistically. But for two dissimilar trajectories, for example, two trajectories facing each other with opposite directions, their Frchet distance can be very large. Hence NTP-USD is unable to spot the infection risk when pedestrians are very close for a certain amount of time but facing back to each other. So, for a trajectory pair, if the number of their sampling point pairs with an unsafe distance is more than the threshold , it is considered as the pedestrian pair with continuous unsafe social distancing. The NPPC-USD is designed to count the number of the above pedestrian pairs, which can reflect the concept of the unsafe distance for a period of time.

The Average Gathering Degree (AGD): to describe the degree of pedestrians gathering in public spaces, the concept of gathering group is defined. As shown in Fig. 6, pedestrians in one gathering group have social distancing with one or more people in the same group less than the safe distance. The social distancing between any two people in different gathering groups is larger than the safe distance. According to the number of pedestrians in a group, the gathering degree is divided into six levels, from 0 to 5 (illustrated in Fig. 6). In order to facilitate unified grading, a single person is also regarded as a gathering group with gathering degree 0. For each frame, the maximum gathering degree is taken as the gathering degree of this frame . The average gathering degree of a video with frames is formulated as Eq. (21): During the pandemic, the larger the number of people in a gathering group, the higher the risk of cross-infection. Therefore, the AGD can reflect the gathering situation of people in a period of time, which is a useful metric for assessing the risk of infection in public spaces.

Fig. 6

The gathering group and the corresponding gathering degree.

Experiments

Datasets

Market1501: the AAM-Softmax appearance feature descriptor network is pre-trained using a large-scale person re-ID dataset Market1501 (Zheng et al., 2015) captured by six cameras. It contains 12,936 images of 751 identities for training, 3,368 images of another 750 identities as the query set, and 19,732 images as the gallery set. The input images are resized to 128 × 64.

MOT16: we evaluate the performance of the proposed tracking method on the MOT16 benchmark dataset (Milan, Leal-Taixé, Reid, Roth, & Schindler, 2016). It consists of 14 video sequences, where 7 videos are used as training and verification sets, and another 7 are employed as test sets. The input sizes of the MOT16 are 1920 × 1080 and 640 × 480. There are front-view scenes taken from moving camera and top-down view scenes captured from surveillance camera. The complex scenes, the large number of pedestrians and the varying illuminations have imposed great challenges in analysing this MOT16 benchmark dataset.

SCU-VSD: we conduct social distancing measurement and analysis experiments on our own dataset, called as SCU-VSD. It includes 8 pedestrian video sequences, which were taken from a pedestrian street with different scenes and perspective views. For each video sequence, the size is 1920 × 1080, the duration is 60 s, and the frame rate is 25 fps (each video gives 1500 consecutive frames).

Implementation details and experimental results

Online hierarchical association based multi-pedestrian tracking

The detection results of the MOT16 benchmark dataset used in the paper are provided by the POI method (Yu et al., 2016). The detector in the POI is Faster R-CNN (Ren et al., 2015) fine-tuned by additional training datasets (including ETHZ pedestrian dataset (Ess, Leibe, Schindler, & Van Gool, 2008), Caltech pedestrian dataset (Dollár, Wojek, Schiele, & Perona, 2009) and their surveillance dataset (Yu et al., 2016)). For the AAM-Softmax loss, the hyper-parameters and in Eq. (3) are set to 30 and 0.006 respectively. The Optimiser is Adam and the batch size is 128. During the training, the learning rate is set to with the first 55,000 interactions, and it decays to in the last 10,000 interactions. For the transition mechanism of the states, the thresholds and are set to 3 and 30 respectively. For hierarchical data association, the appearance threshold in Eq. (6) is set to 0.8 and the Mahalanobis threshold in Eq. (8) is set to 9.49, the hyper-parameter in Eq. (9) is set to 0.2. For the states’ update, the threshold is set to 0.5.

The evaluations of the proposed MOT method are conducted by using following six metrics (Luo et al., 2015). (1) Multi-Object Tracking Accuracy (MOTA): it reflects the overall accuracy according to false positives, false negatives and identity switches; (2) Multi-Object Tracking Precision (MOTP): it reveals the overall tracking precision based on the bounding box overlap between the ground-truth and the prediction position; (3) Mostly Tracked (MT): the percentage of ground-truth trajectories that are covered by the tracker output for more than 80% of their length ; (4) Mostly Lost (ML): the percentage of ground-truth trajectories that are covered by the tracker output for less than 20% of their length; (5) Identity Switches (IDS): the number of times the reported identity of a ground-truth trajectory changes; (6) Fragmentation (FM): the number of times a ground-truth trajectory is interrupted in the tracking result.

The results shown in Table 1 are obtained based on MOT16 benchmark dataset. Compared with other online MOT methods, our proposed hierarchical data association based multi-pedestrian tracking method has achieved overall competitive performance. While maintaining high tracking accuracy and precision (MOTA and MOTP are 61.4% and 79.1% respectively), the IDS of our proposed method decreases to 710, which effectively reduces the number of trajectory ID switches and improves the ability to maintain trajectory ID. The reduction of FM (1913) indicates the decrement of the number of trajectory interruptions.

Table 1

Comparisons of different online algorithms on MOT16 benchmark (with private detectors).

Methoda	MOTA(%)↑	MOTP(%)↑	IDS↓	FM↓	MT(%)↑	ML (%)↓
Config-MOT (Bae & Yoon, 2017)	43.9	76.0	1030		17.4	30.2
MOTDT (Long, Haizhou, Zijie, & Chong, 2018)	47.6	50.9	792	–	15.2
STRN (Xu, Cao, Zhang, & Hu, 2019)	48.5	73.7		–	17.0	34.9
Deep Sort (Wojke et al., 2017)			781	2008
EAMTT (Sanchez-Matilla et al., 2016)			910	–	19.0	34.9
The proposed method						19.9

The values in red and blue represent the optimal and the suboptimal results respectively.

Trajectory-based social distancing measurement and analysis

For each scene in the real world, a rectangular reference area on the ground is selected and its actual length and width are measured. Due to the arbitrary angle of the camera, the rectangular reference area is presented as a quadrilateral in the original perspective video. According to the aspect ratio of the reference area, a reference rectangle is drawn with scaling factor in the bird’s eye view (with size 500 × 500), which corresponds to the calibrated rectangle of the quadrilateral in the original video. Through the coordinates of the four vertex pairs of the quadrilateral and the calibrated rectangle, the perspective transformation matrix of each video can be obtained by using the Eq. (10). By using , the transformed trajectory of each pedestrian in the bird’s eye view can be calculated. The detailed information of the selected rectangular reference area of each SCU-VSD video is shown in Table 2.

Table 2

The information of the selected rectangular reference areas for SCU-VSD Dataset.

SCU-VSD		01	02	03	04	05	06	07	08
In the real world	Width (m)	6	6	7.8	6	6	6	6	7.2
	Length (m)	14.7	12	11	24.5	18	12	15	12.6
In the bird’s eye view	Width (pixel)	60	60	78	60	60	60	60	72
	Length (pixel)	147	120	110	245	180	120	150	126

The comparisons between the original perspective view and the calibrated bird’s eye view for SCU-VSD dataset are shown in Fig. 7. The rectangular reference area in each video is marked as a purple box. Due to the different perspective views of the videos, the reference areas in original videos are presented as different quadrilaterals. In the calibration process, we use four corresponding vertex coordinate pairs of the reference area in the original video and the bird’s eye view to calculate the perspective transformation matrix of each video by Eq. (10), shown in Table 3. The numerical values of each matrix are expressed using scientific notation.

Fig. 7

The comparisons between the original perspective view and the calibrated bird’s eye view for SCU-VSD.

Table 3

The coordinates of the four vertex pairs and the perspective transformation matrix for SCU-VSD.

Dataset	In the original video	In the bird’s eye view	M
SCU-VSD-01	P_1=(695,183),P_2=(328,899)P_3=(1300,940),P_4=(1129,193)	P1′=(225,325),P2′=(225,472)P3′=(285,472),P4′=(285,325)	1.9422e−016.5468e−016.8420e+01−2.1163e−021.4486e+002.1647e+02−1.9038e−052.4575e−031.0000e+00
SCU-VSD-02	P_1=(409,317),P_2=(113,776)P_3=(1199,765),P_4=(1095,310)	P1′=(225,325),P2′=(225,445)P3′=(285,445),P4′=(285,325)	1.5449e−015.8471e−011.3333e+021.4504e−021.4124e+009.7908e+012.2426e−052.1704e−031.0000e+00
SCU-VSD-03	P_1=(397,153),P_2=(51,712)P_3=(1771,1715),P_4=(1510,163)	P1′=(225,300),P2′=(225,410)P3′=(303,410),P4′=(303,300)	7.4208e−023.0369e−011.8572e+02−1.0730e−026.9188e−012.4727e+02−2.5199e−051.1300e−031.0000e+00
SCU-VSD-04	P_1=(781,129),P_2=(320,891)P_3=(1515,940),P_4=(1295,140)	P1′=(225,225),P2′=(225,470)P3′=(285,470),P4′=(285,225)	1.4485e−015.7949e−019.7719e+01−2.3251e−021.4215e+001.2039e+02−1.4727e−052.1771e−031.0000e+00
SCU-VSD-05	P_1=(677,155),P_2=(436,788)P_3=(1529,788),P_4=(1225,155)	P1′=(225,250),P2′=(225,430)P3′=(285,430),P4′=(285,250)	1.4474e−015.2241e−011.1847e+02−8.9224e−161.2690e+001.3379e+02−2.1849e−182.0769e−031.0000e+00
SCU-VSD-06	P_1=(190,355),P_2=(250,913)P_3=(1679,768),P_4=(1146,272)	P1′=(225,325),P2′=(225,445)P3′=(285,445),P4′=(285,325)	1.0968e−013.0204e−012.1112e+026.8670e−029.3750e−011.4407e+028.1190e−051.3861e−031.0000e+00
SCU-VSD-07	P_1=(666,247),P_2=(211,909)P_3=(1613,961),P_4=(1472,263)	P1′=(225,300),P2′=(225,450)P3′=(285,450),P4′=(285,300)	1.0462e−014.1639e−011.3844e+02−9.2389e−039.9577e−011.7481e+024.6381e−061.5342e−031.0000e+00
SCU-VSD-08	P_1=(665,187),P_2=(461,634)P_3=(1569,631),P_4=(1328,186)	P1′=(225,300),P2′=(225,426)P3′=(297,426),P4′=(297,300)	1.5073e−015.3939e−011.1180e+028.1498e−041.2834e+001.7667e+02−5.8150e−072.0913e−031.0000e+00

For social distancing measurement, the safe distance threshold is set to 2 m, the threshold is set to 250, and the scaling factor is set to 0.1. Based on the varying scales of time, the experiments of multi-scale social distancing measurement and analysis in public spaces are performed as follow: (1) for 1/25 s (one frame as a unit), the Euclidean distance between tracking objects in the bird’s eye view is measured to calculate the real-time social distancing between pedestrians, the real-time ratio of pedestrians with unsafe social distancing and the real-time gathering degree; (2) for 10 s (250 consecutive frames as a unit), the ARP-USD and AGD are calculated; (3) for 60 s (1500 frames, this is the entire video duration), the ARP-USD and AGD are calculated; for pedestrians’ trajectories, the NTP-USD and the NPPC-USD are calculated.

The real-time social distancing measurement and analysis for SCU-VSD dataset are shown in Fig. 8. The figure on the left is the original video, and the one on the right is the corresponding bird’s eye view. The tracking pedestrians in the original video are transformed to the trajectory points in the bird’s eye view. The Euclidean distance between tracking object pairs in the bird’s eye view are measured frame-by-frame to estimate the real-time social distances between the pedestrians. If the social distances are less than the safe distance, the tracking bounding boxes in the left figure will change from blue to red, and the corresponding trajectory points in the right figure will change from green to red, with a red line linking pedestrians. The real-time ratio of pedestrians with unsafe social distancing and the real-time gathering degree are calculated, and the results are displayed in the top-left corner of the right figure.

Fig. 8

The real-time social distancing measurement and analysis for SCU-VSD dataset.

Taking 10 s as a unit, each video is divided into 6 periods. For each period, the ARP-USD and AGD metrics are calculated, and the results are drawn using colourmaps. The colourmaps of ARP-USD (on the left) and AGD (on the right) for every 10 s of each video clip are shown in Fig. 9.

Fig. 9

The colourmaps of ARP-USD and AGD for every 10 s.

From Fig. 9, each row of the colourmaps can reflect the changing trend of metrics in different time units of the same video, while each column of the colourmaps can reflect the metrics’ changing tendency of different videos in the same time unit. These trends can be displayed intuitively through colour gradients of the corresponding colorbar. We take the first row and the first column of colourmaps as the examples for analysis. The first row of data represents the ARP-USD and AGD metrics of SCU-VSD-01 clip in different time units. It can be seen that the changing trends of the two metrics of SCU-VSD-01 are roughly identical, rising and then falling, reaching their peaks (80.4% and 1.81 respectively) at the second time unit. The first column of the data represents the comparisons of the two metrics of the 8 video clips in the first time unit. It also can be observed that in this period, the two metric values of SCU-VSD-03 are the minimum (7.07% and 0.06 respectively), while the ARP-USD of SCU-VSD-04 and the AGD of SCU-VSD-01 are the maximum (84.83% and 1.32 respectively). In practical applications, the duration of the time window can be adjusted according to the actual requirement, so that ARP-USD and AGD at different time scales can be obtained.

From the global perspective, the four metrics ARP-USD, NTP-USD, NPPC-USD and AGD for each entire video are calculated, shown in Table 4.

Table 4

The four metrics ARP-USD, NTP-USD, NPPC-USD and AGD for each entire video.

Datasets	ARP-USD (%)	NTP-USD	NPP-CUSD	AGD
SCU-VSD-01	67.01	32	11	1.2
SCU-VSD-02	75.90	10	9	1.0
SCU-VSD-03	47.62	3	2	0.57
SCU-VSD-04	63.84	22	11	1.03
SCU-VSD-05	64.48	14	10	0.98
SCU-VSD-06	49.59	5	4	0.76
SCU-VSD-07	67.64	19	6	0.99
SCU-VSD-08	61.60	6	6	0.92

As shown in Table 4, the NTP-USD, NPPC-USD and AGD of SCU-VSD-01 achieve the maximum values, which are 32, 11 and 1.2 respectively, while the four metrics of SCU-VSD-03 are the minimum, 47.62%, 3, 2 and 0.57 respectively. Comprehensively, it can be concluded that SCU-VSD-01 video has the largest number of pedestrian pairs with unsafe social distancing and the highest average gathering degree. In contrast, SCU-VSD-03 video has the smallest number of pedestrians with unsafe distancing and the lowest average gathering degree.

Conclusion

In this paper, in response to the VSD problem in public places during the pandemic, we first proposed a hierarchical association based online multi-pedestrian tracking method to obtain pedestrians’ trajectories. We then proposed a spatio-temporal trajectory based multi-scale social distancing measurement and analysis method. The proposed VSD method considers both Euclidean distance from a static perspective and Frchet distance from a spatio-temporal perspective to estimate the social distancing and analyse the crowd gathering situations based on a variety of time scales. The multi-scale metrics obtained by the proposed VSD approach can provide the local authorities with guiding information to help them monitor the real-time and overall situations of the social distancing of crowds in public spaces, so as to formulate and take corresponding prevention measures. In addition, for the areas where the pandemic has outbroken, the proposed VSD and analysis scheme can be used to provide useful supporting data for the subsequent epidemiological investigation, such as locating and search of the infection chain.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.