Monitoring COVID-19 social distancing with person detection and tracking via fine-tuned YOLO v3 and Deepsort techniques

In plain words

During an epidemic, keeping people apart is a common measure. This paper describes a camera-based checker for that. A neural network finds the people in each video frame and draws a box around each one, and a tracker keeps the same ID on a person from frame to frame. The system then places every person in a rough three-number space and treats people who stand close together as one group. Each frame gets a score, the number of people divided by the number of groups, which the authors call a violation index. The authors compare three well-known detectors and find that one of them balances accuracy and speed best. The paper does not measure how accurate the distance estimate or the grouping is.

TL;DR

• The paper builds a social-distancing monitor on video: a person detector (YOLO v3), a tracker (DeepSORT), and a pairwise-distance rule that groups people and counts a violation index §IV-A.
• The detector is fine-tuned on 800 Open Images "Person" images for a binary person / not-person task §V.
• In the authors' tests YOLO v3 gives the best balance of mAP and FPS against Faster R-CNN and SSD tab. III.
• There is no official code and no quantitative evaluation of the grouping or the violation index. Use the method as a blueprint, not as a validated tool.

Problem

Social distancing is treated as the main containment measure while no vaccine or effective drug exists §I. The authors want to automate checking of distances among people in surveillance video, and to compare detection and tracking schemes for that purpose §I. They note that manual monitoring is costly and risky for staff §II. The paper gives no measurement of how often manual monitoring fails.

The SIR-type models in §II (equations 1 to 3) describe disease spread with a social-distancing factor. They are background only: the monitor does not use them, and the paper does not fit them to data.

Method

The system has two parts. A detector and a tracker give each person a box and a stable ID in each frame §III-D, §III-E. A geometric rule then uses the box centroids and sizes to group people who are close, and computes a violation index §IV-A.

The workflow in §IV-A has seven steps:

1. Fine-tune an object detector to find people in footage §IV-A step 1 §V.
2. Feed surveillance frames to the detector. Each person gets a box and an ID §IV-A step 2 §III-E.
3. Map each person to (x, y, d): the box centroid plus a depth value computed from the box width and height §IV-A step 3.
4. Compute the pairwise L2 distance between people in this three-number space §IV-A step 4.
5. Mark two people as neighbours when their distance is below a closeness threshold, which the authors set between 90 and 170 pixels and update per location in the frame §IV-A step 5.
6. Put neighbouring people in one group and colour the group §IV-A step 6.
7. Compute the violation index v = n_p / n_g, where n_g is the number of groups and n_p is the number of people in close proximity §IV-A step 7.

The authors also state that the estimated number of violations is the violation index times the number of groups §IV.

# Per-frame social-distancing check  [§IV-A]
# Input:  frame; detector; tracker; threshold_px in [90, 170]
# Output: groups of track IDs; violation index

def check_frame(frame, detector, tracker, threshold_px):
    # Find people and keep their IDs across frames  [§IV-A steps 1-2]
    boxes = detector(frame)                      # (x, y, w, h) per person
    tracks = tracker.update(boxes)               # adds track_id  [§III-E]

    # Map each person to (x, y, d) with a depth value  [§IV-A step 3]
    points = []
    for track in tracks:
        x, y, w, h = track.box                   # centroid and box size
        d = ((2 * 3.14 * 180) / (w + h * 360)) * 1000 + 3   # as printed
        points.append((x, y, d))

    # Pairwise L2 distance over n = 3 features  [§IV-A step 4]
    dist = pairwise_l2(points)                   # dense (N, N) matrix

    # Neighbours are closer than the threshold  [§IV-A step 5]
    neighbours = dist < threshold_px             # may vary per location

    # Paper does not state how neighbour links form groups.
    # Connected components is our assumption  [§IV-A step 6]
    groups = connected_components(neighbours)

    # Violation index: people in proximity per group  [§IV-A step 7]
    n_groups = len(groups)
    if n_groups == 0:
        return groups, None                      # not stated in the paper
    n_people = sum(len(g) for g in groups if len(g) > 1)   # assumed
    violation_index = n_people / n_groups
    return groups, violation_index

Key equations, as printed in the source:

Social-distancing SIR variant (background only, not implemented by the monitor) §II:

\begin{aligned} \frac{dS}{dt} &= -\beta S \frac{I}{N}a(I,N) \\ \frac{dI}{dt} &= -\delta I+\beta I \frac{I}{N}a(I,N) \\ \frac{dR}{dt} &= \delta I \end{aligned} eq. 1

Long-term awareness and short-term awareness factors, with behaviour parameter k ≥ 0 §II:

a = \left(1 - \frac{I+R}{N}\right)^{k} eq. 2

a = \left(1 - \frac{I}{N}\right)^{k} eq. 3

Anchor-box loss for one anchor a, with ground-truth box g, positive indicator 1^obj_a and weights α and β §III-A, Loss Function:

L(a|I;\omega) = \alpha \cdot 1_{a}^{obj} L_{reg}\big(f(g_a|a) - Y_{reg}(I|a;\omega)\big) + \beta \cdot L_{cls}\big(y_a, Y_{cls}(I|a;\omega)\big) eq. 4

Faster R-CNN classification and smooth-L1 regression losses §III-B:

\begin{aligned} L_{cls}(p_i, p_i^{*}) &= -p_i^{*} \log(p_i) - (1 - p_i^{*}) \log(1 - p_i) \\ L_{reg}(t^{u}, v) &= \sum_{x \in \{x,y,w,h\}} L_{1}^{smooth}(t_{i}^{u} - v) \\ L_{1}^{smooth}(q) &= \begin{cases} 0.5 q^{2}, & \text{if } |q| < 1 \\ |q| - 0.5, & \text{otherwise} \end{cases} \end{aligned} eq. 5

SSD localisation loss, with matched-box encoding §III-C:

\begin{aligned} L_{reg}(x, l, g) &= \sum_{i \in pos}^{N} \sum_{m \in \{c_x, c_y, w, h\}} x_{ij}^{k}\, smooth_{L_1}(l_i^m - \hat{g}_j^m) \\ \hat{g}_j^{c_x} &= \frac{g_j^{c_x} - a_i^{c_x}}{a_i^{w}}, \quad \hat{g}_j^{c_y} = \frac{g_j^{c_y} - a_i^{c_y}}{a_i^{h}} \\ \hat{g}_j^{w} &= \log\!\left(\frac{g_j^{w}}{a_i^{w}}\right), \quad \hat{g}_j^{h} = \log\!\left(\frac{g_j^{h}}{a_i^{h}}\right) \\ x_{ij}^{p} &= \begin{cases} 1, & \text{if IoU} > 0.5 \\ 0, & \text{otherwise} \end{cases} \end{aligned} eq. 6

\begin{aligned} L_{cls}(x, c) &= -\sum_{i \in Pos}^{N} x_{ij}^{p} \log(\hat{c}_i^{p}) - \sum_{i \in Neg} \log(\hat{c}_i^{o}), \\ \hat{c}_i^{p} &= \frac{\exp(c_i^{p})}{\sum_{p} \exp(c_i^{p})} \end{aligned} eq. 7

YOLO v3 loss, as printed. The source label is eq9, but the sequential count of display equations in the source gives eight, so cite it by section §III-D:

\begin{aligned} &\lambda_{coord} \sum_{i=0}^{S^2} \sum_{j=0}^{B} \mathbb{1}_{i,j}^{obj} \big[(t_x - \hat{t}_x)^2 + (t_y - \hat{t}_y)^2 \\ &\qquad + (t_w - \hat{t}_w)^2 + (t_h - \hat{t}_h)^2\big] \\ &+ \sum_{i=0}^{S^2} \sum_{j=0}^{B} \mathbb{1}_{i,j}^{obj} \Big(-\log(\sigma(t_o)) + \sum_{k=1}^{C} BCE(\hat{y}_k, \sigma(s_k))\Big) \\ &+ \lambda_{noobj} \sum_{i=0}^{S^2} \sum_{j=0}^{B} \mathbb{1}_{i,j}^{noobj} \big(-\log(1 - \sigma(t_o)) \end{aligned} §III-D

Deepsort state, constant-velocity model, Kalman filter §III-E:

x = [u, v, \lambda, h, x', y', \lambda', h']^{T} §III-E

Deepsort motion gate and appearance gate, with combined cost and joint gate §III-E:

d^{(1)}(i,j) = (d_j - y_i)^{T} S_i^{-1} (d_j - y_i) eq. 10

b_{i,j}^{(1)} = \mathbb{1}\big[d^{(1)}(i,j) < t^{(1)}\big] eq. 11

d^{(2)}(i,j) = \min\{1 - r_j^{T} r_k^{(i)} \mid r_k^{(i)} \in \mathbb{R}^{2}\} eq. 12

b_{i,j}^{(2)} = \mathbb{1}\big[d^{(2)}(i,j) < t^{(2)}\big] eq. 13

c_{i,j} = \lambda\, d^{(1)}(i,j) + (1 - \lambda)\, d^{(2)}(i,j), \qquad b_{i,j} = \prod_{m=1}^{2} b_{i,j}^{(m)} §III-E

Pairwise distance in the grouping step §IV-A step 4:

\lVert D \rVert_2 = \sqrt{\sum_{i=1}^{n} (q_i - p_i)^2}, \qquad n = 3

Depth value used as the third feature §IV-A step 3:

d = \frac{2 \cdot 3.14 \cdot 180}{w + h \cdot 360} \cdot 1000 + 3

Detection and tracking background

Anchor boxes §III-A. An anchor has size p in (0, 1] and aspect ratio r > 0. For an image of breadth b and height h, the authors build an anchor with dimensions b·p·√r by h·p·√r §III-A. Table II gives the settings. The paper does not state how ground-truth boxes are matched to anchors beyond the IoU threshold for SSD eq. 6.

Faster R-CNN §III-B. A Region Proposal Network proposes regions, and a Fast R-CNN head classifies them and refines the boxes. Its loss is eq. 5. The paper uses Inception v2 as the backbone §III intro.

SSD §III-C. A single-pass detector with multi-scale feature layers and default boxes, followed by non-maximum suppression eq. 6, eq. 7. The paper uses Inception v2 as the backbone §III intro.

YOLO v3 §III-D. Uses the Darknet-53 backbone with residual blocks and upsampling. It predicts at three scales per spatial location, uses logistic classifiers instead of softmax for multi-label classes, and scores each prediction with objectness, box and class terms §III-D.

DeepSORT §III-E. Each track keeps a Kalman state (the 8-value vector above). A track is confirmed after a successful association; it is tentative for its first three frames and is deleted if it is not mapped. Association combines a Mahalanobis motion gate eq. 10, eq. 11 with a cosine-distance appearance gate eq. 12, eq. 13, weighted by λ §III-E. Hungarian assignment solves the mapping §III-E.

Backbone comparison (Table I)

The authors compare feature-extractor backbones on ImageNet and pick Inception v2 for the Faster R-CNN and SSD heads because its accuracy-to-parameter ratio is highest in their table §III intro tab. I.

Source: tab. I. The paper's ratio is its own metric, not a standard one.

Hyperparameters

Anchor-box settings, as printed tab. II:

Other values:

Results

Author-reported, on the authors' 800-image Open Images person dataset with a test split and Oxford Town Centre frames, on a GTX 1060 §V tab. III. The paper compares three detectors and does not report a tracking metric or an accuracy figure for the violation index §VI.

The authors call YOLO v3 "balanced" in mAP, training time and FPS §V. The Faster R-CNN results have the best mAP and loss but the lowest FPS §V.

Qualitative output: sample frames of Oxford Town Centre footage show violation indices of 3, 2, 2 and 2.33 §VI fig. 8. These are examples, not a measured accuracy.

Paper vs code

No official code.

Searched, in order:

1. The paper and its arXiv metadata: no code link. The LaTeX bibliography links only third-party tools (a GitHub tutorial repository and the OIDv4 toolkit), which are not the authors' code.
2. A project page: none linked.
3. GitHub repository search for "social distancing yolo deepsort", "violation index social distancing" and "monitoring social distancing Punn" returned only unrelated third-party repositories or nothing. User search for the authors (Punn, Sonbhadra) found accounts, but none had a repository on this topic in its search results.

Pitfalls

• No validation of the core output. The grouping and the violation index are never compared with ground truth. The paper does not measure distance error, grouping precision, or tracking identity switches. Do not claim accuracy for the monitor.
• No metric distance. The intro cites 6 feet as the WHO norm §I, but the pipeline uses pixel distances in a space that mixes pixel coordinates with a depth value. The paper gives no camera calibration, and no conversion between pixels and metres. Implementers need to add calibration before any distance claim.
• Depth formula. The depth value d = (2·3.14·180)/(w + h·360)·1000 + 3 is printed without derivation, units, or a statement of whether w and h are pixels §IV-A step 3. The abstract instead says the feature space uses "dimensions of the bounding box" [abstract]. Treat d as a proxy the authors chose, not as a depth in any unit.
• Undefined group rule. The paper says neighbours "form a group" but never gives the rule for joining neighbours into a group. The pseudocode uses connected components; that is our assumption, flagged in the comments above.
• Violation index is circular. The estimated violations are v · n_g, and v = n_p / n_g, so v · n_g = n_p: the "estimate" is just the count of people in proximity §IV, §IV-A step 7.
• Undefined closeness update. The threshold is said to vary between 90 and 170 px "based on the spatial location", with no rule given §IV-A step 5.
• Which detector feeds DeepSORT is contradictory. §III-E says a Faster R-CNN is trained for person identification and used with DeepSORT. §IV says YOLO v3 is used with DeepSORT. §V chooses YOLO v3 for the video monitor. Use YOLO v3, the final choice, but check this with the authors.
• Backbone statement is ambiguous. §V says all detectors were fine-tuned "with Inception v2 as a backbone", but §III-D describes YOLO v3 as Darknet-53.
• Probability roles reversed in Faster R-CNN loss. The paper calls p_i* the "predicted class" and p_i "actual class" §III-B, which is the reverse of the usual convention. Follow the equation's form, not the prose.
• Equation and typo errors in the source. Eq. 1 uses a(I,N) while the text says a(I,R) §II. The SSD localisation loss sums over k but the indicator uses p eq. 6. The YOLO loss ends with an unclosed parenthesis and does not give λ_noobj §III-D. The DeepSORT state uses λ as the aspect ratio, calls h "the height of the image" §III-E, and uses the symbol b^(1) for both gates eq. 11, eq. 13. The cosine-distance feature is written as r ∈ R² eq. 12. The ResNet v2 row of Table I cites the same reference as Inception v3 tab. I.
• Loss values are not comparable across models. The YOLO v3 total loss is the highest in Table III (0.87) yet is called balanced tab. III §V. The three losses are defined differently, so compare only mAP and FPS.
• Metric and speed setup are missing. The IoU threshold for mAP, the FPS measurement hardware, the batch size and the video resolution are not stated. Figure 7 shows per-iteration losses on the OID validation set, but no numbers are given for that curve §V fig. 7.
• Dataset rights. The Open Images images carry the rights of their individual sources. The paper does not state the image licences, and the Oxford Town Centre footage is cited as an external source §V. Check both before reusing data.
• Figure numbering. The source labels figures fig2 to fig9, but in the printed paper they are numbered 1 to 8 in order of appearance. This card uses the printed numbers; the in-text reference "Fig. 9" is printed as Fig. 8 §VI.
• Tracking assumptions. The paper says the Mahalanobis gate fails when the camera moves, yet the motion model is used with a non-stationary camera §III-E. Camera motion is not tested.
• Cost. The dense pairwise matrix grows with the square of the number of people in a frame §IV-A step 4.

Lineage

• Builds on: YOLO v3, Redmon et al. 2018 §III-D; DeepSORT, Wojke et al. 2017 §III-E; Faster R-CNN, Ren et al. 2015 §III-B; SSD, Liu et al. 2016 §III-C; the Open Images dataset, Kuznetsova et al. 2020 §II.
• Compared against: Faster R-CNN and SSD, with the same person-detection training setup §V tab. III. YOLO v1 and v2 are described but not tested §III-D.
• Superseded by: not stated in this paper.