YOLOP: You Only Look Once for Panoptic Driving Perception

In plain words

A self-driving car has to answer three questions from each camera frame: where the other vehicles are, which part of the road it may drive on, and where the lane lines run. The usual approach runs a separate model for each question, one after another, which is slow on the small computers that cars carry.

YOLOP answers all three with one network. A single shared encoder extracts image features once. Three small decoders then read those features: one draws boxes around vehicles, one marks the drivable road, and one marks the lane lines. Because the features are shared, one pass over the image does all the work.

The idea rests on the tasks helping each other. Lane lines often bound the drivable road, and the road usually surrounds the vehicles, so sharing one set of features tends to make each answer more consistent. The authors also tried training the three tasks in stages instead of together, and found that the single joint training run was enough.

TL;DR

YOLOP is one network for driving perception. From one shared encoder it detects traffic objects (only vehicles are evaluated), segments the drivable area and detects lane lines, trained end to end §3 §3.4. It aims at real time on embedded hardware §1. Author-reported results on BDD100K: vehicle detection mAP50 76.5 at 41 FPS, drivable-area mIoU 91.5, and lane accuracy 70.50 with IoU 26.20 tab. 1 tab. 2 tab. 3. The official code differs from the paper in several loss details, so read Paper vs code before you reimplement the losses.

Problem

Driving perception needs object detection, drivable-area segmentation and lane detection, and needs them both accurate and fast §1. Running each task as its own model is slower, and embedded hardware in a car has little compute §1. The paper argues that the tasks share information: lanes often bound the drivable area, and the drivable area usually surrounds the traffic objects §1. A multi-task network could therefore run faster than three separate models and share features §1. The paper also asks whether joint training is better than alternating training in stages §1 §3.4.

Method

YOLOP has one shared encoder and three decoders with no shared blocks between the decoders §3. The encoder is a backbone plus a neck §3.1. The detection decoder is a single-stage, anchor-based detector modelled on YOLOv4 §3.2.1. The two segmentation decoders have the same structure as each other §3.2.2.

1. Backbone. Use CSPDarknet, chosen for its YOLOv4 detection results and because it reduces repeated gradients and the parameter count, which the authors tie to real-time speed §3.1.1.
2. Neck. Combine a Spatial Pyramid Pooling module with a Feature Pyramid Network, so the features carry several scales and semantic levels. Fuse them by concatenation §3.1.2.
3. Detection head. After the FPN, a Path Aggregation Network passes localisation features bottom-up. Predict on the multi-scale fused maps. Each grid cell has three anchors with different aspect ratios. Each anchor predicts a position offset, a height and width scale, class probabilities and a confidence §3.2.1.
4. Segmentation heads. Both take the bottom FPN layer, size (W/8, H/8, 256) §3.2.2. Three nearest-neighbour upsampling steps restore (W, H, 2): per pixel, the probability of drivable area (or lane line) against background §3.2.2. No extra SPP is added, because the encoder's SPP is shared §3.2.2.
5. Loss. Detection loss is a weighted sum of a classification term, an objectness term and a CIoU box term eq. 1 §3.3. The drivable-area loss is cross-entropy eq. 2. The lane loss adds an IoU term, which the authors say suits the sparse lane class eq. 3 §3.3. The total is a weighted sum of the three eq. 4. The paper does not give the weights.
6. Training. The default is end-to-end training §3.4 §4.2. Algorithm 1 trains in three stages: encoder and detection head first, then both segmentation heads with the rest frozen, then everything together alg. 1. The stage-change rule is convergence below a threshold thr alg. 1.

Key equations, as written in the paper:

\mathcal L_{det} = \alpha_1 \mathcal L_{class} + \alpha_2 \mathcal L_{obj} + \alpha_3 \mathcal L_{box} eq. 1

\mathcal L_{da\text{-}seg} = \mathcal L_{ce} eq. 2

\mathcal L_{ll\text{-}seg} = \mathcal L_{ce} + \mathcal L_{IoU} eq. 3

\mathcal L_{IoU} = 1 - \frac{TP}{TP + FP + FN} §3.3

\mathcal L_{all} = \gamma_1 \mathcal L_{det} + \gamma_2 \mathcal L_{da\text{-}seg} + \gamma_3 \mathcal L_{ll\text{-}seg} eq. 4

Here L_class and L_obj are focal losses, L_box is CIoU, and L_ce is cross-entropy with logits §3.3. The paper says the alphas and gammas "can be tuned" and gives no values §3.3.

The training objective, end to end, as the paper describes it:

# Training objective of YOLOP, end to end  [§3.1] [§3.2] [§3.3] [eq. 4]
# Input:  images (B, 3, H, W); per image: vehicle boxes, drivable mask,
#         lane mask (both 2-class maps)
# Output: scalar loss

def training_step(images, boxes, drivable, lanes):
    # 1. Shared encoder: CSPDarknet, then SPP and FPN  [§3.1.1] [§3.1.2]
    fused = encoder(images)
    # 2. Detect head: PAN, then anchors on three scales  [§3.2.1]
    det_out = detect_head(pan(fused))
    # 3. Two segmentation heads read the stride-8 FPN map  [§3.2.2]
    da_logits = da_head(fused.stride8)       # (B, 2, H, W)
    ll_logits = ll_head(fused.stride8)       # (B, 2, H, W)
    # 4. Detection loss: class, objectness and box terms  [eq. 1] [§3.3]
    l_det = a1 * l_class(det_out, boxes) + a2 * l_obj(det_out, boxes)
    l_det = l_det + a3 * l_box(det_out, boxes)   # box term is CIoU
    # 5. Drivable-area loss: cross-entropy on the 2-class map  [eq. 2]
    l_da = cross_entropy(da_logits, drivable)
    # 6. Lane loss: cross-entropy plus an IoU term  [eq. 3] [§3.3]
    l_ll = cross_entropy(ll_logits, lanes)
    l_ll = l_ll + (1 - lane_iou(ll_logits, lanes))
    # 7. Weighted sum of the three task losses  [eq. 4]
    return g1 * l_det + g2 * l_da + g3 * l_ll

The step-by-step schedule of Algorithm 1:

# Step-by-step schedule  [alg. 1] [§3.4]
# Theta holds the encoder (enc), detect head (det) and segmentation heads (seg)

def step_by_step(model, train_set, thr):
    # Stage 1: encoder and detect head, until the loss is below thr  [alg. 1]
    train_until_below(model, train_set, thr, trainable=["enc", "det"])
    # Stage 2: freeze encoder and detect head; train both seg heads  [alg. 1]
    train_until_below(model, train_set, thr, trainable=["seg"])
    # Stage 3: unfreeze everything and train all three tasks  [alg. 1]
    train_until_below(model, train_set, thr, trainable=["enc", "det", "seg"])
    return model

Hyperparameters

Values the paper states are cited to the paper. Values in the code are cited to the official repository at code_commit. "Not stated" means neither source gives the value.

Results

Author-reported, on the BDD100K validation split, after end-to-end training §4.1.1 §4.2. The detection and drivable-area baselines were retrained on BDD100K by the authors. The paper does not say how the lane baselines were obtained §4.1.3. All experiments ran on one GTX TITAN XP §4.1.3.

Ablations, author-reported. Table 4 compares training schedules tab. 4:

Table 5 compares one model trained for all three tasks with models trained for one task each tab. 5. Times are ms per frame:

Table 6 compares the grid-based design (YOLOP) with a region-based design (R-CNNP, Faster R-CNN with two segmentation heads) tab. 6:

The paper's figures are qualitative: day and night detections fig. 3, a detector comparison fig. 4, drivable-area results fig. 5, a comparison with PSPNet fig. 6, lane results fig. 7 and a comparison with ENet-SAD fig. 8. No figure is reproduced on this card.

Paper vs code

Repository: hustvl/YOLOP, MIT licence, at code_commit 8d8f68df318c71f01d6f813c024df646c7d1978f. The repo contains tools/train.py, lib/models/YOLOP.py and the loss code. Its README points to weights/End-to-end.pth for testing README.md.

• One class, so the classification term is zero. The paper's detection loss includes L_class eq. 1. The code sets model.nc = 1 and applies the class loss only when there are more than one class train.py loss.py. The head is built with one class YOLOP.py YOLOP.py. So the class term is always zero in the code.
• Focal loss is off. The paper uses focal loss for L_class and L_obj §3.3. The config default FL_GAMMA is 0.0 default.py, and the focal wrapper is only used when gamma > 0 loss.py. The code therefore uses plain binary cross-entropy for objectness.
• Segmentation loss type. The paper says cross-entropy with logits §3.3. The code uses BCEWithLogitsLoss on each channel of the 2-channel map loss.py loss.py.
• Sigmoid applied twice. The model applies a Sigmoid to the drivable and lane outputs in forward YOLOP.py YOLOP.py. The loss then passes those outputs to BCEWithLogitsLoss, which applies a sigmoid again loss.py. The training loss is therefore computed on sigmoid(sigmoid(z)). This is what the code does, not a checked result: I did not run training to test its effect.
• Lane IoU term has no gradient. The paper adds an IoU loss to the lane term §3.3 eq. 3. The code computes it from argmax predictions and a NumPy confusion matrix loss.py evaluate.py. The result is a constant, so it does not change the gradient.
• Prior anchors. The paper says anchors come from k-means on the dataset §4.1.2. The code uses fixed anchors YOLOP.py and turns k-means off by default default.py.
• Warm-up for Adam. Warm-up applies to a single optimizer group, so the bias-learning-rate branch is never used utils.py function.py. The warm-up momentum schedule is also a no-op for Adam, because its parameter groups have no momentum key function.py.
• Weight decay. The config sets 0.0005 default.py, but Adam is built without it utils.py.
• Lane accuracy is lane recall. The paper says "pixel accuracy" for lanes §4.2.3. The code's lane accuracy is the recall of the lane class evaluate.py function.py. The drivable-area accuracy is overall pixel accuracy function.py.
• Lane masks. The paper renders training lane lines 8 px wide and test lines 2 px wide §4.2.3. The repository reads pre-rendered lane masks from a folder AutoDriveDataset.py and has no rendering code, so the 8 px / 2 px rule cannot be checked here.
• Shear is off. The paper lists shear among its geometric augmentations §4.1.2. The default shear is 0.0 default.py.
• Stage stopping. Algorithm 1 stops each stage when the loss falls below thr alg. 1. The code has no threshold. Each stage is a separate run with a flag set in the config, and each runs to END_EPOCH default.py train.py.
• Parameter groups for freezing. The code freezes by parameter index: encoder 0-16 (backbone, SPP, FPN), detection head 17-24 (PAN and the detect layer), drivable head 25-33, lane head 34-42 train.py. The paper's "detect head" therefore includes the PAN.
• Input size. The paper uses 640 x 384 §4.1.3. The config default is 640 x 640 default.py, with letterbox padding to a multiple of 32 augmentations.py. For 16:9 frames this gives 640 x 384, which is inferred, not stated in the code.
• Metric naming. The paper's Table 2 says mIoU tab. 2. The code logs both the per-class IoU for the drivable class and the mean IoU over both classes function.py. The paper does not say which one it reports.

No discrepancies were found in the backbone, the neck, the three-scale detection head, the segmentation head structure or the CIoU box term.

Pitfalls

• Target assignment is not in the paper. Matching targets to anchors and grid cells is not described. The code uses YOLOv5-style matching: a target is assigned to an anchor when its width and height ratios are both under 4.0, and to up to five grid cells (its own cell plus up to four neighbours) postprocess.py. The objectness target is the detached IoU of the prediction with its target loss.py.
• Head weighting is in the code only. The code applies per-scale objectness weights loss.py and a factor of 3/no loss.py.
• Padding enters only some losses. The lane IoU crops letterbox padding loss.py. The cross-entropy terms do not crop it loss.py.
• The stopping test sees frozen heads. In Algorithm 1 the test uses the total loss over all heads alg. 1. In stage 1, that total still includes the segmentation terms while those heads are frozen.
• Multi-task time saving is computed, not stated. Table 5 gives 15.7 + 14.8
  • 14.8 = 45.3 ms for three single-task models, against 24.4 ms for one multi-task model tab. 5. The paper calls the saving substantial §4.3.2.
• Speed claim against PSPNet. The text says 4 to 5 times faster than the baselines §4.2.2. Table 2 gives 41 / 11.1, about 3.7 times, against PSPNet tab. 2.
• Lanes: ENet is faster. On lanes, ENet runs at 100 fps against 41 fps tab. 3. The real-time claim is for the three-task network as a whole.
• Table typo. Table 4 prints the ED-W accuracy as "71,2" tab. 4. Read it as 71.2.
• Tables 4 to 6 do not state the AP definition. They say AP without an IoU threshold tab. 4 tab. 5 tab. 6. Only Table 1 names mAP50.
• No variance is reported. The paper gives one number per setting, with no seeds and no repeated runs.
• Evaluation details are external. The lane centre lines come from a conversion of two-line annotations §4.2.3. The conversion is not in the repository, so the lane numbers are hard to reproduce exactly.

Lineage

• Builds on: YOLOv4 for the single-stage detector and its anchor scheme §3.2.1. YOLOv4 is arXiv:2004.10934 (not carded here).
• Backbone: CSPDarknet, cited to Scaled-YOLOv4 §3.1.1. That paper is arXiv:2011.08036 (not carded here).
• Dataset: BDD100K §4.1.1, arXiv:1805.04687 (not carded here).
• Compared against: MultiNet, DLT-Net, Faster R-CNN, YOLOv5s, PSPNet, ENet, SCNN and ENet-SAD §4.1.3.
• Superseded by: not stated on this card.