Synscapes: A Photorealistic Synthetic Dataset for Street Scene Parsing

In plain words

Teaching a self-driving car to recognise roads, cars and people needs a lot of street photos where every pixel is labelled. Labelling real photos by hand is slow and expensive, and some situations are rare. Synscapes is a set of computer-made street scenes instead. Each one is built from a random mix of settings: how many cars and people there are, the sun's height, the weather, the road width. The pictures are drawn with a physically based renderer, the kind used for film visual effects, so light, shadows and camera noise look realistic.

Because the computer knows exactly what it drew, every picture comes with labels for each pixel, each object and its distance. The authors use the set in three ways: to test networks that were trained on real photos, to train networks, and to study where a network succeeds or fails under different conditions. Their main claim is that realism matters: Synscapes beats the two comparison sets in every test they ran. Results are reported by the authors themselves.

TL;DR

• Synscapes is 25,000 path-traced street-scene images with Cityscapes-style class labels, instance and depth maps, and per-actor metadata §3 §3.1 §3.2 §3.3.
• Scenario parameters are sampled independently, so each image has its own scene; this is what enables the bin-by-parameter analysis §3 §6.
• Author-reported: it beats two earlier synthetic sets (GTA-based and Synthia) on validation, as training data, and on self-validation, for both DeepLab v3+ and FRRN tab. 1 tab. 4 tab. 5. The authors conclude that realism matters for synthetic data §7.
• Use the card if you need synthetic street scenes with rich metadata, or need to measure how a model's error depends on distance, occlusion or actor orientation. The dataset's own download location and licence are not in the paper (see Pitfalls).

Problem

Simulators are often proposed as a source of test and training data for perception systems. The problem is whether synthetic sensor output resembles real sensor output closely enough. If it does not, a missed pedestrian in simulation cannot be told apart from a true model weakness §4 §1. Earlier synthetic street datasets relied on simple geometry and real-time rendering (a game engine), or on the legal and technical limits of reusing a commercial game §2. Real datasets, in turn, are costly to label for every attribute, and rare conditions are expensive to capture §6.1.3.

Method

Scene generation. Each image is a procedurally generated scenario. A scenario is one sample from a space of scenario parameters, each paired with a distribution. Parameters include the number of cars and pedestrians, road width, road surface, time of day and weather §3. The paper gives the generation engine's design in an earlier publication, not here §3.

1. Sample every scenario parameter independently from its distribution. Each of the 25,000 images gets its own unique parameter sample, so the parameters stay decorrelated §3 §3.4 fig. 4.
2. Build the 3D world (ego vehicle, agents and the rest of the scene) from that sample §3.
3. Render with unbiased path tracing. Sun and sky radiometry, physically based reflectance, camera optics (a long-tail point spread function) and sensor effects (readout noise, camera response function, colour) are all simulated §3.
4. Write an RGB PNG at 1440 × 720 and a 2048 × 1024 version. For the larger version the sensor simulation runs at the higher resolution, so the noise is not up-sampled §3.1.
5. Write annotations for each image: a single-channel class PNG with Cityscapes class ids, an instance PNG, an OpenEXR depth map in metres of planar z-depth, and a JSON metadata file §3.2 §3.3.

Annotation encoding. The instance PNG stores each instance id across the R, G and B channels with the formula below. It is not numbered in the paper §3.2:

\mathrm{id} = R + 256 \cdot G + 256^2 \cdot B §3.2

Metadata. Three kinds, one JSON file per image §3.3:

• scene parameters under the scene key, described in app. A.2;
• camera intrinsics and extrinsics under camera, constant across the set app. A.1;
• per-instance records for non-static classes (person, rider, car, truck, bus, train, motorcycle, bicycle): 2D box, oriented 3D box, class, fractional occlusion and truncation app. A.3.

Evaluation protocol (validation). Pre-trained Cityscapes networks, and networks trained on the other synthetic sets, are run on each synthetic set's validation split and scored with mean IoU §4.1 tab. 1. Detection uses Faster R-CNN (ResNet-101) with the reference model trained on KITTI, scored with the COCO mAP metric (IoU 0.50:0.95) and per-class AP at IoU 0.50 §4.2 tab. 2 tab. 3.

Evaluation protocol (training). Networks are trained on each dataset with the same hyperparameters, then tested on real data. For segmentation, FRRN and DeepLab v3+ are trained on the synthetic data, and optionally fine-tuned on Cityscapes §5.1 tab. 4. For detection, Faster R-CNN and FastBox are trained on Synscapes, optionally fine-tuned on KITTI, and scored on KITTI and Synscapes §5.2 tab. 6 tab. 7 tab. 8.

Analysis by binning. Predictions are grouped by one parameter or instance property, and IoU is averaged per bin. Because all other parameters stay spread out, a bin's score reflects the binned variable §6 fig. 8 fig. 9 fig. 11 fig. 12. Correlation analysis, for segmentation only, is below.

# Correlate per-class IoU with one scenario parameter  [§6.1.3]
# Input:  predictions and labels for 24,000 training images
# Output: correlation factor and p-value for one (class, parameter) pair

def correlate(images, parameter, class_name):
    # Divide the images into 64 subsets  [§6.1.3]
    subsets = split_into_subsets(images, parameter, count=64)
    # One IoU score per subset for this class  [§6.1.3]
    scores = [iou(subset, class_name) for subset in subsets]
    # Standard linear regression; x-values per subset not stated  [§6.1.3]
    correlation, p_value = linear_regression(subset_values, scores)
    # Keep the pair only if p is 0.05 or less  [§6.1.3]
    return correlation, p_value

Hyperparameters

Dataset generation and annotation:

Segmentation training and analysis:

Detection training and evaluation:

Results

Author-reported. Setup: Synscapes (25,000 images), GTA (Richter et al.) and Synthia as the two comparison synthetic sets, Cityscapes and KITTI as the real sets, DeepLab v3+ and FRRN for segmentation, Faster R-CNN (ResNet-101) and FastBox for detection. Full per-class IoU and AP tables are in the paper tab. 1 tab. 3 tab. 4 tab. 5; only mean scores are copied here.

Validation: Cityscapes-trained reference models on each synthetic validation set, mean IoU tab. 1:

Detection validation, Faster R-CNN trained on KITTI, COCO metrics tab. 2:

Per-class AP at IoU 0.50, car and pedestrian tab. 3:

Segmentation training, mean IoU on the Cityscapes validation set tab. 4:

Self-validation, DeepLab v3+, each model on its own dataset's validation set, mean IoU tab. 5: Cityscapes 78.79; Synscapes 87.00; Richter 63.05; Synthia 57.22.

Detection training, Faster R-CNN, COCO metrics on KITTI validation tab. 6:

Detection training, Faster R-CNN, Synscapes validation tab. 6: Synscapes alone reaches mAP 0.488 (mAP@0.50 0.777, mAP@0.75 0.518).

Car and pedestrian AP at IoU 0.50 on KITTI tab. 7: KITTI 0.534 / 0.847; Synscapes 0.344 / 0.131; Synscapes + KITTI 0.902 / 0.679.

FastBox (KittiBox), AP on 500 KITTI test images tab. 8, Easy / Medium / Hard: KITTI 97.4 / 89.0 / 75.1; Synscapes 68.6 / 53.4 / 44.5; Synscapes + KITTI 98.7 / 90.0 / 77.0 (%).

Paper vs code

The official code is 7dlabs/synscapes-utils (MIT licence, the authors' organisation). It contains three Python files: visualisers and metadata helpers. It has no training, evaluation or analysis code, and no network code, so no segmentation or detection hyperparameter can be checked against code.

• Angle units: the paper gives extrinsic pitch, roll and yaw without units app. A.1; the helpers convert them with math.radians, so they are degrees helpers.py.
• Camera projection: the paper gives intrinsics only app. A.1; the code projects with u = -y·fx / x + u0 and v = -z·fy / x + v0 in the camera frame, with x pointing forward helpers.py.
• 3D boxes: the paper's convention (origin at rear lower right; x forward, y left, z up; vector lengths are extents) app. A.3 matches the code, which adds each vector to the origin and draws the edges from there visualize-synscape-metadata.py.
• 2D boxes: the paper says normalised coordinates app. A.3; the code multiplies by image width and height visualize-synscape-metadata.py.
• Instance ids: the paper gives the R + 256·G + 256²·B formula §3.2. No code in the repository decodes the instance image; it is only overlaid visualize-synscape-metadata.py.
• Directory names: the paper names img/rgb and img/rgb-2k §3.1 but not the class and instance directories. The code reads img/class and img/instance visualize-synscape-metadata.py.
• Occlusion culling: the paper says actors over 99% occluded may be removed from metadata §3.2. The visualiser culls at 90% by default visualize-synscape-metadata.py. These are different tools; neither changes the data.

Pitfalls

• Dataset licence and download are not stated. The paper's licence (CC BY 4.0) covers the paper, not the dataset. Neither the dataset's download location nor its licence appears in the paper. The code repository's MIT licence covers the utilities only LICENSE. Confirm the dataset's licence before commercial use. Until you can, do not redistribute it.
• Split sizes are inconsistent. The dataset has 25,000 images §3.1. The segmentation analysis uses 24,000 training images §6.1. The validation split size is not stated. The detection analysis uses 20,000 Synscapes images §6.2. Detection test sets are 1,000 images in §4.2 but 500 in §5.2.
• Unresolved author comment in a caption. The caption of the KITTI-trained detection table reads "MW: KittiBox or RCNN?" tab. 7. The text ties that table to Faster R-CNN §5.2; the KittiBox results are in tab. 8.
• Text error on fine-tuning gains. The text says Synscapes gives "more than twice the gain compared to both Richter and Synscapes" §5.1. Against the baseline of 76.56 tab. 4, Synscapes then Cityscapes gives +2.29; Richter gives +1.01 and Synthia +0.89. The second name is probably a typo for Synthia; check against the table, not the sentence.
• Realism is argued, not isolated. The paper says the gap is "most likely" due to sensor simulation and feature variation §4.2. The segmentation comparison is "not by itself proof" of a smaller domain shift §4.1. The GTA-based set is the most varied in models and building types §7, so the gaps are not the effect of realism alone.
• Detection training details are missing. Faster R-CNN and FastBox hyperparameters are given §5.2, but the paper does not state the augmentation, anchor settings or input size used, and there is no training code in the repository.
• Per-subset analysis is under-specified. The 64-subset construction, the x-value for each subset and the regression form beyond "standard linear regression" are not stated §6.1.3. Reproducing fig. 10 needs assumptions the paper does not make.
• Occluded-actor metadata is approximate. Occlusion and truncation are reported "within 1%" app. A.3. Actors over 99% occluded can be dropped from metadata while remaining visible §3.2.
• Use the right resolution. The 2k set is rendered at 2048 × 1024 with sensor simulation at that size, not up-sampled §3.1. Do not up-sample the 1440 × 720 set to match it, because the noise profile would differ.
• Rider and mount share one instance. For bicyclists and motorcyclists the instance is the rider and the vehicle together, classed by vehicle type. The class image keeps them apart §3.3. An instance-based metric will count them as one object.
• Static classes have no instance records. The per-instance metadata covers only non-static classes app. A.3. Road, building and similar classes exist only in the class image.
• Correlation is not causation. The parameter correlations fig. 10 show association within one model and one dataset. The authors attribute the motion-blur effect to features being smeared §6.1.3, which is a hypothesis.

Lineage

• Builds on: the procedural engine, described in an earlier publication that this card does not cover §3. Unbiased path tracing is the cited rendering technique §3. Neither is carded here.
• Compared against: Virtual KITTI §2, Synthia §2, the GTA-based dataset of Richter et al. ("Playing for Benchmarks") §2, and the Cityscapes dataset §3 for label convention and training.
• Superseded by: none known from content already in this wiki.