One Thousand and One Hours: Self-driving Motion Prediction Dataset

In plain words

Self-driving cars need to guess where nearby cars, cyclists and pedestrians will be in the next few seconds, and then choose their own path. Learning these skills needs recordings of real traffic, and most of the large recordings are kept inside companies. This paper releases one such collection. A fleet of cars drove the same route in one town again and again, and the cars' perception system logged what it saw for each short slice of time.

The release also includes a detailed map of the road, with lanes, traffic lights and crosswalks, and a top-down aerial photo of the area. The authors ship a Python toolkit to load the data and draw it. They also train two simple learning baselines, one for predicting other vehicles' motion and one for driving the car itself. The baselines get better as they see more of the recordings, which is the paper's main argument for why data of this size matters.

TL;DR

• Releases about 1,118 hours of logged traffic from 20 self-driving cars on one 6.8-mile route in Palo Alto, as 25-second scenes §3.1 tab. 2 fig. 1.
• Adds an HD semantic map (15,242 labelled elements) and a 74 km² aerial map at 6 cm per pixel §3.2 §3.3.
• Baselines (ResNet-50 on 224×224 bird's-eye-view rasters) improve with more training data in open-loop forecasting tab. 3 and in closed-loop planning fig. 8. Both are author-reported.
• The official toolkit (L5Kit) is Apache-2.0 and its planning and forecasting configs differ from the paper in horizon, output, and training size (see Paper vs code).

Problem

Motion prediction and planning progress has depended on large in-house datasets, and open datasets for these tasks are small. The authors cite Argoverse Forecasting as the closest open alternative, at about 300 hours with lane centre lines only §2. Their goal is an open dataset that is larger, carries the full perception output (boxes and class probabilities, not just centroids), and has a detailed map §2.

The paper also argues that open-loop forecasting does not capture planning, because errors accumulate once the vehicle leaves the logged path. Closed-loop evaluation needs data beyond the log §2. Single-route collection is a deliberate choice, since a fleet serving one high-demand route is easier to bound for safety §1.

Method

Data collection. The fleet drove a fixed route in Palo Alto between October 2019 and March 2020, during daytime from 8 AM to 4 PM §3.1. The sensors are seven cameras, three LiDARs and five radars, all on the roof or front bumper. The roof LiDAR has 64 channels at 10 Hz §3.1 fig. 3.

Perception output. Each scene stores the detections of an in-house perception system, not raw sensor data. Each detected object is stored as a 2.5D box with its speed, acceleration, heading and turn rate, plus a class label §3.1. The paper says the detector was trained on a dataset similar to the Lyft Perception Dataset along the same route §3.1. Its architecture and training are not described in detail.

Scene format. Scenes are 25 seconds long and stored as compressed zarr arrays for random access §3.1 fig. 4. The data is split 83 / 7 / 10 % into train, validation and test. Each vehicle's logs go to only one split §3.1.

HD semantic map. Human curators annotated a SLAM localisation map. The position of the vehicle is therefore known to centimetre accuracy §3.2. The map is a protocol-buffer file with lane boundaries, lane connectivity, driving direction, road class, paint, speed limits, lane restrictions, crosswalks, traffic lights, signs, restrictions and speed bumps §3.2 fig. 5.

Aerial map. The aerial map covers 74 km² at 6 cm per pixel. It is provided as 181 GeoTIFF tiles of 10,560 × 10,560 pixels, each about 640 × 640 m §3.3.

Toolkit. L5Kit loads and samples scenes, centres them on the vehicle or on another agent, and rasterises them into bird's-eye-view images with the semantic map, the aerial map or both §4 fig. 6.

Forecasting baseline. The task is to predict (x, y) positions over T = 5 s for each agent, given its current position and optionally past positions. The input is a 224×224 raster centred on that agent. The history is stacked as extra raster frames. The network is a ResNet-50 with an L2 loss, trained for 38,000 iterations with batch size 64 §4 fig. 7.

# Motion forecasting baseline, one training step  [§4]
# Input:  raster (B, C, 224, 224) centred on each agent
#         targets (B, T, 2) future (x, y) positions, T = 5 s  [§4]
#         availabilities (B, T) 1 where the target exists
# Output: scalar loss

def forecasting_loss(model, raster, targets, availabilities):
    # Predict T future (x, y) positions for each agent  [§4]
    # [code: examples/agent_motion_prediction/agent_motion_prediction.ipynb#L108]
    outputs = model(raster).reshape(targets.shape)    # (B, T, 2)
    # Squared error per coordinate, the paper's "L2 loss"  [§4]
    errors = (outputs - targets) ** 2                 # (B, T, 2)
    # Ignore future steps that have no ground truth
    # [code: examples/agent_motion_prediction/agent_motion_prediction.ipynb#L111]
    errors = errors * availabilities.unsqueeze(-1)    # (B, T, 1)
    return errors.mean()

Planning baseline. The authors follow ChauffeurNet, which augments imitation training with synthetic perturbations to mitigate the accumulation of errors §2 §4. The network sees a 224×224 raster centred on the ego vehicle and predicts the ego's future (x, y) positions over a 5-second horizon, with the same architecture and loss as forecasting §4. At test time the network controls the ego vehicle in 25-second test scenes. Other agents replay their logged behaviour, so the simulation is non-reactive. Only errors caused by the ego vehicle are counted: collisions, traffic-rule violations and off-road events §4.

In the official code the perturbation is an Ackermann-model trajectory augmentation. With probability p, it draws a Gaussian lateral, longitudinal and yaw offset, applies it to the state at the current frame, and refits a feasible steering model to the logged future train.ipynb ackerman_perturbation.py.

# Ego-trajectory perturbation for planning training  [§4]
# Input:  logged ego history and future frames
# Output: history and future used as the training example

def perturb_ego_example(history, future, prob):
    # Keep the logged example with probability 1 - prob
    # [code: examples/planning/config.yaml#L57]
    if random_uniform() >= prob:
        return history, future
    # Draw lateral, longitudinal and yaw offsets from zero-mean Gaussians
    # [code: examples/planning/train.ipynb#L126-L127]
    lateral, longitudinal, yaw = gaussian_offsets(std=[0.5, 1.5, pi / 6])
    # Apply the offsets at the current frame and refit a feasible trajectory
    # [code: l5kit/l5kit/kinematic/ackerman_perturbation.py#L57-L114]
    new_future = refit_ackermann_trajectory(history, future, offsets)
    return history, new_future

Hyperparameters

Results

Author-reported. Setup: this dataset, ResNet-50 forecaster, open-loop evaluation, displacement error in metres at each horizon, and ADE. The text compares only training-set size and history length. No other baseline is reported tab. 3.

The paper states that error keeps falling as training data grows, with and without history §4 tab. 3. Adding 1 s of history lowers ADE at every training size tab. 3.

Planning is evaluated closed-loop on 25-second test scenes. The text gives no numbers. The authors state that the SDV's performance increases dramatically with the amount of training data, and show a qualitative run from a model trained on 1,000 hours fig. 8 §4.

Paper vs code

Code: the official toolkit at lyft/l5kit (linked from the paper's footnote §4), commit 589470a158b25c53435032484b33afbb201a4ceb. The licence is Apache-2.0; its text is at docs/LICENSE and the repository has no LICENSE file at its root.

• Planning horizon. Paper: 5 s §4. Planning example config: future_num_frames: 12, which is 1.2 s at step_time: 0.1 config.yaml.
• Planning output. Paper: (x, y) positions §4. Code: X, Y and yaw per step, num_targets = 3 * future_num_frames train.ipynb, with yaw in the loss model.py.
• Loss. Paper: L2 loss §4. Code: MSELoss(reduction="none") with availability weights, then a mean agent_motion_prediction.ipynb train.ipynb model.py.
• Batch size and steps. Paper: batch 64, 38,000 iterations §4. Both example configs use batch 12 and 5 steps agent_motion_config.yaml agent_motion_config.yaml. These look like smoke-test settings, not the paper's run.
• Forecast history. Paper's best rows use 1 s of history tab. 3. Example config uses history_num_frames: 0 agent_motion_config.yaml.
• Pretraining. Not stated in the paper. The example notebook uses ImageNet weights agent_motion_prediction.ipynb.
• Perturbation strength. Paper: "synthetic perturbations", no magnitudes §4. Code: probability 0.5 and Gaussian std of 0.5 m, 1.5 m and π/6 config.yaml train.ipynb.
• Evaluation protocol. Paper does not say how Table 3 was computed. The example notebook evaluates on a "chopped" split: it keeps the first 100 frames of each scene and scores only agents valid in the 100th frame, with future positions held out in a separate ground-truth file agent_motion_prediction.ipynb.
• Dataset size. Text: 170,000 scenes §3.1 fig. 4. Table 2: 162k scenes in total tab. 2. The paper does not explain the gap.
• Collection period. Abstract: "over a four-month period". §3.1: October 2019 to March 2020, which spans about six months.

The code has no sensor or detector configuration, so the sensor layout and detector description in §3.1 cannot be checked against it.

Pitfalls

• Coordinates. The model predicts positions in the agent's frame. To get displacements in world space, transform by world_from_agent and subtract the centroid agent_motion_prediction.ipynb. The paper does not describe this step.
• Raster origin. The example configs set set_origin_to_bottom: True and the code comment warns that models trained before December 2020 need False agent_motion_config.yaml. Check the flag before reusing older weights.
• Loss masking. Unavailable future steps are excluded from the loss in the code agent_motion_prediction.ipynb. The paper does not mention this.
• Metric naming. The paper's displacement error is a Euclidean (L2-norm) distance tab. 3 caption §4. The training loss is squared error. Do not mix up the two when comparing numbers.
• History length. The text says "the past few seconds" §4 while Table 3 uses 1 s. Use the table value.
• Perturbation randomness. The Ackermann perturbation draws from np.random without a seed in the code ackerman_perturbation.py. Results will vary across runs unless you seed it yourself.
• Slow vehicles. The code's perturbation has a TODO saying it does not work when the ego vehicle is slow ackerman_perturbation.py. The paper does not mention this.
• Non-reactive simulation. Other agents replay the log, so collisions with them are not counted and the closed-loop numbers are optimistic about the ego's interactions §4.
• Single route and daytime only. The data covers one route and daytime hours between 8 AM and 4 PM §3.1 §1. Generalisation to other places or night driving is not tested.
• Perception labels. Objects come from an in-house detector that is not described in detail §3.1. Label errors are not quantified.
• Licensing. The CC BY 4.0 licence applies to the paper text. This card does not establish a licence for the dataset, map or aerial imagery, so check the release terms before reuse. L5Kit is Apache-2.0 at the commit above.

Lineage

• Builds on: ChauffeurNet, cited for planning by imitation with perturbations §2 §4 (named only; no card, no arXiv id in the bibliography). ResNet-50 is the backbone §4 (named only; no card).
• Compared against: Table 1 compares KITTI, Oxford RobotCar, Waymo Open Dataset, ApolloScape, Argoverse 3D Tracking, Lyft Perception Dataset, nuScenes and Argoverse Forecasting tab. 1. None has a card here, so nothing from them is used.
• Superseded by: not stated.