DL-Traff: Survey and Benchmark of Deep Learning Models for Urban Traffic Prediction

In plain words

Many research papers propose neural networks that predict urban traffic, for example how many taxis or bikes will enter or leave each zone of a city, or how fast traffic will move on each road sensor. This paper first reviews those models. It sorts them by how they treat space: a city cut into a grid of cells, a road network drawn as a graph, or a set of sensors treated as parallel time series.

Then it runs a set of the models on the same public datasets, with the same split, the same training settings and the same error measures, so the results can be compared fairly. The benchmark code is released in two repositories. The authors' conclusion is that no single grid model dominates, and that a few graph models are consistently strong.

TL;DR

• A survey and a benchmark, not a new model. It covers grid-based and graph-based deep traffic models plus a few multivariate time-series baselines.
• Grid models predict next-interval inflow and outflow; graph models predict 12 future speeds from 12 past speeds.
• Author-reported: STDN has the best grid RMSE on all four grid datasets; Graph WaveNet is best at 60 minutes on METR-LA, MTGNN on PeMS-BAY.
• The released grid code reports only rescaled MSE, and the graph and grid code differ from the paper's stated settings (see Paper vs code).

Problem

Many deep models for traffic prediction exist, but papers use different datasets, metrics and extra data sources, so their results are hard to compare §1. The authors ask for two things: a concise survey of the models, and a benchmark that runs them under the same settings and metrics §1.

The shared data form is a tensor of shape (T, N, C): T time steps, N spatial units (grid cells or graph nodes), C information channels §1. The two benchmark tasks are:

• Grid inflow and outflow, one step ahead. The input is α past steps of an H×W×C tensor (C = 2: inflow and outflow) and the output is the next step §2.
• Graph traffic speed, multi-step. The input is α steps of an N×C tensor (C = 1: speed) and the output is β future steps §2.

Method

This paper has no new model. The "method" is the shared benchmark protocol and the model families it compares.

Model families. Grid-based models convert the 4-D grid tensor to a 3-D tensor by stacking channels, or apply convolutions per cell, and use LSTM, GRU, temporal convolution or attention for time §4 tab. 3. Graph-based models apply graph convolution over an adjacency matrix and a temporal module such as a gated TCN, GRU, LSTM or attention §4.2 tab. 3. The multivariate time-series models (LSTNet, TPA-LSTM, GeoMAN, Transformer) are reviewed but not expanded §4; LSTNet alone is benchmarked, as a graph-task baseline tab. 5.

Protocol. The steps below follow the official graph code. The grid code differs in its split and loss (see Paper vs code).

# Data split and windows for graph speed prediction  [§5.1] [§2]
# Input:  series (T, N) of sensor speeds
# Output: train, validation and test (input, target) pairs

def split_and_window(series, steps_in, steps_out):
    # Standardise each sensor using the whole series (see Pitfalls)
    # [code: workMETRLA/pred_DCRNN.py#L187-L189]
    series = standardise_per_sensor(series)
    # First 80% of time steps are train+val, last 20% are test
    # [code: workMETRLA/Param.py#L14]
    cut = int(T * 0.8)
    # Windows: steps_in past steps predict the next steps_out
    # [code: workMETRLA/pred_DCRNN.py#L26-L28]
    train_val = windows(series, 0, cut, steps_in, steps_out)
    # Test inputs begin steps_in steps before the cut
    # [code: workMETRLA/pred_DCRNN.py#L31]
    test = windows(series, cut - steps_in, T, steps_in, steps_out)
    # Last 12.5% of train+val windows are validation
    # [code: workMETRLA/pred_DCRNN.py#L74]
    train, val = split_at(train_val, 0.875)
    return train, val, test

# Training with early stopping  [§5.1]
# Adam, learning rate 0.001, batch 64, MAE loss, at most 200 epochs
# [code: workMETRLA/Param.py#L6-L13]

def train(model, train, val):
    optimizer = Adam(model.parameters(), lr=0.001)
    best_val = infinity
    wait = 0
    for epoch in range(200):
        # One pass over shuffled training batches with L1 loss
        # [code: workMETRLA/pred_DCRNN.py#L100-L107]
        for inputs, targets in shuffled_batches(train, 64):
            loss = mean_absolute_error(model(inputs), targets)
            optimizer.step_on(loss)
        # Keep the checkpoint with the lowest validation loss
        # [code: workMETRLA/pred_DCRNN.py#L109-L113]
        val_loss = evaluate(model, val)
        if val_loss < best_val:
            best_val = val_loss
            wait = 0
            save(model)
        else:
            wait += 1
            # Stop after 10 epochs without improvement
            # [code: workMETRLA/pred_DCRNN.py#L116-L118]
            if wait == 10:
                break
    return load_best(model)

# Masked error on de-standardised predictions  [§5.1]
# Input:  true and predicted speeds, shape (S, H, N)
# Output: RMSE, MAE and MAPE (%)

def masked_metrics(true, predicted):
    # Values below 1e-5 are set to zero in both arrays
    # [code: workMETRLA/Metrics.py#L12-L13]
    true[true < 1e-5] = 0
    predicted[predicted < 1e-5] = 0
    # Only entries whose true value is non-zero count
    # [code: workMETRLA/Metrics.py#L14-L17]
    keep = true != 0
    error = (predicted - true)[keep]
    # Mean over counted entries
    # [code: workMETRLA/Metrics.py#L17-L20]
    rmse = sqrt(mean(error ** 2))
    mae = mean(abs(error))
    # MAPE divides by the true value, zeros excluded
    # [code: workMETRLA/Metrics.py#L47-L59]
    mape = mean(abs(error / true[keep])) * 100
    return rmse, mae, mape

Grid tasks. The grid protocol uses the same metric names, but the released grid code computes only the rescaled MSE (see Paper vs code). The time window is 6 steps in and 1 step out §5.1 Param.py.

Spatial and temporal blocks named in the paper §4.1, §4.2: ST-ResNet uses closeness, period and trend inputs with residual CNN blocks §4.1; DMVST-Net and STDN combine local CNNs with LSTM and attention §4.1; DCRNN uses a diffusion-style graph convolution inside a GRU encoder-decoder §4.2; STGCN uses a gated TCN with spectral graph convolution §4.2.

Hyperparameters

Values the paper states §5.1 are marked as such. "Code" rows come from the released repositories, not from the paper text.

Dataset specifications (grid: TaxiBJ, BikeNYC-I, BikeNYC-II, TaxiNYC; graph: METR-LA, PeMS-BAY, PeMSD7(M)) are listed in tab. 2 and described in §3. Grid sizes in the code match the table for TaxiBJ 32×32 Param.py, BikeNYC-I 21×12 Param.py and TaxiNYC 10×20 Param.py. The BikeNYC-II size was not checked.

Results

Author-reported. Setup: single-step grid prediction (6 steps in, 1 out) and multi-step graph prediction (12 in, 12 out), with HistoricalAverage and CopyLastStep baselines §5.1. Metrics are RMSE, MAE and MAPE. In the source tables, bold marks the best value per column; bold below is copied from the source.

Grid datasets, single-step tab. 4

Graph datasets, multi-step tab. 5

Headlines, as the authors state them:

• Grid: no grid model is dominant. STDN is generally strongest, and PCRN and DeepSTN+ have the lowest MAE on BikeNYC-I and TaxiNYC §5.2 tab. 4.
• Graph: DCRNN, Graph WaveNet and MTGNN rank in the top three across datasets. Graph WaveNet and MTGNN take most of the best scores §5.2 tab. 5.
• Graph models beat the HistoricalAverage and CopyLastSteps baselines and LSTNet at longer horizons. LSTNet is competitive only at short horizons §5.2 tab. 5.
• Efficiency §5.3 fig. 6: ST-ResNet has the fewest trainable parameters among grid models; STDN and DMVST-Net train slowest because they use LSTMs and per-cell computation. STGCN and Graph WaveNet are the most efficient graph models. The efficiency comparison is shown only as bar charts in the paper.

Paper vs code

Paths prefixed grid/ are in DL-Traff-Grid, commit 8cdf93c382a7fba9400a686116f2b804743f13ed. That repository has no LICENSE file, so its license is unknown and it is cited for settings only. Unprefixed paths are in DL-Traff-Graph (ccc038aeef05ffd43fab42e0752c8f94b90163a7, MIT). Both repositories are linked by the paper §1 §6.

• Loss (grid). Paper: MAE for all models §5.1. Grid code: MSE for STDN Param_STDN.py and for the other grid scripts Param.py. The graph code uses MAE as the paper says Param.py.
• Learning rate and batch (grid). Paper: 0.001 and batch 64 by default §5.1. Grid code: learning rate 0.0001 Param_STDN.py and batch 1024 Param_STDN.py.
• Batch (GMAN). GMAN uses batch 32 in its own file Param_GMAN.py; the paper does not mention this.
• Split (grid). Paper: 7 : 1 : 2 §5.1. Grid code: 0.8 for train+val and 0.2 of that for validation, so about 0.64 : 0.16 : 0.20 predflowio_STDN_noflow.py. The graph code follows 7 : 1 : 2 Param.py.
• Metrics (grid). Paper: RMSE, MAE and MAPE §5.1 tab. 4. Grid code: only the MSE, rescaled by MAX_VALUE squared predflowio_STDN_noflow.py predflowio_STDN_noflow.py. A search of the grid repo finds no MAE or MAPE code. RMSE could be derived as the square root of the rescaled MSE, but MAE and MAPE cannot be reproduced from the released grid code. The graph code computes all three Metrics.py.
• STDN variant. Paper: STDN uses flow gating to add local flow information §4.1. The grid repo contains only the no-flow variant as the STDN run script (predflowio_STDN_noflow.py), which imports STDN_models_noflow predflowio_STDN_noflow.py. I did not find a flow-gated STDN script.
• DCRNN. Paper: diffusion convolution with bidirectional random walks §4.2. Code: forward and reverse random-walk matrices DCRNN.py, expanded by a Chebyshev-style recursion with K = 3, hidden 64, 2 layers DCRNN.py DCRNN.py. The paper gives no K or hidden size.
• STGCN horizons. The paper reports STGCN at 3, 6 and 12 steps tab. 5 without saying how it is trained. In code, the 12-step variant uses only the last horizon step as its target pred_STGCN12.py pred_STGCN12.py, and separate scripts exist for 3 and 6 steps pred_STGCN3.py pred_STGCN6.py.
• ASTGCN. The code asserts that input and output steps are equal, because ASTGCN has a single time parameter pred_ASTGCN.py. The paper does not mention this.
• Runs. The paper's text does not say how many runs are averaged. The source has a commented-out sentence about three runs. The code fixes one seed, 100 pred_DCRNN.py.
• Early stopping. Paper: stop when validation error has not improved for 10 epochs; at most 200 epochs §5.1. Code matches pred_DCRNN.py Param.py. The grid code also uses patience 10 on validation loss predflowio_STDN_noflow.py.

No discrepancy found for the graph split, optimizer, learning rate, or early-stopping settings.

Pitfalls

• Test leakage in normalisation. The graph code fits the StandardScaler on the whole series, including the test period, before splitting pred_DCRNN.py. A clean reproduction should fit the scaler on the training part only. The paper does not say which was used.
• Zero handling is stronger than "ignore zeros". Values below 1e-5 are set to zero in both truth and prediction before the error is taken Metrics.py. Negative or near-zero predictions are therefore clipped to zero and counted as errors against non-zero truths. Only entries whose true value is zero are excluded.
• Metrics mutate their inputs. The metric functions overwrite the truth and prediction arrays in place Metrics.py. Pass copies if the arrays are reused.
• MAPE is unstable near zero. It divides by the true value and is computed only on non-zero truths Metrics.py.
• Grid normalisation is a global constant, not a standardisation. Each city's data is divided by one constant, e.g. 1292.0 for TaxiBJ Param_STDN.py. The reported grid MSE is multiplied back by that constant squared predflowio_STDN_noflow.py.
• DCRNN decoder start. The decoder starts from a zero input, and each step feeds back its own prediction DCRNN.py. Teacher forcing is not used.
• Test windows overlap training time. Test inputs start 12 steps before the test split, so their inputs come from the end of the training period pred_DCRNN.py.
• Hidden costs. The paper reports four GTX 2080Ti GPUs §5.1 and training-time bar charts fig. 6. Those charts are not reproduced in the text, so per-epoch cost is not stated.
• Grid MAE and MAPE cannot be reproduced from the grid code. See Paper vs code. Use the paper numbers as reported, and label them as such.

Lineage

• Builds on: the graph and grid models it reviews, each cited in tab. 1 and §4. The graph models include STGCN §4.2, DCRNN §4.2 and Graph WaveNet tab. 5; the grid models include ST-ResNet §4.1 and STDN §4.1.
• Compared against: HistoricalAverage, CopyLastStep(s), CNN and ConvLSTM baselines §5.1 tab. 4 tab. 5.
• Superseded by: not stated in this card's sources.